white men always had the right to vote.
It should be noted black men only got the right to vote in 1870 symbolically. That's what the civil rights movement was about (partially). Black men didn't really get the right to vote until the Voting Rights Act in the 70s I think (maybe 1960s).
After the Civil War, and the process of Reconstruction had began, blacks did hold public office on the local, state, and national levels of government.
Most White southerners resented the reconstruction governments and were opposed to Blacks voting and holding public office.
Whites began forming secret organizations. The best known were the Knights of the White Camelia and the Ku Klux Klan. Members of these organizations dressed in ghostly costumes rode the countryside at night, terrorizing both Black and Whites who did not approve of their policies, and prevented Blacks from exercising their rights.
In a series of Force Acts in 1870 and 1871, Congress made it illegal to prevent people from voting. President Grant ordered federal troops into most parts of the South, where the violence was most widespread. Hundreds of Klan members were arrested and convicted and the organization was broken. By the end of 1871, the Klan's activity had declined. However, the victory would only be a short one.
Although the secret societies were destroyed, White intimidation of Blacks continued. By 1873, most Blacks were afraid to vote in the South, and White Democrats were gaining control of assemblies. The Democrats were also able, to take back control of Congress, and state legislatures.
By 1876, White "redeemers" had captured all but three of the southern states, which were, South Carolina, Louisiana, and Florida where federal troops were still serving. When President Rutherford B. Hayes removed all federal troops in 1877, the reconstruction came to an end.
The end of reconstruction also marked the end of the nationwide crusade for Black rights. It would not be until the 1960's, until Blacks would be granted the rights they should have been able to enjoy in the 1860's and 1870's.