Why did many poor Southern whites call the Civil War a rich man's war but a poor man's fight?

Because, as with most wars, it was funded and run by the rich but the fighters on the front lines were often poor farmers and other middle class civilians.