What does the quote 'That government is best which governs not at all' from Henry David Thoreau mean?

The context is:

I HEARTILY ACCEPT the motto, - "That government is best which governs least"; and I should like to see it acted up to more rapidly and systematically. Carried out, it finally amounts to this, which also I believe, - "That government is best which governs not at all"; and when men are prepared for it, that will be the kind of government which they will have.

Essentially Thoreau is saying that people are better off when they are left to themselves, without the government's meddling (which is most of what government does). But, he qualifies the statement, it takes a particular sort of person to be able to live in such a society (without it breaking up into hunter-gather bands).