Do most Mormons really wear the garments?

Most do yes. The only ones who tend not to are the ones who question their own worthiness or commitment to the faith. To LDS it is as natural as wearing anything else.