It comes from the fact that a male chicken, a rooster, rules over the females, or hens. The place where birds sit is called a roost, so if you "rule the roost," you dominate the rest of your group. They do what you tell them, and defer to your decisions. Sometimes you see this used humorously to mean that in a marriage, the wife "rules the roost" instead of the husband.