Why did Hester refuse to take The Scarlet Letter off?

She told the Governor and the Reverand (I assume it's this scene you're referring to) that they had had no right to punish her, but also no right to tell her when her punishment was over.