Do priest vestments have to match altar color?

Often, a banner matching priest vestment color is in front of altar, the color being associated with the church time, like purple as it is in Lent. It does not always match; if priest is doing funeral he will wear white but the altar may still be purple.