Why does my blower only operate on 3 and 4?

My first guess would be the blower motor resister. My second guess would be the switch and or plug into the back of the switch.