Why was Colin Baker sacked from playing Doctor Who?

The BBC dismissed him because they wanted to "refresh" the show, the viewing figures were dropping and he got the blame.