Michael Grade insisted that Colin Baker be fired from Doctor Who after Season 23. Grade cited a drop in ratings, the show being "too violent", and a cheap look as his reasons.
All of these are doubtful however. The "cheap look" was due to the shoestring budget that Doctor Who received from the BBC. Ironically, the one man with the power to increase the funding for Doctor Who at the time was Michael Grade. The "violence" issue had always existed, right from the William Hartnell days. In fact, the era deemed most violent by the self-proclaimed "public watchdogs" was the Tom Baker 'Horror' era, which routinely received the highest ratings in the show's history. Lastly, while the ratings had slipped slightly during Colin Baker's era, they were still significantly higher than they had been during the "Classic" eras of the first two Doctors, as well as remaining one of the BBC's highest rated shows in the UK, as well as having a worldwide viewership of over 100 million people.
The real reason was personal. Colin Baker had been married to the actress Liza Goddard. They had an acrimonious divorce. Grade and Goddard were old friends, and he sided with her in the divorce. They moved in together, and later became lovers. Grade was determined to "punish" Baker. Doctor Who production manager Gary Downie stated in an interview with Doctor Who Magazine;
There's a history between Michael Grade and Colin. Liza Goddard was Colin's wife. And she was Michael Grade's best friend. The divorce was acrimonious and she moved into Michael Grade's house while she was getting over the divorce. And I'll say no more. Michael Grade was determined. He did not want Colin working for the BBC.
Producer John Nathan Turner wanted Colin Baker to stay on. Colin Baker wanted to stay on. The fans wanted Colin Baker to stay on. But Michael Grade demanded that Baker be fired. While the Controller of BBC1 (Grade's job) did have the power to make such a decision, no Controller had ever done anything like that before, and it was viewed as an abuse of his position.