A quote form this TARDIS wiki page:
When John Nathan-Turner became producer of Doctor Who in 1981, Baker, Lalla Ward and Christopher H. Bidmead, all angrily protested Nathan-Turner's decisions to take Doctor Who into a different direction. Baker left Doctor Who after 7 seasons after arguments with producers and directors. Baker never liked Nathan-Turner. Tom Baker also hated Matthew Waterhouse and Janet Fielding. The three became friends years later"
fairly self explanatory.