I believe scheduling conflicts. Furthermore, Nicholas Courtney was 78 years old and would have been too old to be running around.
- Courtney suffered a stroke around that time as well, which prevented him from appearing with David Tennant in the "Wedding of Sarah Jane Smith" episode of SJA, shot immediately after production of Series 4 ended. Davies also wanted to establish a new set of UNIT characters to be brought back later (such as the female brigadier seen in "Turn Left" and "Planet of the Dead").
The question asks about "a Brigadier" -- i.e., any officer holding the rank between full colonel and major general -- not "the Brigadier" -- i.e., Alistair Lethbridge-Stewart.