In "Army of Ghosts"/"Doomsday", it was the Torchwood Institute, acting mainly on greed & trying to exploit an energy-source they didn't understand, that opened the way for the Cybus Cybermen to invade this universe's Earth from "Pete's World". In the process, the Torchwood Institute also helped the Cult of Skaro to escape from The Void, bringing with them a Time Lord prison ship, which they called "the Genesis Ark", containing a vast number of Daleks & they invaded the Earth, too. The Doctor has always been strongly against people who endanger lives through short-sighted greed & the Torchwood Institute did this on a huge scale.
In addition, the Torchwood Institute had a total disregard for the rights of non-humans (&, whenever it suited them, for the rights of most humans, too). The Doctor has also always been strongly against people (of any species) who disregard the rights of others.
The Torchwood Institute therefore had two strikes against it at that point.
The third & most personal strike against it was added when sorting out the mess led to the loss of Rose. It was the Torchwood Institute's fault that the Doctor & Rose were parted.