He shouldn't but can and occasionally does. However, nasty things tend to happen. As Rory pointed out in "The Impossible Astronaut": "In all fairness, the universe did blow up."
It may also be difficult, as well as dangerous. For example, when the TARDIS has "slipped a time track" in the past, getting back was sometimes a tricky feat of navigation, so something similar could be in effect here.