He's given slightly varying answers.
In 2008, he almost turned down a role in Hamlet (alongside Patrick Stewart) because of his commitments to the show. The resulting negotiations apparently forced both Tennant and Russell T Davies to think seriously about Tennant's future, and they both realized that if he stayed on much longer, he might never be able to leave without seriously damaging both the show and his career.
Anyway, although he admitted to worries about being typecast, and about endangering the show's future, he claims that his biggest motivation was that he wanted to be able to watch the show as a fan (or, in his words, as "a big fanboy geek") again, and it's hard to believe that he wasn't being pretty honest in those interviews.
In the end, he decided to bow out alongside Russell T. Davies who was leaving after the completion of "The End of Time". He has said if he didn't leave then he might never have left the role. He did have a moment where he wanted to stay and talked to Steven Moffat about what he had planned but stuck with his decision to leave.
- According to Davies' book The Writer's Tale, Tennant had talked about his timing for leaving as early as 2006, though as noted he made his final decision rather late in the day after the Moffat meeting.