The exact timing of when Amy was taken by the Silence and swapped with a ganger has not to date been revealed. The closest clue we have is the Doctor said it happened "Just before America", referencing "The Impossible Astronaut".
Madame Kovarian Edit
Any scene in which Madame Kovarian can be seen peeking through a window indicates that Amy was then a ganger. (Due to a late change in episode broadcast order, "the Doctor's Wife" lacks such a scene.)
Any scene that shows Amy's pregnancy fluctuating in the TARDIS scanner indicates that Amy was then a ganger.