What Freudian concept has been used to explain Hamlet's motivations especially in regard to his interactions with Gertrude?
In the early twentieth century, it became fashionable to apply Freudian concepts to literature. In particular, the Oedipus Complex, which is a theory that men secretly wish to kill their fathers and seduce their mothers, was used to explain Hamlet's relationship with Gertrude. This theory was influencial on Lawrence Olivier's award-winning film of the play in 1948, but is now considered passé.
The "oedipal complex" is a dead-end as far as Shakespearian criticism is concerned, first because Shaskespeare never heard of it, but more importantly because there is no such thing. The Greek myth of Oedipus is about the ritual succession of kingship under the old, pre-Olympian religion, whereby the claimant killed the old king and married the queen, who was the real power. The myth was misconstrued by later Greeks, who were horrified by the idea of female political authority. See Robert Graves, the Greek Myths.
Sigmund Freud, who looked inside himself and seemed to find everyone else in there too, reasoned that because he seemed to have a thing for his mother, every other man must also. He took the later, patriarchal Greek misinterpretation of the original Oedipus story as the proof of his "oedipal complex."