Why did Polyphemus leave his wife on the deserted island?

In the Odyssey Polyphemus had no wife.

In Percy Jackson (Sea of Monsters) he wanted to marry Grover (or Clarissa) but did not succeed.