The person in question is your son.
Think about it this way:
Who is "your father's son"? It's you, because you have no
brothers or sisters. So, you just replace "your father's son" with
"you", and the riddle now reads: "...that man's father is you."
Simple.