Your mother's (mom) nephew is your answer.
The plural of nephew is nephews.
It is correct if you are talking about more than one nephew. If only one nephew is involved, it is nephew's father. My nephew's father is my brother. I am talking about one specific nephew. If you mean to refer to the father of your nephews, then it would be nephews' father.
my nephew
Nephews.
The plural form of nephew is nephews.
Your nephew's children are your great nephews and great nieces (or grand nephews and grand nieces).
Your borther-in-law's children are your nieces and nephews. You are their uncle or aunt, as is your spouse.
Great nephew
You are your nephew's aunt/uncle,,, his children are your grandnieces/nephews,, his grandson is your great-grandnephew.
Your nephew's child is your great niece (if a girl) or your great nephew (if a boy).
Your brother's children would be your nieces and nephews. Your nephew's children would be great-nieces and great-nephews.
Your nephew's son is your grand nephew or, some would say, your great nephew. You are his great aunt or great uncle.