Your nephew's son would call you his great uncle or great aunt. Some would use the terms grand uncle or grand aunt.
Aunt would be the most common term used. It would actually be great aunt.
Great aunt, but in most families it is just aunt.
your nephews
Some would call your nephew's children your grand nephews and grand nieces, others would call then your great nephews and great nieces.
If it is a boy then it is your nephew. If it is a girl then it is your niece.
You would be his Great Uncle and he would be your Great Nephew.
Your nephew's son is your grand nephew or, some would say, your great nephew. You are his great aunt or great uncle.
They are your great nephews and/or great nieces. Alternately, some people say grand-nephews, grand-nieces.
Great nephew
Your brother's children would be called your nieces or nephews.
Your brother's children would be called your nieces or nephews.
Your brother's child's children are your grandnieces or grandnephews.
3rd cousin
Nephews