In most families he would be called your nephew. But there is no real relationship. Unless your brother adopts him, which would legally make him your nephew.
Step-brothers
The relationship of a stepbrother's son to someone would be that of a step-nephew. This is because the stepbrother is not biologically related to the person, but is a relative through marriage. Therefore, the stepbrother's son would be the person's step-nephew, as they are the child of the stepbrother.
If her son is from a previous relationship, then he would be your step-son.
Step Nephew
Great uncle / Great nephew
step brothers
step-granddaughter
No because that would be your cousin
Your step-son is the child of your spouse and another person, and is of no true genealogical relationship to you. So your step-son's children would also be of no true genealogical relationship to you either. However, they would commonly be called your step-grandchildren and you would be their step-grandfather or step-grandmother.
Your grandson is the great-nephew of your brother.
Step father and son
Your mother's new husband is your step-father and you are his step-son or step-daughter.