Who gets the custody of the child if the mother is having an affair?

It depends on the state...most are "no fault" states. This means that an affair has no bearing on the results of the case. Well being is the determining factor. She would have to be proven "unfit" by you to obtain a majority or full custody.