What rights does the father have if the mom is in jail and they had never been to court for custody before?

Generally, if the parents are unmarried the mother has sole custody and control in most states until the father can establish his paternity. Remember, a child's mother can always be identified by medical records. Since the father didn't give birth and he was not legally married at the time of the birth he must seek other means of establishing his paternity and that is done through paternity test. A paternity test can be arranged through the court and once established the father can request visitations, custody or set up child support for the child.
Generally, if the parents are unmarried the mother has sole custody and control in most states until the father can establish his paternity. Remember, a child's mother can always be identified by medical records. Since the father didn't give birth and he was not legally married at the time of the birth he must seek other means of establishing his paternity and that is done through paternity test. A paternity test can be arranged through the court and once established the father can request visitations, custody or set up child support for the child.