What custody rights does a illegal father have if he was never married to the mother who is a citizen?

Immigration status has no bearing on issues such as parental rights and responsibilities in the United States. What does matter is prevailing law where the child legally resides. Since you have not provided the state (or country) in question, an accurate answer cannot be provide.