What are the rights of a father once he signs the adoption papers if the mother decides to keep the baby?

None, the voluntary relinquishment of parental rights is permanent. If a couple are not married the mother is considered to hold full and sole custody of said child with or without the means of a court order, until/unless a court rules otherwise.