When two parents split up but were never married what type of custody does the mother have by default if the father has a visitation order and pays child support?

It's all up to the courts to decide. A mother can request full sole legal custody of the child even if the father has visitation rights and pays child support. The mother does need to prove to the courts why full sole legal custody is in the child's best interest.