How can I prevent my ex husband from having any type of custody rights of our child?

Only if you can proove with legal records that he has abused others, drug use, and or abusive behavior twards you and or the children. If not then no you have to deal with him until they are 18. Unless he wants to sign his parental rights away.