Ask yourself what gives you the right to have custody of the children? The best answer for ALL concerned is to talk it over with your husband and come to an agreement. With this you can then get the agreement written up by lawyers. If you do not feel comfortable talking to your husband about this by yourself I suggest you both meet with a third party - say your lawyer. I suggest you get to a court ASAP and file a custody order, before he does. The first one who files normally has a better shot at winning. And if you can prove abuse, file that also. This is a hard question to answer so you need to act fast.
You could lose your child, if you have joint custody. He can state that you kidnapped the child. I would suggest going to get custody or at least get in contact with a lawyer and see what they say. If there was abuse, usually the justice system won't get involved until someone actually gets hurt. As sorry as that is.
You need to seek legal counsel ASAP, as the laws differ from region to region, country to country.
I was in that very situation about three years ago. By taking my daughter with me, insisting she sleep where I was staying, and controlling the visitations with her father, I was able to set the status quo. The courts, generally, are reluctant to go against the status quo, unless there is some form of concrete proof that it is in the child's best interest.
Just remember, going through a divorce is not the child(ren)'s fault...don't make them suffer! As separated/divorced parents, the ONLY question should be, "How can we be the best co-parents?" Should you go through the entire process of a trial, keep that in your mind as you ask/answer every question. And if your spouse is unable to do the same thing, all the better for you.
If you leave your husband and take your child with you, you better hurry and get temporary custody of your child legally. I lost my son because I left without filing the proper paperwork in court to give me temporary custody. He took him from me during a visitation and I now only get every other weekend and pay 300.00 child support. All because I didn't get a temporary parenting plan. Unless one or both parents have court papers showing custody, either parent can legally take the child. If you don't have papers, your husband could pick your child up from school and there isn't a thing you can do about it. If you call the police requesting visitation or the return of the child they won't do anything unless you have papers in hand. If you want custody, get it through the family court.