Can the mother take the children out of state if the parents are not married and there has been no custody appointed?

If the father of the children has not paid child support and showed no interest in this particular manner (not just good enough he shows up at the door to see the children) then yes, the mother has every right to move where she wants. If the father has been paying support and can prove it, then it's best to seek legal counsel and he should get part custody of the children.