Can a 17-year-old who has moved out of his parents' house enroll himself into school while living with family members in South Carolina?

That depends upon the policy of the school district. It will at the very least require the adult(s) with whom the minor lives to have a notarized document from the parents giving them permission to oversee the minor's affairs but generally a permanent custodial order from the court is necessary.