Who refused to secede from federal union?

Four slave-states of the Upper South - Kentucky, Missouri, Maryland and Delaware.