To get to Oryx's castle, you have to kill every monster listed here:

~Lich

~Ent Ancient

~Oasis Giant

~Phoenix Lord

~Ghost King

~Cyclops God

~Red Demon

~Cube God

~Skull Shrine

~Pentaract

~Lord of the Lost Lands

~Grand Sphinx

~Hermit God

~Ghost Ship

If all of these monsters are killed (They're spread across the realm) Oryx makes an announcement about him closing the realm. After two minutes after said announcement, the realm will start to shake, and everyone in the realm will automatically go to Oryx's Castle.