On the map screen, stand on a defeated castle or fortress, and press L and R simultaneously to re-enter it.