Old town, Peninsula, Castle