Castle of S.Giorgio, Ducal Palace, Castle stronghold