Once a private playground for royalty, there were fears the dilapidated island fortress would be left to crumble.