No, the fortress can never be destroyed.