You can only abandon an Fortification of your alliance in case you are at least a lord. Click on the Fortification and select the "Abandon" option.