No, it can not. Your castle is unique and linked to your account. You can not lose your castle through attacks or raids. It can not be downgraded. When you change position, your castle will always come with you.