But I have been walked in by spirits that tricked me in giving permission and even by spirits that walked in without permission at all.

You might be right that you are in the end always in control because if i'm not mistaken you can always take back your body if you really want to (I was once walked in by a Succubus and she rhetorical asked me - while she was in my body and I was outside of it - if I maybe had plans to take over her body because it took me so long to reclaim my own body).