I think you should let it go. You don't seem to be the right match for each other.
Read what you wrote: "The reservations were basically because we are very different people...He did something I didn't like and I was triggered....He was quite angry at me and I got scared of his reaction....I've had a relationship in the past where we use to fight a lot and I guess that was still in me, because I got really scared and got into a "fight or flight" mode."
You seem to have this pattern in you still you should solve it by being on your own or dating emotionally healthy man.
It's not fair that I tell you this because I'm stuck with this pattern too, so I know it's easier said than done.
