"Maybe I made it sound worse than it really is, but something is telling me that he will get really weird if we break up."
I was raped by a boyfriend who was a lot like the man you have described. He raped me the day I broke-up with him. He never hit me, nor sexually assaulted me before that day. I too felt a weird feeling with him during the relationship, he was possessive, didn't want me to spend time with friends, questioned me, and didn't let me break it off gently (earlier in the relationship), I also tried to get him to break-up with me to no avail. I broke-up with him outside during the day at a busy park. He said "okay, but can we still be friends?". I said yes, he asked me to go watch a movie with him at his house, I went, on the way I felt like I was in danger. I was. He raped me at his house, never saw a movie.
You need to trust your feelings here, if you feel he may get crazy (and some men do, not to scare you but a possessive man who feels overly dependent on a women may try to kill her or injure her in response to the treat of loss) then you should get the proper help/ back-up. If you contact "1-800-799-SAFE" you may get opinions on how to break-up safely. (I did not do this b-4 I broke it off) If he is not dangerous you have really only gained a skill in safe break-ups by calling, if he is dangerous you will have taken steps to protect yourself.
Distancing yourself form this guy is a great idea. Don't try to be friends first, and DO NOT go ANYWHERE with this guy alone after or during the break-up, nor should you let him in your house after (or during) the break-up. Really stay away from him for awhile after the break-up. Let him get over you without you around. Try to have many friends around you (in the immediate area) when you break-up, and do so in a public place. Go home with friends. Don't let him know when you will be home if you live alone. Get advise from professionals on the best way to handle this.
Again, it may be nothing, maybe he won't react in this way. But if he does, you must be prepared. Also, that safe # can give you help if you think he may hurt himself.