#1 - I believe you jumped into a cycle with this man. (referring to how his last relationship ended)
#2 - If you are feeling these doubts, there is definitely something there causing them. It time to seriously re-evaluate your needs and happiness. Is this guy truly fulfilling your needs?
#3 - If you don't feel secure with your man - then there are major issues to be worked through. If he isn't will to work for it, than that lack of action speaks a thousand words on what he thinks of your relationship.