It is the Mirena. Many of us here (including myself) thought we were going crazy. It doesn't help when doctors agree.
Have your hubby come here and read posts with you. He'll see first-hand that the two of you are not alone, and that things do get better, although sometimes gradually.
I am at 4 weeks, 1 day post removal, and most of my symptoms are gone. I am still having inconsistent sleep, and some moodiness recently (my period should be coming soon-- although that's unpredictible). I have a sinus infection, which isn't helping right now, but even with that, I feel better than I did six weeks ago.
Hang in there!