Getting rid of stuff can be difficult but living with it is even worse! It was sooo hard for me to get rid of my clothes. I work at an office in Manhattan and had some pretty nice and expensive stuff. The worst was the handbags :( I actually found mold growing in a couple since the majority of my mold problem was in my closet. It's rough to be a girl and throw out all your pretty things :) BUT - better my handbags than whatever health I have remaining. I used to get the same reaction from my clothes FYI - An itchy achy, almost warm, feeling. Exactly as you said it, like poison.
Now I feel like I can smell mold anywhere. I will actually walk into a store and smell it right away and I have to go outside. Sometimes my friends think I'm a little crazy but oh well :)
Good luck with your appointment with specialist today!