My wife realized a couple years ago that a couple of her "friends" were bringing her down by their constant negativity and problems. Once she stopped being around them, her life brightened tremendously.
Yes, it could be some inherent weaknesses in my wife's being that made her more affected by those friends. However, you won't be able to "look in the mirror" correctly if your spirit is under constant bombardment. Better to free yourself from the immediate threat, find a safe area, and work on those things one needs to.
What I saw in your post I would liken to a swimmer being pulled under the water by a weight. Certainly you could say it's the swimmer's fault for not being strong enough to swim with the extra weight. I say cut the line, swim to safety, and keep practicing the swimming under better circumstances. No reason to drown if you don't have to.