Step away in your mind and ask yourself with him this is what I get and without him this is what happens?
The answer to how this works out is down to you and whatever happens is the right answer. After all you have shown that you have the strength to walk away at any time.