As far as I know, a breakout does not have to be visible for the herpes virus to be contagious. Viruses are much smaller than bacteria and I believe that they can pass through a condom also.
I've read here and there that as long as you're not engaging in skin to skin contact a few days before a breakout and a few days after it heals, it's relatively safe. But, I still think that the possibility of transmission still exists (and for someone who is negative for the virus, that's too great).