I haven't had a problem with dairy. I don't have much of it, and have noticed no ill effect. I'm making clear progress with the candida, so if the lactose is a problem, it isn't much- I'm winning anyway! ;)
This may be one of those individual things about the diet, if you notice a problem, then listen to your own body.