Your self-observations are excellent. You are correct, it is very difficult to overcome AF when you have a severe yeast infection, and almost impossible to recover until you address the cause, which may be a yeast infection, and in some cases it is. I have written a great deal about this very topic in my book, and consider the relationship between AF and candida a "missing link" in recovery of one or either these conditions.
Most all chronic yeast infected patients have AF. But not all AF patients have candida. Strong adrenals can turn a chronic yeast infection into a condition which can certainly recover fast, and I have witnessed this on many occasions in the clinic. But it takes a a balanced approach to get the adrenals right (just like candida). I've always maintained that the best approach is the 80/20 approach. Eighty percent of the focus is lifestyle and diet, twenty percent is with therapeutics (taking stuff). And, eighty percent of lifestyle balance is looking at factors such as overwork, not enough relaxation, poor sleep, emotional stress, relationship hassles, etc. And twenty percent is the diet.
It seems a lot of folks on forums always look at "taking stuff" and "eating the right stuff" but then seem to get side-tracked and don't concern themselves a great deal at all with their emotional state, if the chill enough, exercise (to suit their body type), reducing their anxiety levels, etc. Adrenal fatigue recovery is very dependent on slowing your life down a great deal and learning to say "NO" to people. It may also mean cutting relationships with toxic people and even quitting a job because of a toxic boss or co-worker (more commonly required than you may think). It may mean stopping studying while you try to earn a living when you have two kids, saying "I can't to caring for a sick and elderly relation when you are sick yourself, etc.
"Life is what happens while you are busy making other plans" - John Lennon.