While there is good evidence linking dairy intake to an increased risk for both prostate and ovarian cancers, this link for breast cancer has oddly not been made.
One of the mechanisms proposed for dairy in it's association with an increased risk for prostate cancer is the stimulation of insulin-like growth factor of IGF-1.
Elevated IGF-1 levels are also associated with an increased risk for breast cancer so it would seem that dairy should increase the risk for this cancer as well. For some reason, large meta-analysis have not proven this association.
One of the reasons that breast cancer risks do not appear to go up with dairy intake while ovarian and prostate cancers risks do, could be due to conjugated linoleic acid or CLA.
CLA is naturally occurring in fatty meat and dairy products. CLA appears to have an affinity for breast tissue and might be protective against cancer.
In general, high animal fat intake in diets is associated with an increase in all kinds of cancers including breast. Dairy might be the one exception.
I am not urging anyone to go out and start drinking glasses of milk, remember ovarian cancer risks increase with an increase in dairy consumption. I am encouraging those who have the occasional yoghurt or cheese to eat it full fat and preferably organic or at least from grass fed cows (higher CLA content).
I think dairy consumption can be problematic for many people and for many reasons, but for some, it is digested well and not antigenic. If you're going to partake of dairy consumption, make it high quality.
If you're a vegan or can not tolerate dairy, CLA is available as a supplement. One of the main reasons CLA is taken as a supplement is to reduce visceral and subcutaneous abdominal fat.