If you do not feed your cows long enough, do not brush them, or leave them out in the rain, they will get sick (shown by a purple particle effect) and no longer produce milk. You need to give them Animal Medicine to cure them.
If you want Vanilla Milk, you will have to trade some of your Harvest Festival milk with Johan in the General Store.
Cows give larger, more valuable milk the more they like you. You can improve the relationship by petting them once per day, letting them out when it is sunny, keeping them inside when it is raining or snowing, and feeding them fodder by hand (instead of using a trough).
It is important to note that relationships are per person. If playing in multiplayer, if you want large milk when you milk the cows, you will have to personally raise your relationships.