I agree. But there are some health concerns associated with drinking human milk, which wouldn't be an issue with cow's milk. There are also a lot more unknowns.
And even though there is a double standard, I think 'grossness' in itself is a pretty weak reason not to do something. Health, taste, and so on, are a lot more convincing.