There are many who consider as an injury to themselves any conduct which they have a distaste for, and resent it as an outrage to their feelings; as a religious bigot when charged with disregarding the religious feelings of others, has been known to retort that they disregard his feelings, by persisting in their abominable worship or creed. But there is no parity between the feeling of a person for his own opinion, and the feeling of another who is offended at his holding it; no more than between the desire of a thief to take a purse, and the desire of the right owner to keep it. And a person’s taste is as much his own peculiar concern as his opinion or his purse.
John Stuart Mill, On Liberty, Ch. 4, paragraph 12.
Liberals and libertarians need to think very carefully about the kinds of preferences/harms should be considered valid policy concerns. The are obvious cases: I wrong you when kick you in the shin, but not when I wear clothes you find distasteful. It seems that this is so even when you have a very high tolerance for shin pain and a low tolerance for fashion crimes, and the harm/disutility is equal in each case. Most people find it reasonable that people have a presumptive right not to be physically attacked, but no such right not to be visually offended by poor taste. There is a large grey area in between these two cases. Utilitarianism as a moral theory is incapable of considering this question, or even admitting that it is a problem. This, more than anything else, is why I am not a utilitarian. The Mill of On Liberty was not a utilitarian in this respect either. On some readings, not even the Mill of Utilitarianism was really a utilitarian.
Update: Eric disagrees. Update 2: Not really. See the comments on Eric’s post.