I recommend to you Steve Horwitz's lucid post on anti-racism and libertarianism. A choice excerpt:
What I am interested in is the claim that those who stand in opposition to racism are being accused of being susceptible to using the state to somehow enforce that set of beliefs. First, as Roderick Long argued a few years ago in his “One Cheer for Political Correctness” essay, there's nothing inherently unlibertarian about recognizing the existence of structural racism/sexism etc. nor about standing up and loudly opposing it through non-coercive means. Will Wilkinson offers a different version of a similar theme in the context of the Paul newsletters. Second, throughout the long history of the West and the rest of the world, those who believe in the fundamental inequality of the races and/or believe that “like should stay with like” have been far more willing to use the state to enforce those views than those who have opposed them have.
Yes, legislation like the Civil Rights Act of 1964 involved some interference with private property and the right of association, but it also did away with a great deal of state-sponsored discrimination and was, in my view, a net gain for liberty. In the longer run, it seems quite clear that classical liberalism/libertarianism has sided with the opponents of racism and that those who viewed the races as unequal were much more likely to use the state to enforce that view than were those who saw the races as equal. To suggest that anti-racism libertarians are somehow secret statists because opposition to racism must necessarily lead to state enforcement of those views is both a distortion of the actual arguments people like me have made and flies in the face of a long history of libertarians being both anti-racist and anti-state.
This is our heritage as classical liberals, and it long predates the Old Right of the early/mid 20th century, with its very mixed record on race/ethnicity issues, as a source of inspiration for not just the overall spirit of libertarianism, but its perspective on race in particular.
Right on. If you haven't, please do read Roderick Long's smart essay on political correctness. In my long experience among conservatives and libertarians, one very often encounters a kind of smug, thoughtless anti-PC or counter-PC of the kind Long ably describes. The thing to keep in mind here is that most PC episodes mocked and derided by the right are not state impositions. They are generally episodes of the voluntary social enforcement of relatively newly established moral/cultural norms. Sometimes this process squelches or chills honest discussion and inquiry. I feel that way about the Larry Summers flap. But such mechanisms of indignation and shaming are precisely how societies coordinate on and defend norms of civilization without resorting to force.
Counter-PC attitudes often seem to me to be really expressions of resentment that a new more egalitarian norm has displaced the old racist or sexist norm, which is why devil-may-care un-PC pronouncements about race and gender often really are just thinly veiled expressions of racism and sexism. Anti-PC types folks often cast themselves as especially “brave” for being willing to speak the truth as they see it. But this is often nothing more than the imagined boldness of behaving indecently among decent people. If you really do believe in the equal freedom and dignity of people, you tend to act like it, and if not, not.
I think the really ironic thing is that it is only when it comes to concerted opprobrium aimed against racism, sexism, etc. do many anti-PC types seem to recognize the possibility that non-coercive collective attitudes can be stifling to a sense of independence and the exercise of autonomy.