The defense of free speech often hides a multitude of sins. Since Brandeis University withdrew an honor it had intended to bestow on the author and activist Ayaan Hirsi Ali, many have flocked to her defense in the name of free expression — no matter how offensive. But implicitly they are suggesting that Islam and Muslims are worthy targets of Ms. Hirsi Ali’s scorn. And their preciousness about the right to offend won’t be credible until they advocate extending it beyond Islamophobes — to racists, anti-Semites and homophobes, too.

Ms. Hirsi Ali is no casual critic of Islam; she has built a career and brand railing against what she calls “a destructive, nihilistic cult of death.” She has even come perilously close to justifying the Norwegian mass murderer, Anders Behring Breivik, whose killing spree, according to her, was a last recourse because he felt he had been “censored” by “advocates of silence” — a nebulous group that she insists promotes a dangerous mix of multiculturalism and tolerance of Islam.

Brandeis stated that her planned address didn’t share the “university’s core values” and rescinded an honorary degree; the university’s volte face may have been clumsy, but it wasn’t censorship. In the eyes of Ms. Hirsi Ali’s supporters, however, Brandeis was “kowtowing to the Muslim hordes” and giving in to the pressure of Arab money.

It’s not entirely unreasonable that a liberal arts institution would view this sort of language as discordant with its values. After all, academic institutions are cultural battlegrounds, and they set the tone for contemporary discourse. But the accusations leveled at Brandeis show the perils of not sticking entirely to free speech absolutism.

The university appeared not to have done basic research on Ms. Hirsi Ali’s rhetoric and how strong a reaction she often provokes. She skillfully rebranded the incident as an attack on free speech and an attempt to silence her. This summoned forth a panoply of voices coming to her defense; some went so far as to claim that she’d been metaphorically “honor killed.”

Couching Ms. Hirsi Ali’s defense in the derision of Islam is troubling — and it exposes how selective champions of free speech can be. In a rather hysterical response in The Daily Beast, James Kirchick peppered his defense of Ms. Hirsi Ali with references to Muslim detractors as one step away from terrorists, and drew a tenuous line between not granting someone an award and endorsing murder. He wrote that “forcing a university to rescind its honoring of an acclaimed critic of Islam exists on a censorious continuum that ends with the dismal fate of individuals like Theo van Gogh.” In Mr. Kirchick’s world, petitions are fatwas, and it’s only a matter of time before leafy university campuses are littered with the corpses of academic jihad’s victims.

Hysterics aside, broad generalizations like this suggest that there is no Muslim mainstream made up of people who have the right to object to, and fear, language that stigmatizes them; there are only terrorists and their victims. The implication is that because some Muslims have a record of violence toward critics and apostates, that all Muslims have it coming to them. It’s an argument that boils down not to, “because of freedom of speech” but “because they deserve it.”

Swapping races and religions to gauge if the response to a particular incident would have been different is an imperfect counterfactual game, but in this instance it is instructive. Had Ms. Hirsi Ali been a widely acknowledged homophobe, or white supremacist, would free speech supporters have rushed so readily to their lecterns to defend her? Probably not, which is why the right to offend should be extended to all. Otherwise, our personal preferences will always dictate that there be exceptions.

It is clearly far more palatable, even popular, to muscularly stand up for the right to offend Muslims than it is to back those who offend any other minority in Europe today. Indeed, when the notorious American Islamophobe Pamela Geller was banned from Britain on account of her vitriol toward Muslims, her exclusion was met with a chorus of objections. This selective attitude toward freedom of speech allows such disparities to become entrenched.

The reaction to the Brandeis affair is a troubling harbinger. It suggests that America, like Europe, might also begin to pick and choose who deserves to be protected from offensive speech. Once that door is open, the Trojan horse of libertarianism will smuggle in intolerance.

Those who fancy themselves defenders of free speech must be consistent in their absolutism, and stand up for offensive speech no matter who is the target.