Last Friday, a group of white supremacists marched on the University of Virginia (UVA), spreading a message of hatred and igniting the spark for what would turn into a weekend full of violence and terror. The group of Neo-Nazis carried torches and chanted, “You will not replace us” and “Jews will not replace us.”

White America’s fear of losing what it perceives to be its own is not new; since the United States’ inception, white men in particular have been moved to violence by the threat of receiving anything less than all this country has to offer. It’s not that white men have ceased to be overrepresented in political, economic, and social spheres of power throughout the country; they still have a mighty stronghold on them all. It’s simply that they fear what Luke Charles Harris, co-founder of the African American Policy Forum, calls “diminished overrepresentation.”

This is also not the first time that a college campus has been used as a battleground for accusations of anti-white discrimination. The movement to dismantle affirmative action policies—which permit the consideration of race, sex, and other criteria in an effort to provide opportunities for historically oppressed groups, especially in relation to employment or education—has been waging war against colleges and universities employing such policies for the better part of the 21st century. Though the anti-affirmative action movement bears no torches or Nazi regalia, its agenda is very much aligned with that of the white supremacists who marched at UVA: namely, to dismantle all legal protections put in place to advance racial equity after centuries of state-sanctioned violence and oppression. While protesting Charlottesville, social justice advocates must also vehemently defend affirmative action and vigilantly examine the connections between racist violence at all levels of society.

At the center of the movement to dismantle affirmative action is an unlikely character: Edward Blum, a 65-year-old former stockbroker who has made it his life’s mission to undo laws aimed at combating racial inequality, from voting rights to education to employment. His efforts have resulted in six U.S. Supreme Court cases. Born to a left-wing Jewish family in Michigan, Blum turned to neoconservative legal activism after losing a Texas congressional race to a Democrat in the 1990s.

In 2007, Blum launched the website UTNotFair.com in the hopes of finding a plaintiff to challenge affirmative action policies at the University of Texas (UT). Shortly thereafter, he found Abigail Fisher, a young white woman from Texas who had been rejected from UT Austin. Not a lawyer himself, he pulled together a legal team to represent her, and together, they built a case against the university. The prosecution team saw Fisher’s rejection from UT Austin as a direct result of what they characterized as discriminatory admissions policies that favored applicants of color. In June 2016, the Supreme Court ruled against Fisher, upholding the constitutionality of the race-conscious admissions program at UT Austin.

Now Blum is giving it another go. Back in 2014, Blum launched several nearly identical websites through his organization, Project on Fair Representation, in an attempt to find high-achieving Asian-American students who had been rejected from competitive universities throughout the country and file lawsuits on their behalf. The three websites, HarvardNotFair.org, UWNotFair.org, and UNCNotFair.org, all feature photographs of dejected looking Asian-American students alongside text reading, “Were you denied admission to [this university]? It may be because you’re the wrong race” or ethnicity.

This tactic is not new. For decades, conservatives have used the Asian-American community and the “model minority” myth for their own gain. By rebranding the face of affirmative action, Blum hopes to demonstrate to the United States and to the Supreme Court just how unjust and unconstitutional affirmative action supposedly is, not just to whites but to everybody. By making the case that affirmative action is hurting non-white people as well as white people, he is distancing anti-affirmative action advocacy from the rhetoric of reverse-racism, which has served him poorly in the past.

In 2014, Blum’s organization, Students for Fair Admissions, an offshoot of Project on Fair Representation, officially filed suit against Harvard University. The suit accused Harvard of employing an unconstitutional quota system in their admissions processes that made it significantly harder for Asian and Asian-American students to be admitted than other students.

The lawsuit likened the plight of Asian-American applicants to that of Jewish students in the early 20th century, when Harvard and other elite institutions employed quotas to prevent too many Jewish students from being admitted.

What the lawsuit fails to mention is that these anti-Semitic quota systems predated affirmative action by several decades. Rather than aiming for equality of opportunity, these systems were rooted in racial animus and a desire to keep the student population as Anglo-Saxon and white as possible (Jews were largely not considered white back then). Harvard and its peer institutions were not attempting to equalize opportunity for applicants of all backgrounds; Black Americans and women were not even permitted to enroll. There was nothing remedial about their policies, only discriminatory.

The elimination of affirmative action in the United States would primarily benefit white men. White women, who have historically been among its fiercest opponents, are in fact a primary beneficiary of affirmative action. Affirmative action policies in institutions of higher education and in workplaces throughout the United States have paved the way for white women to rise to positions that would have otherwise remained unattainable to them.

The elimination of affirmative action would also disadvantage the vast majority of the Asian-American community, according to Eric Liu for TIME. He writes, “The ethnic and socioeconomic diversity within Asian America is usually overlooked in the media. Great numbers of Asian Americans do not fit the model minority or ‘tiger family’ stereotypes, living instead in multigenerational poverty far from the mainstream … [Their situation] reveals how the current debate is too narrowly focused on the elite.”

While white women and all people of color would stand to lose a lot if affirmative action were dismantled, Black communities would be hit the hardest. The policies that are in place now are by no means sufficient to level the playing field, but the thought of losing them altogether is catastrophic. In states where affirmative action has been eliminated altogether, we’ve seen enrollment numbers for Black students in institutions of higher education drop significantly.

There can be no plainer proof that post-racialism is an utter fallacy than what’s happening now in Charlottesville, where white supremacists terrorize and murder while the White House claims equivalency between the two sides. Despite what white America likes to tell itself, race-based protections are needed in 2017 as much as they were in 1967. We cannot allow Blum’s framing to manipulate us into viewing college admissions and upward mobility as a zero-sum game for marginalized communities. And we cannot separate the violence in Charlottesville from the structural violence inherent in efforts to maintain whiteness at our educational institutions. If we do, an already bad situation will turn even worse.