When it comes to money in politics, Democrats and Republicans are more than just polarized—they are not even in conversation. Consider last week’s news. At the Conservative Political Action Conference, attendees were treated to an extensive dissection of the IRS, which has been the favorite target of Republican activists since it was revealed in 2013 that the IRS had improperly used political criteria to evaluate applications by advocacy organizations for tax-exempt status. Meanwhile on the left, the Senate Democratic leadership launched the latest in a long-running series of attacks on the Koch brothers and the Koch-funded 501(c)(4) Americans for Prosperity.

I cite these examples not because they were last week’s top stories, but because they barely seem relevant to those on the other end of the spectrum. To most liberals, the IRS affair is a non-event that wrapped up in a few months last summer; to conservatives, meanwhile, attacks on the Kochs were played out by 2010 if not before. Left and right aren’t simply applying different sets of values to current events; they are using very different definitions of what counts as news.

In an ideal political world, this is where neutral arbiters—journalists, nonpartisan policy shops, watchdog groups, and the like—would step in, enforcing adherence to the facts and laying down the basic guidelines that make civilized debate possible. To use Chief Justice Roberts’ famous analogy, these are the people and organizations who call the “balls and strikes” for politicians and advocates. This role is increasingly difficult to fulfill. The age of Walter Cronkite—of mediators who had the authority to lay down “the way it is”—is long gone, in favor of a media environment that, while not nearly as polarized as political elites tend to believe, is increasingly fragmented.

The most consequential and controversial questions in politics are not amenable to fact-checking.

Still, the proliferation of fact-checking services, like FactCheck.org and the Tampa Bay Times’ PolitiFact.com is some indication that neutral arbiters can adapt. While skeptical observers point out that fact-checkers obviously haven’t eliminated falsehoods from politics—something they can hardly be expected to accomplish by themselves – their effect is real and measurable. Political scientists Brendan Nyhan and Jason Reifler have demonstrated that simply reminding state legislators that fact-checkers (in this case, PolitiFact’s state affiliates) are watching decreases their propensity to stretch the truth.

Unfortunately, many of the most consequential and controversial questions in politics are not amenable to fact-checking. These debates revolve not around whether a statement is right or wrong, but whether it is important or trivial. Separating crucial issues from distractions is a form of political debate in itself—and consequently not something above-the-fray third parties can adjudicate.

This is precisely what is happening in the examples I describe above. Liberals and conservatives look at money in politics through wildly different lenses; consequently, they pick out different features of the money-in-politics landscape as worthy of news or public concern. For many progressives, the excessive influence of corporations and the wealthy is a first-order threat to democracy, and the Kochs are an ideal example: to mention them is to invoke everything wrong with American politics, making them a kind of metonym. Likewise, for conservatives, the IRS story is at the confluence of the Obama administration’s most dangerous attributes: lawlessness, government overreach, and the regulation of political speech.

Fact-checkers and other arbiters have done what they can on these subjects, highlighting and dismantling some of the most straightforwardly incorrect claims, such as Rachel Maddow’s tenuous attempt to tie the Koch network to Florida’s law mandating drug testing for welfare recipients, or Mike Rogers’ (R-MI) unsupported claim that the White House is behind the IRS’ targeting of conservatives. But in both cases, the symbolic value overrides the accuracy of the claims: whether or not the Kochs actually have the influence imputed to them by the DSCC or the IRS scandal actually went straight to the highest levels of government is secondary. For partisans who find the Koch/IRS stories compelling in the first place, debunking specific claims is unlikely to make them waver. The idea, after all, is that these stories are just particularly clear instances of something far-reaching and substantively important; if power-mad bureaucrats or right-wing plutocrats are mortal threats to the republic, then disputes over facts look more like quibbles than refutations, since there are always more examples where those came from.

The point is not that left and right are equally hyperbolic and inaccurate in their rhetoric, nor that these are questions of “values” that aren’t amenable to factual debate. The point is that the way that people think and argue about politics is thoroughly shaped by their understanding of which features of the political world are fundamental. Liberals order their thinking around one phenomenon and conservatives another; the Kochs and the IRS are just shorthand.

One way to resolve this tension is to say that all narrative-making of the kind described above is unjustifiable—the product of ideologues forcing facts to fit their prejudices. This posture of objectivity is what media theorist Jay Rosen, channeling Thomas Nagel, mocks as the “View from Nowhere.” Of course, without heuristics or preconceived notions, journalists and other arbiters—let alone ordinary voters—could never sort through vast quantities of political information and spin to identify the important events and trends. That is, broad political engagement requires slogans, summaries, and appeals to shared understandings of the world. Otherwise, participation would be reduced to technical debates between specialists. Some slogans are acceptable and some are misleading, but the way to separate the two is through political argument—in other words, debate about what matters.

Neutral arbiters try to see through these slogans, and to avoid such uncomfortable debates. In fact, many join the debate without acknowledging it by advocating for ostensibly centrist “solutions,” applying the heuristic that the truth must lie somewhere between left and right. Brendan Nyhan cites as an example reporters who suspend their rules of objectivity to scold both Democrats and Republicans for failing to reduce the deficit. Journalists, too, make it their work to build narratives on underlying ideas about which values are important (in this case bipartisanship, deficit reduction, and compromise for compromise’s sake, among others). At least liberals and conservatives are typically more explicit about the ideas underpinning their activism.

Political commentators have a tendency to blame polarization on structural features such as gerrymandering and closed primaries, but this is not borne out by the evidence. If that were the case, process reforms could do a great deal to reverse polarization, but no such luck. Instead, left and right will continue to be separated by a chasm for the foreseeable future and will resolve disagreements by argument, power politics, or not at all. Meanwhile, neutral arbiters can and should do their best to fact-check those arguments and police political competition, but there is little they can do about the fact that liberals and conservatives have drastically different ideas about what matters. Their jobs will be easier if we acknowledge that.