The most successful Internet companies, it seems, all learned the lesson of Tom Sawyer. By outsourcing labor to their users, it is as if they have cordially invited friends and neighbors to delight in painting their fence. Instagram does not need to hire photographers; users snap, post, and comment on photos, and their hashtags are a filing system, a taxonomy as meticulously curated as a filing cabinet. Instagram’s parent company, Facebook, relies on all of us to keep each other amused, as does the Chinese platform Weibo. Even Google, Amazon, and Alibaba, operating far afield from social media, rely on combined consumers and producers (“prosumers”) to let them know when their algorithms are helpful. Every society depends on free labor—work that is vital but which goes unpaid. They also allow customers to police the quality of products and websites so that they don’t have to pay someone to do that work. Airbnb, Uber, and eBay apply similar methods, shifting the burden of quality control by having both sides in commercial transactions rate one another. We all give up small treasures—our data—to paint the fence of platform capitalists.
We may soon do the same for government. Every society depends on free labor—work that is vital but which goes unpaid. Smart governments realize that they need to strike some balance between market activity and the free labor that supports families and communities. Policymakers promote business and growth, but they also realize that if every moment were commodified, the foundations of social reproduction would wither away. Index funds may prove a better investment than children. And if you don’t get credit for being civil, paying attention in class, or taking care of your aging parents, why would you?
There are standard solutions to such problems. Courts can drain the bank accounts of “deadbeat dads.” Churches and civil society groups can stigmatize deviants, and the carceral state can further scare scofflaws. But these approaches take resources. The perfectly efficient neoliberal state would cut out the middleman. It would learn from Silicon Valley that you can motivate people not only to rate and rank one another, but also to positively enjoy the power and responsibility that rating (and being rated) entails.
The Chinese government is now implementing just such a system at the national level. Called the Chinese Social Credit System (SCS), it has some familiar foundations. Its early iterations (pioneered by private firms) allow users to share images of their scores with one another. As with financial credit scores used by many lenders, the system rewards people for repaying debts promptly. But the SCS does not stop with credit; it factors in court judgments, criminal records, academic dishonesty, jaywalking, moving violations, and failing to pay transit fares.
The Chinese Social Credit System factors in court judgments, criminal records, academic dishonesty, jaywalking, and more.
Surveillance, software, and relatively simple artificial intelligence can supply a fearsomely panoptic dossier. But this monitoring alone does not address the concern of Chinese Communist Party authorities that cornerstones of their authority are eroding. Thus the SCS will also dent your score for posting “unreliable” information or engaging in nebulously defined negative interactions online. Conversely, the system will reward volunteer activity and “filial piety”—devotion to one’s parents, grandparents, and perhaps other relatives. To paraphrase Margaret Thatcher, scoring is “the method; the object is to change the heart and soul.”
How can a government judge the relative value of working in the market versus visiting a lonely aunt? For the architects of the SCS, these spheres diverge: cash rules commerce, and a new currency will govern culture. That currency is reputation, a single score to express a person’s social value. As China’s SCS approaches its full implementation around 2020, the scoring of activities will spread, assigning points for a wider range of antisocial and social behaviors. Eventually China may make a Great Leap to Commensuration, in which every activity (or inactivity) is judged and converted to points, giving lived reality the feel of a never-ending video game.
The Chinese government claims that the SCS simply reflects the values now embodied in Chinese families, schools, and courts. But with no appeal mechanism—a basic aspect of due process in any scored society—the SCS’s relentless logic of commensuration threatens to supplant, rather than supplement, the authority of families, schools, and courts. The SCS could easily end up serving as a quant-driven power grab, enabling its authors to assert authority over vast swathes of social life in a way they could never achieve via legislation. Such quantitative governance of culture is a paradox: the very effort to articulate the precise value of manners and emotions threatens to unravel them entirely, as spontaneous affections and interactions are instrumentalized into points.
•••
The real innovation is the ‘spiderweb’ effect: a misdeed in one area of life can have consequences far beyond it.
What the SCS lacks in legitimacy, it makes up for in efficacy. Aspects of it are already being felt. Millions of people with low scores have been blacklisted from travel. The SCS may also eventually rely on peer scoring, a method patented (though not implemented) by Facebook in the United States. That is, if an activist criticized the government or otherwise deviated from prescribed behavior, not only would her score drop, but her family and friends’ scores would also decline. This algorithmic contagion bears an uncomfortable resemblance to theories of collective punishment.
The real governance innovation in the SCS, however, is what might be called a “spiderweb” or “ripple” effect: a misdeed in one area of life can have consequences far beyond it. For example, imagine a small firm pollutes a river, and environmental regulators find out. In old governance systems, an administrator or court would assess the infraction and decide whether to impose civil or criminal penalties on the firm’s owner or managers. While the outcome could be devastating for an executive, it ended there.
The new algorithmic governance of the SCS would make the pollution a problem throughout the executive’s life. He might be denied the right to fly or to use the Internet—or he may need to pay more to do so. His children might be denied a place in preferred schools. Travel abroad could be rejected outright. Loans might cost more, and access to social services restricted. The SCS’s stated aim is to enable the “trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step.” Every “misdeed” is another step toward the internal exile of “discredit.” Every merit expands the scope of one’s world.
Yet this system also relies on someone defining each and every misdeed and merit. Such a system, as it expands, will eventually run into classic dilemmas of metrics. For example, every objective scoring system must embrace risk adjustment. It must give competitors some compensation for adversity beyond their control and must ensure that the lucky are not given credit merely for circumstance. A hospital in a war zone is going to have a higher mortality rate than one in Beverly Hills. So a risk adjuster tries to imagine what the average hospital copes with, and then adjusts any given hospital’s score up or down based on whether it faced unusual adversity or ease.
Imagine trying to find the “average” for filial piety, one of the Chinese scoring system’s cornerstones. At what point does a nagging father deserve a shortened visit from an exasperated son? Should one of three children have less obligation to spend time with her parents than a singleton? How much more does a rich person owe to his parents compared to a poor person? Each question—and countless more—will need to be answered in a quantifiable and standard way.
To the extent that the system is a black box, there is no way for the victim to find out about the defamation.
Without having even reached this dilemma, however, there is early anecdotal evidence that the SCS may be failing on its own terms. For example, a bank may submit false information to blackball its best customer, in order to keep that customer from seeking better terms at competing banks. To the extent that the system is a black box, there is no way for the victim to find out about the defamation.
This basic concern about data quality and integrity undermines arguments, such as the one posed by the Financial Times, that “Chinese AI companies, almost wholly unfettered by privacy concerns, will have a raw competitive edge when it comes to exploiting data.” If guarantees of due process are limited or nonexistent, how strong can promises of data quality and integrity be? For true believers in big data, the signal of ever larger data sources will eventually drown out the noise of gamed and manipulated feeds; critics caution there is little basis to believe so.
Legal scholars in Hong Kong are already raising concerns about the SCS’s accuracy and fairness. It is crucial to note, however, that if reformers focus only on legalistic concerns, their push for algorithmic accountability may miss the forest for the trees. The commensuration at the heart of the SCS is an irremediable flaw. In Spheres of Justice (1983), Michael Walzer defined tyranny as the domination of one mode of distribution in realms where it does not belong. It is easiest to see this domination in the realm of money and illicit commodification. If a billionaire wants to buy a beautiful handcrafted couch, few mind. However, we rightly object if the same billionaire wants to buy an election, or be first in line to receive a donated kidney, or purchase a license to travel at twice the speed limit. All those rights and privileges are supposed to be governed by some other standards: democracy, or medical necessity, or strict equality. Money becomes tyrannical when it overturns the standards of independent fields.
Algorithmic governance such as the Chinese SCS allows the government (and its private partners) to consolidate and thus dominate spheres of reputation that should be decentralized and private. It is prone to a tyrannical granularity, a spectrum of control—following Gilles Deleuze’s critique of “control societies”—that is more exacting and intrusive than older forms of social order. The Chinese SCS allows the government to dominate spheres of reputation that should be decentralized and private. The Chinese SCS both uses and encourages data feeds, encouraging a spiral of surveillance. One school aspires to so minutely judge citizens’ lives that not even evanescent emotions are safe from scrutiny. In an experimental “intelligent classroom behavior management system” described by Business Insider, cameras scan classrooms every thirty seconds and record “students’ facial expressions, categorizing them into happy, angry, fearful, confused, or upset . . . [as well as recording] student actions such as writing, reading, raising a hand, and sleeping at a desk.” There are no limits to the regimentation a scoring system might produce. Marketed as “affective computing,” these methods could easily compute optimal affect, prescribing expressions and thoughts to match them.
•••
Sadly, the rush to monitor and measure goes well beyond the SCS. A global educational technology industry has pushed for behavioristic testing and ranking of students, schools, and teachers. The same monitoring technologies may also dominate hospitals, nursing homes, and daycare facilities. Wherever there is “soft” care to be done, untrammeled by the Taylorist impulse to measure and manage, methods like the SCS may spread. Reputational currency is a way to rebrand repression as rational nudging.
Yet if we are worried about failures in this circuit—from children raised on YouTube videos to teachers who cheat to get an edge on high-stakes exams—the answer is not to double down on performance-based ranking systems designed to shame supposed shirkers. Rather, it lies in paying good wages to teachers and caregivers, merging the spheres of society and economy that schemes such as the SCS split. Leaders must put their money where their mouths are, rather than guilt-tripping or penalizing their subjects into a zero-sum rat race for reputational currency.
Caregiving, which is notoriously underpaid labor, is especially vulnerable to neoliberalism run amok.
Caregiving, which is notoriously underpaid labor, is especially vulnerable to neoliberalism run amok. Mike Kelley, an artist and sculptor, illustrated the discomfort society often feels when it bothers to stop and think about what it might mean to “repay” those whose work is to love and care for us. In More Love Hours Than Can Ever Be Repaid (1987), one of his eeriest and most affecting pieces, he assembled a wall hanging out of knitted afghans and handcrafted stuffed animals that he had purchased at thrift stores. The handicrafts exist in an uncanny netherworld. Care was poured into them, feeding affections and cathexes. They meant enough for someone to stitch them together and give them away—but not enough to be kept by the recipient.
To some, the patterns for these stuffed animals and shawls serve as a cliché of the culture industry, plied by Reader’s Digest or First as the soft, cloth equivalent of a Betty Crocker cake. But they were channels for the expression of love. The quantification of “love hours” flattens the texture of real relationships, just as the wall hanging distorts three-dimensional stuffed animals into a flat medium. The viewer’s discomfort is only alleviated by realizing that social provisioning, rather than interpersonal emotional debit accounts, can cure whatever “debts” the stuffed animals evoke. Indeed, if the state wants something done, let it pay. The real economy can then put some constraint on the quantitative governance of culture, behavior, and thought itself.
