Gamifying Patriotism
December 16, 2015 1:34 PM Subscribe
The crew at Extra Credits take a break from their usual videogame-focused content to provide a video overview of Alibaba's gamified Sesame Credit system. The system is one of eight government-overseen pilot programs to establish a "social credit score" for Chinese citizens.
While the system is opt-in for now, by 2020 at least one such system will become mandatory for all citizens.
Previously on MeFi.
While the system is opt-in for now, by 2020 at least one such system will become mandatory for all citizens.
Previously on MeFi.
Black Mirror is not supposed to be a how-to, dammit.
posted by Artw at 1:43 PM on December 16, 2015 [15 favorites]
posted by Artw at 1:43 PM on December 16, 2015 [15 favorites]
Controversially, the company does not hide that it judges the types of products shoppers buy online.Consider your purchases carefully, citizen! In your buying habits The State can see your every weakness and vice.
"Someone who plays video games for 10 hours a day, for example, would be considered an idle person, and someone who frequently buys diapers would be considered as probably a parent, who on balance is more likely to have a sense of responsibility," Li Yingyun, Sesame's technology director told Caixin, a Chinese magazine, in February."
posted by Existential Dread at 1:46 PM on December 16, 2015 [1 favorite]
Haha, fooled you, those diapers were for me so I could play my MMOs all day without having to get up!
posted by kmz at 1:50 PM on December 16, 2015 [43 favorites]
posted by kmz at 1:50 PM on December 16, 2015 [43 favorites]
Related: recent comments on the present and future of China's internet, from the head of China's Cyberspace Administration, a former journalist named Lu Wei:
And China is one big, happy family, and everyone must be a good child of the Mother Land.
posted by filthy light thief at 1:54 PM on December 16, 2015 [1 favorite]
- At a recent news conference, Lu denied that China censors the Internet — even though it's a widely known fact. He said that China manages the Internet, as does every country's government.
- China blocks many websites without which the Internet would be unimaginable in the West — including Twitter, Facebook, The New York Times and YouTube — because the government cannot control their content.
And China is one big, happy family, and everyone must be a good child of the Mother Land.
posted by filthy light thief at 1:54 PM on December 16, 2015 [1 favorite]
It seems like there's two related things going on here. The first is the creation of a classification system that, in theory, could build trust between people by identifying those who are financially and legally in good standing. And the second, much more frightening thing is the expansion of that system by adding in things that society and the government considers desirable, like buying diapers, not criticizing government policies, and so on, which makes it a tool of social control.
The first thing isn't so bad as an idea. In a country as chaotic as China, where regulations aren't always enforced and criminals hide in the anonymity of the crowd, it would probably be nice to know right away that the person you're meeting has a dozen unpaid parking tickets, or that their degree in civil engineering was issued by a diploma mill instead of an accredited university, or that the person with the dreamy photo on the dating site has massive debt from bad loans, and so on.
But the problem, so far as I can see, is that there's no transparency in the system. The standards it's enforcing are the ones valued by large corporations, and soon the government. In a top-down situation like that it's almost inevitable that the system will become a method of social control. And that seems to be happening right at the start, with Sesame Credit judging your worthiness based on your purchases.
Something more whuffie-like might have been a grand experiment, but it sounds like they're not doing that.
posted by Kevin Street at 2:25 PM on December 16, 2015 [1 favorite]
The first thing isn't so bad as an idea. In a country as chaotic as China, where regulations aren't always enforced and criminals hide in the anonymity of the crowd, it would probably be nice to know right away that the person you're meeting has a dozen unpaid parking tickets, or that their degree in civil engineering was issued by a diploma mill instead of an accredited university, or that the person with the dreamy photo on the dating site has massive debt from bad loans, and so on.
But the problem, so far as I can see, is that there's no transparency in the system. The standards it's enforcing are the ones valued by large corporations, and soon the government. In a top-down situation like that it's almost inevitable that the system will become a method of social control. And that seems to be happening right at the start, with Sesame Credit judging your worthiness based on your purchases.
Something more whuffie-like might have been a grand experiment, but it sounds like they're not doing that.
posted by Kevin Street at 2:25 PM on December 16, 2015 [1 favorite]
I'm confused. Are we saying this doesn't happen in the US?
posted by anotherpanacea at 3:12 PM on December 16, 2015
posted by anotherpanacea at 3:12 PM on December 16, 2015
Openly.
posted by entropicamericana at 3:27 PM on December 16, 2015 [3 favorites]
posted by entropicamericana at 3:27 PM on December 16, 2015 [3 favorites]
I can see someone like Rick Scott bringing this in as a mandatory thing for people on any form of government assistance. It will be run by someone he knows, the subscription costs (and there will be) will come out of the poor's pockets, and it will eventually spread upwards as stores, banks, etc offer discounts and conveniences to those who opt in (like with our current data mining economy).
Also, a twist on this would make for some great dystopian sci fi. A slow AI (think flatworm more than human) emerges from the consensus society interactions around improving metric X. The propagation of ideas and impulses harmful to metric X are selected against. Feedback ensues, with the agents of power at varying levels unable to comprehend that a larger, alien will is asserting itself through subtle, day to day actions by millions of people all worried about metric X.
Hopefully, this is simply relegated to the garbage and the people of China won't have to suffer through it.
posted by Slackermagee at 3:35 PM on December 16, 2015 [3 favorites]
Also, a twist on this would make for some great dystopian sci fi. A slow AI (think flatworm more than human) emerges from the consensus society interactions around improving metric X. The propagation of ideas and impulses harmful to metric X are selected against. Feedback ensues, with the agents of power at varying levels unable to comprehend that a larger, alien will is asserting itself through subtle, day to day actions by millions of people all worried about metric X.
Hopefully, this is simply relegated to the garbage and the people of China won't have to suffer through it.
posted by Slackermagee at 3:35 PM on December 16, 2015 [3 favorites]
Interesting story idea! It sounds a little like David Brin's The Giving Plague, albeit with a computer virus instead of a biological one.
posted by Kevin Street at 3:48 PM on December 16, 2015
posted by Kevin Street at 3:48 PM on December 16, 2015
Despite the denial here, I've read elsewhere that Sesame also builds a social network and decrements your rating by association. So if you're a hard working guy with a wife and two kids, but your social network is populated with folks who post to MeFi all day, your rating will take a hit without ever having personally posted to MeFi.
Much in the same way factories group workers into "self-managing" teams and punish the group collectively for individual team member errors. Social pressure is more effective than managerial pressure
So when they say its a social credit score, they don't mean finance credit, they mean remaining a "credit to your society"
Orwellian thought control.
posted by Fupped Duck at 3:58 PM on December 16, 2015 [3 favorites]
Much in the same way factories group workers into "self-managing" teams and punish the group collectively for individual team member errors. Social pressure is more effective than managerial pressure
So when they say its a social credit score, they don't mean finance credit, they mean remaining a "credit to your society"
Orwellian thought control.
posted by Fupped Duck at 3:58 PM on December 16, 2015 [3 favorites]
openly
Covert gamification seems like it would be missing the point.
posted by aspo at 3:59 PM on December 16, 2015 [1 favorite]
Covert gamification seems like it would be missing the point.
posted by aspo at 3:59 PM on December 16, 2015 [1 favorite]
In addition to the obvious insane problems, there is, contrary to the video, the opportunity for exploitation.
What will happen when messy acts of violence get perpetrated by people with high scores? What happens when someone with a very very low score starts friending everyone left and right? What about when low scores become a badge of honor with some groups?
posted by JHarris at 4:20 PM on December 16, 2015
What will happen when messy acts of violence get perpetrated by people with high scores? What happens when someone with a very very low score starts friending everyone left and right? What about when low scores become a badge of honor with some groups?
posted by JHarris at 4:20 PM on December 16, 2015
Also, a twist on this would make for some great dystopian sci fi. A slow AI (think flatworm more than human) emerges from the consensus society interactions around improving metric X. The propagation of ideas and impulses harmful to metric X are selected against. Feedback ensues, with the agents of power at varying levels unable to comprehend that a larger, alien will is asserting itself through subtle, day to day actions by millions of people all worried about metric X.
If you haven't read Karl Schroeder, I feel like you might enjoy some of his stuff.
Covert gamification seems like it would be missing the point.
So of course we have all sorts of implicit and explicit mechanisms of social surveillance and control in the US. It has a material impact on my life possibilities whether my credit score is good and I can pass a background check. I pay more or less for insurance based on a driving record that follows me around. Dozens (hundreds? thousands?) of corporations and a number of three-letter agencies maintain some form of cumulative profile of my behavior. Lots and lots of people, whether they're right or not, have strong beliefs about the social and economic signaling of things like grades and certifications. The database is rewriting civilization in its own image.
Still, as fucked up as our own polity and economy certainly are along these lines, it sure does elide some important differences to say that we're "doing this" in the US. I'm as critical of the situation here as just about anyone, but I feel like we lose a whole lot of ability to think coherently about this stuff when we act like there's no meaningful distinction between the Chinese state and the American one.
posted by brennen at 4:27 PM on December 16, 2015 [2 favorites]
If you haven't read Karl Schroeder, I feel like you might enjoy some of his stuff.
Covert gamification seems like it would be missing the point.
So of course we have all sorts of implicit and explicit mechanisms of social surveillance and control in the US. It has a material impact on my life possibilities whether my credit score is good and I can pass a background check. I pay more or less for insurance based on a driving record that follows me around. Dozens (hundreds? thousands?) of corporations and a number of three-letter agencies maintain some form of cumulative profile of my behavior. Lots and lots of people, whether they're right or not, have strong beliefs about the social and economic signaling of things like grades and certifications. The database is rewriting civilization in its own image.
Still, as fucked up as our own polity and economy certainly are along these lines, it sure does elide some important differences to say that we're "doing this" in the US. I'm as critical of the situation here as just about anyone, but I feel like we lose a whole lot of ability to think coherently about this stuff when we act like there's no meaningful distinction between the Chinese state and the American one.
posted by brennen at 4:27 PM on December 16, 2015 [2 favorites]
I'm confused. Are we saying this doesn't happen in the US?
It's all stick here, no carrot. I know people who have gotten fired or turned down for jobs either for not having social media accounts, or for stuff other people posted about them on their own accounts.
I don't know anyone getting expedited visas or lower interest rates for having a "good" social media presence.
At this point, i'd say it makes more sense to focus on how scary this is and that it is coming, rather than snarking about lol US police state.
posted by emptythought at 5:05 PM on December 16, 2015 [3 favorites]
It's all stick here, no carrot. I know people who have gotten fired or turned down for jobs either for not having social media accounts, or for stuff other people posted about them on their own accounts.
I don't know anyone getting expedited visas or lower interest rates for having a "good" social media presence.
At this point, i'd say it makes more sense to focus on how scary this is and that it is coming, rather than snarking about lol US police state.
posted by emptythought at 5:05 PM on December 16, 2015 [3 favorites]
> Something more whuffie-like might have been a grand experiment, but it sounds like they're not doing that.
oh my god we've landed in a universe where whuffie seems like a comparatively good idea. Please tell me one of you guys is working on repairing the dimension-shift portal, because I'm not sure how much more of this place I can take.
posted by You Can't Tip a Buick at 5:34 PM on December 16, 2015 [7 favorites]
oh my god we've landed in a universe where whuffie seems like a comparatively good idea. Please tell me one of you guys is working on repairing the dimension-shift portal, because I'm not sure how much more of this place I can take.
posted by You Can't Tip a Buick at 5:34 PM on December 16, 2015 [7 favorites]
This isn't a police state. It's far more sophisticated than that. We're never going to need forced coercion when technology will amplify our herd instincts to levels that would make Jim Jones envious.
posted by Beholder at 7:37 PM on December 16, 2015
posted by Beholder at 7:37 PM on December 16, 2015
The end result is the same.
posted by entropicamericana at 7:38 PM on December 16, 2015
posted by entropicamericana at 7:38 PM on December 16, 2015
"Don't fight it son. Confess quickly! If you hold out too long you could jeopardize your credit rating." <- from movie only 30 years old
posted by RobotVoodooPower at 9:24 PM on December 16, 2015 [5 favorites]
posted by RobotVoodooPower at 9:24 PM on December 16, 2015 [5 favorites]
I really enjoyed the conversation on Charlie's Blog about this - the comments are absolutely worth a good read as well.
posted by longbaugh at 3:30 AM on December 17, 2015
posted by longbaugh at 3:30 AM on December 17, 2015
I'm about to go off on a tangent here but I still think it's relevant...
With language...the ways Chinese dissidents have tended to get around internet restrictions and censorship in particular have been with language creativity playing off of the tonal nature of the (Mandarin). Which is awesome. "Grass-mud horse" which translates to (apologies here) "fuck your mother" went viral, even infiltrating stuff for kids in its more kid-friendly form of an actual, fake "mythical creature", the grass-mud horse. On the more serious and less vulgar side, once the word "censorship" was being (un?)ironically censored from the internet people started calling it "harmonization" (a euphemism). Once that started getting censored it became "river crab", a word phonetically similar to "harmonization"... and which eventually led to Ai Weiwei (who's been censored and jailed for his art) doing an art installation of 3,000 porcelain river crabs in the US/UK.I thought I'd read something about him also giving out river crabs to people in China as a kind of protest but I'm having a hard time finding it AHA nevermind, I found it - Ai Weiwei gives out 10,000 river crabs to his supporters.
It reminds me of Cockney rhyming slang, where for example, "money" becomes "bees and honey", or the word "stairs" becomes "apples and pears", so you get a sentence like "he went up the apples and pears". You're going multiple levels down to the point where without enough context it's impossible to know what's being said, though it's still not clear how or why rhyming slang developed... probably not as an invention to get around censorship, but maybe. I don't think it's really used that way today at any rate, but I could be wrong.
Anyway, these aren't the only examples, just the ones that became really popularly known. And not that grass-mud-horse and river crab was language that ultimately helped protect anyone (probably), it at least it allowed people to discuss these things for a while and fostered/allowed non-government-approved conversations.
But when the government catches on invariably, even with something like grass-mud horse when it managed to become a nationwide cultural phenomenon, it gets shut down. And when the government is straight-up tracking and analyzing data rather than language I guess there's not really much you can do to avoid it. Using rapidly-evolving coded language isn't going to protect you from, say, your credit score and purchase history, and the social impact that might have for you. I mean people have always been clever with measures against surveillance and authoritarianism (facial recognition stuff for example) but tapping into social media--where you have a web of information where even just who you're friends with gives away a ton (for example I have my facebook profile on lockdown, but none of my friends do, so there's still a lot out there on me)--is so easy and such a rich vein for monitoring most aspects of life for a lot of your citizens.
Interesting that it's "opt-in for now". And this is pretty concerning to me because this has never been terrible for anyone before: [...] the "new system will reward those who report acts of breach of trust", where "trust" seems to be defined as 'not talking about things the government doesn't want you to talk about'--"social credit will "forge a public opinion environment that trust-keeping is glorious".
Also it's interesting how "convenience" comes up in this discussion. One of the people interviewed said she didn't mind it because booking a hotel was so convenient with the Sesame system. And I mean, I've given up a lot in the name of convenience. Kind of disconcerting.
But yeah, not happy to hear about this, would be interesting to see what my Chinese colleagues think of this... I know one would be totally chill with it but another colleague who's voiced general support for some measure of censorship before, I'm wondering if this might be too much for him.
I mean none of it's actually a surprise... but in a way it's still kind of surprising to me.
posted by suddenly, and without warning, at 7:55 AM on December 17, 2015 [5 favorites]
With language...the ways Chinese dissidents have tended to get around internet restrictions and censorship in particular have been with language creativity playing off of the tonal nature of the (Mandarin). Which is awesome. "Grass-mud horse" which translates to (apologies here) "fuck your mother" went viral, even infiltrating stuff for kids in its more kid-friendly form of an actual, fake "mythical creature", the grass-mud horse. On the more serious and less vulgar side, once the word "censorship" was being (un?)ironically censored from the internet people started calling it "harmonization" (a euphemism). Once that started getting censored it became "river crab", a word phonetically similar to "harmonization"... and which eventually led to Ai Weiwei (who's been censored and jailed for his art) doing an art installation of 3,000 porcelain river crabs in the US/UK.
It reminds me of Cockney rhyming slang, where for example, "money" becomes "bees and honey", or the word "stairs" becomes "apples and pears", so you get a sentence like "he went up the apples and pears". You're going multiple levels down to the point where without enough context it's impossible to know what's being said, though it's still not clear how or why rhyming slang developed... probably not as an invention to get around censorship, but maybe. I don't think it's really used that way today at any rate, but I could be wrong.
Anyway, these aren't the only examples, just the ones that became really popularly known. And not that grass-mud-horse and river crab was language that ultimately helped protect anyone (probably), it at least it allowed people to discuss these things for a while and fostered/allowed non-government-approved conversations.
But when the government catches on invariably, even with something like grass-mud horse when it managed to become a nationwide cultural phenomenon, it gets shut down. And when the government is straight-up tracking and analyzing data rather than language I guess there's not really much you can do to avoid it. Using rapidly-evolving coded language isn't going to protect you from, say, your credit score and purchase history, and the social impact that might have for you. I mean people have always been clever with measures against surveillance and authoritarianism (facial recognition stuff for example) but tapping into social media--where you have a web of information where even just who you're friends with gives away a ton (for example I have my facebook profile on lockdown, but none of my friends do, so there's still a lot out there on me)--is so easy and such a rich vein for monitoring most aspects of life for a lot of your citizens.
Interesting that it's "opt-in for now". And this is pretty concerning to me because this has never been terrible for anyone before: [...] the "new system will reward those who report acts of breach of trust", where "trust" seems to be defined as 'not talking about things the government doesn't want you to talk about'--"social credit will "forge a public opinion environment that trust-keeping is glorious".
Also it's interesting how "convenience" comes up in this discussion. One of the people interviewed said she didn't mind it because booking a hotel was so convenient with the Sesame system. And I mean, I've given up a lot in the name of convenience. Kind of disconcerting.
But yeah, not happy to hear about this, would be interesting to see what my Chinese colleagues think of this... I know one would be totally chill with it but another colleague who's voiced general support for some measure of censorship before, I'm wondering if this might be too much for him.
I mean none of it's actually a surprise... but in a way it's still kind of surprising to me.
posted by suddenly, and without warning, at 7:55 AM on December 17, 2015 [5 favorites]
This sort of thing...
...social credit will "forge a public opinion environment that trust-keeping is glorious".
It's like a Virtual Maoism... A Velvet Cultural Revolution.
Who needs brutal social repression with student uprisings when you can combine capitalist desire-induction, convenience and social persuasion without the heavy hand.
All looked over by the machines of loving grace...
posted by symbioid at 8:44 AM on December 17, 2015
...social credit will "forge a public opinion environment that trust-keeping is glorious".
It's like a Virtual Maoism... A Velvet Cultural Revolution.
Who needs brutal social repression with student uprisings when you can combine capitalist desire-induction, convenience and social persuasion without the heavy hand.
All looked over by the machines of loving grace...
posted by symbioid at 8:44 AM on December 17, 2015
without the heavy hand
I think you can be pretty sure that somewhere in the chain there are still a bunch of people getting shot in the head.
posted by brennen at 9:10 AM on December 17, 2015 [2 favorites]
I think you can be pretty sure that somewhere in the chain there are still a bunch of people getting shot in the head.
posted by brennen at 9:10 AM on December 17, 2015 [2 favorites]
And in my FB feed I see Tencent just purchased Riot. Which I guess reduces competition in this market, now.
Who gamifies the gamifiers?
posted by symbioid at 9:12 AM on December 17, 2015
Who gamifies the gamifiers?
posted by symbioid at 9:12 AM on December 17, 2015
Oh - I see they owned a majority share, but now own it outright. I misremembered last night thinking Riot had their own system in competition with Tencent. NVM.
posted by symbioid at 9:17 AM on December 17, 2015
posted by symbioid at 9:17 AM on December 17, 2015
So, let's play devil's advocate here--
With the knowledge what whatever you say over this social network can hurt your credit score, who would ever use it to express "incorrect" opinions? How would the network come to know about what you say outside of its confines?
What is to stop people from being social media hypocrites, using the system to say things the government wants to hear, but everywhere else saying what they really think?
How does the system detect other negatives? If playing video games is a negative, how does it tie your play history to you? If it only does through through social media games (like the Sesame Credit version of Farmville), then what company in their right mind would make such a game under this system?
posted by JHarris at 3:11 PM on December 17, 2015
With the knowledge what whatever you say over this social network can hurt your credit score, who would ever use it to express "incorrect" opinions? How would the network come to know about what you say outside of its confines?
What is to stop people from being social media hypocrites, using the system to say things the government wants to hear, but everywhere else saying what they really think?
How does the system detect other negatives? If playing video games is a negative, how does it tie your play history to you? If it only does through through social media games (like the Sesame Credit version of Farmville), then what company in their right mind would make such a game under this system?
posted by JHarris at 3:11 PM on December 17, 2015
« Older A brief history of sending letters to Santa | Myth and reality of the Hardanger fiddle and... Newer »
This thread has been archived and is closed to new comments
posted by entropicamericana at 1:36 PM on December 16, 2015 [2 favorites]