*34c3 intro* Herald: Okay, so ladies and gentlemen
Katika Kühnreich. So did you all thank the angels today? No, yes, maybe?
[Applause] Herald: Did you all took your prosĂm two
today? No? Okay, see, Katika will tell you about the social credit system from China,
and will tell you a little bit more about how a human being can be valued. She is
the magista in political science and modern and ancient Chinese science. She
not only has her theoretical knowledge from books, but she always also combines
it with the street study. So please welcome, with a very warm hand of
applause, Katika! [Applause]
Katika: Hey guys, I'm happy to be here, and first I want to thank all the people
who made that possible. This conference and this talk; so thanks to all you, and a
special thanks to all the people translating.
[Applause] Katika: Before we start, I like to say a
few points. The first was the thank you, we already have. The second is I have to
apologize: English is not my mother tongue so you will have some funny mistakes that
I make, so the Internet has a bit more fun. Because the world can be (nice). The
second thing, is I will take you on a journey through China's social credit
system - its current state. And because we only have half an hour, more or less,
because I'd really like to spend a lot of time with you guys on talks, on discussion
and QA, we have to leave a lot of interesting sites out. So there were some
excellent talks on the CCC about technological aspects of Internet and
Internet usage in China. I will concentrate really hard on this social
credit system. But if you have questions, please ask in the Q&A. The Q&A will not be
recorded, so you can feel free. And if you have more questions, just contact me after
the talk. And, it's a China topic and often when you talk about China, people
say "oh yeah, oh yeah: that's China." This is no China bashing session - I'm sorry
for anyone who expected that. China has a government, and a government wants to
control something. So this talk is about social credit systems in China but it is
about a tendency I see internationally. It's the tendency to answer social
problems with technological solutions, or at least it's the try. So please keep in
mind: it's happening in China, but it's not that we are safe in the West or cozy
or anything else. So it's meant as an example of an international tendency. And
I prepared some cards, so I will stick to the topic. I have to look at them from
time to time. This talk presents my current state of research; it's not the
truth. And this talked sadly represents my current state of my cold as well, so I'm
not totally fed please apologized that as well. For a little journey I prepared a
couple of stops. The first one will be a video from 2015 made by Extra Credit on
social credit systems in China. So if you don't like the talk, you have seven and a
half minutes to just vanish - nobody will notice. The next point will be "Updates
and Corrections" - it's a video from 2015, and there's some faulty information
in it. And to do that, I will give you a quick look at the Chinese Internet
landscape. The next point will be the current state of social credit systems. If
I say SCS: it's the same - I'm just lazy. And there's one question often not
answered when talking about the Chinese credit system - it's why is it happening.
And again it's not because it's an evil government. It's a government. So we'll
dive in some ideological aspects of Chinese politics today. And in the end,
I'm really looking forward to that point because I can shut a bit up and listen to
your thoughts and we're really interested in that. So, everyone ready? Cozy? That's
wonderful, thanks! Then we can start with the video; that isn't it shown right now.
Now we need some technical support. It's much more fun when you see the pictures.
But we have time, so. Herald: Do we have the (tv guys) here?
Katika: We got free water! Herald: Gentlemen, I think we have a
technical problem here, so while the VOC and my stage manager is trying to get some
people from there. Yes he will do this - he's a good guy. He's the best guy!
Gentlemen, ladies, applause for the VOC! [Applause]
Herald: So while we're waiting to have the video; and trust me it looked really good
from here. Let's play a little game - later we will have a Q&A session but
unfortunately we are not allowed or we we can't record the Q&A session, and we will
also not video stream the Q&A session. So I would like you to have a quick hand of
applause for all of you, because you will have a great discussion with this nice
lady and therefore we try to make applause - we test the applause.
Katika: Okay [Applause]
Herald: No no no, that should be louder - come on!
[Applause] Herald: Okay we still have no video, but I
think over there I did not hear any applause. Could you, could you please try
it again? [Applause]
Herald: Yeah, better, better. I think these guys are way better. Are they? Come
on! [Applause]
Katika: Otherwise I can use the time to give you some information I'd present you
after the video. Thank you. The information you get in the video is based
on a press release of the American Civil Liberties Union, and that contains some
mistakes. One thing - when you want to know something about China, the government
is really open in publishing what they're doing, and what they're planning to do.
And it's seems to be much more comfortable to just copy, but maybe have a look into
what the government says because it's a good way to find out what they are doing
and they're scaring the open. It works with a lot of governments. And I realized
that because this video had a lot of impact, and there was a lot of news report
on that scary system in China but when you looked at the news report and you had the
video before, you found out that it was more or less the same, the very same, as
in the video even years later. So please if you want to find something out - check
for facts. They might be online too. So that's the picture, and that's my break.
Video: Okay, so one you recently brought this to our attention, and when we first
started looking into it, we couldn't believe it. Even now looking at this I
keep thinking that there's got to be something about this that I just don't
understand, like, I must be reading this wrong. It's just so clearly something out
of dystopian science fiction. And yet, here it is in our world, and people are
actually embracing it. Apparently China has gamified being an obedient citizen.
[Music] Video: Going under the innocuous name of
Sesame Credit, China has created a score for how good a citizen you are, and that's
one of the scariest things I've heard in quite a while. It's jointly run by
Tencent, yes that Tencent - the one that owns Riot Games and has a significant
share in Epic and Activision-Blizzard, and also the ascendant Chinese competitor to
Amazon, Alibaba, hence the name Sesame. So the owners of China's largest social
networks have partnered with the governments to create something akin to
the US credit score. But instead of measuring how regularly you pay your
bills, it measures how obediently you follow the party line. They dredge data
from your social network, so if you post pictures of Tiananmen Square, or share a
link about the recent stock market collapse, your Sesame Credit score goes
down. Share a link from the state- sponsored news agency about how good the
economy is doing and your score goes up. The Alibaba and Tencent are also the
largest online retailers in China so Sesame Credit is also able to pull data
from your purchases. If you're making purchases the state deems valuable like
work shoes or local agricultural products, your score goes up. If you import anime
from Japan, though, down the score goes. And this score has real-world
consequences. Like many games, Sesame Credit has tiers and levels, and having a
high score gives you special benefits - like making it easier to get the paperwork
you need to travel. Or making it easier to get a loan. Now currently, there are no
consequences for having a low score but there's been talk about implementing
penalties once the system becomes mandatory in 2020. Penalties like slower
Internet speeds for low scoring citizens. Or even restricting the jobs that a low
scoring person's allowed to hold. But there's one more layer to Sesame Credit,
and here's where this goes from being repulsive to downright insidious. Because
this is all part of a social network it also scans your friends -
so you will lose points for having friends with low obedience scores. And it tells
you this. At any point, anybody can check anyone elses score. And when you check
your own score, Sesame Credit provides a handy map of your friends to show you who
is dragging your score down. Have you ever had that thing where you play a game with
somebody who wasn't doing very well and you've tried to change their behavior to
make them do better? Or maybe after a while you just sort of stopped playing
with the people who are holding you back. That's at the heart of how this system
works, and it's also what makes this one of the most terrifying tools of
authoritarian oppression I've ever read about. Because like mass censorship, jail
time, assassinations - those are all big messy implements for keeping a population
in line. That messiness and severity foster resentment - eventually rebellion.
They're expensive, they're unwieldy - in the end, those tools are impossible to
maintain. But social pressure? Ostracization? Those things are free -
they happen on their own. And as a government tool they don't have nearly the
same potential for going embarrassingly disastrously wrong. With a system like
this in place, a government doesn't even have to tell neighbor to spy on neighbor;
to rat each other out. Because that's all built into a seemingly innocuous game
system. The government need not step in. Re-education will be handled for them by
friends, classmates, and relatives who want to maintain a high score. And if that
doesn't work then potentially dangerous ideas still end up quarantined by the
social isolation this game system causes. Express or help to spread to many radical
ideas, and people will stop associating with you. And not because some jackbooted
thugs showed up at the door with threats, but simply because associating with
somebody with those ideas will lose them all the privileges they've worked so hard
to obtain. It recontextualizes obedience to an authoritarian regime.
In the past, you obeyed such powers because you were afraid. Fear kept you
motivated. But fear is negative. It fosters resentment. The world we're
stepping into instead uses positive reinforcement to promote being subservient
to the will of the regime. It's Big Brother's kinder, gentler hand. And the
things that make this scary is that we've seen the efficacy of this only too well in
games. You may not actually know this, but when World of Warcraft was in its early
stages of development, it had an unrested penalty mechanic that started limiting
experienced gains for players who had played too much. And players hated it -
they resented it they complained about it every day. So after brainstorming on this
for a while, Blizzard had the idea to simply change how the mechanic was
presented. Without changing any of the existing numbers or systems, they started
referring to the unrested experience state as normal, and made it the default. And
they started calling the original normal experience gained state as rested. That's
all they changed, and everybody loved it. People would log on every day just to get
that bonus. Positive reinforcement works wonders. But unlike World of Warcraft,
which built a system just to get people to embrace meaningless grinding, Sesame
Credit has built a system to get people to enjoy falling into line. Now the system's
not mandatory yet - it's opt-in right now. But it's going to be mandatory in 2020,
and there's a terrible brilliance to phasing that in. The early adopters are
going to be people excited about this system. People who are already patriotic
and are eager for anything that'll help display that patriotism to the world. And,
as early adopters they're gonna talk it up. They're gonna give it an air of being
positive and fun. Then it'll be foisted on the society as a
whole. More than that though, the early adopters are gonna compete. Already you
can see hundreds of thousands of tweets displaying people's high scores or showing
off the new milestones they've hit. Giving a hard numbers to their patriotism and
giving them bragging rights for being the most patriotic, most right-thinking person
they know. And that's gonna set the tone for how Sesame Credit is intended to be
used - as a competition to see who can agree with the government the most. We've
talked about propaganda games on this channel before but for all the time we've
spent examining and deconstructing terrible games that espouse hate and for
all the studies on propaganda games James has done, this is the use of game systems
that frightens me the most. Because to most people Sesame Credit will seem
benign. Perhaps even fun - it's a conversation starter, something to share
with your friends. But it's making heavy use of all the psychological motivators
that we Gamemakers deploy in scoring systems, and ladders and levels. Systems
that we built to shape play habits and to keep people coming back. Like I said, I'm
still kind of in a state of disbelief looking at this. If any of you are
watching this from China, please tell me if I'm misunderstanding this thing,
because I would love to be wrong about it. If not, well, I hope this episode can do
some small amount to help the fight to keep such a system from becoming
mandatory. For everyone in the rest of the world, I hope this helps remind us all how
important it is to be aware and to be vigilant. All of these gamification
techniques we've learned through making games offer incredible opportunities for
making this world a better more engaging place. But every great tool carries with
it the potential for misuse. And it's on us as a community who understands this
amazing new medium, to do what we can to stop that. We'll see you next week.
[Applause] Katika: So, we had the shedule, and thank
you to Extra Credit for this video, and thank you for concentrating on
gamification. Because a lot of time, when I talk to social scientists, without a lot
of technological knowledge, it was like okay - it's a new system. And the whole
gamification thing triggered me, and I wondered about a lot of people who didn't
seem too interested in it. Because I said that the video contains some faulty
information. First, I'd like to concentrate on key aspects they told us in
with this video. There is a social rating system in China now. The system will
become mandatory in 2020. That's all right. It works on Big Data algorithms,
artificial intelligence, and your online and offline data. And it uses technologies
like gamification and nudging. And if you hear it's an Orwellian system - no it's
not. 1984 was about force. Gamification is about feeling good and cozy. So, in the
updates and corrections, I promise you a quick look at the end Chinese internet
landscape. What most people already know. From the beginning the Chinese government
started to control the internet as much as it could and its users. So some
vocabularies you might have heard are the Great Firewall, or its new extra the
Golden Shield, the Great Cannon, which is a DDoS weapon. It would be really
fascinating to dive in there, but we have to watch from far. Another point that's
really interesting for us looking Social Credit System is true name or real name
registration. Which means you can't use a lot of services if you don't give them
your ID number. And the mandatory system in 2020 will work on ID numbers as far as
we know. So, we watched the other side: the users, their millions of users - it's
one more than 1.4 billion people in China. And ICTs are really popular. Much more
than in Germany, for example. And another thing that please keep in mind - cash is
vanishing in China. A couple of years ago, for example, when I
lived in China 2004: I paid my rent in cash half a year in advance. When I paid
for anything I paid cash. You could couldn't find a lot of places accepting
cards. But now China didn't have this development of cards - they just jumped
right through paying with mobile apps. And you can buy noodles or snack or magazine
with your mobile phone. And the Chinese users found multiple ways to deal with
censorship, and to trespass it. It's a wonderful thing to look up.
And now, the big players of the Chinese Internet, because we learned some names in
the film. I made something very Chinese - I made something colorful. Okay, you have
three big players: its Alibaba and Tencent - we heard about them, and Baidu is the
third biggest player. Each millions of users, or billions. So to explain this, I
will use some American companies names to help understanding. And Baidu is known for
that; was the wrong; yeah, Baidu is known for Baidu Barker - this is like the
Chinese Wikipedia, it's just a bit much controlled. And a have Baidu Pay, and
Baidu is huge in artificial intelligence. So the next one is Tencent - we learned
about the gaming thing. And developed QQ - this is a messenger than some of you might
remember: ICQ. Yeah, it's just a bit bigger. Another app a lot of people don't
think it's Chinese, it is Chinese, it's called WeChat. It's I can say they even
mmm WhatsApp, right? And guess what WeChat developed - WeChat Pay. And there's
another little company Tencent holds some parts in its SnapChat - it's twelve
percent owned by Tencent. Alibaba has Alipay which is one of the biggest payment
apps. And you can even use it in Germany now I think, in some shops. It's running
Taobao Wang, which is like eBay just a bit bigger. And Sina Weibo. Weibo mains micro
blogging or twittering. And Sina is the biggest company in China, and they hold
more or less a third. Now we come to something really interesting - it's Ant
Financials. Because Ant Financials created something called Sesame Credit. Sesame
Credit is Ali Baba's child, not Tencent, and Ali Baba's. Tencent developed their
own system. So, updates and corrections without colors. 2014 was the year the
Chinese state council released a planning outline on the construction of social
credit systems in China. And it gave us a lot of information but only very few
specific information. So, they started, and in 2020 it will be managed mandatory.
But they worked on for as long as at least 2005.
It allows governmental and private SCS. So you have both. And eight companies are
allowed to form their own private social credit systems. We learned Tencent and
Alibaba are two of them. In different regions, Hainan for example, it's an
island - you have social credit systems run by the government. So you see you have
a lot of different systems running at the same time in different regions and
sometimes you can choose between different social credit systems at the same place.
And Sesame Credit we have learned is the biggest and run by Alibaba. And Sesame
Credit is not the one who will be mandatory. I have one other point that I
forgot it might come back later. So, to give you a bit more information, let's
have a quick look at recent developments that I thought of would be really
interesting for you when you have that scenario of a society that is formed by a
writing system. In 2016 there were some new guidelines on the SCS. New
cybersecurity laws and you might expect it they are getting a bit harder and harder
every time. And there was a police robot - I don't know if you heard about it I was
going to write an article but then we had a really nice conference that made us a
lot of work and my article died in the way. So you might have watched Wall-E -
what was the movie couple of years ago. You might remember Eve - there was a
little cute white robot that had a specialty - it could taser. Someone in
China must have watched that movie too, because guess what - it looks like Eve and
it can tazer. You find pictures and clips online. And the People's Bank of
China announced its own digital currency to come soon. They are working on it for
years. And the real name registrations for online comments. So no real name
registrations, no online comments. And in 2020 - the mandatory Social Credit System.
The next question is usually: how does the social criticism work? And we take a quick
look at Sesame Credit because they added more and more services that Extra Credit
guys could know about. So we learned a lot of data mining they do.
For example, when you use AliPay it goes to your scores. So your score gets higher
and higher the more you shop with AliPay. Then other Alibaba services - we know
there are different and millions of services. And now it's getting special
because it draws information from officials as well. So we have information
from courts, debtors registries, and because love always makes everything
better, you get your data from the biggest dating app in China - it's Baihe. So you
can see if the person you want to fall in love with have a good score. And of
course, they're including more and more services. And there are penalties now. If
you forgot to pay your child support for a couple of month or years or; I have heard
people can forget that for quite a long time. There are penalties: for example,
you can't take high-speed trains or first- class flight tickets. Which is a very
intelligent way. Again, it's a very intelligent way to add penalties because
if you start with people nobody likes, nobody likes openly, it's clever. And of
course, the companies do not tell you a lot about the technologies like they use
in the systems. So and then people want to know about the mandatory system, what it
can do, how will it look and so on, and there's a very simple answer - we don't
know. The Chinese government doesn't give you a lot of information on how it will
look. But we can guess. In having a look on how the government worked in the last
20-30 years. So, we can have a look how they prepare the mandatory system. Like I
said, different systems at the same time, and they will learn from this. And my
guess is they will have like a little medley of the best features of the
existing systems and get rid of the faulty ones. So, by best, I mean best from the
point of view of the government of course. So now we are bit later because of our
technological difficulties, thanks again for solving. Why do you have a Social
Credit System? Let's dive a bit back in history. Some
people might remember the year 1989. Before you had a lot of countries that
called themselves socialists. But after, you had only a few and China is one of
them. And how did they make it? In the 80s they had something called the era of
reform and opening-up, where they opened up to the West mostly, and to capitalism.
And you might know what that means. China's called the transformation society.
And with some people getting rich quickly, and a lot of people getting poorer and
poorer and watching the rich - you have a lot of social tension. And remember, China
is a huge country. A vast amount on people, and know people are really good in
getting on each other's nerves. So you have problems. I give you one number that
I really like - in 2014, the official count of mass incidents everyday was
nearly 250. And the number is rising, so it will be much more today. And a mass
incident can be anything from pensioners holding up banners demanding more money
because they worked their whole lives, I'm sorry, I really have to concentrate on not
swearing, to like, riots, full-grown riots, petrol bombs, burning buildings.
For example, G20 in Hamburg, you might have heard. That's a mass incident. That's
nothing to get crazy about, that's just a little thing that happened. Hamburg is
still standing. And, thank you, hello Hamburg.
[Applause] And keep in mind organized societies are
controlled on different levels. That's not only China. And social control is usually
done by groups, so it's much easier if we control each other. And it has the long
history in China too, and a lot of socialist countries develop new forms of
social control. So in China, you have a Hukou - that means you can only get social
services where you're permitted to live. I just give you a really really short
version of everything it's not it's a bit more complex as you know. The Tang on
which you can say it's an analog version a lot of your stuff what you've done goes in
there. You needed a marriage permit if you wanted to get married and it wasn't easy
to live without a married permit and as China became more and more digitized you
might know what comes next. Oh too anyone control became digitized as well and as
people are using ICTs more and more they create a huge amount of data. And the
bloody thing with data is you don't feel, smell or see anything vanishing, do you?
And people it's helpful as humans to
develop something towards something we can't see smell or fear. And more and more
technologies are used in civilians and social control. But that still leaves the
question: why is China doing it? Let's go to another point I'd really interested
in: ideological influences the way we see the world and the way we think about it
is formed by theories, by ideologies and what's the Chinese flag ? You see it
again, maybe you remember, some of you, it's red with yellow stars - that usually
is a code for socialists and this is Mao Zedong. He was kind of big in China. And
so you have - here you have three socialist symbols. It really helps to
know a bit about Marxism in researching China and especially in Mao Zedong
thought it's not called maoism it's called more Zedong thought it's about what
he thought and in this reform of opening up - I'm trying to feed you a bit of
Chinese history - the Chinese government opened up to more. For example they opened
up to nationalism. And if you know a bit about socialism, usually you wouldn't
put socialism and nationalism in one room, because they didn't like each other. The
Chinese government found a way they like each other. They got really interested
in PR and they really research other countries how governments there stood in
power. And some are not so well known to most of us this is Confucius.
Confucianism shaped and molded the Chinese societies for more than 2,000
years. And if you have that big influence, you can't just get rid of it.
It's in people's memories. It's in the whole society, and Confusianism came back
as an ideology, where some parts of the Socialists ideology vanished. Confucianism
came back and Confucianism is about a society where everyone is on her or his
place and her place is always below. And with all these new ideological
influences, the way of communication changed as well.
Before it was the masses and now it's the public. All campaigns and programs had
revolutionary names or poetic forms like hundred flowers yet the Great Leap
Forward, and all these campaigns molded our picture of China, or the picture
most people have in their hearts people in bloom, red flags. But now it's changed you
have new campaigns and programs and they have other names as well. You have
socialist harmonious society as a goal for the society and a couple of years ago
before, you couldn't put socialism and harmony in one room either. And you have
the Chinese dream. Yes, it sounds a bit like American dream, yeah. And we learned
that all governments are interested in control and feedback because you can't
control something, if you don't pay attention and can't blow up anyway. So
like we have an autonomous nervous system that tells every part what's going on, you
have one electronic autonomous nervous system now in many societies not only
China. People using ICTs and a social credit
system just creates a wonderful opportunity of feedback and on the same
time on influence. So, we come to the end of this talk and to the summary. So, again
this I think is really important: it's an international tendency to solve social
problems with technological solutions. And as far as I am concerned this is not the
best way maybe. And the Chinese government - they do not count and think in years,
they think in decades at least. And they looked at the internet and found out
that this is a good tool and they're using it. Successfully. But we know at least
since Edward Snowden, that other governments are capable of using the
Internet as a tool as well. A social credit system exists of the use of big
data, algorithm, ICTs and gamification. And it will become mandatory. And this
game, when it becomes mandatory, will define your life. And not only your life,
it will define the life of the people surrounding you as well. The schools your
children for example will go to will be chosen on your rate. And there's the
opportunity to put this together with a national cryptocurrency. So, thank you
all for listening that what's my current state of research and my current state of
my code, we both presented to you this talk and I'm really looking forward to QA
and discussing. Thanks. *34c3 outro*
subtitles created by c3subtitles.de in the year 2018. Join, and help us!