*35C3 preroll music*
Herald Angel: Alright. Then it's my great pleasure to introduce Toni to you
She's going to talk about "the Social Credit System," which is, kind of, feels to me
like a Black Mirror episode coming to life. So, slightly nervous and really
curious what we're going to learn today. So please give a huge, warm round of
applause and welcome Toni! *Applause*
Toni: Good morning, everyone! Before I'm going to be talking I'm going into my talk
I'm just going to be presenting the Chinese translation streams for everyone
who doesn't speak English.
*speaks chinese*
*Applause*
So because today's talk is about China we figured it would be good to have it
in Chinese as well. And, I'm going to be talking today about
the Social Credit system in China, where "the" Social Credit system that you
always hear about in Western media actually doesn't really exist
and most of my talk will actually be talking about what all we don't know.
Which could fill an entire hour or even
more. But I'm just going to be focusing on some of the most interesting things for
me. First of all, a little bit about me. I'm an economist, but I'm not I'm not only
concerned with money. I'm kind of looking at economy, at economics as the study of
incentives, which means that what I'm really interested in is how humans respond
to different kind of incentives. I don't believe that humans are completely
rational. But I do believe that humans do try to maximize what they think is their
best interest. Now, some words about me: I studied math, economics and political
science in a couple of different cities all around the world. I spent overall 19
months in China. Most recently I was there in July on a government scholarship, which
was really, really interesting, because while there I read all of these Western
newspaper articles about the Chinese Social Credit system, and I went to a
pretty good university and I asked them: So what do you think about this system?
And most of them basically looked at me blankly, and were like: What system, I
haven't even heard of this! So that was kind of an interesting experience to me
because in the West it's like this huge, all-encompassing system. And in China,
most people that aren't directly -- that aren't directly in touch with it actually
don't know anything about this. I'm broadly interested in the impact of
technology on society, life, and the economy, obviously, and in my free time I
do a lot of data science and machine learning with Python and R. So, I thought
it was quite interesting to look at the Social Credit system, also from this point
of view because you always heard that it's like this big data initiative, and then
when coming down to it, what you actually see is that, they don't actually use
machine learning all that much. They have, basically, a rule based catalog where, if
you do this you get 50 points, if you do this you get 50 points, and then they
actually have a lot of people that are reporting on other people's behavior. I'm
going to be talking about how exactly it looks, later on but I was very, very
surprised after reading a lot of the Western newspaper articles that were
basically "Oh, this is this big dystopia, Orwellian, with big data working." And
then, you read what's actually happening and they have huge lists of "if you
jaywalk, you get 10 points detracted from you," this kind of thing. If you want to
get in touch with me you can use Twitter but you can also use different e-mails
either my professional e-mail or my personal e-mail address, that you can both
see there. If you have any thoughts on that or are interested in this a little
more I can give you more resources as well, because obviously today's talk will
only be scratching on the surface. So, perceptions of the Social Credit System.
One of the interesting things that I've talked about before was how, in the West
and in China, the perception is completely different. So in the West, which is from
financialtimes.com, you see this huge overwhelming guy, and he basically puts
every Chinese person under a microscope. They're all kind of hunched over, and
everyone has this score attached to them, and they seem pretty sad and, like, very,
very Orwellian concept. Whereas, in China, this is actually from a Chinese state
media, and what it says is, well, we can all live in harmony with this new system
and all trust each other. And interestingly Chinese people actually
believe that, to some degree. They believe that technology will fix all this current
problems in society, especially because, in China currently, trust is a rare
commodity. And this new system will lead to more efficiency and trust, and a better
life. And I have a really, really interesting quote from a Western scholar,
that really summarizes the Western perspective: "What China is doing here is
selectively breeding its population to select against the trait of critical,
independent thinking. This may not be the purpose, indeed I doubt it's the primary
purpose, but it's nevertheless the effect of giving only obedient people the social
ability to have children, not to mention successful children." This, basically,
plays with the idea that if you have a low score, currently, in the cities that are
already testing this system, what happens is, your children can't attend good
schools. What happens is, you cannot take trains, you cannot take planes. You cannot
book good hotels. Your life is just very, very inconvenient. And this is by design.
This is kind of the plan. The Chinese government, they say it's a little
different, the idea is about changing people's conduct by ensuring they are
closely associated with it. One of the main things about this system is, there
isn't very much new data being generated for the system. Instead, what's happening
is, all the existing data that is already collected about you is, basically,
combined into one big database for each and every person by your ID number. So, in
China, once you're born, you get an ID number, which is similar to a Social
Security number in the U.S. We don't really have a similar concept in Germany,
and it used to be that your ID number was only necessary for public -- like for
government stuff, but now you need your ID number for getting a bank account, you
need your ID number for buying a cell phone, even if it's a prepaid cell phone,
you still need your ID number. So all your online activity that happens with your
cell phone is associated with your ID number, which means you can't really do
anything anonymously, because it's all going back to your ID number. There's a
couple of predecessors, some of them going actually back to the 1990s, that are
supposed to be integrated into the new system. One of them, or like two of them
are blacklists. One of them is a court blacklist. So in China, courts work a
little bit differently. They tend to like giving you fines, as they do in other
countries, but they also like giving you "apologies to do." So one of the things,
if you do something, for example you're a company, your food safety wasn't up to par
– you have to pay a fine. But in addition to this fine you also have to write a
public apology letter in the newspaper, how you are very sorry that this happened
and it won't happen again, and it was a moral failing on your part, and it won't
happen again. And if you don't do that, you go on this blacklist. Similarly, if
you take out a line of credit and don't pay it back within three months, or like
don't don't do any payments for three months, you go on this debtors blacklist.
If you're on this blacklist, which again is associated with your *shēnfènzhèng*, so
your ID number – what happens is you cannot take trains you cannot take planes.
Your life basically becomes very very inconvenient, your children can't go to
good public schools, your children can't go to private schools, your children can't
go to universities, all of these issues are suddenly coming up. There is also a
company database that's called Credit China which is basically similar to the
public debtors blacklist but it's basically a credit system a credit score
for companies. And then there's the credit reference center of the People's Bank of
China which is a credit score. It was supposed to be like Schufa or like the
U.S. FICO for individuals. But one of the big problems in China is that there are a
lot of people that aren't part of the formal economy. A lot of people are
migrant workers. They get their money in cash. They do not have bank accounts. They
do not have anything… they do not have rent or utilities or anything like this because
they live in the country. So they own their own home which they built themselves
so they didn't even finance it and their home isn't officially theirs because in
China you can't actually own property. Instead the government leases it to you.
So there were a lot of people that were not covered in this system, and I think
the last data that I had was that less than 10 percent of Chinese adult citizens
were actually in the system and had any sort of exposure to banks, which is very,
very little. And that meant that people couldn't get credit because banks would
only give credit to people that were in the system or people where they had
some sort of handling on whether they would be paid back. Now, the
implementation details of the new system are very very scarce, but the basic idea
is that Chinese citizens are divided into trustworthy individuals and what the
Chinese call "trust breakers". Sometimes you have five different groups, sometimes
you have two different groups, but in general there's sort of this cut-off:
above this line it's good and beyond this line it's bad. This is one graphic from
the Wall Street Journal that just shows some of the inputs that go into the
system. And one of the things that we see is that the inputs are _crazy_ crazy
varied. So it is: do you pay income taxes? Do you pay your utility bills on time?
Do you respect your parents? However they measure that. Do you have a criminal
record? Do you pay for public transportation or have you been caught
not paying? What about your friends?
Do you retweet or use WeChat to distribute sort of information against
the party, which they call reliability. In actuality it's not about whether it's
factual, it's about whether it's against the party or not. Where do you buy and
what do you buy, apparently if you buy diapers it's better than if you buy
videogames. For your score. Because you know if you buy videogames obviously
you're not very responsible. And if you buy diapers you have a kid, you are sort
of conforming to the societal ideal. And then your score is supposed to go into all
these different categories, you're supposed to have better access to social
services if your score is good. You're supposed to have better access to internet
services. So in theory the idea is that at one point if your score is too bad, you're
not allowed to use WeChat anymore. You're not allowed to use Alibaba anymore. You
can't become a government worker. You can not take planes and high speed trains. You
can not get a passport. And your insurance premiums will go up. So it's supposed to
be this really really big, overwhelming system. But in actuality what they say
their stated goals are, is "it's a shorthand for a broad range of efforts to
improve market security and public safety by increasing integrity and mutual trust
in society." So one idea is to allocate resources more efficiently. Resource
allocation in China is a pretty big problem, because people grow up with:
There's 1.3 billion people. So there's - it's always going to be scarce. And a lot
of stuff is – people grow up with this idea that it's just very very scarce, and
current distribution strategies, which are mostly financially based but also often
*guanxi*-based, don't really seem fair. For example, public transport in China is
highly subsidized, which means that the price does not reflect whether – does not
reflect true scarcity. So currently the way it works is in theory it's first come
first serve, in practice there's people that are buying up all the tickets for,
for example, the high speed train from Shanghai to Beijing and then selling it at
a profit, or selling it to certain companies that have good ties to the
government. That seems very unfair. So the new system is supposed to distribute them
more fairly and more efficiently. The other thing is restoring trust in people.
Perceived inter-personal trust and trust in institutions is extremely low in China.
If you're from Germany, you might have heard that there is Chinese gangs
basically buying up German milk powder and selling it in China. This is actually
happening, because in 2008 there was a big scandal with laced milk powder. And ever
since then, anyone who can afford it does not use Chinese milk powder, because they
don't trust the government, or the regulations, the firms, enough to buy
Chinese milk powder so they are actually importing this. And the big irony is:
sometimes this milk powder is produced in China, exported to Germany, and then
exported back to China. The Social Credit system is then supposed to identify those
that deserve the trust. And the third point is sort of a reeducation of people.
The idea is: they want to make people in the image that the Communist Party thinks
people should be. And one additional way to the punishments and rewards this could
work, is the feeling of being surveyed. Because you can't do anything anonymously,
you will automatically adapt your behavior because you know someone is watching you
all the time, and this is how a lot of the Chinese firewall actually works, because
most people I know that are sort of more– more educated, they know ways to
circumvent the Chinese firewall, but they also know that they're always being
watched, so they don't do that because, you know, they're being watched, so they
self– they censorship– they censor themselves. As I said before, allocation
of scarce resources so far is mainly through financial *guanxi* channels. *guanxi*
is basically an all permeating network of relationships with a clear status
hierarchy. So if I attend a school, everyone who also attended this school
will be sort of in my *guanxi* network. And there's this idea that we will have a
system where we are all in-group, and in- group we trust each other and we do favors
for each other, and everyone who's outside of my immediate group I don't trust and I
don't do favors for. And in some ways the *guanxi* system right now is a substitute
for formal institutions in China. For example if you want a passport right now.
You can of course apply for passports through regular channels, which might take
months and months. Or you can apply for a passport through knowing someone and
knowing someone, which might take only two days. Whereas in Germany you have these
very regular, formal institutions, in China they still use *guanxi*. But,
increasingly especially young people find that *guanxi* are very unfair, because a lot
of these are: where you went to school, which is determined by where you're born,
who your parents are, and all these things. Another thing that's important to
understand because: the system works through public shaming. And in a lot of
western society we can't really imagine that, like, I wouldn't really care if my
name was in a newspaper of someone who jaywalked for example. It would be: oh
well, that's okay. But in China this is actually a very very serious thing. So
saving face is very very important in China. And when I went to school there I
actually – we had this dormitory, and it was an all foreigners dormitory, where the
staff that were responsible for the dormitory felt that foreigners were not
behaving in the way they should. So their idea was to put the names, the pictures,
and the offenses of the foreigners in the elevator to shame them publicly. So for
example if you brought a person of the opposite sex to your room, they would put
your name, your offense and your room number in the elevator. And of course this
didn't work because for a lot of western people it was basically like: "oh well I'm
going to try to be there as often as possible because this is like a badge of
honor for me" and the Chinese people they figured "well this is really really shame
and I'm losing my face". She brought alcohol. So this didn't really work at
all. But this is kind of the mindset that is behind a lot of these initiatives. As I
said there's a lot of problems with – we don't really know what's going to happen.
And one of the ways that we can see what might happen is actually to look at pilot
systems. China has – or like ever since the Communist Party took hold – the
Chinese government has tried a lot of policy experimentation. So whenever they
try a new policy, they don't roll it out all over, but they choose different pilot
cities or pilot districts, and then they choose "oh well this is the district where
I'm going to be trying this system and I'm going to be trying another system in
another district or city". And this is also what they did for the, or what
they're doing for the Social Credit system. Now I have three systems that I
looked at intensively for this presentation, overall there's about 70
that I know of - the Suining system, Suining is a city in China, the Rongcheng
system, another city in China and Sesame Credit. Sesame Credit is a commercial
system from Alibaba - I assume everyone knows Alibaba, the're basically the
Chinese Amazon, except they're bigger and have more users and make more money,
actually. And they have their own little system. One of the problems with this kind
of system that I found when I tried modeling it, was that it's a very very
complex system and small changes in input actually changed the output significantly.
So when they try– usually when they try this pilot system they basically have a
couple of pilots, then they choose the pilot that is best and they roll it out
all over. But for this kind of thing, where you have a lot of complex issues, it
might not be the best way to do that. The Suining system is actually considered the
predecessor of all current systems. It had a focus on punishment, and it was quite
interesting. At the beginning of the trial period they published a catalogue of
scores and consequences. Here is an example. This is basically taken from this
catalog. So if you took out bank loans and didn't repay them, you got deducted 50
points. Everyone started with 1000 points for this system. If you didn't pay
back your credit cards you also got deducted 50 points. If you evaded taxes,
also 50 points. If you sold fake goods, 35 points were deducted. And actually the
system was abolished I think in 2015, 2016, because all the Chinese state media
and also a lot of Internet citizens talked about how it's an Orwellian system and how
it's not a good system, because it's all very centralized and everything that you
do is basically recorded centrally. But Creemers writes: "Nonetheless, the Suining
system already contained the embryonic forms of several elements of subsequent
social credit initiatives: The notion of disproportional disincentives against rule
breaking, public naming and shaming of wrongdoers, and most importantly, the
expansion of the credit mechanism outside of the market economic context, also
encompassing compliance with administrative regulations and urban
management rules." So one of the things that is difficult for especially German
speakers is that credit in Chinese, *xìnyòng*, means credit as in "loan", but
also means credit as in "trust". So the Social Credit System is one way of trying
to conflate those two – the economic credit and the trust credit – into one big
system. But the Suining system basically failed. So, they adapted the system and
are now practicing a new kind of system, the Rongcheng system. Whenever you read a
newspaper article on the social credit system in the west, most people went to
Rongcheng because they just received a couple of awards from the Chinese
government for being so advanced at this social credit thing. But it's very
difficult to call this "one system" because there's actually many many
intertwined systems. There is one city level system, where city level offenses
are recorded. For example tax evasion, and there's a couple of rules. If you evade
taxes your score goes down 50. But then if you live in one neighborhood your score
might go up for volunteering with the elderly. If you live in another
neighborhood your score might go up for, for example, planting some trees in
your garden or backyard. So depending on your neighborhood, your score might be
different. If you work for a– if you work for a taxi cab company, for example, they
also have their own little score system and your score might go up if you get good
reviews from your drive…, from your passengers. Your score might go down if
you don't follow traffic rules, these kinds of things. There are designated
scorekeepers at each level. So, each district chooses a couple of people who
are responsible for passing on the information to the next higher level,
about who did what. There is supposed to be an official appeals procedure, so
whenever you score changes you're supposed to be notified, but apparently that's not
happening at this point for most people. Again, it's a system of data sharing, and
one thing that they haven't really disclosed yet is what kind of data is
shared. Are they only sharing the points, so if I'm in a district and I plant some
trees, does the central system get the information "person A planted some trees,"
or does the central system get the information "person A got 5 points?" We
don't know at this point. And it would mean something very different for how the
system could be used. But still the end result, at this point, is that there's one
score. So you have one central score and it's kind of– there's all these different
smaller systems that go into this score. But at the end, everyone has one central
score, and currently about 85 percent of people are between 950 and 1050. So you
start off with a thousand – and those are basically the normal people – and then
anyone above a 1050 is considered a trustworthy person, and anyone below 1050
is considered a trust-breaker. And, as I've said before, with the naming and
shaming and all these things, what you can actually see here is a billboard with the
best trustworthy families in Rongcheng. So these are the families that have the
highest scores, for example. Sesame Credit is a little different. It's the only
system that actually uses machine learning and artificial intelligence to determine
the outputs. In Rongcheng, for example, they have artificial intelligence, they
have computer vision, for the most part, and the computer vision cameras they
decide– they try to recognize you when you jaywalk. And then when they recognize you
when jaywalking, you get a small SMS; "well, we just saw you jaywalking, your
score is now dropping." But how the score develops, depending on your jaywalking,
isn't really determined by machine learning or artificial intelligence.
Instead, it's determined by rules. You know: one time jaywalking deducts five
points, and this is stated somewhere. Sesame Credit doesn't work like that.
Instead it uses a secret algorithm, and the way– I talked to some people that work
for Sesame Credit or for Alibaba, and the way they described it was; they basically
clustered people based on behavior, then gave scores to these clusters, and
then afterwards, did basically reverse engineered their own score, using machine
learning, so that whenever something new happens, you can move to a different
cluster. This Sesame Credit was actually refused accreditation as a credit score in
2017, so banks are not allowed to use the Sesame Credit score for your– to use the
Sesame Credit score to determine whether they give you loans or not. Because Sesame
Credit is quite ingenious – obviously Alibaba wants to keep you within their
platform – so if you buy using Alibaba and using Alipay, your score goes up. If you
buy using Weechatpay, which is a competing platform, your score goes down. This uses
many of the same rewards mechanisms of the official government systems, and this is
just an illustration of what kind of scores you can have, apparently your
scores can go between 350 and 850, and in Chinese there's basically five different
levels. So 385 is a "trust-breaker" or "missing trust". And then 731 is "trust is
exceedingly high". So one way I tried to approach this issue was through agent-
based modeling. Social Credit System is individual level, but what we're really
interested in, or what I'm really interested in, is actually societal-level
consequences. So if everyone gets this score, what does that mean for society?
And agent-based modeling works quite well for that, because it allows us to imbue
agents with some sort of rationality, but with a bounded rationality. What does
bounded rationality mean? Usually in economics people assume agents are
completely rational, so they are profit maximizers, they have all the information.
But in reality, agents don't have all the information, they have a lot of issues
with keeping stuff in their mind. So a lot of the time, they won't choose the best
thing in the world, but they choose the best thing that they see. And bounded
rationality allows us to account for this thing. It allows us to account for
heuristics and these things. And what I did is I took the propensity for specific
behavior from current state of the art research, mostly from behavioral
economics. For example, I looked at tax evasion, and I looked at who is likely to
evade taxes in a system, and then obviously there was some stochastic –
some chance element. But the distribution that I chose is related to
the current research. And I also checked that my model has similar results to the
Rongcheng model, which I modeled at at the beginning. So on average 87% of my users
have a score of within 10 percent of the original score, which is also the data
that Rongcheng city actually publishes. Now, for the most part, I compared design
choices in two axes. One of them was a centralized system versus a multi-level
system, and a rule-based system versus a machine learning system. The centralized
system is basically: you have a central – all the information is kept centrally, and
everyone in China, or wherever, in Rongcheng has the exact same scoring
opportunities. Now, if you have a centralized system the clear expectations
were pretty good. But, at the same time, the acceptance from the population was
really, really low, which they found during the Suining experiment. And
there's also the problem of a single point of failure. Who decides the central
catalog, and, depending on who, sort of, has the power, it kind of, just,
reproduces power structures. So because you have this central catalog, the same
people that are in power centrally, they are basically deciding some sort of score
mechanism that works for them very well, so that they and their family will have
high scores. And multi-level system has the advantage that local adaptation kind
of works, and there's sort of many points of failure. But in my model, when I
allowed locals to basically set their own rules, what happened was that they
competed. So, it started out being this district of Rongcheng, for example, and
this district of Rongcheng, they compete for the best people that they want to
attract, and suddenly you have this kind of race to the bottom, where people want
to move where they wouldn't be prosecuted, so they move to places where there's less
cameras, for example. At the same time, there's many points of failure, especially
the way it's currently set up, with people reporting data to the next high level.
And, a lot of the time, what we have actually seen in Rongcheng, was that they
reported data on people they didn't like more than data on people they did like.
Or, their families got better scores than people they didn't know. So it also kind
of reproduced these biases. The rule based system has the advantage that people were
more prone to adapt their behavior, because they actually knew what they
needed to do in order to adapt their behavior. But the score didn't really
correlate with the important characteristics that they actually cared
about. And, as opposed to in this machine learning system, you know how in Germany
we don't really know the Schufa algorithm. And I, for example, don't exactly know
what I could do in order to improve my Schufa score. And this is a similar system
in China with the Sesame Credit score. A lot of people don't really – they say,
"well I really want to adapt my behavior to the score, to improve my score, but
when I tried doing that my score actually got worse." And you can have different
biases, that I'm going to be talking about in a little bit. There's also this big
problem of incentive mismatch. So, the decentralized, rules-based systems like
Rongcheng, which is the system that I analyzed the most. Why, because I believe
this is the system that we're moving towards right now. Because Rongcheng won a
lot of awards. So the Chinese government, the way they usually work is, they try
pilots, then they choose the best couple of systems, they give them awards, and
then they roll out the system nationwide. So I assume that the system that's going
to be – the system in the end will be similar to the Rongcheng system. Now, one
problem that I actually saw in my simulation was that you could have this
possible race to the bottom. There's also this conflict of interest in those that
set the rules, because a lot of the time, the way it works is, you have your
company, and your company, you, in combination with your party leaders,
actually decide on the rules for the score system. But the scores of all your
employees actually determines your company's score. If you employ a lot of
people with high scores you get a better score. So you will have this incentive to
give out high scores and to make sure that everyone gets high scores. But at the same
time the government has an incentive for scores to be comparable. So there's a lot
of incentives mismatch. The government also has the incentive to keep false
negatives down, but they actually, the way the Chinese system currently works is,
they emphasize catching trust-breakers more than rewarding trust-follow... or
trustworthy people. So, false positives, for them, are less important, but false
positives erode the trust in the system, and they lead to a lot less behavioral
adaptation. I was actually able to show this using some nudging research that
showed that as soon as you introduce an error probability and you can be caught
for something that you didn't do, your probability of changing your behavior
based on this score is actually lower. And in Rongcheng, one of the perverse things
that they're doing is, you can donate money to the party or to, like, party
affiliated social services, and this will give you points, which is kind of an
indulgence system. Which is quite interesting, especially because a lot of
these donation systems work in a way that you can donate 50000 renminbi and you get
50 points, and then you donate another 50000 renminbi and you get another 50
points. So you can basically donate a lot of money and then behave however you want,
and still get a good score. And the trust in other people can actually go down even
more in this system, because suddenly you only trust them because of their scores,
and the current system is set up so that you can actually look up scores of
everyone that you want to work with, and if they don't have a score high enough
then suddenly you don't want to work with them. The trust in the legal system can
also decrease, actually. Why? Because trust in the legal system in China is already
low, and a lot of the things, like jaywalking, they're already illegal in
China, as they are here, but no one cares. And suddenly, you have this parallel
system that punishes you for whatever. But, why don't you just try to fix the
legal system, which would be my approach. Suddenly, illegal activity could happen
more offline, and this is one of those things that is quite interesting. In
countries that we've seen that have moved towards mobile payments, and away
from cash, you see less robberies but you don't actually see less crime. Instead you
see more new types of crime. So, you see more credit card fraud, you see more phone
robberies, these kinds of things. And this is also where things could move in the
Chinese case. One major problem is also that this new system – I've talked a
little bit about this one, but – it can introduce a lot of new bias, and reproduce
the bias even more. So, for example, China is a country of 55 minorities. The Han are
a big majority, they have about 94 percent of the population. So any computer vision
task, we've shown, that they are really, really bad at discriminating between
individuals in smaller ethnic groups. In the U.S., most computer vision tasks
perform worse for African-Americans, they perform worse for women, because all of
the training sets are male and white, and maybe Asian. In China, all of these tasks
are actually performing worse for ethnic minorities, for the Uyghurs, for example.
And one way that they could try to abuse the system is to basically just – what
they're also doing already in Xinjiang is – to basically just identify, "oh this is
a person of the minority, well I'm just going to go and check him or her more
thoroughly." This is actually what happens in Xinjiang. If you're in Xinjiang and you
look like a Turkish person, or like from Turkmenistan, from a Turkish people, you
are a lot more likely to be questioned. You're a lot more likely to be stopped and
they ask you or require you to download spyware on your phone. And this is
currently what happens and this new kind of system can actually help you with that.
I've said that it can reproduce these kind of power structures, and now obviously we
all know neutral technology doesn't really exist, but in the Chinese case, in the
social credit case, they don't even pretend – they always say "well, this
is neutral technology and it's all a lot better," but actually it's the people
currently in power, they decide on what gives you point and what deducts points
for you. Another problem, currently the entire system is set up in a way that it
all goes together with your *shēnfènzhèng*, with your I.D. card. What if you don't
have an I.D. card? That's foreigners for one. But it's also people in China that
were born during the one child policy and were not registered. There's quite a lot
of them, actually. They're not registered anywhere and suddenly they can't do
anything, because they don't have a score, they can't get a phone, they can't do
anything, really. And part of the push with this social credit system is to go
away from cash, actually. So if you need to use your phone to pay, but for your
phone you need your *shēnfènzhèng*. If you don't have a *shēnfènzhèng*,
well, tough luck for you.
And currently the system in Rongcheng is set up in a way that you can check
other people's scores and you can also see what they lose points for. So you can
actually, sort of, choose to discriminate against people that are gay, for example,
because they might have lost points for going to a gay bar, which you can lose
points for. Another big issue, currently, is data privacy and security. Personal
data is grossly undervalued in China. If you ask a Chinese person, "what do you
think, how much is your data worth?," they say "what data? I don't have data." And,
currently, the way it works is, if you have someone's ID number, which is quite
easy to find out, you can actually buy access to a lot of personal information
for a small fee. So you pay about 100 euros and you get all hotel bookings of
the last year, you get information of who booked these hotels with them, you get
information of where they stay, you get train bookings, you get access to all of
the official databases for this one person. And for another 700 renminbi you
can actually get live location data, so you can get the data of where this person
is right now, or where his or her phone is right now, but if you've ever been to
China you know that where the phone is, usually, the people aren't far. Supchina
actually did an experiment where a couple of journalists tried buying that, because
it's actually these kind of services are offered on weechat, pretty publicly. And
you can just buy them, quite easily. So one additional thing that I looked at is,
because one of the things that is quite interesting is, you have this idea of
credit as twofold. Credit is trust credit but credit is also loan credit, and what
if credit institutions actually use this unified credit score to determine credit
distribution? The idea is that it's supposed to lead to reduced information
asymmetry, obviously, so fewer defaults and overall more credit creation. New
people are supposed to get access to credit, and there's supposed to be less
shadow banking. But what actually happens? I'm not going to be talking about how I
set up the model but just about my results. If you have this kind of score
that includes credit information but also includes morally good – or measures of
being morally good – what you have is, in the beginning, about 30 percent more
agents get access to credit, and especially people that previously have not
gotten credit access suddenly have credit access. But the problem is that this
social credit score that correlates all of these different issues, it correlates only
very, very weakly with repayment ability or repayment wishes, and thus suddenly you
have all of these non-performing loans. You have – and what we see is sort of
like – we have non-performing loans. Banks give out less loans because they
have so many non-performing loans, and then the non-performing loans are written
off, and suddenly banks give out more loans. But you have this oscillating
financial system, where you give out a lot of loans, a lot of them are non-
performing, then you give out a lot of loans again. And this is very, very
vulnerable to crisis. If you have a real economic crisis during the time where non-
performing loans are high, then a lot of banks will actually default, which is
very, very dangerous for a financial system as nationed as the Chinese one.
Now, what are some possible corrections? You could create a score that basically is
the same as the Schufa score. So that it looks only at credit decisions, but
suddenly, you lose a lot of incentives for the social credit score, if the social
credit score doesn't matter for credit distribution anymore.
Another thing, and this is, I think, the more likely one,
is that you have a blacklist for people that have not repaid a loan
in the past. So you basically get one freebie, and afterwards
if you didn't repay your loan in the past then you will not get a loan in the
future. You will still be part of the social credit system, and your social
credit score will still be important for all of these other access issues, but it
won't be important for access to loans anymore, once you've been on this
blacklist. Which is probably something that the Chinese government could go
behind, but it's also more effort to take care of it; then you have to think about,
"well, you can't leave them on the blacklist forever, so how long do you
leave them on the black list? Do they have to pay back the loan and then they get off
the blacklist? Or do they have to pay back the loan and then stay not in default
for a year, or for five years?" There's a lot of small decisions that, in my
opinion, the Chinese government hasn't really thought about, up until now,
because they're basically doing all these pilot studies, and all of these regional
governments are thinking of all these small things, but they're not documenting
everything that they're doing. So, once they – they want to roll it out by 2020,
by the way, nationwide – once they've rolled it out there's a pretty big chance,
in my opinion, that they'll have a lot of unintended consequences. A lot of things
that they haven't thought about, and that they will then have to look at. So, I
believe that some sort of system is likely to come, just in terms of how much energy
they've expended into this one, and for the Chinese government at this point, for
the party, it would be losing face if they did not include any such system, because
they've been talking about this for a while. But most likely, it would be a kind
of decentralized data sharing system. And when I ran my simulation... By the way I
will make public my code, I still need some, basically, I used some proprietary
data for my model, and I still need the permission to publish this. Once I publish
this one I will also tweet it, and we'll put it on GitHub for everyone to play
around with, if you want to. And some of these implementation details that were
very important in determining model outcomes where "do we have a relative or
absolute ranking?" So far, all of the systems I looked at had absolute rankings,
but there's a point to be made for relative rankings. Do we have one score,
where, basically, if you're a Chinese person you get one score? Or do we have
different sub-scores in different fields? Do we have people reporting behavior, or
do we have automatic behavior recording? How do you access other people's scores?
How much information can you get from other people's scores? Currently, if
someone is on a blacklist, for example, if you have their ID number, again, you can
put it into this blacklist, and then they will say "oh, this person is on this
blacklist for not following this judge's order," and then it says what kind of
judge's order it was. So, most likely, it will be something like this. The idea is
that the Social Credit system isn't only for individuals, but also for firms and
for NGOs. So, what kind of roles will firms play in the system? I haven't looked
at that, in detail, at this point, but it would be very interesting. Another idea
that western people often talk about is, do people also rank each other? Currently,
that's not part of the system in China, but it might be at one point. And lastly,
where does the aggregation happen? So I've said that a lot of it is actually data
sharing in China. So what kind of data is shared? Is the raw data shared? "Person A
did something." Or is the aggregated data shared? "Person A got this score." At this
point, most of the time, it is actually the raw data that is shared, but that also
has sort of these data privacy issues, of course, that I've talked about. OK,
perfect! No there's 10 more minutes. Thank you for your attention! If you have
questions, remarks you can ask them now or you can catch me up later. You can tweet
to me or send me an e-mail, whatever you're interested in. Thank you very much!
*applause*
Herald Angel: Hello! As Toni said, we have 10 minutes left for questions. If you have
a question in the room, please go crouch in front of our five microphones. If you're
watching the stream, please ask your questions through IRC or Twitter, and
we'll also try to make sure to get to those. Let's just go ahead and start with
mic one. Question: Good! Thank you very much for
this beautiful talk. I was wondering how did the Chinese government, companies, and
most of all, the citizens themselves, respond to you doing this research, or,
let's put it differently, if you would have been in the system yourself,
how would your research affect your social credit score?
*laughter*
Answer: So, um... There's actually two different responses that I've seen. When I
talk to the government themselves, because I was there on a government scholarship,
and mentioned that I'm really interested in this, they basically said oh well this
is just a technical system. You don't really need to be concerned with this. It
is not very important. Just, you know, it's just a technicality. It's just for us
to make life more efficient and better for everyone. So I assume my score would
actually go down from doing this research, actually. But when I talk to a lot of
people at universities, they were also very – they were very interested in my
research, and a lot of them mentioned that they didn't even know that the system
existed! Herald: Before we go to a question from
our signal angel, a request for all the people leaving the room, please do so as
quietly as possible, so we can continue this Q and A. The signal angel, please!
Signal Angel: Jaenix wants to know, is this score actually influenced by
association with people with a low score. Meaning that, is there any peer pressure
to stay away from people with bad scores? Answer: The Sesame credit score definitely
is influenced by your friends' scores, the Rongcheng score, so far, apparently, is
not influenced, but it is definitely in the cards, and it is planned that it will
be part of this. I think WeChat, which is the main platform – it's sort of like
WhatsApp, except it can do a lot a lot more – WeChat is still not connected to
the Social Credit Score in Rongcheng. Once they do that, it will most likely also
reflect your score. Herald: All right, let's continue with
mic 3. Q: I have a question about your models.
I'm wondering, what kind of interactions are you modeling? Or actions, like, what
can the agents actually do? You mentioned moving somewhere else. And, what else?
A: Okay so the way I set up my model was, I set up a multilevel model. So I looked
at different kinds of levels. I started out with, basically, they can evade taxes,
they can get loans and repay loans, they can choose where to live, and they can
follow traffic rules or not follow traffic rules. And because these were, sort of,
four big issues that were mentioned in all of the different systems, so I started out
with these issues, and looked at, what kind of behavior do I see? I used some
research that – some friends of mine actually sent out surveys to people and
asked them "well, you're now part of the system. Did your behavior change, and how
did it change depending on your responses, depending on your score, and depending on
the score system that exists?" And I, basically, used that, and some other
research on nudging and on behavioral adaptation, to look at how likely is it
that someone would change their behavior based on the score.
Herald: All right let's do another question from the interwebs.
Q: Yeah, it's actually two questions in one. How does this system work for Chinese
people living abroad, or for noncitizens that do business in China?
A: Currently the system does not work for noncitizens that do business in China,
because it works through the *shēnfènzhèng*. You only get a *shēnfènzhèng* if you're a
Chinese citizen or you live in China for 10 or more years. So everyone who is not
Chinese is currently excluded. Chinese people not living in China, if they have a
*shēnfènzhèng*, are on this system, but there's not a lot of information.
Herald: All right, mic 4. Q: Well, we've come a long way since the
*Volkszählungsurteil*. Can you tell us anything about the dynamic in the
time dimension? How quickly can I regain credit that was lost? Do you have any
observations there? A: So in the Suining system what they
actually did was they had a very, very strict period. So if you evaded taxes your
score would be down for two years and then it would rebounce. In the Rongcheng
system, they did not publish this kind of period. So, my assumption is that it's
going to be more on a case by case basis. Because, I looked at the Chinese data, I
looked at the Chinese policy documents, and they didn't really, for most of the
stuff, they didn't say how long it would count. For the blacklists, which was kind
of the predecessor that we look at currently, the way it works is you stay on
there until whatever the blacklist – until whatever the reason for the
blacklist is has been resolved. So, you stay on there until you send off this
apology that the judge ordered you to. And then, usually, you still needed to apply to
get off. So it doesn't – for blacklists, it does not work that you automatically
get off. You need to apply, you need to show that you've done what they've asked
you to do, and then you can get off this blacklist. And I assume it will be a
similar sort of appeals procedure for the system.
Herald: All right. Let's go to mic 2. Q: Thank you. I just wanted to if looking
up someone else's data in details, like position et cetera, does affect your own
score? A: Currently, it apparently does not, or
at least they haven't published that it does. It might in the future, but most
likely it's actually behavior that they want. So they want you to look up other
people's scores before doing business with them. They want you to, basically, use
this to decide who you're going to associate with.
Q: Thank you! Herald: All right, do we have another
question from the Internet, maybe? Signal: Yes, I do! Standby... The question
is, how is this actually implemented for the offline rural population in China?
A: Quite easily; not at all at this point. The idea is, by 2020, that they will
actually have all of this is implemented. But even for the offline – or let's say
offline rural population in China is getting smaller and smaller. Even in rural
villages you have about 50-60% of people that are online. And most of them are
online via smartphone, and their smartphone is connected to the
*shēnfènzhèng*. So it's not very complicated to do that for everyone who is online. For
everyone who's offline, off course, this is more problematic, but I think the end
goal is to not have people offline at all. Herald: All right. Let's jump right back
to microphone 2, please. Q: Thank you for the very good and
frightening talk, so far. At first I have to correct you in one point. In Germany we
have a similar system because we have this tax I.D., which is set from birth on and
rests 30 years after a person's dead. Yeah. So we have a lifelong I.D.
A: You're right. I just... I don't know mine, so I figured… *dismissive sound.*
Q: No problem! But, at least we could establish a similar system, if we have a
government which would want it. A question for you: you mentioned this "*guanxi*." Is
it a kind of a social network? I didn't understand it, really.
A: Yes, it is a kind of social network, but one that is a lot more based on
hierarchies than it is in the West. So you have people that are above you and people
that are below you. And the expectation is that, while it's a quid pro quo, people
that are above you in the hierarchy will give you less than you will give to them.
Q: Aha, okay. Herald: OK, all right. Unfortunately, we
are out of time, so, please give another huge applause for Toni!
*applause*
*postroll music*
subtitles created by c3subtitles.de in the year 2019. Join, and help us!