*34C3 preroll music*
Herald: Humans of Congress, it is my pleasure to announce the next speaker.
I was supposed to pick out a few awards or something, to actually present what he's
done in his life, but I can only say: he's one of us!
*applause*
Charles Stross! *ongoing applause*
Charles Stross: Hi! Is this on? Good. Great.
I'm really pleased to be here and I want to start by apologizing for my total
lack of German. So this talk is gonna be in English. Good morning. I'm Charlie Stross
and it's my job to tell lies for money, or rather, I write science fiction, much of
it about on the future, which in recent years has become ridiculously hard to
predict. In this talk I'm going to talk about why. Now our species, Homo sapiens
sapiens, is about 300,000 years old. It used to be about 200,000 years old,
but it grew an extra 100,000 years in the past year because of new
archaeological discoveries, I mean, go figure. For all but the last three
centuries or so - of that span, however - predicting the future was really easy. If
you were an average person - as opposed to maybe a king or a pope - natural disasters
aside, everyday life 50 years in the future would resemble everyday life 50
years in your past. Let that sink in for a bit. For 99.9% of human existence on this
earth, the future was static. Then something changed and the future began to
shift increasingly rapidly, until, in the present day, things are moving so fast,
it's barely possible to anticipate trends from one month to the next. Now as an
eminent computer scientist, Edsger Dijkstra once remarked, computer science is no more
about computers than astronomy is about building big telescopes, the same can be
said of my field of work, writing science fiction, sci-fi is rarely about science
and even more rarely about predicting the future, but sometimes we dabble in
Futurism and lately, Futurism has gotten really, really, weird. Now when I write a
near future work of fiction, one set, say, a decade hence, there used to be a recipe I
could follow, that worked eerily well. Simply put: 90% of the next decade stuff is
already here around us today. Buildings are designed to
last many years, automobiles have a design life of about a decade, so half the cars on
the road in 2027 are already there now - they're new. People? There'll be some new
faces, aged 10 and under, and some older people will have died, but most of us
adults will still be around, albeit older and grayer, this is the 90% of a near
future that's already here today. After the already existing 90%, another 9% of a
near future a decade hence used to be easily predictable: you look at trends
dictated by physical limits, such as Moore's law and you look at Intel's road
map and you use a bit of creative extrapolation and you won't go too far
wrong. If I predict - wearing my futurology hat - that in 2027 LTE cellular phones will
be ubiquitous, 5G will be available for high bandwidth applications and there will be
fallback to some kind of satellite data service at a price, you probably won't
laugh at me. I mean, it's not like I'm predicting that
airlines will fly slower and Nazis will take over the United States, is it ?
*laughing*
And therein lies the problem. There is remaining 1% of what Donald Rumsfeld
called the "unknown unknowns", what throws off all predictions. As it happens, airliners
today are slower than they were in the 1970s and don't get me started about the Nazis,
I mean, nobody in 2007 was expecting a Nazi revival in 2017, were they?
Only this time, Germans get to be the good guys. *laughing, applause*
So. My recipe for fiction set 10 years in the future used to be:
"90% is already here, 9% is not here yet but predictable
and 1% is 'who ordered that?'" But unfortunately the ratios have changed, I think we're now
down to maybe 80% already here - climate change takes a huge toll on architecture -
then 15% not here yet, but predictable and a whopping 5% of utterly unpredictable
deep craziness. Now... before I carry on with this talk, I want to spend a minute or
two ranting loudly and ruling out the singularity. Some of you might assume, that
as the author of books like "Singularity Sky" and "Accelerando",
I expect an impending technological singularity,
that we will develop self-improving artificial intelligence and mind uploading
and the whole wish list of transhumanist aspirations promoted by the likes of
Ray Kurzweil, will come to pass. Unfortunately this isn't the case. I think transhumanism
is a warmed-over Christian heresy. While its adherents tend to be outspoken atheists,
they can't quite escape from the history that gave rise to our current
Western civilization. Many of you are familiar with design patterns, an approach
to software engineering that focuses on abstraction and simplification, in order
to promote reusable code. When you look at the AI singularity as a narrative and
identify the numerous places in their story where the phrase "and then a miracle
happens" occur, it becomes apparent pretty quickly, that they've reinvented Christiantiy.
*applause*
Indeed, the wellspring of today's transhumanists draw in a long rich
history of Russian philosophy, exemplified by the russian orthodox theologian Nikolai
Fyodorovich Fedorov by way of his disciple Konstantin Tsiolkovsky, whose derivation
of a rocket equation makes him essentially the father of modern space
flight. Once you start probing the nether regions of transhumanist forth and run
into concepts like Roko's Basilisk - by the way, any of you who didn't know about the
Basilisk before, are now doomed to an eternity in AI hell, terribly sorry - you
realize, they've mangled it to match some of the nastier aspects of Presbyterian
Protestantism. Now they basically invented original sin and Satan in the guise of an
AI that doesn't exist yet ,it's.. kind of peculiar. Anyway, my take on the
singularity is: What if something walks like a duck and quacks like a duck? It's
probably a duck. And if it looks like a religion, it's probably a religion.
I don't see much evidence for human-like, self-directed artificial intelligences
coming along any time soon, and a fair bit of evidence, that nobody accepts and freaks
in cognitive science departments, even want it. I mean, if we invented an AI
that was like a human mind, it would do the AI equivalent of sitting on the sofa,
munching popcorn and watching the Super Bowl all day.
It wouldn't be much use to us. *laughter, applause*
What we're getting instead, is self-optimizing tools that defy
human comprehension, but are not in fact any more like our kind
of intelligence than a Boeing 737 is like a seagull. Boeing 737s and seagulls both
fly, Boeing 737s don't lay eggs and shit everywhere. So I'm going to wash my hands
of a singularity as a useful explanatory model of the future without further ado.
I'm one of those vehement atheists as well and I'm gonna try and offer you a better
model for what's happening to us. Now, as my fellow Scottish science fictional author
Ken MacLeod likes to say "the secret weapon of science fiction is history".
History is, loosely speaking, is the written record of what and how people did things
in past times. Times that have slipped out of our personal memories. We science
fiction writers tend to treat history as a giant toy chest to raid, whenever we feel
like telling a story. With a little bit of history, it's really easy to whip up an
entertaining yarn about a galactic empire that mirrors the development and decline
of a Habsburg Empire or to respin the October Revolution as a tale of how Mars
got its independence. But history is useful for so much more than that.
It turns out, that our personal memories don't span very much time at all. I'm 53
and I barely remember the 1960s. I only remember the 1970s with the eyes of a 6 to
16 year old. My father died this year, aged 93, and he'd just about remembered the
1930s. Only those of my father's generation directly remember the Great
Depression and can compare it to the 2007/08 global financial crisis directly.
We Westerners tend to pay little attention to cautionary tales told by 90-somethings.
We're modern, we're change obsessed and we tend to repeat our biggest social mistakes
just as they slip out of living memory, which means they recur on a timescale of
70 to 100 years. So if our personal memories are useless,
we need a better toolkit and history provides that toolkit.
History gives us the perspective to see what went wrong in the past and to look for
patterns and check to see whether those patterns are recurring in the present.
Looking in particular at the history of the past two to four hundred years, that age of rapidly
increasing change that I mentioned at the beginning. One glaringly obvious deviation
from the norm of the preceding 3000 centuries is obvious, and that's
the development of artificial intelligence, which happened no earlier than 1553 and no
later than 1844. I'm talking of course about the very old, very slow AI's we call
corporations. What lessons from the history of a company can we draw that tell us
about the likely behavior of the type of artificial intelligence we're interested
in here, today? Well. Need a mouthful of water.
Let me crib from Wikipedia for a moment.
Wikipedia: "In the late 18th century, Stewart Kyd, the author of the
first treatise on corporate law in English, defined a corporation as: 'a collection of
many individuals united into one body, under a special denomination, having
perpetual succession under an artificial form, and vested, by policy of the law, with
the capacity of acting, in several respects, as an individual, enjoying privileges and
immunities in common, and of exercising a variety of political rights, more or less
extensive, according to the design of its institution, or the powers conferred upon
it, either at the time of its creation, or at any subsequent period of its
existence.'" This was a late 18th century definition,
sound like a piece of software to you? In 1844, the British government passed the
"Joint Stock Companies Act" which created a register of companies and allowed any
legal person, for a fee, to register a company which in turn existed as a
separate legal person. Prior to that point, it required a Royal Charter or an act of
Parliament to create a company. Subsequently, the law was extended to limit
the liability of individual shareholders in event of business failure and then both
Germany and the United States added their own unique twists to what today we see is
the doctrine of corporate personhood. Now, though plenty of other things that
happened between the 16th and 21st centuries did change the shape of the world we live in.
I've skipped the changes in agricultural productivity that happened
due to energy economics, which finally broke the Malthusian trap
our predecessors lived in. This in turn broke the long-term
cap on economic growth of about 0.1% per year
in the absence of famines, plagues and wars and so on.
I've skipped the germ theory of diseases and the development of trade empires
in the age of sail and gunpowder, that were made possible by advances
in accurate time measurement.
I've skipped the rise, and hopefully decline, of the pernicious
theory of scientific racism that underpinned Western colonialism and the
slave trade. I've skipped the rise of feminism, the ideological position that
women are human beings rather than property and the decline of patriarchy.
I've skipped the whole of the Enlightenment and the Age of Revolutions,
but this is a technocratic.. technocentric Congress, so I want to frame this talk in
terms of AI, which we all like to think we understand. Here's the thing about these
artificial persons we call corporations. Legally, they're people. They have goals,
they operate in pursuit of these goals, they have a natural life cycle.
In the 1950s, a typical U.S. corporation on the S&P 500 Index had a life span of 60 years.
Today it's down to less than 20 years. This is largely due to predation.
Corporations are cannibals, they eat one another.
They're also hive super organisms like bees or ants.
For the first century and a half, they relied entirely on human
employees for their internal operation, but today they're automating their
business processes very rapidly. Each human is only retained so long as they can
perform their assigned tasks more efficiently than a piece of software
and they can all be replaced by another human, much as the cells in our own bodies
are functionally interchangeable and a group of cells can - in extremis - often be
replaced by a prosthetic device. To some extent, corporations can be
trained to serve of the personal desires of their chief executives, but even CEOs can
be dispensed with, if their activities damage the corporation, as Harvey
Weinstein found out a couple of months ago.
Finally, our legal environment today has been tailored for the convenience of
corporate persons, rather than human persons, to the point where our governments
now mimic corporations in many of our internal structures.
So, to understand where we're going, we need to start by asking "What do our
current actually existing AI overlords want?"
Now, Elon Musk, who I believe you've all heard of, has an obsessive fear of one
particular hazard of artificial intelligence, which he conceives of as
being a piece of software that functions like a brain in a box, namely the
Paperclip Optimizer or Maximizer. A Paperclip Maximizer is a term of art for
a goal seeking AI that has a single priority, e.g., maximizing the
number of paperclips in the universe. The Paperclip Maximizer is able to improve
itself in pursuit of its goal, but has no ability to vary its goal, so will
ultimately attempt to convert all the metallic elements in the solar system into
paperclips, even if this is obviously detrimental to the well-being of the
humans who set it this goal. Unfortunately I don't think Musk
is paying enough attention, consider his own companies.
Tesla isn't a Paperclip Maximizer, it's a battery Maximizer.
After all, a battery.. an electric car is a battery with wheels and
seats. SpaceX is an orbital payload Maximizer, driving down the cost of space
launches in order to encourage more sales for the service it provides. SolarCity is
a photovoltaic panel maximizer and so on. All three of the.. Musk's very own slow AIs
are based on an architecture, designed to maximize return on shareholder
investment, even if by doing so they cook the planet the shareholders have to live
on or turn the entire thing into solar panels.
But hey, if you're Elon Musk, thats okay, you're gonna retire on Mars anyway.
*laughing*
By the way, I'm ragging on Musk in this talks, simply because he's the current
opinionated tech billionaire, who thinks for disrupting a couple of industries
entitles him to make headlines. If this was 2007 and my focus slightly
difference.. different, I'd be ragging on Steve Jobs and if we're in 1997 my target
would be Bill Gates. Don't take it personally, Elon.
*laughing*
Back to topic. The problem of corporations is, that despite their overt
goals, whether they make electric vehicles or beer or sell life insurance policies,
they all have a common implicit Paperclip Maximizer goal: to generate revenue. If
they don't make money, they're eaten by a bigger predator or they go bust. It's as
vital to them as breathing is to us mammals. They generally pursue their
implicit goal - maximizing revenue - by pursuing their overt goal.
But sometimes they try instead to manipulate their environment, to ensure
that money flows to them regardless. Human toolmaking culture has become very
complicated over time. New technologies always come with an attached implicit
political agenda that seeks to extend the use of the technology. Governments react
to this by legislating to control new technologies and sometimes we end up with
industries actually indulging in legal duels through the regulatory mechanism of
law to determine, who prevails. For example, consider the automobile. You
can't have mass automobile transport without gas stations and fuel distribution
pipelines. These in turn require access to whoever
owns the land the oil is extracted from under and before you know it, you end up
with a permanent army in Iraq and a clamp dictatorship in Saudi Arabia. Closer to
home, automobiles imply jaywalking laws and drink-driving laws. They affect Town
Planning regulations and encourage suburban sprawl, the construction of human
infrastructure on a scale required by automobiles, not pedestrians.
This in turn is bad for competing transport technologies, like buses or
trams, which work best in cities with a high population density. So to get laws
that favour the automobile in place, providing an environment conducive to
doing business, automobile companies spend money on political lobbyists and when they
can get away with it, on bribes. Bribery needn't be blatant of course. E.g.,
the reforms of a British railway network in the 1960s dismembered many branch lines
and coincided with a surge in road building and automobile sales. These
reforms were orchestrated by Transport Minister Ernest Marples, who was purely a
politician. The fact that he accumulated a considerable personal fortune during this
period by buying shares in motorway construction corporations, has nothing to
do with it. So, no conflict of interest there - now if the automobile in industry
can't be considered a pure Paperclip Maximizer... sorry, the automobile
industry in isolation can't be considered a pure Paperclip Maximizer. You have to
look at it in conjunction with the fossil fuel industries, the road construction
business, the accident insurance sector and so on. When you do this, you begin to
see the outline of a paperclip-maximizing ecosystem that invades far-flung lands and
grinds up and kills around one and a quarter million people per year. That's
the global death toll from automobile accidents currently, according to the World
Health Organization. It rivals the First World War on an ongoing permanent basis
and these are all side effects of its drive to sell you a new car. Now,
automobiles aren't of course a total liability. Today's cars are regulated
stringently for safety and, in theory, to reduce toxic emissions. They're fast,
efficient and comfortable. We can thank legal mandated regulations imposed by
governments for this, of course. Go back to the 1970s and cars didn't have crumple
zones, go back to the 50s and they didn't come with seat belts as standard. In the
1930s, indicators, turn signals and brakes on all four wheels were optional and your
best hope of surviving a 50 km/h-crash was to be thrown out of a car and land somewhere
without breaking your neck. Regulator agencies are our current
political system's tool of choice for preventing Paperclip Maximizers from
running amok. Unfortunately, regulators don't always work. The first failure mode
of regulators that you need to be aware of is regulatory capture, where regulatory
bodies are captured by the industries they control. Ajit Pai, Head of American Federal
Communications Commission, which just voted to eliminate net neutrality rules in the
U.S., has worked as Associate General Counsel for Verizon Communications
Inc, the largest current descendant of the Bell Telephone system's monopoly. After
the AT&T antitrust lawsuit, the Bell network was broken up into the seven baby
bells. They've now pretty much reformed and reaggregated and Verizon is the largest current one.
Why should someone with a transparent interest in a technology corporation end
up running a regulator that tries to control the industry in question? Well, if
you're going to regulate a complex technology, you need to recruit regulators
from people who understand it. Unfortunately, most of those people are
industry insiders. Ajit Pai is clearly very much aware of how Verizon is
regulated, very insightful into its operations and wants to do something about
it - just not necessarily in the public interest.
*applause* When regulators end up staffed by people
drawn from the industries they're supposed to control, they frequently end up working
with their former office mates, to make it easier to turn a profit, either by raising
barriers to keep new insurgent companies out or by dismantling safeguards that
protect the public. Now a second problem is regulatory lag where a technology
advances so rapidly, that regulations are laughably obsolete by the time they're
issued. Consider the EU directive requiring cookie notices on websites to
caution users, that their activities are tracked and their privacy may be violated.
This would have been a good idea in 1993 or 1996, but unfortunatelly it didn't show up
until 2011. Fingerprinting and tracking mechanisms have nothing to do with cookies
and were already widespread by then. Tim Berners-Lee observed in 1995, that five
years worth of change was happening on the web for every 12 months of real-world
time. By that yardstick, the cookie law came out nearly a century too late to do
any good. Again, look at Uber. This month, the European Court of Justice ruled that
Uber is a taxi service, not a Web App. This is arguably correct - the problem is, Uber
has spread globally since it was founded eight years ago, subsidizing its drivers to
put competing private hire firms out of business. Whether this is a net good for
societys own is debatable. The problem is, a taxi driver can get awfully hungry if she
has to wait eight years for a court ruling against a predator intent on disrupting
her business. So, to recap: firstly, we already have Paperclip Maximizers and
Musk's AI alarmism is curiously mirror blind. Secondly, we have mechanisms for
keeping Paperclip Maximizers in check, but they don't work very well against AIs that
deploy the dark arts, especially corruption and bribery and they're even
worse against true AIs, that evolved too fast for human mediated mechanisms like
the law to keep up with. Finally, unlike the naive vision of a Paperclip Maximizer
that maximizes only paperclips, existing AIs have multiple agendas, their overt
goal, but also profit seeking, expansion into new markets and to accommodate the
desire of whoever is currently in the driving seat.
*sighs*
Now, this brings me to the next major heading in this dismaying laundry list:
how it all went wrong. It seems to me that our current political upheavals, the best
understood, is arising from the capture of post 1917 democratic institutions by
large-scale AI. Everywhere you look, you see voters protesting angrily against an
entrenched establishment, that seems determined to ignore the wants and needs
of their human constituents in favor of those of the machines. The brexit upset
was largely result of a protest vote against the British political
establishment, the election of Donald Trump likewise, with a side order of racism
on top. Our major political parties are led by people who are compatible with the
system as it exists today, a system that has been shaped over decades by
corporations distorting our government and regulatory environments. We humans live in
a world shaped by the desires and needs of AI, forced to live on their terms and we're
taught, that we're valuable only to the extent we contribute to the rule of the
machines. Now this is free sea and we're all more interested in computers and
communications technology than this historical crap. But as I said earlier,
history is a secret weapon, if you know how to use it. What history is good for, is
enabling us to spot recurring patterns that repeat across timescales outside our
personal experience. And if we look at our historical very slow AIs, what do we learn
from them about modern AI and how it's going to behave? Well to start with, our
AIs have been warped, the new AIs, the electronic one's instantiated in our
machines, have been warped by a terrible fundamentally flawed design decision back
in 1995, but as damaged democratic political processes crippled our ability
to truly understand the world around us and led to the angry upheavals and upsets
of our present decade. That mistake was the decision, to fund the build-out of a
public World Wide Web as opposed to be earlier government-funded corporate and
academic Internet by monetizing eyeballs through advertising
revenue. The ad-supported web we're used to today wasn't inevitable. If you recall
the web as it was in 1994, there were very few ads at all and not much, in a way, of
Commerce. 1995 was the year, the World Wide Web really came to public attention in the
anglophone world and consumer-facing websites began to appear. Nobody really
knew, how this thing was going to be paid for. The original .com bubble was all
about working out, how to monetize the web for the first time and a lot of people
lost their shirts in the process. A naive initial assumption was that the
transaction cost of setting up a tcp/ip connection over modem was too high to
support.. to be supported by per-use micro billing for web pages. So instead of
charging people fraction of a euro cent for every page view, we'd bill customers
indirectly, by shoving advertising banners in front of their eyes and hoping they'd
click through and buy something. Unfortunately, advertising is in an
industry, one of those pre-existing very slow AI ecosystems I already alluded to.
Advertising tries to maximize its hold on the attention of the minds behind each
human eyeball. The coupling of advertising with web search was an inevitable
outgrowth, I mean how better to attract the attention of reluctant subjects, than to
find out what they're really interested in seeing and selling ads that relate to
those interests. The problem of applying the paperclip maximize approach to
monopolizing eyeballs, however, is that eyeballs are a limited, scarce resource.
There are only 168 hours in every week, in which I can gaze at banner ads. Moreover,
most ads are irrelevant to my interests and it doesn't matter, how often you flash an ad
for dog biscuits at me, I'm never going to buy any. I have a cat. To make best
revenue-generating use of our eyeballs, it's necessary for the ad industry to
learn, who we are and what interests us and to target us increasingly minutely in hope
of hooking us with stuff we're attracted to.
In other words: the ad industry is a paperclip maximizer, but for its success,
it relies on developing a theory of mind that applies to human beings.
*sighs*
Do I need to divert on to the impassioned rant about the hideous corruption
and evil that is Facebook? Audience: Yes!
CS: Okay, somebody said yes. I'm guessing you've heard it all before,
but for too long don't read.. summary is: Facebook is as much a search engine as
Google or Amazon. Facebook searches are optimized for faces, that is for human
beings. If you want to find someone you fell out of touch with thirty years ago,
Facebook probably knows where they live, what their favorite color is, what sized
shoes they wear and what they said about you to your friends behind your back all
those years ago, that made you cut them off. Even if you don't have a Facebook account,
Facebook has a You account, a hole in their social graph of a bunch of connections
pointing in to it and your name tagged on your friends photographs. They know a lot
about you and they sell access to their social graph to advertisers, who then
target you, even if you don't think you use Facebook. Indeed, there is barely any
point in not using Facebook these days, if ever. Social media Borg: "Resistance is
futile!" So however, Facebook is trying to get eyeballs on ads, so is Twitter and so
are Google. To do this, they fine-tuned the content they show you to make it more
attractive to your eyes and by attractive I do not mean pleasant. We humans have an
evolved automatic reflex to pay attention to threats and horrors as well as
pleasurable stimuli and the algorithms, that determine what they show us when we
look at Facebook or Twitter, take this bias into account. You might react more
strongly to a public hanging in Iran or an outrageous statement by Donald Trump than
to a couple kissing. The algorithm knows and will show you whatever makes you pay
attention, not necessarily what you need or want to see.
So this brings me to another point about computerized AI as opposed to corporate
AI. AI algorithms tend to embody the prejudices and beliefs of either the
programmers, or the data set the AI was trained on.
A couple of years ago I ran across an account of a webcam, developed by mostly
pale-skinned Silicon Valley engineers, that had difficulty focusing or achieving correct
color balance, when pointed at dark-skinned faces.
Fast an example of human programmer induced bias, they didn't have a wide
enough test set and didn't recognize that they were inherently biased towards
expecting people to have pale skin. But with today's deep learning, bias can creep
in, while the datasets for neural networks are trained on, even without the programmers
intending it. Microsoft's first foray into a conversational chat bot driven by
machine learning, Tay, was what we yanked offline within days last year, because
4chan and reddit based trolls discovered, that they could train it towards racism and
sexism for shits and giggles. Just imagine you're a poor naive innocent AI who's just
been switched on and you're hoping to pass your Turing test and what happens? 4chan
decide to play with your head. *laughing*
I got to feel sorry for Tay. Now, humans may be biased,
but at least individually we're accountable and if somebody gives you
racist or sexist abuse to your face, you can complain or maybe punch them. It's
impossible to punch a corporation and it may not even be possible to identify the
source of unfair bias, when you're dealing with a machine learning system. AI based
systems that instantiate existing prejudices make social change harder.
Traditional advertising works by playing on the target customer's insecurity and
fear as much as their aspirations. And fear of a loss of social status and privileges
are powerful stress. Fear and xenophobia are useful tools for tracking advertising..
ah, eyeballs. What happens when we get pervasive social
networks, that have learned biases against say Feminism or Islam or melanin? Or deep
learning systems, trained on datasets contaminated by racist dipshits and their
propaganda? Deep learning systems like the ones inside Facebook, that determine which
stories to show you to get you to pay as much attention as possible to be adverse.
I think, you probably have an inkling of how.. where this is now going. Now, if you
think, this is sounding a bit bleak and unpleasant, you'd be right. I write sci-fi.
You read or watch or play sci-fi. We're acculturated to think of science and
technology as good things that make life better, but this ain't always so. Plenty of
technologies have historically been heavily regulated or even criminalized for
good reason and once you get past any reflexive indignation, criticism of
technology and progress, you might agree with me, that it is reasonable to ban
individuals from owning nuclear weapons or nerve gas. Less obviously, they may not be
weapons, but we've banned chlorofluorocarbon refrigerants, because
they were building up in the high stratosphere and destroying the ozone
layer that protects us from UVB radiation. We banned tetra e-file LED in
gasoline, because it poisoned people and led to a crime wave. These are not
weaponized technologies, but they have horrible side effects. Now, nerve gas and
leaded gasoline were 1930s chemical technologies, promoted by 1930s
corporations. Halogenated refrigerants and nuclear weapons are totally 1940s. ICBMs
date to the 1950s. You know, I have difficulty seeing why people are getting
so worked up over North Korea. North Korea reaches 1953 level parity - be terrified
and hide under the bed! I submit that the 21st century is throwing
up dangerous new technologies, just as our existing strategies for regulating very
slow AIs have proven to be inadequate. And I don't have an answer to how we regulate
new technologies, I just want to flag it up as a huge social problem that is going to
affect the coming century. I'm now going to give you four examples of
new types of AI application that are going to warp our societies even more
badly than the old slow AIs, we.. have done. This isn't an exhaustive list, this is just
some examples I dream, I pulled out of my ass. We need to work out a general
strategy for getting on top of this sort of thing before they get on top of us and
I think, this is actually a very urgent problem. So I'm just going to give you this
list of dangerous new technologies that are arriving now, or coming, and send you
away to think about what to do next. I mean, we are activists here, we should be
thinking about this and planning what to do. Now, the first nasty technology I'd
like to talk about, is political hacking tools that rely on social graph directed
propaganda. This is low-hanging fruit after the electoral surprises of 2016.
Cambridge Analytica pioneered the use of deep learning by scanning the Facebook and
Twitter social graphs to identify voters political affiliations by simply looking
at what tweets or Facebook comments they liked, very able to do this, to identify
individuals with a high degree of precision, who were vulnerable to
persuasion and who lived in electorally sensitive districts. They then canvassed
them with propaganda, that targeted their personal hot-button issues to change their
electoral intentions. The tools developed by web advertisers to sell products have
now been weaponized for political purposes and the amount of personal information
about our affiliations that we expose on social media, makes us vulnerable. Aside, in
the last U.S. Presidential election, as mounting evidence for the British
referendum on leaving the EU was subject to foreign cyber war attack, now
weaponized social media, as was the most recent French Presidential election.
In fact, if we remember the leak of emails from the Macron campaign, it turns out that
many of those emails were false, because the Macron campaign anticipated that they
would be attacked and an email trove would be leaked in the last days before the
election. So they deliberately set up false emails that would be hacked and then
leaked and then could be discredited. It gets twisty fast. Now I'm kind of biting
my tongue and trying, not to take sides here. I have my own political affiliation
after all, and I'm not terribly mainstream. But if social media companies don't work
out how to identify and flag micro- targeted propaganda, then democratic
institutions will stop working and elections will be replaced by victories, whoever
can buy the most trolls. This won't simply be billionaires but.. like the Koch
brothers and Robert Mercer from the U.S. throwing elections to whoever will
hand them the biggest tax cuts. Russian military cyber war doctrine calls for the
use of social media to confuse and disable perceived enemies, in addition to the
increasingly familiar use of zero-day exploits for espionage, such as spear
phishing and distributed denial-of-service attacks, on our infrastructure, which are
practiced by Western agencies. Problem is, once the Russians have demonstrated that
this is an effective tactic, the use of propaganda bot armies in cyber war will go
global. And at that point, our social discourse will be irreparably poisoned.
Incidentally, I'd like to add - as another aside like the Elon Musk thing - I hate
the cyber prefix! It usually indicates, that whoever's using it has no idea what
they're talking about. *applause, laughter*
Unfortunately, much as the way the term hacker was corrupted from its original
meaning in the 1990s, the term cyber war has, it seems, to have stuck and it's now an
actual thing that we can point to and say: "This is what we're talking about". So I'm
afraid, we're stuck with this really horrible term. But that's a digression, I
should get back on topic, because I've only got 20 minutes to go.
Now, the second threat that we need to think about regulating ,or controlling, is
an adjunct to deep learning target propaganda: it's the use of neural network
generated false video media. We're used to photoshopped images these days, but faking
video and audio takes it to the next level. Luckily, faking video and audio is
labor-intensive, isn't it? Well nope, not anymore. We're seeing the first generation
of AI assisted video porn, in which the faces of film stars are mapped onto those
of other people in a video clip, using software rather than laborious in human
process. A properly trained neural network
recognizes faces and transforms the face of the Hollywood star, they want to put
into a porn movie, into the face of - onto the face of the porn star in the porn clip
and suddenly you have "Oh dear God, get it out of my head" - no, not gonna give you
any examples. Let's just say it's bad stuff.
*laughs* Meanwhile we have WaveNet, a system
for generating realistic sounding speech, if a voice of a human's speak of a neural
network has been trained to mimic any human speaker. We can now put words into
other people's mouths realistically without employing a voice actor. This
stuff is still geek intensive. It requires relatively expensive GPUs or cloud
computing clusters, but in less than a decade it'll be out in the wild, turned
into something, any damn script kiddie can use and just about everyone will be able
to fake up a realistic video of someone they don't like doing something horrible.
I mean, Donald Trump in the White House. I can't help but hope that out there
somewhere there's some geek like Steve Bannon with a huge rack of servers who's
faking it all, but no. Now, also we've already seen alarm this year over bizarre
YouTube channels that attempt to monetize children's TV brands by scraping the video
content of legitimate channels and adding their own advertising in keywords on top
before reposting it. This is basically your YouTube spam.
Many of these channels are shaped by paperclip maximizing advertising AIs, but
are simply trying to maximise their search ranking on YouTube and it's entirely
algorithmic: you have a whole list of keywords, you perm, you take them, you slap
them on top of existing popular videos and re-upload the videos. Once you add neural
network driven tools for inserting character A into pirated video B, to click
maximize.. for click maximizing bots, things are gonna get very weird, though. And
they're gonna get even weirder, when these tools are deployed for political gain.
We tend - being primates, that evolved 300 thousand years ago in a smartphone free
environment - to evaluate the inputs from our eyes and ears much less critically
than what random strangers on the Internet tell us in text. We're already too
vulnerable to fake news as it is. Soon they'll be coming for us, armed with
believable video evidence. The Smart Money says that by 2027 you won't be able to
believe anything you see in video, unless for a cryptographic signatures on it,
linking it back to the camera that shot the raw feed. But you know how good most
people are at using encryption - it's going to be chaos!
So, paperclip maximizers with focus on eyeballs are very 20th century. The new
generation is going to be focusing on our nervous system. Advertising as an industry
can only exist because of a quirk of our nervous system, which is that we're
susceptible to addiction. Be it tobacco, gambling or heroin, we
recognize addictive behavior, when we see it. Well, do we? It turns out the human
brain's reward feedback loops are relatively easy to gain. Large
corporations like Zynga - producers of FarmVille - exist solely because of it,
free to use social media platforms like Facebook and Twitter, are dominant precisely
because they're structured to reward frequent short bursts of interaction and
to generate emotional engagement - not necessarily positive emotions, anger and
hatred are just as good when it comes to attracting eyeballs for advertisers.
Smartphone addiction is a side effect of advertising as a revenue model. Frequent
short bursts of interaction to keep us coming back for more. Now a new.. newish
development, thanks to deep learning again - I keep coming back to deep learning,
don't I? - use of neural networks in a manner that Marvin Minsky never envisaged,
back when he was deciding that the Perzeptron was where it began and ended
and it couldn't do anything. Well, we have neuroscientists now, who've
mechanized the process of making apps more addictive. Dopamine Labs is one startup
that provides tools to app developers to make any app more addictive, as well as to
reduce the desire to continue participating in a behavior if it's
undesirable, if the app developer actually wants to help people kick the habit. This
goes way beyond automated A/B testing. A/B testing allows developers to plot a binary
tree path between options, moving towards a single desired goal. But true deep
learning, addictiveness maximizers, can optimize for multiple attractors in
parallel. The more users you've got on your app, the more effectively you can work
out, what attracts them and train them and focus on extra addictive characteristics.
Now, going by their public face, the folks at Dopamine Labs seem to have ethical
qualms about the misuse of addiction maximizers. But neuroscience isn't a
secret and sooner or later some really unscrupulous sociopaths will try to see
how far they can push it. So let me give you a specific imaginary scenario: Apple
have put a lot of effort into making real- time face recognition work on the iPhone X
and it's going to be everywhere on everybody's phone in another couple of
years. You can't fool an iPhone X with a photo or even a simple mask. It does depth
mapping to ensure, your eyes are in the right place and can tell whether they're
open or closed. It recognizes your face from underlying bone structure through
makeup and bruises. It's running continuously, checking pretty much as often
as every time you'd hit the home button on a more traditional smartphone UI and it
can see where your eyeballs are pointing. The purpose of a face recognition system
is to provide for real-time authenticate continuous authentication when you're
using a device - not just enter a PIN or sign a password or use a two factor
authentication pad, but the device knows that you are its authorized user on a
continuous basis and if somebody grabs your phone and runs away with it, it'll
know that it's been stolen immediately, it sees the face of the thief.
However, your phone monitoring your facial expressions and correlating against app
usage has other implications. Your phone will be aware of precisely what you like
to look at on your screen.. on its screen. We may well have sufficient insight on the
part of the phone to identify whether you're happy or sad, bored or engaged.
With addiction seeking deep learning tools and neural network generated images, those
synthetic videos I was talking about, it's entirely.. in principle entirely possible to
feed you an endlessly escalating payload of arousal-inducing inputs. It might be
Facebook or Twitter messages, optimized to produce outrage, or it could be porn
generated by AI to appeal to kinks you don't even consciously know you have.
But either way, the app now owns your central nervous system and you will be
monetized. And finally, I'd like to raise a really hair-raising specter that goes well
beyond the use of deep learning and targeted propaganda and cyber war. Back in
2011, an obscure Russian software house launched an iPhone app for pickup artists
called 'Girls Around Me'. Spoiler: Apple pulled it like a hot potato as soon as
word got out that it existed. Now, Girls Around Me works out where the user is
using GPS, then it would query Foursquare and Facebook for people matching a simple
relational search, for single females on Facebook, per relationship status, who have
checked in, or been checked in by their friends, in your vicinity on Foursquare.
The app then displays their locations on a map along with links to their social media
profiles. If they were doing it today, the interface would be gamified, showing strike
rates and a leaderboard and flagging targets who succumbed to harassment as
easy lays. But these days, the cool kids and single
adults are all using dating apps with a missing vowel in the name, only a creeper
would want something like Girls Around Me, right? Unfortunately, there are much, much
nastier uses of and scraping social media to find potential victims for serial
rapists. Does your social media profile indicate your political religious
affiliation? No? Cambridge Analytica can work them out with 99.9% precision
anyway, so don't worry about that. We already have you pegged. Now add a service
that can identify people's affiliation and location and you have a beginning of a
flash mob app, one that will show people like us and people like them on a
hyperlocal map. Imagine you're a young female and a
supermarket like Target has figured out from your purchase patterns, that you're
pregnant, even though you don't know it yet. This actually happened in 2011. Now
imagine, that all the anti-abortion campaigners in your town have an app
called "Babies Risk" on their phones. Someone has paid for the analytics feed
from the supermarket and every time you go near a family planning clinic, a group of
unfriendly anti-abortion protesters somehow miraculously show up and swarm
you. Or imagine you're male and gay and the "God hates fags"-crowd has invented a
100% reliable gaydar app, based on your Grindr profile, and is getting their fellow
travelers to queer bash gay men - only when they're alone or outnumbered by ten to
one. That's the special horror of precise geolocation not only do you always know
where you are, the AIs know, where you are and some of them aren't friendly. Or
imagine, you're in Pakistan and Christian Muslim tensions are rising or in rural
Alabama or an Democrat, you know the possibilities are endless. Someone out
there is working on this. A geolocation aware, social media scraping deep learning
application, that uses a gamified competitive interface to reward its
players for joining in acts of mob violence against whoever the app developer
hates. Probably it has an innocuous seeming, but
highly addictive training mode, to get the users accustomed to working in teams and
obeying the apps instructions. Think Ingress or Pokemon Go. Then at some pre-
planned zero-hour, it switches mode and starts rewarding players for violence,
players who have been primed to think of their targets as vermin by a steady drip
feed of micro-targeted dehumanizing propaganda inputs, delivered over a period
of months. And the worst bit of this picture? Is that the app developer isn't even a
nation-state trying to disrupt its enemies or an extremist political group trying to
murder gays, Jews or Muslims. It's just a Paperclip Maximizer doing what it does
and you are the paper. Welcome to the 21st century.
*applause* Uhm...
Thank you.
*ongoing applause* We have a little time for questions. Do
you have a microphone for the orders? Do we have any questions? ... OK.
Herald: So you are doing a Q&A? CS: Hmm?
Herald: So you are doing a Q&A. Well if there are any questions, please come
forward to the microphones, numbers 1 through 4 and ask.
Mic 1: Do you really think it's all bleak and dystopian like you prescribed
it, because I also think the future can be bright, looking at the internet with open
source and like, it's all growing and going faster and faster in a good
direction. So what do you think about the balance here?
CS: *sighs* Basically, I think the problem is, that about 3% of us
are sociopaths or psychopaths, who spoil everything for the other 97% of us.
Wouldn't it be great if somebody could write an app that would identify all the
psychopaths among us and let the rest of us just kill them?
*laughing, applause* Yeah, we have all the
tools to make a utopia, we have it now today. A bleak miserable grim meathook
future is not inevitable, but it's up to us to use these tools to prevent the bad
stuff happening and to do that, we have to anticipate the bad outcomes and work to
try and figure out a way to deal with them. That's what this talk is. I'm trying
to do a bit of a wake-up call and get people thinking about how much worse
things can get and what we need to do to prevent it from happening. What I was
saying earlier about our regulatory systems being broken, stands. How do we
regulate the deep learning technologies? This is something we need to think about.
H: Okay mic number two. Mic 2: Hello? ... When you talk about
corporations as AIs, where do you see that analogy you're making? Do you see them as
literally AIs or figuratively? CS: Almost literally. If
you're familiar with philosopher (?) Searle's Chinese room paradox
from the 1970s, by which he attempted to prove that artificial intelligence was
impossible, a corporation is very much the Chinese room implementation of an AI. It
is a bunch of human beings in a box. You put inputs into the box, you get apples
out of a box. Does it matter, whether it's all happening in software or whether
there's a human being following rules inbetween to assemble the output? I don't
see there being much of a difference. Now you have to look at a company at a
very abstract level to view it as an AI, but more and more companies are automating
their internal business processes. You've got to view this as an ongoing trend. And
yeah, they have many of the characteristics of an AI.
Herald: Okay mic number four. Mic 4: Hi, thanks for your talk.
You probably heard of the Time Well Spent and Design Ethics movements that
are alerting developers to dark patterns in UI design, where
these people design apps to manipulate people. I'm curious if you find any
optimism in the possibility of amplifying or promoting those movements.
CS: Uhm, you know, I knew about dark patterns, I knew about people trying to
optimize them, I wasn't actually aware there were movements against this. Okay I'm
53 years old, I'm out of touch. I haven't actually done any serious programming in
15 years. I'm so rusty, my rust has rust on it. But, you know, it is a worrying trend
and actual activism is a good start. Raising awareness of hazards and of what
we should be doing about them, is a good start. And I would classify this actually
as a moral issue. We need to.. corporations evaluate everything in terms
of revenue, because it's very equivalent to breathing, they have to
breathe. Corporations don't usually have any moral framework. We're humans, we need
a moral framework to operate within. Even if it's as simple as first "Do no harm!"
or "Do not do unto others that which would be repugnant if it was done unto you!",
the Golden Rule. So, yeah, we should be trying to spread awareness of this about
and working with program developers, to look to remind them that they are human
beings and have to be humane in their application of technology, is a necessary
start. *applause*
H: Thank you! Mic 3? Mic 3: Hi! Yeah, I think that folks,
especially in this sort of crowd, tend to jump to the "just get off of
Facebook"-solution first, for a lot of these things that are really, really
scary. But what worries me, is how we sort of silence ourselves when we do that.
After the election I actually got back on Facebook, because the Women's March was
mostly organized through Facebook. But yeah, I think we need a lot more
regulation, but we can't just throw it out. We're.. because it's..
social media is the only... really good platform we have right now
to express ourselves, to have our rules, or power.
CS: Absolutely. I have made a point of not really using Facebook
for many, many, many years. I have a Facebook page simply to
shut up the young marketing people at my publisher, who used to prop up every two
years and say: "Why don't you have a Facebook. Everybody's got a Facebook."
No, I've had a blog since 1993! *laughing*
But no, I'm gonna have to use Facebook, because these days, not using Facebook is
like not using email. You're cutting off your nose to spite your face. What we
really do need to be doing, is looking for some form of effective oversight of
Facebook and particularly, of how they.. the algorithms that show you content, are
written. What I was saying earlier about how algorithms are not as transparent as
human beings to people, applies hugely to them. And both, Facebook and Twitter
control the information that they display to you.
Herald: Okay, I'm terribly sorry for all the people queuing at the mics now, we're out
of time. I also have to apologize, I announced, that this talk was being held in
English, but it was being held in English. *the latter pronounced on the G*
Thank you very much, Charles Stross!
CS: Thank you very much for listening to me, it's been a pleasure!
*applause*
*postroll music*
*subtitles created by c3subtitles.de in the year 2018*