31C3 Title, no sound
Alright, welcome! So, welcome again from me. It's great to be here!
So many people, even to this late hour. I've been told, this is the prime time.
That is awesome, at 11 p.m. I'm David, I'm a Computer Scientist from Bonn.
And we just can start with the things that happened so far at the congress.
If you happened to be here at the congress
or watched sessions on stream - welcome again
to the colleagues on the internet - then there will be always devices that one
does not like so much to use. [Laughter]
Who participated in the sessions of Tobias Engel and Karsten Nohl, does indeed use
his mobile phone less confident. And who was with starbug afterwards, will
not like to use iris scanners or finger print scanners anymore and may use gloves
more frequently now. So here a little disclaimer:
If someone has an intimate relation to his photocopier
and tends to keep it like that, should refrain from participating this session.
We will do three things during this session. First of all we will
get to know one of most prevalent and dangerous bugs of the last years.
Secondly, we will comprehend the bug. That is in a manner
nerds and muggels will understand. And last but not least, for the activists
among us - may be some present here - we will deduct some rules
that may apply to a single person that will handle a powerful opponent,
just like a global player. But in your case
it can be something completely different. That's why I will describe precisely
how this dispute evolved over time and
what kind of mistakes I made. The talk's kind of structured
like a novel. First, there's a prologue, for the conspiracy theorists
among you. The year is 2008.
In summer 2008 the US were having the primaries for presidential election.
Barack Obama was in the running against Hillary Clinton. In the US, like here,
there's lots of intrigue in politics. So there were a few anonymous emails,
that should benefit Mrs. Clinton. Those mails claimed, among other things,
that Obama had been born in Kenia als a Kenian citizen. That would make him fomally
unfit to be president. To become president of the US, you have to be
'natural born citizen' of the US. What exactly a 'natural born citizen' is
the Americans themselves even don't really fully know. But there's a whole
Wiki article about the controversy, where you can read all about it.
Two things generally acknowleged: First, one's to be American.
Second, one's to also be that at time of birth. So when I come to the US,
newly naturalized, that doesn't work. That Obama's second name
is Hussein was somewhat suboptimal too in that context.
(laughs). Obama obviously had an interest in
ending that 'argument' as quickly as possible. So he made his birth certificate
publicly available. I say 'short birth certificate' because,
when he was born, a short and a long one were made. The short one ist shown here on
the left, you see it behind me. And I in front of me.
But good conspiracy theorists aren't distracted by facts.
(laughter and applause)
Immediatly, there are accusations the birth certificate's faked.
Supposedly, there was a stamp missing, and ... and ... and. Whatever you can
come up with. You all can come up with it.
On the right, you see a few car stickers by Obama's enemies.
The lowermost explicitly calls for the birth certificate. The theory that Obama
shouldn't be allowed to be president, is rather wide-spread in the US.
Obama won the primaries, and the following election, but the dispute
simmered on. There was a whole scene of birthers
that wanted to prove Obama's actually not American.
After the whole thing hadn't calmed down fo two and a half years - Obama already being
president for some time - in 2011 he had all of it. He published the scan of
the long version of the birth certificate, on the right in the picture. You can already see
there's much more information in it, and you could think: They'll leave him alone now.
But far from it. Shortly after the release there were accusations
the birth certificate was a clumly forgery. Let's take a closer look.
The left picture is a strong enhancemt of the red box in the right picture.
The numbers six and four are visible. These numbers have sharp, pixel-perfect edges.
Yes, it's even visible on the projector. And the numers are uniformly colored.
On their right side the number one is blurred and colored unevenly.
The one is as you would expect a scan in reality. Why is there such
a difference between two numbers in one and the same row of numbers?
A few more examples. Again one can see numbers with sharp edges
or these ticking boxes in contrast to normal, slightly blurred numbers
and boxes. I drew some red boxes the ticking boxes
and the 'and'. There one can see a kind of shift. And it does really look
as though somebody drew this using Paint. Meaning the ancient one, I am sure you remeber
from your childhood. MS Paint on Windows 3.11. I used to sit at my father's workplace
at work and stole his working hours. Or this one,
particulary beautiful. This section of the frame is from the stamp at the bottom.
There's a typo, in the stamp. Yeah sure, makes sense. We have heard that one before,
typo in the stamp. I mean of course one would think it's a fraud
the way it looks. And at the same time think that the intern
at the White House is too stupid to use Photoshop.
Laughter
Concerning PR this was a massive failure of course. According to a Gallup poll
in 2011, 5% of Americans believed, Obama was
definitely not born in the US. And a further 8% thought, that he was 'probably not'
born in the US. Well that didn't work out. The White House had to
back up pretty badly. To this day they get requests because of this. This was the prologue.
We will now move on to the main trial and jump in time to 2013.
On the 24th of June 2013 a company, I was friends with, called me
The had two big Xerocs Workcentres. Xerocs Workcentres are
those giant buisness copiers, that stand everywhere nowadays. They are connected via WIFI,
can scan, print, copy, mail and cost as much as a small car.
These printers aren't the ones your grandma uses, but have
a few hundred users per device, maybe more. In this picture
you can see a construction plan. The black areas aren't original, I just
cencored those afterwards, since I would not have been allowed
to use it. I marked three spots in yellow on the plans.
These spots are standardized blocks containing the squarefootage
of the room. These spots will become more important soon. The company
told me: "Hey David, when we scan a construction plan
the numbers change. Could you take a look at it?"
Laughter
On the left side, that's me. Laughter
Applause
At this point I have to add, that the relationship with them is really good. I worked my way
through my computer sience degree. Of course my parents also
contributed, I won't deny that. But I did IT-Service for the company and
they were really nice all the time and of course I thought they were screwing with me.
For sure. Copier changes numbers?? Of course, makes sense. We've heard that before.
They said: "Yes, come over and take a look at it.
We need the device, it has to work."
So I drove over there and took a look. Still being a bit
on the watch for the joke. They have a Xerox Workcentre 7535.
Here are the three marked spots in the original, before scanning.
I am not sure how good you can read it, so I will read it out loud.
On the top it says 14.13 sqm (square meter) in the middle it's 21.11 sqm,
and at the bottom 17.42 sqm. So I put the plans in the Workcentre
and scanned it. And here are the same spots after the scan.
Laughter and Applause
Interesting. Suddenly all rooms are 14.13 sqm big.
I thought this can't be right. Completely impossible. This isn't happening.
I was still thinking they are screwing with me. (laughs)
While scanning the - to clear that out from the beginning, since I
got that question a dozen times in the internet- While scanning the text detection
was turned of. The number substitution takes place in the raw pixel data.
The company also had a second Workcentre, the 7556.
Thats bigger and faster. Aside from these two kinds of Workcentres,
that I mention here in the beginning, there are a lot more. It is
a gigantic family of devices. In contrast to the smaller device
which spat out the same numbers every time,... (laughs)
the larger one gave out different ones every time. (Laughter)
It is bigger and has more CPU power.
(Laughter)
Look at those rows and how the values change. At "Stelle 2",
that is the middle row, first and last it's 14.13 sqm.
And in the middle 21.11, once. That would have been the correct value btw.
There is a chance to get it right. (Laughter)
In the other rows it looks similar.
In case one of you needs one of those NSA random generators....
(laughs)
Applause
Keep in mind, that actually this is no...
I am laughing as well, but it is no laughing matter.
Note that the numbers are set into the layout perfectly. The error
was only noticed, because an obviously bigger room had
a smaller square footage than a smaller one next to it.
There's a broom cupboard with 100 sqm and next to it a ball room
with 4 sqm. (Laughter)
It hardly gets any meaner. The layout looks perfect.
I do realise that the writing is really small. Don't you
thinks this is some mean corner case and I was working on
for three month, just to finally stick it up to Xerox.
We will look at other examples. This is the original case
in which the bug was originally noticed, and I didn't want to keep it from you.
Here's the next one. This is an expense register.
(Laughter)
Two sixes became eights.
It's funny, I released the picture it on my website,
and I said: " Here a six became an eight."
Then I get an e-mail: "No, on the top there's another."
(loud laughing and applause)
Again perfectly set. Why was it noticed this time?
Because the numbers are supposed to be sorted by size.
What I want to say is
it is impossible to notice. If I give you some columns of numbers
that don't make any noticable sense. Then you could obviously
not see, that there's wrong numbers. It's always around there being
semantic criteria, to make it noticable. To make it
obviously implausible. Otherwise you have no chance to notice.
Slowly I became a little worried.
The neck length increases. To not let this be some random events, I started
working to reproduce the error on purpose. IT guy style
invested a night and generated number columns in different
sizes and fonts. I scanned those and experimented for
a few hours. And, indeed, the error accurs again.
These are my random numbers. We will be able to work with those
some more. The eights marked in yellow
should be sixes and do not belong there. Let's stay ourselfes shortly.
I promised you in the introduction, that I would
lay out the entire interaction with Xerox, that would follow, over time
and tell you, how I felt at the corresponding times and emphasize the things
that according to my experience are extremely important
when confronting a giant opponent. And I will keep that promise.
I will tell you why at all times. But now I will
say one thing up front. This thing I will discuss in different ways through the entire presentation.
What never helps in my point of view is unfriendly twittering and hating.
(self-concious applause)
It's really nice that you are applauding, I wasn't sure that would happen.
(laughter)
I have nothing against twitter as such. Nothing at all.
But if you want to achieve something, you make yourself vulnerable
with such behaviour. And above all you won't be taken seriously.
You can always be accused of not wanting a proper discussion.
That won't fit in 140 letters, no matter what any of you say.
(applause)
Secondly you can always be accused of seeking attention
for yourself. Because almost everything is public on twitter.
At the most twitter is useful for establishing first contact, when you
ask for an e-mail adress or a phone number. If I don't recommend twitter,
what do I recommend? Much more serious and straight foreward
is erverything, that is not public. That way one shows willingness to work
rationaly and not urge to scream around. That's mail or phone calls.
So we called the Xerox support. Several times ...
Often ... We phoned uo all the levels up to the top level
in Dublin - nobody knew anything.
We also sought personal contact. Staff from the local Xerox retailer
came over. That's not Xerox themselves, but a retail and support company.
Thay were shocked - of course, right? And then they tried to reproduce it
themselves. Zack! They reproduced it...
(laughter and applause)
That was .. we are laughing now. They were standing there
heads hanging low. You are standing there selling these things
and suddenly you question your existence.
That's not cool at all. At Xerox - not the support company,
but the entire, big Xerox, 140.000 employees,
there was surprise, but no efforts were made
to help us or the retail company. Meaning they were cautious of the problem.
(laughs) (laughter)
So there were no signs at all of greater interest
and no advice, as for solving the problem. Then one guy came
from Xerox Central, who updated the software, we had an acient one
installed. He installed the new software, problem was still there.
I thought: "Great, now we know the problem existed in the fimware
three years ago until today." Hmmm.
When for more than a week nothing happened on Xerox's side
that promised hope, I thought: "Now you have been accommodating enough!"
So I wrote a blog article in German and English
about what I just told you about. In this article I offered
test documents to download. The readers can print, scan and check whether
they are affected or not. With that the spread of the story started.
I have to add, my blog is not really huge , really not. It has around
500-1000 readers per day. That's not a huge amount, but also not nothing
and the most readers are computer scientists of some form, I know that from the e-mails
I get. On the bottom of my slides from now on you can see a line.
This line will continuously move further to the right. Thats a
plot of the klicks. It's not meant to show off with clicks, but
in context it's great to see, at what time one gets attention in what way
and also to see how fast it fades. We will show that immediately.
This small bump - yes, it's visible. The line
moved to the right and there's a peak of 3000 hits/hour.
Those numbers are from Google Analytics, I have been told, one has
to multiply them by two, but for order of magnitude it's enough.
On the 2nd and 3rd of August the story hit on several tech-blogs.
At this point I declare the long-known fefe as tech-blog.
(laughter)
I know, I know, there's the first protest. But I will agree on the fact,
that fefe is read by a lot of IT-poeple. Alright, I am
not hearing any more protest. The peak you see here is because of blog.fefe.de .
The message spreads, and I get more and more mails from readers
that are affected. The most concerning is that I get e-mails with confirmations
for a lot of Xerox-Workcentres that I don't even know.
(laughter)
I told you before these things are one giant family of products. Very slowly
I realise, that this could turn into something bigger eventually.
Lesson learned: It was good to release the test-documents online
with the article. Would the users not have been able to check for themselves
using the test-documents, the story would never have had an impact like it would soon have.
On the 4th of August the story arrived in tech-portals around the world.
In the slide is Hacker News by Y-Combinator, that's one of the biggest
of this kind, you probably know it. From now on I get hundreds of technically
versed e-mails a day. I say "technically versed", because there were also others
that were less technical. Over the entire time I
spend days to channel and sort the news
I get. This enabled me to continue the reporting
in a professionaly and to get to the roots of the bug with professional help.
The whole thing becomes an avalanche and I am not allowed to sleep any more.
Cause the US press is on the phone constantly. You must not think that US- journalists
ever realise, that there's a thing called time zones ....
(laughter and applause)
Here's another anecdote. One would think the US media journalists are
competitors. Meaning if one had a special information he would not pass
it on to the others, right? As soon as the colleague from ABC had my phone number
ALL of them had it. I tell you, it's incredible! (laughs)
Lesson learned: Write these things in multiple languages! Important are English
for the international space. Also the language of the home market of the company,
you are confronting. In my case thats the USA, so English, again
two birds with one stone. By the way: in the US Xerox is so strong
that "to copy" is called "to xerox" there. They really say that
in everyday conversation. The same way we say: "Hand me a Tempo! (cotton tissue)",
just to give you an impression of how much repute the company and the brand
has there. And when in the world of technology something like this goes around
what's next? Mass media (some laughing)
And there you get the whole package. We'll just click through here to
illustrate it. This list is in no way complete, there were thousands of
articles suddenly, all over the world. And if I show an article, then
- just as a disclaimer - it doesn't make a statement about the date of publishing
statement about the date of publishing, I just make it in a way that's good for the show
(some laughter)
Browsing, here is Heise, of course that joys me as a computer scientist,
they covered the whole story in five articles or so.
ZDF Hyperland, yes? I'm demonstrating the german press a bit here.
The german press was very reserved. The most articles
were in fact from abroad. Therefore the comment
about the "home market". But here a small anecdote about the german press.
A journalist told me that he wanted to bring the story to the "Tagesschau".
They told him "Yeah, hmm, it's alright. But for this we want it to happen
during real copying, and not just during scanning!"
(laughter and applause)
If anyone from the "Tagesschau" is watching, this applause is for you!
(laughter)
So I think: You geniuses! Pro Tip: If you print a scan,
then you have a copy! (laughter)
With the difference, that such a saved scan can cause
harm even years later. But please! So I thought,
no "Tagesschau" story, it's going around the world already anyways,
not my problem if they are the only ones not covering it.
Lesson learned: Stay professional and sovereign. Don't just bloat things
out of thirst for attention. Every one of you can probably name
some affaire, that went rather well
for whoever made it public, and then in the
decisive moment he tasted blood and made something up.
That's bad of course. Oh well. The Economists, that's really
vintage, I liked this title: "Lies, damned lies and scans"
That comes from Tom Sawyer: "Lies, damned lies and statistics"
Now PR wise, we're at a point where it's expensive.
The Economists has influence. ABC News - even more expensive.
There are the colleagues with their phones.
BBC, CNBC. Suddenly, it was everywhere.
My powerpoint is lagging, here it is again. Business Week,
that is a popular economy magazine. I'll recall here,
until now, no reaction from Xerox. Yes, three days in business,
worldwide. No reaction! And when you take that long, the tone gets
really rough. I quote: "On the scale of things, that are too terrible
to imagine, document altering scanners are somewhere
up there with meat eating bacteria."
(laughter)
They are actually writing this in the Business Week! (laughs)
So I was called my a friend of mine, listen you have to
read this. Great! Imagine, there's Peter Coy, he's editor there,
that we will see again a few more times over the course of this talk.
So, my blog article is now at about 100.000 visitors per day.
And still, no feedback from Xerox. In the meantime
I was able to explain, with the help of many reader-mails,
what's happening at all. And that's what I am telling you now,
so we make a small excourse about image compression.
Here we have a test image, that I made. It's a
sundew, with a fly on it, that's a plant. The fly as well as the
text belong to this test image. For us to have a nice variety of pictures.
Data transfer costs time, money and storage. Image consist,
compared to text, of a great amount of data. And to send and save pictures
completely uncompressed would be really expensive.
And images are sent everywhere, yes? The use is there for every one
of us. I tell you, it goes to the highest possible scenarios.
Just recently there was a giant coverage, and even an
investigation by the government, just because a former member of
the parliament transferred pictures. (laughter)
(laughs) So now, this member of the parliament
can't wait for his pictures forever, so we have to compress the image data.
(laughs again)
Listen here! (laughs stupidly)
(applause)
Now we have two parts of my test image. One image part
and one text part. And I enhanced it so much you can see individual
pixels. This is so we can see what go wrong with different compression
methods. There is lossless compression. Here the
image data stays as is, it is just somehow stored more
efficient. Or we accept losses, so, changes in the image data,
to "squish" the data and make it even smaller.
Here are the popular GIF-images.
Can I have a small hand sign, who thinks that GIF has lossy compression?
Wow, that's a lot! Almost everyone.
GIF is a lossless compression method.
The downside is, it only supports 256 colours.
The here shown lower quality stems not from the image being saved
as a GIF, but from the colour reduction.
To be able to see it better, I reduced the colour amount to 16.
Here you see it nicely, uiuiui. So.
The finished image is saved pixel for pixel, and then LZW compressed.
LZW is an old compression algorithm, similar to ZIP.
GIF is very suited for graphics with few colours. And because pixels are still
saved completely one by one, sharp edges are well
represented. You can see, the text looks pretty good. It's less good
in photographs, as you can see. Most widespread are JPEG images. And JPEG
is lossy. The original image doesn't get saved pixel for pixel anymore,
but instead gets split into 8x8 pixel blocks. And every block then
gets approximated with cosinus-waves. How exactly this works mathematically,
we can spare ourselves from here. But it is good to know, that this
kind of compression, it's good for pictures, but bad for sharp edges,
as you can see in the letters, yes, you can see artifacts, you can see
some stains around it. But usually this would be full of artifacts,
the image. I can hold up my notebook or so.
Long story short. Depending on the type of image, certain compression
methods are good, and others aren't.
That's why there is the JBig2-fomat. This is one of the special words, that I
wrote down in three variants for the translators.
Here you can dissect one image in multiple sub images. The red
circled here as an example. These are sub images. These sub images we call
"patches", english for "Flicken". As we see, there are parts of the image,
that don't belong to any patch. That's pretty cool, because
the data for these won't need to be saved at all.
You just say, background white. The joke here is, these seperate patches, you can
compress these with multiple compression methods.
The text patches, for example with GIF, I'll show it just very roughly here.
You probably can't use GIF in JBig2. But the principle stays.
And the photo patch for example with JPEG. Every patch its suited compression method.
That's a real advancement. I probably won't have to explain anyone here,
that with this you will know, which patch contains what, get a good
quality, and probably a smaller file size. So,
if you dissect the image into patches anyway, you might as well use a
completely new high tech compression method. You can dissect the original image
much finer, and have every individual letter as its own patch.
That's a lot of patches. A whole lot of patches.
And you can do this with text pages and books. And its used,
I didn't just make that up now.
So next we see, which patches are similar to each other.
This step is called "pattern matching". I have marked four patches with arrows
here. These patches are very similar. No wonder, you will say.
All of them are small "e"s. They are only different by a few pixels.
Through this pattern matching, you get a group of similar symbols.
For this group, you only really save one of those symbols, and that is
used over and over in the compressed image.
Instead of his brothers. From these four marked "e"s, only one would be
really saved, and then replaced all the other ones. This way you can really
save a lot of data, with minimal quality loss.
Here is the final product. Looks still good, doesn't it? No artifacts
visible. Takes a lot less data than without pattern matching.
Did you see that? The pattern matching thinks the I is similar to the small L,
so you can replace that with it. This happens, when pattern matching
works inaccurate. Did you see this too?
These are incredibly dangerous mistakes.
Usual compression errors are not so bad. Then one letter is
unreadable. You see it, and you know that something went wrong, "scan again please".
But here you have actual wrong data, that looks flawless. And they get layoutet in
perfectly because of the similarities. You have to actually read this, to
notice the mistake. And even then, you can only see the mistake,
when the document becomes obviously implausible, like in the blueprint.
I don't know about you guys. But I don't read through all of my scans,
that I take, just to see if it has any mistakes.
But my friends, a politician that would have to gloss over this,
he would say: "Scan a medicine dosing with a Xerox-device
in a retirement home, and there is a high chance that in no time
you'll relieve the pension funds." (laughter)
(applause)
Now it is clear, that this also related to security. Until now, you could have
asked, why does David hold a speech about copying machines on the congress?
But this is actually about a severe failure of a company,
that is a serious security issue. Is anyone here from Berlin?
Maybe a hand sign?
What did the blueprints for the airport get scanned with?
(laughter and applause)
But you know what? Airports, medicine, rockets, airplanes...
As big as this is, that's all trivial. It gets interesting at the question,
where those scans got used in court as evidence, that
can be reexamined now. Or the other way around,
if one of you sues me with a Xerox-scan, from now on I'll just
tell you: "Ah, you know what, it's faulty!" (laughs)
Now you can look for the original first, to prove me
otherwise. I can't prove anymore, that that part of the scan also
comes from the part of the paper that you expect it to be from.
The legal value is zero! There's hundreds of thousands of industrial copiers
worldwide. Those are business devices, every machine has many users, even more
documents that were made by it, that were distributed whereever. And so you can
have an idea, a large company called me, their letter processing works so,
that incoming leters just get scanned immideately by machines,
and from there on they only exist electronically. Have fun, if
those contain errors. So, we come back to the implications later again.
But for now, back to the story. It's the 5th of August. We are three days after
the first impact, and on the third day god created, finally yes, a life sign
by Xerox. Now, they are watching after all man! (laughs)
(applause)
Thank you (laughs)
The PR of Xerox Germany calls me. The talk is very unproductive.
They can't do anything without the americans. At first,
they though it was a joke. I say, it's not. And then
we said, we will stay in contact. (laughs)
(laughter and applause)
And so, the day after, 6th of August, for the first time it really had a punch.
In the morning, I get a screenshot by a reader, from
one of the details from the admin panel of his Xerox-copiers. There they talk
about letter replacement. Aha! For the record, now. We can all learn this
here: There are three PDF compression levels.
These are called "Normal", "Higher", and "High". Very marketing appropriate.
So, "Normal" is the mode, that compresses the most. The reader says:
on "Normal", the error occurs, in the higher levels it doesn't.
My tests seem to comfirm this. I say it extra vague here, more on it later.
(pauses to drink)
I promised you to show you the moods over this situation,
in case something like it ever happens to you. And really: In the first moment
my heart dropped into my gut. I was scared shitless, to be the idiot
that didn't read the manual, yes? (laughter)
Because there is still no official Xerox-statement, and I got
a tip from the press, that Xerox says exactly this in their statement.
Lesson Learned: What's the difference between inside and outside view?
Exactly this. No? Surely you think: "Hello? Why is David so agigated,
it's clear that this type of document error should
never have happened, not even unknowingly." But from the inside...
It looks different. Despite being scared, it's important: Stay calm, act rational.
Because of anxious moments like this, it's important that previously you
never screech and de-escalate. Never rabble beforehand.
If you were always sovereign, you can appear confident,
and in doubt, calmy and publically ask: "Well, boys? Why did the
support not tell me this two weeks ago, eh?"
Lesson Learned: Appear professional from the start, never hate. I'll repeat
that again. So, now, defense to the front. I presented
the screenshot as a possible workaround and advised: Turn compression
on "Higher". Additionally I wrote, that I was wondering a bit,
why the support couldn't say this to me over the course of a whole week.
I also criticized, that the setting is called "Normal". (laughs)
And the possible consequences I showed to you, of course those stay,
because on the scan you can't see, that it might
contain errors. The goal was, to give the thing a spin, before Xerox fights back.
It follows a telephone conference with Rick Dastin. (murmur)
I see, he is known in the audience, the vice president worldwide of Xerox.
And Franics Tse, one of their chief engineers, that
was handling the image compression. Guys, the boss does support himself!
(laughter and applause)
Rick Dastin was in fact the first person that work at Xerox,
that I got officially told by, that the letter replacement was
in fact already known by Xerox. So, if you'd like to know, what the
support can't tell you after a week, then you say: "I want to
talk to Rick Dastin!" (laughter)
And here, it was revealed that the theory, that the pattern matching
was at fault, was true. Dastin also confirmed, that the pattern matching
is only used in "Normal" mode. So after a bit of discussion, it was
also clear, that the support fucked up, and the name
"Normal" might be badly chosen. I then suggested "Experimental".
(laughter and applause)
Maybe here: I'm really in a good mood, and this is a lot of fun,
and we are all laughing, but in that moment I was
just more nervous. Not that you think it would be different for you.
There I'll be completely honest. And then comes a clear "RTFM" from Xerox.
First: "Normal" mode, David, is not even a factory setting!
Dear customers, you're all stupid. Who would set it to such thing!
Second: That letters can get swapped, that is explained in the manual,
on two seperate occasions. Dear customers: double stupid!
For the factory setting: Of course that's only a half truth. For the
customer, factory setting is, what the device gets delivered with. Xerox doesn't
supply to big customers. Those sales go over third parties.
If you order a Xerox-copier, you do it over another company,
that isn't Xerox, and they will advise you and there you can configure
whatever before they ship it. And for the manual: The notice is in some manuals
indeed. But then I looked closer: On page 107 and 328 in the text, yes?
Now we are all old enough to know, how many people will read a 300 page
manual, before handling a printer. (laughter)
I also thought, that copiers generally shouldn't be designed in a way, so
those errors can occur at all. That can't be, no one expects that.
(applause)
The answer was: "Yes, it can be!" (laughter)
"The market wants it this way, errors would just..."
(laughter)
That was indeed a statement, that was said exactly like this. I quote here,
but of course that only related to small file sizes. And errors
would also be very rare. But I would be right, you can't prove, that a
document is free of errors. So, all in all the talk had a nice
atmosphere. They really didn't try to squash me legally or so.
They listened very nicely, the talk was super long too, 45 minutes
or so. And then I let myself get caught by them, like an amateur.
You have to consider, I had never done anything on a scale like this.
And with a company like Xerox, they have professionals. I was already wondering,
why we were talking so peacefully for such a long time. Dastin is the vice
president of a worldwid operating company after all. And he probably
has other stuff to do. And now it turns out, during the phone talk,
Xerox published a statement. Not bad at all. During that time
I couldn't react after all. And it had the beautiful title "Always listening
to our customers"... right at the moment! (laughs)
And they write in their statement, for error free files, please
use a compression setting of at least "Higher", and the error
would be written about in the manual. RTFM. Lesson learned: Have someone
watch the side of the enemy. So I wrote my own article,
about the contents of the phone call, the one that
I just told you about. Well, and then I also wrote,
that I don't think they're off the hook yet. And now?
This could've been over here. When a single blogger goes up
against a giant company, it usually ends one of three ways, when
the company shoots back: Either the blogger gives in after,
or the public sides with the company, or the public
loses interest, when the company shot back.
Every one of you can now think of three stories, where it was like this.
But none of this happened. You see the giant increase at the bottom. The
story was on the cover of Slashdot. And the press, luckily,
also had their attention on me. Here for example, Heise writes, that I
offered the workaround even before Xerox. (laughs)
(laughter and applause)
I'll exceed my time limit a bit.
Or also, bone dry, "Spiegel". They wrote: "So so, Xerox knew about the problem
for years?" (laughs dumbly) That's really... If you sit in
PR of a company, and this happens to you, I guarantee
you don't need to take vacation for the rest of the year.
But it gets real funny, when the story arrives at internet humour.
I won't withhold this from you. I don't know who of you has lived in
the US before. In german, we have the vulgar saying: "Now the shit is
steaming". And the americans say "Shit hits the fan".
The day after this story is on the front page of Reddit. The circled
comments brings the most eloquent version of "Shit hits the fan", that I
have ever seen. (laughter)
Yes, but what he says, is true. I already said it earlier.
When a company is depending on document digitalization, and you think about it,
who isn't these days, then we have a problem. They can shut down the
company, if they are unlucky. For example, I was called by the management
of a state archive. They created their archive with Xerox devices, and what did
they do then? They thew away the originals. Ye?
(spiteful laughter)
Now they stand there, with an empty gaze in front of their scanner fleet, and then
they can check all their documents for plausability. But even otherwise the
internet humour is amazing. (laughter)
(applause)
Even the involved provide the humour themselves.
If you, as the Xerox vice president, get the same interviews all day,
maybe mistakes happen. This one's pretty good. You
don't need to read, I'll read it out real quick. Of all things, in front of BBC
Dastin tried to explain. He said: "You know, all this is
half so bad, this "Normal" compression mode, it can
produce errors, but almost no one uses that, only the military or some
oil drilling platform." (laughter and applause)
Yeah, what could go wrong? (laughs childlike)
So, now we have... (laughter)
(laughs) Now we all noticed,
that errors on oil drilling platforms in the USA were a bit neglected
lately. Now we all laughed. And I did say - I want to keep my
word - laughing is ok, but malice is inappropriate,
even malice is hating. And, try to imagine you in Dastin's shoes. If you were
interviewed about the same thing for 14 hours, you'd make a mistake too.
And of course, that mistake will be talked about. Dastin said to me afterwards,
they misquoted him, and I don't have any reason not to believe him.
Just to protect him a bit here: He probably didn't have
a good day. So, let's continue.
This tech-portal is glad that catpics don't seem to
be affected. (laughter)
Notice the way it's written, as if they make sure, yes, as if they don't
know really, maybe catpics are affected after all.
(murmur) And here's a new press statement
by Xerox. The public pressure was so big, that Xerox said:
"Ah well, you know what, maybe we should rather do a patch
where we remove pattern matching". Legally recognizing the mistake however,
they never did. Even until now. Since it was in the manual.
That's how it is by the way. If it's in the manual, it's ok. For
microwave, it's written, you can't dry your cat in this.
Here is another newspaper article. And when you waited so long,
even a patch won't save you from mockery. Now the newspapers start
including misprints in titles on purpose.
(laughter)
Let's go back to Xerox's statement, because they write
a clear, important declaration. You will not see letter replacement,
if you set your compression to at least "Higher", at minimum 200 dpi.
Xerox published documents, in which it is clearly stated, that pattern matching
is only used in "Normal" compression mode, and not in the two higher ones.
But now here this whole time I've been thinking, I'm sure I also
saw it in the higher modes. Different readers
told me as well. But I just can't reproduce it on my two local
devices. But one thing is for sure:
If letters get replaced in higher modes as well, then absolutely everyone
would be affected. And Xerox would have miscommunicated. Then we would
have a much bigger problem worldwide. So I don't just publish my worry as
a rumour. Decency also dictates that. So, but now one of my
friends in a company in Bonn, my former living place, looked at
his Xerox Workcentre 7545. I'll look up the numbers later! (laughs dumbly)
And because it was my former place of residence, we went there and
took my test numbers, and scanned them in the mode "Higher",
that's the factory setting, and we even chose 300 dpi as a resolution,
for text, you'll agree with me, that's quite generous.
Zack - The yellow numbers are wrong. (laughter)
That's not all by the way. I just marked a few here that I saw.
I won't go through 500.000 numbers and mark all wrong ones.
But you see, how common the errors are. I repeat:
In compression mode "Higher" with 300 dpi. Now we take the blue rectangle and
enhance it. Here are groups of numbers marked in red - oh, you only see it
in light pink now, but you see it - that are identical to the pixel.
Such thing is very unlikely. If you scan the same number multiple times,
it will almost always look slightly diferent. So, pixel identical numbers
in a high quantity means, that numbers get reused, that's
a clear sign of pattern matching. So different from Xerox's statement,
we also have pattern matching that's used here. One reader once even told
me of an interactive visualization, that makes same numbers visible.
Yes, let's see if it... - Yes! - there it is.
And now I can hover over it here with my mouse pointer, and
we'll make everything red, where a number was reused.
I won't make it too long, I'm already a bit in overtime.
It's because you always applaud so nicely. Which I enjoy. (laughs)
(applause)
But here you can see, how many numbers can really be wrong.
From here on it's clear: Hundreds of thousands of devices, on factory
settings are affected, and the fun is really over. With this you can really
hit a company hard. And I didn't want to publish this without
searching a talk first. And I wanted to make sure, that I
didn't make a mistake. I didn't want to be able to be
sued for millions in stock price here. So I recorded the whole process
of the wrong number generation on video, and put it on youtube as an
unlisted video. I sent the link to Francis Tse, one of the chief
engineers that I mentioned earlier. And of course they were
thunderstruck. From here on the thing is really all encompassing. Francis
confirmed over phone, that I did all right indeed. And Xerox was cooperative,
but they also wanted me to wait until they reproduced the error.
But I also remembered, that during our last telephone call,
I felt a bit fucked with. So I said, my people,
it won't be like last time now. "I have the blog article done,
and the video is already uploaded." (laughter)
(laughs) And when you...
(applause)
"Don't take offense, but I request to be included from now on,
because I also treat you fair." So we agreed on the thing,
and now you see what it brings to not hate in advance. If
you shat on them beforehand on Twitter, it's clear they say "Come, screw you!"
After that, there was about six hours back and forth calls. We had
calls over and over. They tried to reproduce the error with my help.
For me it was evening, I spent the night on the phone in the office
and didn' eat anything but the cookies that layed around. At some point Francis
calls again, and says completely dumbfounded "Yep, we reproduced it."
Errors on factory settings, then there was silence on both sides.
We were just all shocked. And you know what was found parallel?
The Code for the compression scan is eight years old. That's how long the bug
was out in the wild. Eight years.
Yes, they were a bit dumbfounded. And I said: "Here's
my blog article, please read it and confirm, what legal safety
I have for publishing this."
(laughter and applause) (gasps of laughter)
No, so... this error is extremely dangerous.
I didn't want to wait any longer. Here's the article, and that's what
they did. And I was allowed to publish the article before them, even. That's
pretty unique. And you will agree with me, don't hate: If that's what
you reach with this, then that's good. A conversation between adults.
Lesson learned: Negotiate in the right moment. This is the next
Xerox press statement. I'll increase my speed a bit.
Xerox, of course, commented right after this as well.
They retract their earlier communication, thank me, and
say, that now first of all they'll see, how big the thing really is.
And from there on they were always nice in the statements, and
overall the climate was very constructive. This is the next Slashdot article.
It's getting surreal, just look at the titles!
After the back and forth, it doesn't matter for be with Slashdot
what Xerox says, but what they confirm to me. (laughter)
And here again is our snappy Peter Coy from Business Week.
But now... One more, I do have on more.
I mean, a compression mode! (laughter)
Doesn't really matter now. But on August 11th the proof for the
error also occuring on "Highest" mode succeeds.
Even a quality conscious user in the last eight years, that wanted to
produce beauttiful PDFs, couldn't avoid it. And to be honest,
after my informations the error doesn't occurr on TIFFs.
I don't want to make it look worse than it is. No one takes TIFFs, of course,
they're gigantic. On August 12th Xerox admits publically, that it's a matter of
an eight year old system error. And announces the patch again.
But of course they are deep in the whole thing, legally. And when it's
midday in the USA, it's night time here. And so in the middle of the night, when
visitors of this speech are usually awake, Dastin and Tse called me on
my phone, and wanted to tell me first, which I have to say, I found incredibly
nice of them, that they found the bug, and they'll roll out new
software. And there you can see that the relation really
got better. This is the patch download page by Xerox. Here you can see
how many devices are affected. Note the "X"e, that's whole
device families! (laughter)
So, the press is reporting again. The computer magazine CT writes
an article, and calls the whole thing "Scannergate". And here is
one last kick from our beloved Peter Coy.
He sounds so sarcastic, but unfortunately he's completely right.
Eight years of production of scanned, archived documents could contain
these errors, and cause harm until forever. Hundreds of thousands
of deviced and companies worldwide. We live in a society, where now,
as we are speaking, the transition from a world of paper into a mix of
paper and digital is happening. And the translator
between the two worlds, that's deviced by Xerox workcentres.
It'll be with us for a long time. Now the most important thing:
I already said, that Xerox has a decentralized supply over third parties.
Personally, I have no reason to believe that the patch reached
a lot of devices. So: Spread the word! At the end of this talk there will be URLs,
where you can get more info and see more. It's almost
the end... Besides all the "Lessons learned", there's one "Lesson" that I
haven't mentioned yet. I always got disbelieving looks,
that I didn't take any money for the thing. One manager even said,
I'm "pretty dumb". About that, two things. First, it's generally hard to make money
with something like this. Even if you want With no proof you won't be taken serious.
And with the proof, you'll mostly just find the bugfix directly, and then
you won't get any money either.
And second: Companies don't know friends. If I had taken money,
it would've somehow been made public and could've been used against me.
And it would've brought be in a position hard to negotiate.
But I wanted this error to be fixed. And last but not least,
the community helped me, and they didn't get money either.
I'd do it like this again, but...
(cheering)
...at the end of the day, everyone has to decide that for themselves. If you
would do it differently, then that's ok. I just want to say in advance,
you bring yourself in a weaker negotiation position. That's all
the "Lessons learned" again. I won't reiterate them again now.
They're here so you can download the presentation, and still have them.
And now we close the circle to the start, and with that we are done.
At the start, there's the prologue with Obama's birth certificate. Here it is,
the "long form birth certificate". Shortly after the Xerox-saga, journalists from
the "Reality Check" USA wrote me, if the Xerox bug could've been
the reason for the "forgery". And they did a whole lot of
detective work. For example, the Obamas published their tax documents, shortly
before the birth certificate. It was scanned by a Xerox Workcentre 7655.
Tja, and further technical attributes spoke for
a Xerox scanner. And the "Reality Check" guys asked me, if
I could ask Xerox about it, since I had such good contacts. And Xerox...
(laughter) And Xerox asked for understanding,
that they really didn't want to deal with this now... (laughs)
...and I left it alone. And now I'll prepare for my
congress speech, for this talk today, yeah, I look in the PDFs again,
and there's the exact copied, yeah the exact letters in there, that were
a sign by Xerox for pattern matching back then. And I look on the internet
pages, and there it also says something about letter doubling. Here's two exact
same boxes. Notice the indents on it. Now, make your own image here
But I think it could be, that this conspiracy
is hereby over and done. And with this, it only remains for me to say thanks, for
spending a whole hour with me!
(applause)
If everyone keeps clapping, it'll take even longer!
So... (laughs)
Up there you'll find another link for the Xerox saga. Pass it on!
And down here a link to my page. There I'll publish
the presentation online. Maybe tomorrow. I won't go into the WIFI here! (laughs)
(laughter)
And take care of evil copiers!
Herald: Okay, thanks first of all, for this amazing talk!
I think it was very interesting for everyone.
Everyone on the way out, please hurry and close the doors after.
And be quiet.
For the questions, I'd like to start with the ones from the internet.
From our Signal Angel.
Signal Angel: Thanks! And a great applause from the internet,
you couldn't hear it now. But there was a lot of positive feedback.
And also the plea to publish the presentation, especially
the symbol images were well recived.
Daniel: It will happen, on my page, latest tomorrow. Definitely.
Signal Angel: Very good, thanks. Two questions from me.
The first question is, does Xerox have a technical difference between
Scanning, Printing and Copying? Or is it always the same thing?
Daniel: So, scanning, there paper comes in and for printing it comes out, ne?
(laughter)
No, so, for printing, you just recieve the printing data.
I don't know about anything being compressed afterwards again.
Scanning - here there are different modes. The PDF modes, there are three, that
I mentioned earlier. And copying - In my view it's not like this, that it always
happens during printing, because there you don't compress. You see how I mean it, yes?
I'm sure I would have recieved some reports if it was like that.
And that's why I don't think the process of copying itself is affected. But
that wouldn't be so bad anyways, because there are no documents that get archieved here.
Signal Angel: Okay, and the second question: Are there any definitive
harms that happened because of this bug?
Did you ever recieve and feedback regarding this?
Daniel: I have feedback, the ones that I named earlier.
And of course a few more. I'm of course not going to say any names.
But... So, I can only say this much:
You have to imagine yourself in the place of the company that's affected here.
Your files might be good for the trash.
Will you make this public? No, you will request compensation
from Xerox in silence, and not write any of this on your
website, because then it will fall back to you, that our data
is faulty. No one will ask you, if that was a Xerox copier now.
So I don't expect there to be a grand reveal now, if it can be
avoided. If some random bridge on a highway collapses now
that would of course be a different matter.
Signal Angel: Okay, thanks again! Daniel: Gern!
Herald: Good, then I'd suggest we continue at microphone 2,
at the first person.
Question: Just a short question. This is probably a technique that gets used
by many. Did you ever try this with devices
from other companies?
Daniel: I had a great quantity of reports from other companies. But if you
take on a thing of this scale, you'll become a victim of spin doctoring.
And all of it turned out to be false, Here, again:
Stay sovereign, don't just pump out rumours. Here none of it was true,
and in concrete cases it wasn't the compression method itself,
but the fact that there was indeed another bug.
Herald: Good, then 3 please!
Question: Hello? Thanks for the talk, it was pretty cool.
I just wonder about the thing, the bug being there somehow for eight years.
Did you look on search engines, did others... I mean, I can't
imagine that for eight years no one saw it, because
as you say, on a blueprint, there you can see pretty quickly, so...
or maybe other people messaged you, because they had seen it before,
or maybe they said, hey I noticed this before, Xerox said,
yes, higher compression, then they were lucky and it worked.
Daniel: So, it was, first of all hard to discover. Second of all,
it was known for the mode "Normal". It was on purpose, they even knew about it
And that's why it was hard to recognize the real bug, because Xerox...
The support that knew - mine didn't know - always blamed it on the
"Normal" setting. And then it's plausible, then I tell you:
"Yes, you used the "Normal" setting, take another one, then
the error will occur less, you'll probably be lucky there"
So I think, that indeed, that the bug was discovered for the first time...
Question: So, no one contacted you, with "Hey, I've seen this before" or so?
Daniel: No, no one. In the whole storm, no.
Herald: Okay, next up again from the 2 please.
Question: Moin, thanks for the presentation from me as well. Was very cool.
Daniel: Sure.
Question: Short question, you said, you didn't do it for money...
Daniel: Correct.
Question: ...and somehow... I find it very noble, very cool. But
did they ever offer you something from their side?
Daniel: No, they didn't. No one there...
Question: Not even a job or anything?
Daniel: Well, there I can in fact hold Xerox a bit. They didn't offer
me anything. I couldn't have accepted it anyways
by that logic. That's why it was totally fine. In that long night, where we
had the phone call, they were ready to have me fly in. But
I honestly don't know anything about copiers either. Not my main job.
I can show them the bug, but I can't repair it. So...
Question: Ok, but if they would have flown you in, why not
work with them together and try to solve the thing?
Daniel: Jo, I could've done that. But I couldn't have contributed anything.
Because, they have to find the bug in their code themselves. It was clear that
something happened. I can't help with that. I'd just sit around.
So I also said it just like that.
Question: That makes sense.
Daniel: Yes, and flying 2x intercontinential for that... I don't know.
Question: Yes, but if they paid I would've done it.
Daniel: I admit, I also overthought it again. But I had
also stuff to do job wise, and it wouldn't have worked out.
Herald: Good, next up 3 again.
Question: Well, I have a copier at home, and I have a very
intimate relationship with it. Are there any reports, that
some tried it with their home copiers,
and then went "Oh Sh...?"
Daniel: I don't know of any reports like that. It only affected the things that
I just showed. Workcentre, ColourCube. All big things.
basically. Question: Okay.
Daniel: This JBig2 in Hardware, that's also
I think very expensive to implement.
Question: Okay, thanks! Daniel: Jo!
Herald: And 3 again please!
Question: Maybe a cool crows research task
Is maybe to look through those manuals,
to collect. Who had access, which year does it show
up in the documentation at all, is it really
that old, so eight years, or maybe only four years?
They only noticed four years ago, and thought, hm, it's cheaper, we print
new handbooks, and leave the software as it is. Because it's more expensive,
to roll out new firmware.
Daniel: There's a theory, that here a bug was declared a feature.
I can confirm that. But I don't have proof for it. I want to say that very
clearly. But seriously, who would design a scanner,
that swaps around numbers? Only if it was just for the military (laugsh)
Herald: Okay, I think one last question. Then 2 again.
Question: Not really a question, but more of a suggestion for the presentation,
in case you present it again. It's really great.
You have this scale, with accesses to your website at the bottom.
I wondered, during the talk, if maybe you could also do that
with the stock price of Xerox? (Daniel laughs)
Daniel: It wasn't that bad. I mean, that PR section of them
handled it pretty well despite the world wide attention they had.
I mean, that's really an error, where you could think, this is
a danger for the whole company. It's their bread and butter business.
But it didn't turn out that way. We will see, I could've put
such a live stock price curve in the presentation. I don't know,
what's happening on the internet right now. But good suggestion, thanks!
Herald: Okay, we also have questions from the internet.
Therefore I'd also like to...
Signal Angel: I just have one more question from the internet. Are
there are statistics or numbers, about how high
the likeliness of such an error is?
Daniel: Well, you saw the page I told you about. That was the case
with font size 7 or 8. I don't know anymore, where I got it really
niceöy reproduced. But when...
Signal Angel: But... Numbers, thatr's not a normal page now is it?
Daniel: It was all numbers, but of course it's also possible with
similar letters. But it can happen too. I don't have any statistics.
For the numbers the 6 and 8 are affected the most. But real
error percentages, I don't have. But you can see, what's possible.
So I have... I didn't try for hours on end, until I found the
page with many yellow points. I scanned ONE page, and then it
was like that. Yeah? So it's not like you have to look for it forever.
Question: Yes, thanks!
Herald: Alright, I think we are done then.
Then please another big applause for the lecturer!
(applause)
Daniel: Thanks!
(longer applause)
31C3 Credits with no audio
Subtitles created on amara.org in the year 2017 - 2022 by multiple collaborators