♪ (preroll music) ♪
Angel: After half a year, Volkswagen committed to tweaks to their emission readings.
Those two boys, Daniel Lange and Felix Domke here on my left,
will share some insights with us.
Daniel will not only focus on the ECUs,
which is the acronym for the Electronic Control Unit,
and I think we're seeing one over here already,
whereby Felix will show us some tricks to extract and tweak the firmware.
On both sides we will see how many people have been involved in the entire process
and we would get an idea what everything is involved in there.
So, you applause and I'm gonna take over the Bildschirm.
Good luck!
Felix: Alright. Hello? Okay.
Hey, so, I'm Felix Domke.
Do we see the video output yet? No.
Anyway, I'm Felix Domke.
I'm here on my own because I was personally interested
in how Volkswagen is cheating on their emission control.
And maybe we get video at some point.
I want to stress that I was self-funded.
I did this with my own money because I was personally interested in this.
So I did not do this on behalf of anyone else.
Daniel: Let's start the Keynote again and see whether that one works better then.
F: I am sure we figure this out.
D: Oh, it worked before?
Yes, that's because one of us two wanted to use a Mac.
*audience laughs*
F: But I wanted to use Keynote, I don't care which operating system.
D: This one works.
F: Anyway. So I will now hand off to Daniel which will give the first part of this talk
and after that I will give the second part of this talk.
D: Okay, thanks Felix. My name is Daniel.
I used to work for a big Bavarian auto manufacturer which is not Audi...
*audience laughing*
...for 14 years.
I've been running the IT strategy, I've been doing IT architecture.
And most relevant to this talk,
I've been responsible for the process chain electronics and electric.
I've done the rollout of Connected Drive in China.
So I kind of have quite deep insight into how the automotive industry works
and I'd like to share a bit with you.
I'm an Engineer by training, I guess many of you are.
And I want to share how the engineers think
inside such a big corporation like Volkswagen,
and what pressures, what boundary conditions they are working on.
I have my own company now which makes my life a bit easier
than Felix's, as you see in the legal disclaimer.
Those are folks from the UK.
They're called "Brandalism", I hope you notice the "McDonald's"-M at the end.
Those are folks who started a few years ago to reclaim the public space.
They were just annoyed by all of that advertising.
And when the Paris negotiations for the eco treatment came,
they just felt a big invitation to use the opportunity
that Volkswagen has created for all of us
and make advertising in their style, but perhaps not in the message
they would usually have conveyed.
I'm a strategist.
So what is the thing that defines how the automotive indusry works today?
We are in a saturated market.
In the developed countries, so everywhere in Europe, in the North Americas,
everybody has a car that wants one. Some have two.
So when you want to sell another car
you're basically talking about replacing an existing car with another one.
The only growth you have is in the BRIC states.
BRIC is: Brazil, Russia, India, and China.
And, here, especially in China.
You have a big overcapacity. There's just too many automotive manufacturers
and there's too many plants they have.
So all of them basically struggle to get the loads on the plants,
to produce enough cars and to have those cars sold at some point in time.
Because the queueing in between production and sales
is actually the big parking spaces you see in Bremerhaven or so
where there's ten thousands, in some countries even hundreds of thousands of cars
basically stored in between production and sales.
Ten years ago, fifteen years ago, that didn't exist.
The cars were basically sold off the factory.
But people have been moving away very, very slowly from cars.
They have, as I said, a saturated market.
It's just not that easy to sell a car anymore.
That is because of social shifts.
When I was young, there was "The Dukes of Hazzard",
there was "General Lee", this car that basically is the star of the show.
There was "Knight Rider" and nobody watched it for David Hasselhoff, not even the girls.
They watched it for KITT.
When I was young, I wanted to own a car.
I wanted to have KITT, possibly.
And when I grew old enough, I found out I can get a car that looks like KITT
but, you know, all the fun stuff is not in there,
so I turned to computers.
The next thing is organization, megacities.
We live in very condensed spaces in those cities.
If you talk about a place like Beijing where there's like 21 million people
in an area that is one city, where there's nothing big inbetween,
there's no river, there's no forest,
it's just like one city.
If you go to Tokyo, Yokohama, you can drive on the motorway
for nearly three hours when you enter the city before you leave the city.
And you're driving on the motorway, you're driving on an elevated road
for which you paid toll so you actually can drive.
But it's three hours before you leave the city again.
And in these cities owning a car and operating a car
is about the worst thing you can do. You just don't want to do that.
The average speed of a car in Beijing these days is 12 km/h.
If you're a good runner you can beat that.
And incidentally this is exactly the speed that a horse carriage makes.
*audience laughs and applauds*
We have managed to undo all of the innovation of the last 200 years,
it's just the interior is a little bit more comfortable.
The actual getting from A to B is the same as with a horse carriage these days.
And then there's technology shifts.
The problem is, there are big things, big visions that everybody follows,
like electric mobility. Electric mobility means:
You buy a car that's one and a half times the price of your standard car,
you lug around 300 kg of batteries for no apparent reason to do so,
and you now need to install something in your garage —
which you most probably don't have, look at megacities —
to be able to recharge the car because it only goes a hundred miles.
So it's currently not a very compelling thing to sell to the end customer.
There's self-driving cars, which is kind of a great, big vision.
But I would really call that a "vision".
A "vision" is something that's not being implemented in my lifetime.
And then there's downsizing.
Downsizing means ...
Everybody wanted to have the biggest engine,
everybody wanted to have the biggest car, let's say, 10 years ago.
You wanted to have that six cylinder that was giving you status.
But now the automotive industry has an overall cap
on how much emissions the average new car fleet my have.
And they can only reach that if they manage to sell smaller engines to you.
Because for everybody who buys a really big engine
that will never ever make that emission cap,
they need somebody whom they've sold a small car to —
preferrably an electric car, because they even have statistical advantages
to make them a bit more attractive —
to set that off.
So very literally the poor guy with the small car needs to exist
for the rich guy that drives the eight cylinder and doesn't give a shit.
Strategy-wise, there's only two things that an automotive car company is driven in.
And that's really everything there is.
There is "reach a target ROCE".
ROCE is: Return on Capital Employed. That is just two numbers:
your EBIT, which is your Earnings Before Income Tax,
and the amount of money you have in your company, the employed capital,
which you got from people that lent it to you
or from your stakeholders, from your investors.
And that is what the company is measured against.
Every automotive company basically runs like this.
Just this one figure, it's a percentage like "30%."
"30%" means: On the money you have you made a 30% return during that year.
The downside of measuring in ROCE is that everytime you use that Euro or Dollar
it counts again because the money works for you.
That means what you're looking at
is a company that gradually moves from a very industrial type of application
to something that tries to move faster,
that tries to be quick and regain money faster.
And then there's "outperform the competition."
You have to understand the situation
that there's a good dozen companies
and everybody has the same strategic position:
"We will outperform the competition."
So statistically, you will know that half of them are going to fail
because that won't happen, right? Somebody has to be the lower half.
But the only thing I have seen in about five or six companies
where I know the strategy in detail, is: the sequence.
Is the first or is the latter the more important one?
And sometimes that depends on markets.
There's this new emerging market and you want to outperform the competition,
you want to grow more.
And then there's this laggard market somewhere in the European Union
where you just look at the money, you know, how much money are we making on this.
But that's all, that is how an engineer is basically steered, that is the strategy.
And that means when you break that down through the levels of hierarchy, what is counting is:
How much money do you need to make this?
How much money are you gonna make on this?
Those two divided will be contributing to the ROCE.
And do you deliver anything that can help us outperform the competition?
You notice that there is a lack, which is, you know: What does the customer want?
Or: What is good for associates? Or something like that.
Just in case you hadn't noticed before.
Okay, I'd like to do a bit of a quiz with you before you all fall asleep after lunch.
Eleven million.
"Eleven million" in the context of the exhaust emission scandal.
What is that number?
Audience: Cars affected!
Correct! Cars affected.
Eleven million is actually the Volkswagen cars
which need to be recalled world-wide
to get this little filter thing fixed and their software updated
to meet the emission targets which they had been produced against.
1500 ...?
A: Number of engineers!
Number of engineers? No, not correct.
Number of engineers would be above 10000 for a car in Volkswagen Group. Sorry?
A: Cost for fixing it per car?
Cost for fixing it per car? No, that's maximum 600, we're gonna see later.
*unintelligible suggestion from audience*
No? Well that was too difficult then, and that was a bit intentional.
That's the amount of hard disks they collected from the associates.
*audience laughs and applauds*
Now the thing is, we've had in the press that there is maximum 13 managers
which are responsible for this emission scandal within Volkswagen.
But then they collect 1500 hard disks and USB sticks from 380 associates,
and that number is a month old because they haven't reported newer numbers.
So something is mismatching there, right? Something is mismatching there.
So the first number we have is for how many associates
are actually somehow affected by this is 380.
Because you come to work somewhere in Wolfsburg I think, right?
And then there's this nice chap coming up and telling you,
"Uhm, actually we took the hard disk off your PC,
you're gonna get a new one from IT, we guess tomorrow,
they're a bit behind with, you know ..."
6.7 billion ...?
Just shout!
*unintelligible suggestions from audience*
Fine? No that will be less, much less.
*unintelligible suggestions from audience*
Yes, you're getting close. It's the money they put back,
they set aside to actually pay for the recall and the legal fees.
Now if you divide that by 11 million you get about €600 per car.
So it's not that much money per car.
In Europe, the plan is basically that you go to the dealer and get a software update.
In the States, people already got $1000 in cash and in coupons
as a goodwill measure.
So something I learnt from Martin Haase here going to the CCC Congress all the time
is that we need to read text really well.
So the upper one is the original in German, the lower one is my English translation.
The English translation is as accurate as possible, so it's not good English.
Please excuse that, it is so you get the gist in case you can only read the English.
So that is Mr. Pötsch, he's the president of the Volkswagen supervisory board.
He is the poor guy that now has to sort it all out.
He used to be the CFO. We're gonna see why that is important a little bit later.
And he has made this analysis:
It was "individual misbehaviour", so it's not an organizational problem,
it's "weaknesses in particular processes",
and it's "the attitude in particular sub-partitions" ...
"Teilbereiche des Unternehmens", it's impossible to translate in English,
it's actually impossible in German, but, you know, the legal team came up with that.
So the "attitude in particular sub-partitions of the company to tolerate rule violations."
Now, if we go through this very quickly: It's not a rule violation,
you violated the fucking law.
The other thing is, if you have particular processes, you have particular associates,
and you have particular sub-partitions of the company,
That tells you something, right? That just tells you something.
This was probably two days' work of somebody in the legal team,
and I guess you noticed, right? I guess you notice.
"Legal team" is probably these people.
Jones Day is a big American lawyer company
and they've asked them to help with sorting out this.
Now the funny thing is, there's public prosecutors all over the planet interested in Volkswagen
but Volkswagen thinks it's not really clever
to have those people come in and find all the info,
it's better to have Jones Day, their own kind of bought-in legal team,
ask the associates first.
Now the problem is, whenever the let's say German prosecutors wake up and go in there
and say, like, "We would like to see what has happened,
so please hand over the material, please hand over the hard disks,"
they would get a very, very nice reception, be greeted with coffee and shown a room
where all of the hard disks and everything is stored,
"We collected it for you."
I have no idea whether they gonna show everything to them
I have no idea whether there may be some material lost in between.
We've heard from Anna earlier in Germany it seems to be
that things hit the shredder and hard disks get lost and everything.
So if it works like that in the government I have no idea how it works in companies.
But if I was on the prosecutors I'd probably see that I speed up a little
because otherwise you'll get all pre-prepared material.
And because Jones Day can't do all of that —
you have to interview all of those people, and you have to look through the hard disks —
they asked Deloitte to come in and help them.
Now Deloitte are a very good company, they have very, very good forensic teams,
so that's a very good choice. But the important thing here is:
Out of the four big consulting companies that do finance analysis and stuff
those are the only Americans. The others are headquartered somewhere else.
So what it tells you here — American legal teams, American auditors —
that's where Volkswagen looks. Volkswagen is actually afraid of America.
They are not that afraid of Europe or some other country in some other continent.
Now, let's talk text a bit again.
"We have no findings on the involvement of the supervisory board
or the board of management presented."
Now, again, "no findings", okay, "presented", right?
It's not "we don't have any findings" or "there is nothing",
it says "we have no findings presented."
And the other thing is "involvement", that's an odd term.
In German, "Involvierung", that's not even German, right?
If you look it up, "Involvierung", nobody of you talks of "Involvierung"
when you talk to your family or when you do something at work.
The trick here is, the supervisory board has a reason for existing: supervision.
*audience laughs and applauds*
The board of management has a reason for existing and that is: decision.
They are the deciding body.
None of them are ever "involved", right?
When you work on something in a big hierarchical company
there is no "involvement" of your board member,
there is no "involvement" of your supervisory board member.
So per definition, they cannot have an involvement, right?
If he wanted to be straight he would have said:
"I, as a former board of management director and now as the head of the supervisory board,
I guarantee there was no involvement of my or my colleagues in this.
And if there was, I would pay back my salary, I will go to jail, I will ... whatever."
Right, sacrifice a goat?
But that would have been straight communication.
But this is not straight communication, this is... bullshit.
Okay, quiz time! 10 ...?
You remember, this guy here told us there's no involvement in anything fishy, right?
It's all those small engineers, all those bad, bad people down there.
But they are gonna hunt 'em down, right?
So there's no involvement with anything fishy here.
So, in that context, what is "10"?
A: Board members!
10 board members? Close, they have a little more.
A: Levels of hierarchy.
Levels of hierarchy — quite good. It's, I think, eight or so, but you're quite close.
No, it's actually the amount of planes that Volkswagen owns.
All of them are jet planes.
Because if you're a board member you have to, you know, fly in style.
And because there's nothing ever fishy at Volkswagen
it's run by Lion Air Services out of the Braunschweig airport.
And obviously, Lion Air Servives is registered in Georgetown on the Cayman Islands.
*applause and laughter*
Nothing fishy ever in that company.
Okay, let's get back to topic. I have about another ten minutes
before I want to get Felix the chance to show you what he has done on the ECUs.
So I need to get you up to speed about how all of this context here works.
And this here is called the NEDC, it's the New European Driving Cycle.
This is what your car is tested against for emissions.
It works like that: You condition the vehicle a day before.
Which means you really drive it hard on the Autobahn
so the exhaust is really free and everything.
And then you do these cycles here where you basically accelerate the vehicle,
slow down, accelerate the vehicle, slow down, accelerate the vehicle,
slow a bit down, slow a bit more down, and then you cycle again.
And the last, cycle 5, is an optional one, depending on what you measure,
that is actually going to the autobahn
and you're going up to a top speed of 120 km/h for a very short period of time.
The people that have detected the tweaked emissions
in the VW Jetta and Passat they looked at, they have called this
"a very light usage cycle," and they called it "unrealistic".
Because basically nobody drives the car like this, it's a very artificial thing.
And that is the problem for the engineer, right?
The engineer looks at this and says, "Yeah, you know, it's a standard.
It's something we do to measure against."
But nobody drives like this. It's not realistic, right?
So if you fake the data in this we're not actually faking something our customer uses
because no customer drives like this, it's very artificial.
And there's a very good report by ICCT which is, "Mind the Gap".
Which is what you hear in London when you go into the Tube.
And what they mean is the gap between what gets out when you measure emissions like this
and what gets out when you actually drive the car.
And that gap is widening year by year by year.
Because engineers get better and better at optimizing for this cycle.
The cars on the street? Phhh, they do get better as well, but less, right?
That's why the gap widens.
And trickery on those tests is very common.
I'm sorry you can't probably read that in the stream
and probably can't read that when you're back down there.
But that's an original slide I had to take from Transport & Environment,
from that report which I just named.
And what it says there is what tricks people are doing
to actually drive down the emissions.
For example, they blow up the tyres
by 3 bars more than you could actually use them on the road.
Now when you do, the bottom of the tyre looks like this, right?
So that means you only have a very, very small portion of the tyre
that still touches the ground, so your resistance gets reduced.
They put diesel into the oil beause diesel is lighter than the oil which you are using
inside the vehicle, so friction gets reduced.
They take off the mirror, the side mirror on the passenger side
because that is not legally required to be existing,
so, you know, it's resistance, so get away with it.
They tape close all of the openings of the vehicle
because obviously when the wind goes over it
it goes much smoother once you have everything taped.
Now all of these things are either okay or they are kind of borderline grey area.
And they do this. This is how actually emissions are tested.
So this is why an engineer, when he looks at this, says,
"Yeah, it's an optimization problem. They want me to get a low number
and I have pretty clever ideas, which involve diesel and sticky tape and everything,
to reduce the number."
*sighs*
The results are this.
That's from a 2012 report from — a 2013 report, I'm sorry — from ADAC,
the German MRT company.
And what you see, the lighter blue ones are actually the emissions
which the car produces in this cycle.
The darker blue ones are the ones which are produced
when you just go on the motorway and drive them.
And you see that there is a discrepancy
which is ten times, twenty times, thirty times what is the measured data.
So what you need to understand is that even in the past nobody ever thought,
nobody in the industry ever thought that the data which was measured
had any real connection with reality, right?
The only connection was, you knew that what you're measuring
within the duty cycle NEDC is definitely less
than what you would ever see in any realtime use.
But that's it, that's it. That's no secret, right?
It's something that has been out there for years.
Now the folks at Deutsche Umwelthilfe, which are actually people that helped
find out what Volkswagen did, they wanted to see that others do it as well.
And because I wanted to give you as much information as possible
we are going to look at this product here now, which is not a Volkswagen as you may see.
And when you measure this car it actually looks like this.
So that means when the car is thinking it is running an NEDC —
because it is conditioned to do so, it is the right temperature,
it is the right setup — it actually delivers the blue bars.
And if you run it because you just run it and you don't do the conditioning
it delivers the grey bars.
Now there's many things you can say about how they measured this
because, obviously, this is not science to the best level of accuracy.
But you do see a pattern here, and you do see the pattern
of the 30-, 35-fold emissions. And that is what you always see
because this is what an engine like the one in this car —
a 1.6 l diesel engine if I remember correctly —
actually does when it's just operated normally.
And the lower ones are the ones which you get when the engineers did all the good tweaking.
Now why has all of this ... Oh sorry, so this is just one test, right?
And you see that this test, when the vehicle is cold,
you get fresh air with a nice rose smell out of the exhaust.
And when the vehicle is operated normally you basically get what you expect,
you get the combustion products out of burning diesel.
Now why is all of this now a problem?
This is now a problem because of the American legal system.
The American legal system is very, very different
from what people in the European Union are used to.
In America, there are two things which are a bit strange perhaps
to somebody who's accustomed with a German legal system.
The first thing is, there's jurys.
So there's common people that actually decide about what's right or what's wrong.
And that means, what they award as compensation
to people that have had a disadvantage are often astronomic figures.
Now these figures are sometimes reduced again by the judges,
but it's not uncommon that if something hurt you or you got into an accident
you're awarded million dollar sums.
In Germany, if somebody shoots your eye out, you may be getting €100,000.
So there's a huge discrepancy there.
And the other thing is, in America there are punitive damages.
"Punitive damages" means: You did something wrong, you did it on purpose,
and you're punished for it.
In Europe, a company basically is, you did something wrong
so now you have to compensate the disadvantage somebody else had.
So to a certain extent, a company that doesn't try to trick
actually kind of loses an opportunity because if they are not detected to be tricking
they have just saved money.
There's no punitive element, there's no "You will go to jail for this."
At least in this context of environmental regulation.
Now in case you couldn't read that, that's actually a sign I took in california.
You go into a store and it tells you that basically everything you see there
and touch there is giving you cancer and your unborn children will be damaged.
This is what it says there: Belts, shoes, jewellery, handbags,
all products with metal, and everything
causes cancer, birth defects, and other reproducive damages.
So this is America, right?
Their view of protecting the consumer is completely different from Europe.
And this is why Volkswagen goes and says, "We will show good faith.
We will give you, American Volkswagen owner, a thousand dollars
because we just wanna make sure that you at least know we care."
It's important that you care because the jury will say,
"Well at least they awarded $1000, maybe a little too little,
but at least they did something."
The jury would say that. A professional judge in Germany would say, "Pshh, why?"
So this is why as a European customer you actually go to the dealership,
and if that guy is really nice you may be getting a coffee
while you wait the hour that he flashes your car.
So that's the only thing you're currently supposedly getting in Europe.
Okay, now the problem is: What they did hurts.
And it hurts because, if you do the statistics ...
Very nice people have published a publication here, a real scientific publication
where they did the maths, and they say:
59 people may be dying earlier in the United States
because of the additional emissions in the environment
which they took in and which may damage their body.
The social cost of treating those people — because they may be developing cancer,
they may be going to a hospital, and so on —
is about 450 million Euros. Now that's statistics, right?
"Lies, damn lies, and statistics." Mark Twain is often quoted with that.
But the problem is: That is a real cost, it is a real damage.
If you do violate emission laws it is something that is damaging people's health.
It may be something that is difficult to prove statistically,
but it is something which you don't only do to save money here or there,
it is something which you do to actually hurt people.
Okay, I need to speed up a bit. Very sorry, skip this, that's the next quiz.
15.9 million is actually the salary of this guy here.
That's a lady from BMW, I just wanted to put that out there.
She says, "It shouldn't be called Dieselgate, it's Volkswagen-Gate.
We never did anything wrong at BMW."
And the SZ, actually, yay, they follow, right? In November, it was "Abgasskandal",
in December, it's "Volkswagen- Abgasskandal".
The only problem is that even in 2000, BMW was cought cheating on the Motorrad.
So this is 15 years ago. 15 years ago BMW actually put the same code
which we are now seeing in Volkswagen into their ECUs for the F 650 motorcycle.
And we will see again here the same 34, in this case, -fold increase
in between real use and test bench use.
Now, honestly, they've been caught, they've been caught earlier, and they fixed it.
So in 2001, they actually brought a new version
and apparantly that didn't have this cheat code anymore.
But here we see a pattern again: too little time for development,
too little money willing to be spent on this,
so engineers try to trick.
When you get caught, and you get caught early
nobody probably of you remember this here.
It's fine, it kinda fades away into history.
If you're Volkswagen, if you have 11 million cars out of there,
you have a big problem.
Okay, I'll skip this one, it's really nice, you can see it in the slides.
But I have to go to this here to give Felix enough time.
So how does component development work?
There's a huge set of legal frameworks. It's a very structured top-down process.
You get requirements from the people that represent the market in the company,
you get requirements from the CFO, from the finance director.
And these are broken down into documents which are more than a thousand pages long.
And there's every single detail that could exist in this ECU written out.
There's a piece of paper for everything it does.
Everything. There's not a bit in this thing which is not pushed down
into a very hard set of requirements.
This is then put into a tool, often Rational DOORS by IBM or something,
and then every time something changes this is documented.
There's a complete paper trail, right?
So that means unless there will be a cover-up,
unless we're not given all the information as a public,
there's no way Volkswagen cannot find out who did exactly what at what point in time,
which level of management was involved.
Because every step of the development goes through a Q-Gate, a Quality Gate.
There's managers sitting there and they're approving everything it does,
every progress that has been made, and they're getting reports,
at least bi-weekly, on the progress.
And these reports go up the ladder, they are copied to the next levels of management.
So this is a fully transparent process and this is a fully top-down driven process.
It is completely impossible that you have an engineer that sits there and says, like,
"Well, I wanna cheat," and does the code. There's no motivation for him to do either.
He doesn't get any money for it, he would only be risking his career, so he won't do.
And this is why we have paper trails, and this is why engineers have written down,
"I'm doing this because my manager told me to do this."
And this is why you have Bosch sending a letter in 2007 to Volkswagen which says,
"We delivered you this code you requested. We're your supplier, we do.
But if you send it into production it will be illegal."
And they did.
So this is how actually this exhaust system works.
And this is a little bit important to understand what Felix is now doing
and showing you how the ECU that manages this all works.
To the left would be the engine, to the right is the exhaust,
the end of the exhaust where the remainders come out.
And the first thing is, you have diesel oxid cathalytic
and it basically takes out ... The interesting stuff here is CO,
so carbon oxide, and PM, the particle mass, through 98%, 50%.
The hydrocarbonides before that, they just kind of don't go through
the rest of the process anymore.
Then you have a filter that basically traps all of the diesel particles,
the stuff that causes cancer in your lungs.
But you have to burn them out at some point in time, about every 700 km,
when there have been enough collected.
So it's a bit a trick, right?
The trick is: You collect them so they don't exit the exhaust
but at some point in time you have to burn them again, so they do exit the exhaust.
Now the positive thing here is, they get larger, and the larger they are,
the less risk they — at least as much as we know — cause as a health hazard.
So this is the DPF here. And then at the end, this is the really interesting thing,
this is what most of the scandal now focuses on:
There's a selective catalytic reduction.
And what this thing does is, it does reduce the particle mass,
it does reduce the particles. That's nice.
But the interesting thing is NOx. It goes against this to about 90%.
So this is what it is made for.
It basically injects urea into the airflow and helps to reduce the NOx content
by creating by-products which are mostly water
that comes out the end of the exhaust.
And this is the system, this is a very complex technical system
that has to be managed, and this is managed by an ECU.
This ECU which they selected to do this, and everybody does, is the engine ECU.
Because to the left of the diagram before was this big engine, you didn't see it,
it fell off the diagram, but that's actually the fan blowing into the system.
So this is what you want to manage to actually control what happens there.
Now this thing is quite a sophisticated processor,
it's about the most complex device outside multimedia and entertainment
which we find in the car, and it is a very proprietory thing
because it contains a physical model of engines.
So there have been hundreds, if not thousands of engineers sitting there
and modelling how an engine works, really physically modelling it.
And the things that an OEM — an original equipment manufacturer,
a car maker — can actually tweak are variables.
They can say, "My engine has this and this size,
my combustion cycle looks like this and that."
But the code itself is opaque to the OEM.
It's a proprietory product which you can buy from Continental, or Bosch, or so.
And there's about 20,000 variables which you can tune.
And this thing is simulated and tested to death. Because it is hugely important.
Because you have this machine here that has like 100, 200 horsepowers
and if you steer it wrong it will blow up, and it will blow up really hard.
So this is why this thing is about the best tested piece of software you will ever find.
Which also again means there's everything documented, everything is written down,
everything is seen by everybody who's working with these,
whether it's in development, whether it's in integration,
whether it's in the plants that flash these things, and so on.
There's nothing secret here in this, right?
The functions which are there are actually there to be seen,
well, seen if they are named apparantly, and that is something that Felix will talk about.
*audience applauds*
F: Thank you. Hey, okay. So I will do the second part of this talk.
I'm Felix, by the way.
So my motivation with this was a little bit different.
I'm curious, and, I mean, we can find a lot of source material for this whole scandal.
We can find a lot of information in the press,
a lot of information in the Volkswagen press releases.
However, it should be easier because all the cars are there,
the 11 million cars are out there that have the cheat code in them.
And we are hackers, and we know code, and the truth is in the code.
So my approach was, well, let's take a car, let's take it apart,
let's take the firmware out of it, let's throw it in a disassembler,
maybe get some measurements, and then look at what the car is actually doing
instead of relying on all of this second-hand, third-hand information.
So what do we need for this approach?
So first of all, we need a car that's affected.
You need to drive that car somehow, and driving a car on an open road
can be dangerous if you have to follow particular driving cycles.
So there's a "dyno" you can put the car on and then you can just drive
without the car physically moving.
The wheels are moving, but the car isn't moving.
And this is what other people have done,
and they have taken very interesting measurements out of this.
However, we as hackers, we can go one step further.
We can take a look at the ECU itself.
And not only that, we can also ask other people who worked with these things
and may be able to get more information about them.
I will talk about this in a minute.
So first of all, this is my car, luckily that car was affected by the recall.
So I was very happy when I got the letter telling me I have to go to the shop in January
and get a firmware update because firmware updates are exciting, right?
I love updating things, so updating a car seems great.
Yeah, it sucked that my car was putting out more emissions than it should have,
but otherwise, it gave me the chance to actually look at the car.
I mean, I could have rented a car or something, but that makes it much easier.
I also went on a dyno with my car. On a dyno, there are no speed limits
or no people to run over when you just have to keep a constant speed or something,
so it makes things much easier.
And I talked about ripping apart my car and disassembling it.
I didn't really want to do that, so what I did instead was what I always do:
I go to eBay and I bought an extra ECU.
Here it is, maybe you can show it?
You can go here after the talk and take a look at it.
This is the ECU. This here is the main CPU that also includes the flash.
On the other side there are the power drivers that drive the actual stuff in the car.
And then there's other watchdog circuits and so on.
Okay, thank you.
So, the ECU was built by Bosch, it's an EDC17C46,
that's the name of the hardware.
And it can easily be obtained on eBay, and you can put it on your desk,
you apply 12 volt to it and then it boots.
It will complain about a lot of sensors being missing and so on
but you can see it executing code.
And it doesn't have the very same firmware as my car, but it's very close.
The flash chip is unfortunately in the same pakage as the main CPU,
which is an Infineon TriCore chip,
which is apparantly only used in automotive equipment,
or at least I'm only aware of it being used there.
And I was able to dump the flash by attacking the hardware
and exploiting a bug in the hardware that I haven't found documented anywhere,
but it was not that complicated.
And then I had a firmware dump, I had a 2 megabit binary,
and I throw it in a disassembler.
And what we see is interesting because the code is written very different
from other code that we know.
So usually, code has a lot of flow control
and usually more or less resembles spaghetti code.
This was the exact opposite.
It's more like someone took electrical schematics and put them into code.
There's a set of input signals, there's a set of processing on it,
and there's a set of output signals.
That gets updated every 10 ms or once per rotation depending on processoids.
Really interesting way of writing software and building this.
Also it's very data-driven, so a large part of the firmware is not code but is data.
All of the computations, they don't use constants at all,
they always refer to something from the data section.
As Daniel said, Bosch writes this code, the code is not directly visible to Volkswagen,
but they have visibility into this data, and they know what the data does.
They have tools to change the data.
Volkswagen and other companies can customize this,
really they cannot just customize it,
they can change the whole behaviour of this ECU
by changing just the data, not the code.
The ECU really is a small embedded machine in your car that takes care of the engine,
it's an Engine Electronic Control Unit, there are multiple names for it.
The most important thing that it does is that it takes sensor input,
for example the throttle, and then it applies control to the system.
For example it calculates the amount of fuel to inject, the amount of air to inject
to make the motor running at the speed you want it to run.
These days it's much more complicated.
One important thing the ECU does these days is emission control.
This is why we would expect to find the "cheat code", the code that cheats
that Volkswagen used to cheat in the whole thing,
we would expect to find it in the ECU.
Now taking a look at two megabyte firmware binaries
that doesn't have any visible strings in it,
it's kind of painful if you're just suscepting a code analysis.
So what I did was to do realtime logging.
You can actually read data from your ECU by plugging into this OBD-II port
which is next to your steering wheel.
And while the engine is running you can read out certain data.
Usually you can read out boring data like RPM, and speed,
and some things that the vendor wants you to see.
But there's also a mode that's a little bit hidden,
but you can get pretty easily into it,
where you can read by address, where you can just read the whole memory.
Well, not everything. Some security data is locked out.
But the data we are interested in, we can read that memory.
Now we still need to understand where the interesting stuff is.
We can disassemble the firmware, and that's all fine.
We can also get a little help from something called "A2L files".
The chip tuners use them extensively when they change the mappings,
they want to optimize an engine for a different goal,
for example for more power instead of long lifetime, or something.
They change things in the ECU firmware.
They do reverse engineer a lot, but they also got these files.
And I'm not sure how they got them, but they are out there.
And if you use the right Google terms you will find them.
They are specific to each firmware.
I wasn't able to find one for my actual firmware
but I was able to find one for firmware that is close to mine.
And if you look into this file, what you see is the symbol names,
it's basically a fancy map file.
You see the symbol names, you see a mostly German description of that symbol,
you see a real-use unit, and you see the adress in memory that we can read at.
So with the help of these files we can read out almost any internal state in the ECU.
We still have to make sense out of that, but at least we know where the data is
and what to look for.
It's surprising how complex an ECU is. For example, this thing, what does it display?
Everybody would say it's a function of RPM, it shows you how fast the engine is running.
Well, it's not quite the case, and if we look careful we see that
this code is post-processing the RPM signal.
It's 12 kilobyte of densely written code that has a lot of internal state
that tries to make the RPM value,
convert it to something that the customer wants to see.
For example, you want your idle speed to be stuck at 780, you don't want it to oscillate.
But in reality it does, and this code takes away all of that
and makes it flat 780.
You realize probably at this point that there is a lot of cheating that could go on here
without most people noticing.
You don't really believe that the speedometer in your car displays your actual speed, right?
It displays something related to speed ...
But let's get back to topic.
Selective Catalytic Reduction is the process of, well, if you don't have it
you get a lot of NOx, of nitrogen oxides at the end of the exhaust.
That's bad, you don't want that.
There is one way of getting rid of this, is to add an SCR catalyst.
And the SCR catalyst — I simplified this a lot,
you can find a lot more information about this —
SCR is a process that reduces the NOx using something called DEF,
or AdBlue is a term for it. It's some fluid that you put in there.
Basically it's an Urea/water solution.
And the AdBlue, at a high temperature, converts to Ammonia
and then it reacts with the NOx to nitrogen and water.
Which is great because that's not in any way harmful to us.
However, there's a problem here because the dosage of the AdBlue needs to be correct
and it's very hard to do.
If we dose too little of that the conversion is not perfect
and we will still get a lot of NOx at the output.
Which is better than not doing anything.
It's not perfect, but it's not more harmful than before.
However, if you put in too much of the AdBlue
what you get at the output is ammonia, and you really don't want that.
So the primary goal of emission control is, if you have the SCR system,
is to eliminate as much as possible of the NOx
and minimize the amount of ammonia that comes out of the exhaust pipe.
Ammonia is NH3.
Calculating the right dosage works with a model again.
They modeled everything that happens in the exhaust process,
they have a model of the catalyst, they have a model of the internal state,
they do have a number of sensors and outputs from the other models
that tell them a lot of values.
And the model uses this with a lot of internal storage, internal state.
And the model then calculates the amount of AdBlue to dose
to convert as much NOx as possible without leaking any ammonia.
The way things usually work in an ECU is, there's one system that controls things
and there's another system that monitors things.
It's independent from the main system, it tries to be as independent as possible.
It's still running on the same hardware but it's not sharing a lot of code.
There is an efficiency monitoring scheme that, if the conversion is not good enough anymore,
it will flag this as an OBD-II error
and you will see your "check engine" light going on,
and then you go to the shop, and the shop will diagnose your car and will fix this,
for example if your catalyst is broken.
Based on the test results we would have expected this efficiency monitoring
to actually flag the inefficiencies. But it didn't.
It turns out the main model doesn't always work.
There are some operating conditions where the main model is not sufficient,
it has certain bounds where it works, and outside of these conditions —
for example if the engine is too hot or if the exhaust mass is too large —
the model doesn't produce meaningful results.
It may overdose the AdBlue, and we don't want that.
There's an alternative model which is much, much simpler,
and takes only a few sensory inputs,
and doesn't rely on as many variables to be perfect.
It will still calculate an AdBlue dosage.
However, the main goal of this alternative model is to make the exhaust processing work
in all situations without ever overdosing the NH3.
They're calculating both of these models and then they are selecting one of the models.
The output of the selection then controls the AdBlue dosage,
the pump that injects the AdBlue into the exhaust.
There's code that controls which of the models to use.
There's also a statistics model that counts how often each mode is selected.
Again, all of this model selection depends on the data.
It's code that does the selection but it depends on a lot of data,
there are parameters tought of this.
Let's take a look at the selection criteria for this alternative model.
We see that a lot of these parameters are dummy variables,
things that can never happen.
For example, the athmospheric pressure can't be negative, that can never happen.
Or the air temperature ... I hope it's never larger than that,
or smaller than 0.1K, right?
However, one thing stuck out,
and that was a check if the engine condition is larger than negative temperature.
Which does not exist, the temperature is always positive.
That last one is always true,
so the model that would be selected would always be the alternative model.
That sounded weird and I was looking at the firmware.
Maybe I understood it incorrectly, or maybe I looked at the wrong place
when looking at these parameters?
But if we look at the intermediate results there is a bit at a certain location
that tells us which model was selected, and that bit is indeed always set.
That is weird, it sounds fishy.
Let's take a look at the statistics, the car counts what model you're in.
20% of the cases my car does not do dosing at all.
So I drove some time and then looked at the values.
And the 20% where it doesn't do anything is mostly the warm-up cycle.
But everytime it does something, it's actually the alternative model
which we know does underdose NH3 because it doesn't want to leak ammonia.
And that makes sense because my car uses much less than expected of the AdBlue.
The expected value is roughly 2.5 liters per 1000 kilometers, of the AdBlue.
In my case it only used 0.6 liters per 1000 kilometers.
Which is great for me because I don't have to refill this tank very often.
In fact, I never had to do it, the shop always does it when I'm there.
But this is fishy, and let's take a look at this.
What we also see is that sometimes the regular model is active,
so there must be something more.
If we look at the selection criteria we find that there's an additional term there
that I haven't found before.
There's an additional condition that has to be true
in order to go to the alternative model that underdoses.
We look at the particular conditions and we find a lot of stuff
that is related to diagnostics, things they can do in the shop.
So that's definitely not happening on the street.
But one of the criteria, that really was weird
because it looks if the engine and fuel temperature is larger than 50°C,
it looks at the athmospheric pressure and if it's lower than 750m,
that must be satisfied.
If all of these conditions are satisfied it will move back to the main model
that does the proper exhaust processing. And one thing was really weird.
There were seven curves, not all of them used,
that define an upper and a lower bound
on the distance driven after a certain amount of time.
This is how it looks in disassembly. I'm not sure if you can read this.
But the comments are from this A2L file and they call it "acoustic function".
I'm not sure if this has anything to do with acoustics.
I tried to find all the usages, and there was nothing related to sound or anything.
I think it's just a name for it.
Now if we go and take a look at these upper and lower bounds, we see this:
These are three curves that are defined, each of them has an upper and a lower bound.
It's basically the distance
that you need to have driven after a certain amount of time.
And if you ever fall out of one of these curves we're switching back to the alternative model
that underdoses NH3 and causes the inefficiencies.
This is weird, and I didn't really know what this is.
Let's get back to something completely different, which is the NEDC.
We've seen this slide before, the NEDC mandates you how to drive.
One thing is also interesting: It mandates you that ...
You want this test at "cold-start", and what's better for a cold start
than heating the car to 20°C and keep it that warm until you start.
That's the "cold-start", that's the cold start as defined in the law: 20°C.
This is speed over time, so to get distance over time we need to integrate this.
And we get this graph.
And if we overlay what we found in the firmware we get this.
*audience laughs and applauds*
What we can see here is that if you drive the driving cycle correctly
you will exactly be in the bounds of one of these curves.
And you can do this on the street, you can do this everywhere.
As long as you satisfy the distance over time and your car is warm enough
it will detect this in some way.
Well, you can drive this on a street, but it's really dangerous
because you have to follow a given speed pattern.
So i did this on a dyno, I put my laptop in there,
I logged the data in real-time and then displayed it.
Basically, this is what it looks like. In the middle you see a bar.
You have to drive and keep this middle bar in the middle,
which means you are well within this upper and lower bound, and not try to escape it.
And as long as you do, one of the other green boxes will tell you
that the car is still detecting this as being in this cycle.
Then what I did in the end, I stayed in the cycle for a while and I logged all the data.
At the end I would just hit a constant speed
which would eventually get me out of the conditions.
This is the log that I made.
On the first graph you see the vehicle speed,
you see how I tried to follow the NEDC more or less successfully.
On the second graph you see the distance over time,
you see that I stay within the bounds enforced by the firmware.
You an also see on the third graph—
this is the actual signal at the AdBlue pump—
that it actually doses quite a lot of AdBlue.
It calculates the amount of AdBlue to dose based on the model output
which you see in graph 5 and 6.
By the way, graph 4 is the actual NOx emitted by the engine based on their model.
That's the RML, their mission model then calculates the amout of the dosing to happen.
As we see, as long as we stay within the limits enforced that match the NEDC
everthing is good and a lot of AdBlue is dosed.
And then, in the end, I drove too fast.
And you can see in the second graph that I crossed the upper bar,
the blue line goes over the red line, right?
You can see that the car immediately detects this,
that I'm no longer in the driving cycle.
The interesting part you see here is the effect on the AdBlue dosing, which is here.
It immediately stops doing the dosing. And you can see in the model below
the model still calculates that AdBlue should be dosed.
But after they have the max, after they switch the model
and switch to the alternative model, the alternative model just outputs zeroes,
it doesn't dose anything.
This shows that when we're following the cycle
everything is fine, enough Urea is dosed,
and then once we leave the cycle, there's a severe reduction in the dosing.
And it's all based on detecting this driving cycle.
Two more slides. A: Two more slides.
F: Two more slides. A: Two more slides, here we go!
*audience laughs and applauds*
I have to be clear on the limitations here.
All of this was looking at disassembled code and so on,
I could have done something wrong here, so take this with a grain of salt.
We couldn't do NOx measurements on the dyno, unfortunately.
And I have to stress: We looked at one particular car that uses SCR processing,
not all of the affected cars are doing this,
there are some other mechanisms in the other cars.
And I looked at a car for the German market,
at least the curves have to be different for the other markets.
Let's reenumerate the results— and this is my last slide.
Most of the time, on a regular car, a nonstandard treatment mode is active
that is not as efficient as the real mode that is implemented.
We can show the code that is responsible for this:
This is this negative temperature limit that they look at
which doesn't make any sense and always selects the alternative mode.
And we can see, in the logs, the state selection bit,
we can see the counters that count that the alternative model is active.
We can see that there's an AdBlue underdosing in this state
which causes the inefficient NOx conversions,
that's what we've seen before when people put the car on the dyno.
We know that the efficiency checks are only enabled in the main mode
and the car does exceed the limits.
This shows how the alternate model is selected where it doses too little AdBlue
and causes the inefficient conversion.
We can see that if we follow the driving cylce,
the minimum temperature and the distance over time,
we will see that it switches to the main model
that should have been active all the time.
We can show the code that's responsible for that,
the driving cycle detection that uses the upper bound and the lower bound.
We can extract the exact limits, overlay the NEDC data and see that there's a match.
We can, if we do this actually on a dyno, we can see how it switches the SCR state.
We can show the effect on the DEF dosing, on the AdBlue dosing.
As you've seen on the slide before, as soon as we switch out of the driving cycle
into the street mode, the dosing will get close to zero.
Once you're back in the main model all the efficiency checks are enabled,
for example to take better Urea.
So the efficiency checks are there, but they are not active
because the car is forced to run in the alternative model.
These results are all in line with the Volkswagen press releases.
These are basically just the details as extracted from the firmware
to show you the background.
Thank you.
*audience applauds*
A: Wow! Thank you very much, Daniel and Felix.
*audience applauds*
I'm really sorry, but we have to clear the stage.
There is not going to be time for the Q&A session.
Do that down there. I'm sure that a few people just come down,
grab you and ask questions. Unfortunately, we can't do that.
I have to close it in exactly four seconds over here because we have to go off the stream.
Thank you very much Felix, thank you very much Daniel.
F: Thank you.
♪ *postroll music* ♪