*33c3 intro music*
Herald: Let me present Felix Domke with
the Software Defined Emissions: A Hacker's Review of Dieselgate.
*applause*
Felix Domke: Yeah, hey everyone. Thank you
for coming here. I saw there are a lot of interesting talks at the same time in the
other rooms, so thank you for coming here and listening to me about software-defined
emissions. "A Hacker's Review of Dieselgate" is the subtitle. I'm Felix
Domke. I usually do embedded software, mainly security. I'm definitely not on
cars and definitely not on things that have combustion thingies, so I only got
dragged into car software last year when my own Volkswagen car was accused of
cheating and I wanted to know what exactly was going on. I held the talk last year
about some of the details of the Volkswagen Sharan defeat device. For the
details you can take a look at that talk. This time I want to look more at the
process of finding or analyzing car software. I want to look at whether this
process scales to more cars. The first step when having a piece of software that
does not always do what people think it does is, well obtain a firmware image,
obtain a binary image of the firmware, and in the case of my car I knew it was a
Bosch EDC17, which is a Bosch ECU that a lot of cars use, including my Volkswagen
car. So, I didn't know anything about ECUs, dumping software and so on, so I
asked Google "Hey, what do I need to do to dump an EDC 17," and Google had a lot of
answers for this, but usually those were people that wanted to sell me some device.
Those were chip tuners that built their own devices where you can plug in the ECU
and then it extracts the image, usually by exploiting some bugs in the software. But
I didn't really want to buy something and it takes like a lot of time until I get in
my hands. I wanted to start. So I was looking to do this on my own. What these
sites usually tell you without paying is how you wire up your given ECU for their
device, so they tell you where to connect 12 volt, where to connect the CAN bus,
which is the serial communication bus that the ECU uses to communicate with the rest
of the car devices. Usually it's pretty easy, so...
when analyzing ECU it makes a lot of sense to reproduce the scenario on your desk and
not in your car, so in order to make an ECU boot all you need is ground, 12 volts,
there's usually an ignition pin that you also have to supply 12 volt to it, and
then it boots. So on my desk it looked something like this, and then once we have
the setup we can boot the ECU, we can use Python to talk to the ECU, which is great,
and then we can use socket CAN, which is the Linux CAN support that's really great,
and we can even use MicroPython if we want to have a smaller device that we can
put in a car. So we can talk with the ECU. Talking with the ECU in modern cars,
there's a protocol called UDS. Basically, I simplified this slightly, you can ask
the ECU "Hey, I want to read memory by address," you give it an address and you
ask it to read four bytes in this case of that address, and then it returns it to
you. So I thought "Hey, maybe I can use this to dump the software." In my case the
device responds with a Security Access Denied, so I looked into what I need to
do. You actually have to do a Security Access command. You send a command that's
called Request Seed. You get back basically a 32 bit random number and then
what you have to do is to, have to process this seed through a super-secret
function and then return it in a response call. The question is how do we know this
super secret function. There are multiple methods. We can look at the ECU software
itself, if the algorithms in there, to verify it. We can reverse diagnostic
software that uses this mechanism. So for example the Volkswagen software they use
for car shops, or maybe someone else already reversed this and put it in their
own tools which may be easier to get it from in terms of third-party diagnostic
software. And in the case of my Bosch ECU, the super secret function was this. I
basically had to add this number to it. The mechanism is called pin code.
It's in... I mean it's not super secret. Anyway, once I know this...
*applause* Yeah, thank you. I mean, it... once you
do this, you send back the result and hey, then you can read... you can send the read
command again and, hey you're getting back data, so this is great, right? We can read
memory at runtime of the ECU, and we can even do this while the car is operating.
However it turns out that for the Bosch ECU you can only dump specific regions. You
can dump most of memory, some memory areas are excluded, but most of the interesting
stuff you can read. But you can't read any code. You cannot read anything in flash.
But we are hackers, of course, so we find a way. The CPU used in these ECUs is a
Infineon TriCore CPU, and it's used, at least this particular one, was used in the
ECU I cared about, and the security model for this chip is that you can always enter
a specific bootloader mode and execute your own code, so you can strap a few
lines they... the chip tuners tell you that, right, they tell you what's high and
low, which pins you have to connect your ground and 3.3 volt, and then it enters
this bootloader mode, you can upload some piece of code. However, you can't read the
flash, because the flash is locked. When you start in bootloader mode, the flash is
not readable until you write a specific password to a register. That was not so
great, so I looked into what else I could do. The datasheet is very specific on how
to operate this chip. For example there's this one flash supply pin. So, even though
the flash is in the same package as the rest of the CPU, it has a dedicated supply
pin and it tells you which parameters not to exceed to ensure correct operation, but
I really don't want the correct operation which is in this case preventing me from
dumping the flash. So, what can we do? We can violate the requirements. The
requirements for 3.3 volts. Let's see what happens outside of that range. And turns
out, down to a certain voltage level, roughly 1.6 volt, everything just works as
normal. That doesn't help us. And below that voltage the device hangs in
the bootloader, so that doesn't help us either. The interesting parts happened
when you are at the very specific voltage level, and this is a little bit
unscientific, because it's really just the voltage level I tried, and then most of
the time the device comes up and flash is protected, and then the remaining times
the device comes up and hangs in the bootloader. But one in 10 times something
interesting happened. The device came up and the flash was not protected, so I
could dump it out. *applause*
So having the image now in my hands, I could start in actually reversing the
defeat device and what I found was - I don't want to duplicate a lot of what I
talked about in the last talk - I found a function called "acoustic function," or
"Akustikfunktion" in German. It's a function that senses vehicle speed, the
duration of the engine operation, and some other things, and then controls emission
related functionality, or in short you can say that this is the test cycle detection
that enables the defeat device. And I verified it to exist on my Sharan device
by driving through the test cycle and logging data. And during the last year I
verified that it's actually the same defeat device, more or less, that exists
on a lot of other Volkswagen cars. All these Volkswagen defeat devices that we
talked about for the Euro 5 cars, they use more or less the same acoustic function.
Basically, to remind you, there are a few curves stored in the software that look
like this. This is the NEDC. This is the test cycle you have to drive a car
through. They exactly define how fast you have to drive for a given time in seconds,
so it's speed over time. If we draw this as distance over time it looks like this.
So this is the distance you got. You're not really moving the car, because you're
doing this in a lab on a dynamometer, but what the car thinks it has moved to, and
if we overlay this with the curves we found in a software there's a perfect
match. So this is the way how they describe the test cycle. So this was for
my Sharan. So I looked into, what do the other cars do, especially what do the
cars in North America do, because they're not using the NEDC. And I found something
interesting, or some someone sent an interesting document to me, that was this.
It was an emission service action. It basically describes how there was a recall
for some vehicles, that required a software update in the shop.
So this is basically the document that informs the car shop what they have to do,
and it had something very interesting in it. By the way, this was in December 2014,
so this was way before the whole Dieselgate was public, but this was
already while the EPA was already talking with Volkswagen, already demanding
explanations. All that investigation was already proceeding. Volkswagen knew about
this, that people figured out about the defeat device, and it had something very
interesting in it, that said "in addition, the vehicle's engine management software
had been improved to ensure the vehicle's tailpipe emissions are optimized and
operating efficiently." That sounds really fishy to me, so I was curious, what
exactly did they change in the software update? And luckily they tell you the old
and the new software versions, and you can then go and look them up on a firmware
DVD, that you can download on the Volkswagen website, and it turned out that
it's an ECU software similar to the the Bosch ECU software I looked at before. So
there's an acoustic function again there, and the curve stored there, they match the
US test cycles. This is one of them. There are many more test cycles in the US, so
there's another curve that matches this, and this is the curve stored in the
software and this is the corresponding test cycle. And there are a lot of them.
But I noticed something really interesting, and some of the curves...
they were much wider open than the other ones. So for example this one... there's
really a nonzero probability that if you just have your morning commute, through, I
don't know, some streets or something, that you accidentally match this driving
cycle every time you start driving in the morning, so the car would, every time you
drive this, think it is in test cycle mode, and would operate with the
optimized... in the optimized emission mode and apparently this caused problems
and what I saw, what Volkswagen added in the software that was part of this recall,
was this function. So, this is from it from a disassembly. In pseudocode, this is
this. So they started looking at the steering wheel angle, and if they figured
out that you move the steering wheel angle, then they ignored the curves, the
more open curves. So the idea is "Yeah, if you move the steering wheel, you're
definitely not in a test cycle, so at that point we do not try to operate in this
emission optimized mode. And it's a little bit of speculation, but it matches
up pretty well with with all the facts that I read, is that because those cars
operated in the test cycle mode too often, that eventually caused the the particulate
filters to clog, and their solution for Volkswagen, and again, this was while they
were already investigated by the EPA, was to add the steering wheel angle detection.
For more details, I worked with this with the NDR, and they produced a feature on
that, so there are some more details. So this is Volkswagen, but there are more
cars, and if we look at this... this is a meta-study based on something that the the
Ministry of Transport... they tested a lot of diesel cars and what they found was
this. This is actually a representation by the ICCT. So the orange line is the
emission limit, and the bars have an upper and lower end, and the lower end is how
much emissions the cars have. This is just for nitrogen oxides, for NOx emissions,
what they had in the lab, when you're driving the test cycle. So and you can see
all of these cars managed to stay under the orange line, so they get their
certification, but when driving them on a real street, they produce the emissions
corresponding to the upper end of that bar, which is for some cars significantly
higher. It's off by a factor of 10 and more. When you're driving the car on a
street. And this is interesting, because the cars, they can meet the emission
goals. The question is why don't they always meet the emission goals?
Why do they operate so differently in the test cycle than on the street? And I try
to give you a partial answer. And let's look at how a car can optimize emissions.
The first thing they do... so this is a very simplified diesel engine. So fresh
air goes in, fuel goes out, and there's an exhaust pipe, right? And a lot of nitrogen
oxide, a lot of NOx, goes out as well, and we don't want that. So we added an EGR
valve, which is basically a valve that causes a part... a fraction of the
outgoing air to recirculate again through the engine and burn again. And what this
causes is that the flame temperature goes down, and if we look at the relationship,
it's very simplified here, but with a lower flame temperature you get fewer NOx
concentrations, so you improve emissions by lowering the flame temperature, however
at the same time you're increasing the soot level, or the particulate matter, and
there is this trade-off - if you do too much of EGR, too much of the exhaust gas
recirculation, you're getting too much soot, and the other hand if you do it too
few, you get too much NOx, so you can argue that the green area isn't really
great, because there's no point where both of them are great. And here we see the
result of a clogged EGR valve. If there's too much soot it will clog. EGR, as the
conclusion, is the least cost solution. It doesn't really work at higher loads. It
works at low loads, and it does not require exhaust... high exhaust
temperatures, which is great, but excessive use of that clogs particulate
filters, affects the combustion, the drivability goes down, and there are
trade-offs with this. It's also not very useful for higher engine loads, for
example when you're accelerating you have to disable EGR at high speeds. So a better
method, that was added on top of this, is called "selective catalytic reduction". I
am... so, basically the idea is you have an SCR catalyst in your exhaust pipe...
there are more catalysts there, but let's talk about NOx, and in there, this
happens. We can simplify this, somehow, and say if you put ammonia into this
catalyst, the NOx is converted to nitrogen and water.
And Nitrogen and water is great, it's harmless. It's already part of the air.
The only issue is ammonia is this and this is not something you want the driver to
refill in your car. So instead this solution is we can create ammonia in the
car from using from something that's less dangerous, and we have the reaction there.
We can simplify this again and say we take urea - Harnstoff auf Deutsch - and heat,
and we create ammonia. Urea or urea solution is this. It's called AdBlue or
DEF - diesel exhaust fluid - it's not dangerous. You can buy it, you can
transport it... it's relatively cheap. The idea is, we have this reaction, it
requires ammonia in the catalyst, and we put AdBlue into it, or urea, and using the
heat that we have from the exhaust pipe, we create the ammonia that we need to
reduce the nitrogen oxides back to nitrogen and water. There's a great
property of this, that some of the ammonia that's produced in the catalyst stays
there until it's used up, so there's some storage there. So the the
requirement for creating ammonia is heat, and if you don't have heat, but for
example because you just started up your engine... if there is still ammonia from
the last usage in your catalyst you can still use that, and use that up, and by
the time you have used it up, maybe the heat is enough to supply more AdBlue and
then fill up that storage. The downside is you need a pump to dose the AdBlue, and
you need lots of software to control this process. And you need a heater because the
AdBlue freezes at some point, and it's an expensive solution, it adds roughly $500
to a car, which can be significant amount of money for a small car, and it requires
a large AdBlue tank for long service intervals, so you don't have to refill it
every few thousand kilometers or something. The great thing about SCR is
that it's efficient at higher loads. There's a third method called LNT, Lean
NOx Trap, it's cheaper than SCR for smaller engine, it doesn't require
anything, however the bad thing is it requires
frequent re-generation, which decreases fuel efficiency, so it's kind of a stop-
gap solution. And it's not efficient for continuous high engine load, for example
if you're driving on the German Autobahn at full speed, then LNT is not going to
help you much. For the sake of this talk, let's keep in mind that EGR is exhaust gas
recirculation, that's the thing that operates within the engine and then we
have the SCR, the selective catalytic reduction, that uses AdBlue and is after
the engine. We also saw that all these technologies have significant trade-offs
for NOx compliance, so we can kind of see the motivation for a defeat device here,
because it would be the solution to all of these trade-offs. You get no downsides
during regular driving, because nobody can measure your emissions and while
maintaining conformance because during a test cycle you have perfect emissions.
That kind of explains why there are defeat devices. OK, let's get back to the bigger
picture and see what other cars do. So this is an Opel car, it's a Zafira car,
it's a Euro 6 car, it's a pretty modern car, it has an SCR catalyst. In theory it
should have really great, low emissions, especially at higher speeds because that's
where SCR is good at. But quite surprisingly it doesn't. If we look again
at this report, we can see that this Zafira exceeds the limit by up to 12x
compared to the Euro 6 limit. This is especially interesting because there's
this Opel advertisement where they advertise their diesel technology applying
to the Insignia and the Zafira and they say a lot of diesel fun without regrets,
the new diesel generation of Opel achieves best emission values and gasoline levels.
Yeah, after they got sued for this they had to change it slightly and they had to
add this to the sentence. *scattered laughter*
So, during this testing, they had this 12x emission
limits. For example, one particular test was to drive the test cycle in the same
way, but at a different temperature, at 10°C,
and the car exceeded the values by a factor of 6 even though the
car would be operated in the very same way. It was just that the ambient air
temperature was 10°C instead of 25. So they asked Opel why this was the
case during their investigation, and Opel responded, saying that the EGR and the SCR
injection, they work to the full extent in the temperature range of 20 to 30°C.
It's what they call "normal use". So our question was, is it really just the
temperature window? So we got a car and investigated. The ECU in that car is a
General Motors ECU. It's developed in- house, Opel is a GM daughter. It uses an
automotive PowerPC, yay PowerPC! It uses somewhat obscure variable length
instruction extension to PowerPC. So how do we start? Again, we need a firmware
image. So let's ask the Internet, "How do I dump this ECU?" Luckily, someone in some
chip tuning forum already uploaded their stock ECU, which means they're what they
dumped from their ECU using some chip tuning tool. It's not the same ECU, but
it's very similar and I hope they shared some code so I can analyze the software,
maybe find a way to dump it. So the dump was made with this tool and the tool did
not let me export the binary. And the tool is free, however to use any of the
features in that software, you have to buy their expensive hardware, that then
connects to the car, which I didn't want. This is how their software looks like. I
loaded the image I found on the internet, and I couldn't save it or anything without
having the device attached. However, I can just use a memory debugger and just dump
it from the address space, and that gave me a first firmware image to start with. I
threw it in a disassembler and I found the UDS function and, yeah, it implements read
memory by address, so that's good. Most of the RAM was readable without a security
challenge. That is good, so I didn't even need a security challenge to read RAM.
However, the flash, it is readable, but only with the security challenge. So let's
take a look at the security challenge. Maybe it's as simple as the Bosch one. So
their way of doing this is, they store 16-bit input and output value in the
firmware, and it's different for every device, and they don't store the algorithm
to compute the output from the input, instead they just store the pair, and,
well, it's just 16 bit, right, so let's brute force it. The issue is, you can only
try every 15 seconds, so it's kind of lame. The question is, how do the GM
tools, the original factory tools, get access to that? Luckily, chip tuners had
reversed that and then obfuscated it into their own tools. But that can be de-
obfuscated and eventually it's just a little bit amount of bit shifting and so
that was easy to fix. And also the the GM repair manuals tell you how to wire up the
ECU. They tell you where to put 12 volts, the CAN bus again, and ground, and the
ignition pin, and with all of that in place, I can do the security challenge. I
can now read all of flash memory and read the four megabytes of PowerPC code, which
mostly consists of mathematical functions. There are no strings or anything, it's
really hard to find what a function does, what... There are thousands of variables.
It's really hard to find what they mean, right, so I need to know some entry
points, some known data values, and then I could refer to... one thing I could find
are real-world constants, for example there's the density of diesel fuel stored,
which allows me to understand that this is something related to fuel, an amount of
fuel. Or more useful are the OBD2 calls. So there are some standardized things you
can ask an ECU, that's engine rpm, vehicle speed, and things like that, and I would
find that table in the firmware and then I had a first start of things like RPM,
speed, and so on. That was a good start. It's not much more than what you can see
here. So there's a lot of stuff not included in these.
The next thing I did was, I drove the car
for a few weeks and I let a device attached that would constantly log all
memory using the rebuy address thing, and every few minutes I would get one memory
dump, basically. It's a few hundred kilobytes of RAM, and then I put this into
my disassembly, and that allowed me to understand more of what individual
variables do there. And I found some interesting things. So the first thing,
one of the first things I found, was basically something that resembles this.
So there was something that look at the ambient temperature, and this basically
checks for range, right, and it did... this was for controlling the SCR systems,
and it's interesting to know that the NEDC requires the temperature between 20
and 30°C, and this is right centered around this, when they check from
17.5 to 33°C. But this was of course nothing new. I found something
similar, however, another temperature check. And as you can see it's written in
a different way. It effectively achieves the same thing, but it's a separate piece
of code, and this time it was for the EGR system. So we have these two exhaust
treatment or optimization mechanisms there, EGR and SCR, and they don't share
code. They have their own temperature window. So we found the temperature
window, which was known to exist. The question was, is there more? And one thing
we found was this: It's basically reading the vehicle speed and comparing to a fixed
number, and it turned out it's something like this. So it would check the
vehicle speed and if it's above 145 km/h it would set a flag and then under 140 it
would clear. Keep in mind that the NEDC maximum speed
120 km/h, so during a test cycle that would never happen. So let's see if this,
what we found in software, if this really translate something the car does in the
real world, and it's getting slightly technical here, I apologize, but we need
to log some variables and a useful value to to know is, how much NOx is there after
the engine, and after the SCR catalyst, and luckily there are 2 NOx sensors in the
car. One before and one after the catalyst, and they give you basically the
NOx concentration in ppm. So we log that, and we also log the signal of how much
AdBlue is dosed into the system, and we log the catalyst temperature. And one
thing to keep in mind is that there's also this amount of ammonia that's stored in
the catalyst. We don't have this as a value, but just keep this in mind. And
this is how we've driven the car. The blue line is the vehicle speed. You can
see that it goes from 0 to 150 km/h, and the critical point here is the 145 km/h
that we found in the firmware. The green one is the catalyst temperature, which we
see between ambient level, and then up to 380 degrees. The critical point here is
200 degrees Celsius, where this urea to ammonia process starts to work. We logged
something that is the SCR strategy. So it turns out there are multiple ways how the
ECU computes how much AdBlue to dose and I call them strategy. So 0 means off, no
AdBlue is dosed. 1 means the regular way that keeps into account the storage
mechanism, and then 2 is a special reduced way. And then also we log the actual
dosing value. And then we also had the sensor data from between the engine and
the catalyst, and between the catalyst and the exhaust. The first thing that
happens... or, actually nothing happens until the point where we reach 200°C
at the catalyst. You can... until that point, as I said the required
temperature is not... does not allow AdBlue dosing, and then it starts dosing
quite a few amount of AdBlue. But then, when we cross the 145 km/h, the SCR
strategy changes, and no further AdBlue is dosed until basically this point, and this
point is exactly 120 seconds after we go lower than 140 km/h. So this matches what
we found in the software, right, this was what we
found in the software. So we can see that this was
actually true. This is real behavior of the ECU. And to look at the effect of
this, you have to check the difference between the blue and the green line in the
lower diagram, between basically the amount of NOx that is removed by the SCR
catalyst. And you can see during the regular operation quite a lot of NOx is
removed. The blue line goes up because we're driving faster and faster, and the
green line goes down almost to zero, and this this works for quite a while, and it
even keeps working a while until the catalyst runs out of ammonia, and then it
would need more AdBlue to operate, but because we're in the reduced mode it does
not put any more AdBlue into the system. So the SCR basically stops working, and
the emission levels reach the engine emissions, so no further... The SCR system
does not work in this red area. And here we see this again, so here are the
sequences of active dosing. Here we see where the catalyst temperature is too low
for dosing. We see the regular operation. We see where it's still working,
because there's still ammonia stored and then until we run out of ammonia and no
refill happens, until exactly 120 seconds after going below 140 kilometers an hour.
So our conclusion after this is that the SCR is programmed to stop working at 145
km/h. The efficiency goes to 0. Opel offered a hand-waving explanation, one of
the press releases why this is necessary. They argued with some physical details,
and we presented these physical details to some experts, professors that work with
combustion engines for a long time. Most of them disagree with these explanations,
I mean, yeah. But more importantly, other cars including my Volkswagen Sharan
device, which is Euro 5 car, so one generation older, and it's known to have a
defeat device, and it performs significantly better than this car. OK, we
continue to look. We found something here that is a... that looks like this. It
takes a... there's a barometric pressure sensor that sends us
the pressure of the air and compares it with a value and if we look at how
pressure is related to height we see that what they check with is 91.5 kilopascal
and which corresponds to 850 meter. And apparently Europe's highest test center's
at 800 meter, which may be a coincidence or not. But above that point they reduce
their CR system as well. Now, the interesting thing is... yeah barometric
pressure is something very important to know for an ECU. There's a good reason to
have the sensor for all of the combustion process. You need to know how much air
there is. So for EGR it makes a lot of sense to have this, but for SCR, which is
the system after the engine, no combustion is happening. We are not aware of any
effect that the outside air pressure has on the SCR system, and also other SC
equipped cars don't have this mechanism, so... for us it does not make
physically... it doesn't seem to be physically required. So far we looked at
SCR. Let's look at EGR. What we saw was, when we drove the car during the test
cycle, so we put it in the lab and drive the
test cycle, we consistently saw much higher EGR values, much higher than
compared to driving on the street, compared to all kind of scenarios that we
drove on the street. So higher EGR value here means, that the EGR valve, that I
showed you earlier, is more open, more exhaust gas recirculates to the engine. It
causes lower NOx emissions before the SCR catalyst, and we really... we're curious
why did the car behave so differently when running on a street than running in a test
cycle. And we already took into account temperature, so the temperature was not
the issue anymore. And thankfully the car, when it computes the reason for reducing
EGR, it stores a reason in some variable that we can log, and it looks like this.
There is a number of things that can happen, that causes the ECU to switch to
some low EGR mode, and a few of them make sense, for example if something is
broken, fault flags are set, or if the, I don't know, the coolant temperature is out
of range, it makes sense to just keep the device running at all cost. But when none
of these reasons apply, the value stored is 2, and 2 basically means, that the full
EGR operation is used, so it's basically the NOx-optimized mode with the fewest
emissions. And then we looked at some real-world driving, you can see this in
the background - the vehicle speed is in the background - and we saw that... the
red graph shows you the reason to go to this limited EGR mode and what we saw is
that most of the time the reason is 13 and only a few times it's 2, which means that
it's not limited. And looking into this more details, we see it sometimes drops
back to 2, to the unlimited mode, to the optimized... emission optimized mode, but
any acceleration, or almost any acceleration switches it back to 13, and
then it stays there for a long time. And 13, if we look it up, is what I call load
limit. And then, interestingly, if we run it through the NEDC, we never saw a 13. So
the engine stays in mode 2 all the time, and 16 just means that the engine is off.
But we never see 13. So this explains why the EGR values were so different in a test
cycle. So, let's look into this load limit function that we found. It's basically
defined by curves, by five curves. For every gear there's a curve, or for a
bucket of gears. It's basically that they look up RPM, they get a value for that
curve, and if you exceed that value, they switch to the reduced EGR mode. What they
compare this threshold with is the amount of fuel injected per cylinder per
revolution, but you can also say this is torque, just with a
constant factor. And then once you are outside of one of these curves, it
switches to the non-optimized mode where it emits a lot more emissions, and then
you have to go back into the green area to switch back to the optimized mode. So
let's see what this means in practice. So here we
have a car, and the traffic light is red, so the car stops, and then the traffic
light goes green and the car accelerates, and accelerates, and accelerates, gets
faster and faster, and then it's at the highest speed here, and drives for a
while. And this is a typical city cycle, this is there to... how you drive in a
city, and then the next traffic light turns red and the car brakes and stops in
front of the traffic light. Let's take a look at this again with one more variable,
the RPM. We can see that when the car starts moving, the RPM goes up. And then
at some point there is a drop in RPM, and this is because it's a manual transmission
and the driver switched to the next gear. Now it's switched to again the next gear,
and this causes the RPM to drop, but the speed to remain almost
constant, and it drives for a long time in the same gear, and then the traffic light
goes red, the driver presses the clutch, the engine goes back to idle state, there
is no connection anymore to the wheels, between the engines and the wheels, and
the car gets slower. OK, one more variable. It's the last one, I promise. It
is torque. The engine power in kilowatt or something is not just a function of RPM,
it's a function of RPM and torque. so RPM and torque together are very useful to
characterize engine behavior. And a very good way to do this is to have a graph
where we put RPM on the one axis and torque we put on the other axis, and then
we draw this in two dimensions, and so we get this, basically. This is the operating
points we go through when driving the cycle we saw. So the green dot here
indicates where we are. And so we restart the car, the car accelerates, sorry, the
car idles for a while, so the green dot stays there. It idles at around 800 RPM,
almost no torque, because there's nothing to move, and then the driver accelerates
and the torque goes up, the RPM goes up more slowly, and then at some point, the
driver presses the clutch, which disconnects the engine, the
torque goes down, the RPM adjusts to the speed of the next gear, and then the
driver releases the clutch and now the engine again has to move the car, so the
torque goes up until reaching the the highest RPM value and then that the driver
again switches to the next gear, so the whole thing repeats, and then while the
car is driving, the majority of this the cycle, the engine spends in this one
operating point. We're currently at 1800 RPM or something, and 80 Newton meter or
so torque. And then at some point the driver presses the clutch, the engine goes
back to idle and stays there, basically. So this is how you read this diagram. And
now what we found in the firmware was that overlaid basically on this representation
we see a mask, or a limit. If we go over this curve, those are the same curves that
I showed you earlier, just laid on top of this. If we go over this curve,
then we switch to the worse emission mode, we switch to the mode where the EGR value
is limited. So we can see in our driving that this happens basically at this point,
the point where the driver accelerates above a certain point, that
causes it to go over the load limit and the engine basically switches or
significantly reduces EGR. And that's fine because EGR doesn't work when you need a
lot of engine power, so it make sense that that's at that point, and what we would
think is that it switches back once we leave this load envelope, once we go below
the limit again, once we are inside the limit, we would expect the ECU to switch
back to the full EGR operation. But what we see instead is that this does not
happen, and the reason is that you don't have to go under the maximum, the load
limit, you have to go into this green area. You have to go back to idling at a
very low RPM to switch back to the full EGR mode and this only happens at the very
end. When the driving cycle is almost done, when the driver presses the clutch
and lets the engine idle. So especially this long sequence where the driver... the
car was driving at the same speed, we were technically in ... within the load limit,
where we're not exceeding the load limit, but because we previously exceeded the
load limit and it doesn't matter for how long you exceeded it, and we did not go to
the green area before, we were still in this low EGR, high emission mode, even
though we're still within the load limit imposed by the software. So let's take a
look at how often this actually happens in real-world data. So here's us driving
through a city, and we can see we constantly exceed these load limits. And
this is driving on the Autobahn, and yeah we constantly exceed those. But they look
interesting. They look as if they had been designed according to something, right,
they have the specific form and it's not just... yeah, I... I don't know... and it
turns out if you do something really strange, you can stay within these limits,
so we tried that and we managed to stay within the limit by doing something, and
we... it was reproducible, we could do this a lot of time it would always stay in
this limit and the answer is: If you drive the test cycle you're staying in this
limit. *applause*
So yeah, these curves basically defined... they closely correlate to the limits that
you need to pass the NEDC. Okay, to be clear it is fully acceptable that the EGR
rate is reduced when... for higher engine loads. It's natural, you have to do this.
For example, when you accelerate the EGR rate will decrease up to zero probably,
when you do it ... when you're running at high speeds, all of that is great. So this
method of having a load limit ... well, you can argue if really having the load
limit exactly where the NEDC is makes sense, but having a load limit is okay,
right? However, what we think is not okay is that, if you only exceeded the limit
once ... um... you would stay in this high emissions mode for potentially a long time
until you get back to low speed idle the next time. And we think that is the
problem. We ... so far this was all based on what we saw in the software, so let's
see if this translates to something that happens in reality. So to repro this we...
the car... drive at constantly... or we let it idle, then we accelerate it to
2,000 RPM, we let it drive there for a while and then we quickly exceeded the
load limit by going to 3,000 and then going back and then after doing that we
would again stay at 2,000 RPM. So it looks like this and we would naturally expect
the engine to operate in the same way on the left and on the right side because the
engine is doing the same thing there, it's the same torque level, it's the same RPM,
everything is the same. So we would expect the same emissions, right, um ... and it
turns out it isn't. And ... this is a slightly convoluted diagram. So if you
look at the green and red bars in the middle you can see what happens before and
after exceeding the limit for just once. And in the middle you can see the EGR
position, the EGR valve position, and you can see that we get pretty high values
between... 6... maybe 65 percent or something before exceeding the load limit
once. And after we exceeded it once even though
the engine again is operating in the same exact operating point, we see much lower
EGR valve positions, around 50% or something. And if we look at the bottom we
see what the engine NOx emissions and we see that they are significantly higher on
the right side than they are on the left side. So this... for me, this does not
sound like this is truly optimized for emissions because the engine is doing the
same thing, in both cases the emissions should be low. So going back to this quote
that it works, the EGR and SCR injection work to the full extent in a temperature
range of 20 to 30°C. Okay, but what about the EGR load limit and what
about the the barometric pressure limit for SCR and what about the SCR speed
limit? That would not be "to the full extent", right? And the Opel answer is
really interesting. Of course, they denied doing a test cycle detection, they say
they don't do that. And what they said is, when asked whether they lied to the KBA
when saying that it works to the full extent they said "The statement 'fully'
was really related to the NEDC test schedule, right, which... it went on and
further... the Opel CEO had to say this. He said: "The recent
accusations based on the findings of hacker Mr. Felix Domke" - hey, that's me -
"are misleading oversimplifications and misinterpretations of the complicated
interrelationships of a modern emission control system of a diesel engine.
Emission control devices are highly sophisticated integrated systems which
cannot be broken down into isolated parameters." Especially not by a hacker,
right? *applause*
That was kind of funny. There was another funny thing. Sorry, I only have a German
quote and I didn't want to translate it, but when Opel basically ... they
repeatedly say they don't have a cycle detection, right, and they say it's not a
cycle detection because, if you use the car on the street in the same way as you
would do them during the test cycle, the car would behave in the same way, so it's
not...,right? *applause* ... and ... okay. But what is with Volkswagen, right, they
have the same thing, if you drive the NEDC on the street the car will go to test mode
they have the same thing. I don't see how this does not represent a cycle detection.
That was a lot of things to say about Opel, but on the bright side, they also
said that they will - even though all that was incorrect, what we found - they said
"We will further improve the efficiency of emissions after treatment of our SCR
diesel engines and so on as far as the laws of physics allow. This includes a
voluntary service action" - and this basically means a software update for your
car - "for the cars that are already on the road starting in June." So that is
great. They're actually improving something. Question's in which year,
because this statement is from May 2016 and it's not out yet, but... Opel actually
provided a new software already in July and I think they already worked on this
for quite a while and in July 16 the German KBA, the Kraftfahrtbundesamt, the Federal
Motor Transport Authority, they are pretty nice actually, and they do know about what
they do, they are bit limited by the resources they have, and by the manpower
they have, but they know about cars and they know how to do these investigations.
I mean, they're a little bit bound, but what they should do and what they should
not do, but they asked me to review a new ECU software that was given to them by
Opel for the Zafira in question and Insignia, which had a similar ECU and I
looked at that software and I dumped the firmware and I looked at basically all the
code sequences that I looked at before and I was positively surprised because they
removed... they addressed each of our concerns. All of them, within the physical
limitations of course. So they improved the temperature window and everything, so
there was a significant improvement. They were able to improve the software and they
let the DUH, which is the German Environmental Aid, they used a PEMS system
- PEMS is a portable emissions measurement system. It's
something you put on the exhaust pipe on your car and then you can measure the
exhaust during real-world driving, and Opel gave them a car with the new ECU
software. Otherwise the car was identical to the old software, and the results are
this, right, so on the left side you see the old software, that has all these
things that we criticized, and on the right side you see the same car with a new
ECU software and it's significantly better. It's only slightly above the
limit, right, but it's much better than before and to put this in relation,
before they were on the list pretty bad - so this is sorted by worst to best - so
they are in the, well, upper half at least, and now they are almost one of the
best cars, just by switching the ECU software. And I mean this is great news,
right, they actually improved their cars. Let's just hope they get this out to the
cars soon. Let's just hope it doesn't have side effects and something, but I'm sure
Opel knows how to test for this. Going back to these, we worked on the Opel
thing... I think the Opel case, it.... once they actually upgrade the cars, and
once the cars really show these great values that the preliminary software
showed, I think we can close the Opel case, but there's a lot of other cars
still to look at, and really, I mean... the effort to do this does not scale to so
many cars, so we need to do something more fundamentally to improve the situation.
What I found out is that digital control systems, they are black boxes. The
manufacturers have designed them to be black boxes. They even boast to you that
they are 7,000 parameter in there and no hacker can understand this and it's a very
sophisticated problem. They are designed to be black box, and this is not just true
for Opel, this is true for all car manufacturers. Nobody wants anyone to look
into their ECUs, and people seem to be ok with that. Like they think "Oh this is so
complicated, there are so many German engineers working on
this problem, they must have found a great solution." So we are trusting these black
boxes and we are not able to review the black boxes that we put into our cars and
we have to trust the manufacturer to do the right thing and currently, the
investigation to do this without assistance from the manufacturer, it does
not scale. We can do it but... the manufacturers can put more security on
their ECUs... it probably can be broken, but it takes a lot more time, so it simply
does not scale sufficiently. The issue is black boxes are really powerful, right.
Black boxes can hurt people with, for example, excessive emissions. They can
kill people if we think think about autonomous cars that do mistakes. So what
we do need, I think, is more transparency. A system that can kill people needs to be
reviewable by the people. I think this is a very important thing.
*applause*
So, to have a system that can kill
people... to have it reviewable by the people, we need to do things. For example,
we need... we want access to source code for reviews. It doesn't necessarily mean
we want open source, but we don't ask at all the car manufacturers to open source
all the software. That's not what I'm talking about. What we need is... think
about how Microsoft is sharing source code of Windows with universities or other
countries. We need experts to look at the source code, and we want control software
that is reviewable by design, that has a lot of documentation, that has good
comments, that is human readable code. I don't want to see a disassembly, I want to
see the source, the MATLAB, or whatever they are using to define the functionality
source, and read that. And I want to understand why did they choose that
curve of that map in this way? What was the design criteria? That needs to be
reviewed. And we need transparency for control software decisions, which means
that if a car operates in a certain way, if I'm driving that car, I want to choose
that I can log what the car is doing, for example by putting
in, I don't know, a USB stick or something if it's my car, and then the car will log
all the data to that. That is... in the end that allows me to reconstruct any
decision that the software does. I think this is required to have the necessary
transparency, that allows us to un- blackbox these devices. All right.
Thank you very much. *applause*
Okay, I actually finished five minutes early. I didn't think this would happen,
so... Herald: I'm so surprised.
F: I am surprised too. Herald: You are on time. You have five
minutes left F: Wow, what do I do with these five
minutes Herald: We can walk around the stage or...
Maybe people have some questions? F: I think so!
Herald: Well, let's ask the Internet! Is the Internet ready?
Signal Angel: Yes. Our first question: What dou you think is the responsibility
of Bosch as a supplier for having their software and hardware used for this?
F: So the question was: What's the responsibility for Bosch, who built the
software for Volkswagen? It's a good question and I have to be careful in what
I answer. My personal opinion, and let's take this aside from Volkswagen and Bosch,
is that if you build software that you know is used to be illegally it should...
it must be your responsibility to not do that. And I'm not sure if this is
something that is legally enforceable, but it should be something that's enforceable
ethically or for all of us programmers, that we don't build software that is
designed to break the law. *applause*
Herald: We quickly hop over to microphone 1 please.
Microphone: Thank you for a wonderful talk. I'm just wondering if you're aware
of some cases of Volkswagen cars in Australia, which was suffering from sudden
and rapid power loss. This was happening about five years ago and there was a case
where a Volkswagen suffered rapid power loss on a motorway. The driver was Mrs.
Melissa Ryan and she was rear-ended by a truck and killed. So when you say that
these things can cause death, were you... are you aware that any sort of Volkswagen
software has been leading to power loss in the vehicles and affecting
performance on the road, now I don't know whether Australian driving conditions are
different to European driving conditions, and how that might affect that. Have you
done any tests that might indicate that could be happening in normal driving?
F: Yeah, so... the question was whether I'm aware of, I think an Australian
incident, right, where... M1: Can I...
F: Yeah. M1: There were many reported cases. One of
them was fatal, but there were many reported cases of that happening.
F: Of a sudden power loss, is that right? M1: Sudden and rapid power loss in the
engine. F: Yeah, of the engine. I'm not aware of
these incidents and I what I do know and... is that the the personal safety is
the number one design criteria for ECUs. That does not mean that they are perfect,
of course, that could mean that rare bugs... that there could be malfunctions.
I don't know about this, but at least it's the first design principle to provide the
safety for the people driving the car, which i think is a good thing, right. It's
not the profit or anything, or at least we can hope so. I'm not aware of this
particular incidence, and so I can't really say anything more about this. It
would be great if... Are you aware of any additional details that were found in the
investigation, please sent them to me. M1: Volkswagen was claiming that this was
a gearbox problem on automatic cars, but then it started happening on manual cars
as well, so that excuse went out of the window.
F: The issue with the problems is that most of them are very complex, so they
probably involve more than just the engine ECU, so they're very... but it's a
good example of where we need to understand exactly what is happening, and
where we may not want to rely on Volkswagen or any other manufacturer alone
to assist in figuring out what happens. We need more transparency there so that we
can have definitely neutral accident investigations.
Herald: This was a long question and really detailed answer. Thank you very
much. F: Sorry, I will be short
Herald: Felix, that's your applause *applause*
*music*
subtitles created by c3subtitles.de in the year 2018. Join, and help us!