*35C3 preroll music*
Herald: So the next talk Benjamin Kollenda and Philipp Koppe - they will refresh our
memories because they already had a talk on 34C3 where they talked about the micro
code ROM and today they're gonna give us more insights on how micro code works. And
more details on the ROM itself. Benjamin is a PhD student and has a focus on
software attacks and defenses and together with Phillip they will now abuse AMD
microcode for fun and security. Please enjoy.
*Applause*
Benjamin: Thank you. So as mentioned we were able to reverse engineer the AMD
microcode and the AMD microcode ROM and I'm going to talk about our journey. What
we learned on the way and how we did it. So this joint work with my colleagues at
Ruhr Universtat Bochum and a quick outline how are we going to do it. We're going to
start with a quick crash course on micro architectural basics and what microcode
actually is. Then I talk about how we reconstructed the
microcode ROM and what we learned
along the way. Then I quickly give some examples of the applications we
implemented with the knowledge we gained from second step. And lastly I talk about
a framework we used. How it works and what we can do with it. And also this framework
is available on GitHub along with some other tools so you're free to continue our
work. OK. So when I'm talking about microcode you can think of it essentially
as a firmware for your processor. It handles multiple purposes for example
you can use it to fix CPU bugs that you have in silicon and you want to fix later
in the design phase. It is used for instruction decoding - I cover this one a
bit more. It is also used for exception handling. For example, if an exception or
interrupt is raised, microcode has a first chance of modifying this interrupt
ignoring it or just passing it along to the operating system. It's also used for
power management and some other complex features like Intel SGX. And most
importantly for us microcode is updatable. This used to patch errors in the field.
Everyone remembers Spectre / Meltdown patches and there's
a microcode update. So your
x86 CPU takes multiple steps to execute an instruction. The first step is decoding
a x86 instruction into multiple smaller micro ops.
These are then scheduled into the pipeline
From there, they are dispatched to the different functional units
like your ALU / AGU
multiplication division units
For our purposes the decode step is the
most interesting one. In the decode step you have a instruction buffer that feeds
instructions to some decoders. You have short decoders that handle really simple
instructions. There are long decoders that can handle some more advance instructions.
And finally, the vector decoder. The vector decoder handles the most complex
instructions with the help of microcode. So the microcode engine is essentially the
vector decoder.
The Microcode engine in essence is compromised out of a microcode
ROM that stores the instructions for the microcode engine. Think of it as your
standard instructions. Then there is also a writeable memory the microcode RAM. This
is where the microcode updates end up when you apply microcode updates. And of course
around the storage has a whole lot of things that make it actually run. For this
talk, you only need to know what is a Match Registers. Match Registers are
essentially breakpoint registers. So if we write an address from inside the microcode
ROM inside a Match Register whenever this address is fetched, execution, control is
transferred to the microcode RAM so our patch gets executed. And the microcode
updates are usually loaded by the BIOS or by the kernel. Linux has an update driver,
sometimes the BIOS updates it with a pre-installed version and they have a
pretty simple structure, a partially documented header, and followed by the
actual microcode that is loaded inside the CPU. And so microcode is organized in
something called triads. Each triad has three operations essentially x86
instructions, but based on differences. And lastly, you have a sequence word. The
sequence word indicates which microcode instructions should be executed next. We
have options of executing just the next triad, executing another one by branching
to it, or just saying OK, I'm done with decoding this instruction continue with
x86 code. These updates are protected by some weak authentication which we were
able to break so we can create our own. We can analyze existing ones and we can apply
these to your standard laptop and desktop. However there can only ever be one update
loaded at the time and when you reboot your machine this update will be gone.
Also for the talk we are going to look at some microcode and we will present this
microcode using a register transfer language. It is heavily based on x86. I'm
just going to cover the differences between these two. Most importantly the
microcode can have three operands for an instruction in comparison to x86 which
usually only has two. So you can specify a destination and two source operands.
Also,
microcode has some certain bit flags that need to be set and these we do we see with
these annotations for example ".C" means says instruction also updates a carry flag
based on the result. Then you have the instruction "jcc" which is a conditional
branch and the first operand denotes the condition up on which this branch is
taken. In this case branch if the carry flag is one and [the] second operand
indicates the offset to add to the instruction pointer. Then we also have
some sequence word annotations: "next", "complete", and "branch". Also it should
be noted that the internal microcode architecture is a load-store architecture.
You can't use memory operands in other instructions like you can on x86 you
always need to load and store memory explicitly.
Now we are going to talk about
how we manage to recover the microcode ROM. The microcode ROM is baked into your
CPU, you can't change it anymore. It is defined in the silicon during the
fabrication process and in this picture you can see a die shot taken with a
electron microscope and this is one of three regions that contains the bits for
the microcode operations. And if you zoom in a bit more, each of these regions
consist out of four arrays and these are further subdivided into blocks. Really
interesting is "Array 2" which is a bit smaller than the other ones but it has
some structures above it which are of a different visual layout. This is SRAM
which stores the microcode update. So this is one-time reprogrammable memory that is
still pretty fast. So the microcode RAM is located right next to the microcode ROM
which also makes sense from a design standpoint.
Just an overview of how we
went ahead and how we went about. We started with pictures and then we used
some OCR-ike process to transform them into bit strings which we can then further
process. These bitstrings were then arranged into triads. We could already
gather that we got individual triades right because there were data dependencies
all over the place, but between triads, there were no or very few data
dependencies so the ordering of the triades was still wrong and this was a
major problem when we went ahead and what we had to reverse engineer and this is
mapping a certain physical address of a triad that we gathered from the ROM
readout to a virtual address that is used inside the microcode update or the
microcode ROM. But after reverse engineer this, you can just do a linear sweep
disassembly of the microcode ROM and arrive at human readable output. But this
recovery was a bit tricky because we required physical virtual address pairs.
But gathering these is a bit harder because we worked there through the
available updates, but we could only find two pairs of them. These pairs were
actually easy to find because every update replaces a certain triad inside your
microcode ROM and this triad is usually also placed in the microcode update. So by
matching the address this update replaces with a microcode ROM readout. You can just
get your two data points. But we had to get more data points so we generated these
mappings by matching semantics of triads in the microcode ROM readout and the
semantics when we force execution of a certain microcode address. And gathering
the semantics of the read-out microcode, we implemented a simple microcode
simulator. Essentially it works on triad level, so you give it an input state and a
triad and it calculates the output state of it. Input and output state are
comprised out of the x86-state which is your standard registers and also the
internal microcode registers. There are multiple temporary registers that get
reset for every new x86 instruction that is executed, but they can also be modified
by microcode of course. Our emulator supports all known arithmetic operations
and we have a white-list of operations that do not form or produce any observable
change in state just so that we could process more triades and give them more
data points. In total we gathered 54 additional data-address pairs which turned
out to be enough to recover the whole mapping. This mapping, essentially you
have the four different arrays that map to individual blocks and these blocks in
these arrays or then again permuted a bit and then the triads inside these blocks
have some table-based permutations. So this is not an obfuscation. This is just
from a hardware design standpoint it can make sense to reroute it a bit differently
Also now that we can actually map a certain address to the microcode ROM
readout and we know the addresses of different x86 instructions from our
earlier experiments, we can look at the implementation of instructions. So let's
start with a pretty simple one. Shift- Right-Double which essentially takes a
register, shift it by a given amount and shifts in bits from another register. So
of course you would expect a lot of shifts and rolls in its implementation and this
is exactly what we're seeing here. You have two shift-right operands and you can
see regmd6 and regmd4. These are place holders. The microcode engine can
replace certain bit combinations with the registers that are used in the x86
operation. For example this one would be replaced by ECX or EAX depending on what
you wrote in x86. And at this point we can also already gather more information about
microcodes than we previously knew because we know "OK, so this is source, this is
also a source and this is a destination". But this source which indicates the shift
amount, this one was previously unknown, because it is a high temporary microcode
register and we found out that these usually implement specific different
purpose. They are not - if you write to them, sometimes the CPU behaves
erratically, sometimes it crashes, sometimes nothing happens. But in this
case, this seems to be the shift count, and the shift count is given by a third
operand in the instruction. So in this case, we already learned "OK, if you want
to read the third operand of an instruction, we need to read t41". And
this is how we went about recovering more and more information about microcode. The
rest of the implementation is essentially concerned with implementing the rest of
the semantics of the x86 instruction and updating the flags correctly. OK, so now
let's look at a instruction set that is a bit more complicated. If you check out
rdtsc. rdtsc returns a internal cycle counter in EDX and EAX, so the upper part
ends up in EDX, lower part in EAX. So in the end we want to see writes to these
registers, potentially with a shift somewhere in there. But somewhere the CPU
needs to gather the cycle counter. So in the beginning we have two load-style
operations. This one is a proper load which we identified and this one is
unknown. But despite that we do not know the instruction, we know the target
because the result of this instruction will end up in t9 and the result of this
instruction will end up in t10, so we can follow the uses of these two registers. So
for simplicity I'm going to start with t10 and t10, which we later found out, this is
another register which essentially denotes a specific internal register. And if you
play around with these bits you notice that this combination encodes cr4. The x86
will just see cr4. You can also address cr1 and cr2. And if you look further, t10
is then ended with this bit mask and if you look in the manual you find out that
this bit in cr4 denotes the bit that determines whether oddity C is
available from user space or not. So this is the check if this instruction should be
executed. So now let's just keep in mind that t9 holds some other loaded value from
some other internal register and we will come back to this one a bit later. For
now, let's follow execution. This triad is essentially a padding triad. It is a
common pattern we see. So let's look at where this branch takes us.
And this branch
takes us to a conditional branch triad. And if you look a bit up, this end
instruction actually updated this flag. So this is a conditional branch that
determines whether this check was successful or not. So it branches toward
the error triad or the success triad. But here we already see the exit. We see a
write to RDX or EDX in this case with a shift from t9 by 32 bit, which is exactly
what you would expect to write the time stamp counter on the upper 32 bits of the
time stamp counter to edx. And you have an unknown instruction, but we know, okay, we
move something from t9 to eax, which is the lower 32 bits. But we're not done
here, because we can still look at the error pass that is taken if the access is
denied. So if you scroll a bit down we can see a move of an immediate into a certain
internal register. And this is immediate actually encodes a general protection
fault interrupt code. D denotes to the exception handler that this was a general
protection fault. And later this triad branches to this address, and if you look
at the uses of this address we can find other immediates that also correspond on
to x86 instructions. So now we learned
how we can actually raise our own interrupts. We
just need to load the code we want into the specific register and branch to this
address. And now we learned a lot about how we can actually write microcode, but
it's also interesting to see how certain instructions are implemented. So let's
look at a pretty complicated one: wrmsr (Write MSR). wrmsr essentially writes some
data it is given to a machine specific register. This machine specific register
differs between CPUs, between vendors, sometimes between revisions. And these
implement non-standard extensions or pretty complex features. For example, you
trigger a microcode update by writing to a machine specific register. The register
addresses you want to write to is given in ecx. And now we can see ecx is read and
it is shifted by sixteen bits to t10. So again, we follow uses of t10 and we see
it as XOR'd with a certain bitmask. And this bitmask is C000, which actually
denotes a namespace of the model specific registers. In this case this should be an
AMD-specific namespace. And, of course, this one again sets some flags, and you
can see your conditional branch depending on these flags to what should be the
handler for this namespace.
Next one: We have another XOR that uses a different bit
mask — in this case C001. C001 is the namespace where the microcode update
routine is actually located in. So again, we branch to this handler. And if you just
continue on, there are more operations on rcx, followed by more branches, and this
continues until everything is dispatched to the correct handler. And this is how,
internally, wrmsr is implemented, and also Read MSR is going to be implemented pretty
similar, because it implements some kind of similar thing.
OK, so now I showed you
how we actually went ahead of reconstructing the knowledge we
currently have. And now I'm going to show you what we can actually do with it. And
for this I am going to quickly cover what applications we wrote in microcode. We
wrote a simple configurable rdtsc precision.
This means a certain bit mask is AND'd to
the result of rdtsc, so you can reduce the accuracy of it, which can
sometimes prevent timing attacks. We also implemented microcode-assisted address
sanitizer, which I'll cover quickly in a second. We also have some basic microcode
instruction set randomization. Some microcode-assisted instrumentation. What
this means is, you can write a filter for your instrumentation in microcode itself.
So instead of hooking an instruction, instead of debugging your code or
emulating it, you can just say whenever the instruction is executed filter if this
is relevant for me, and if it is, call my x86 handler — entirely in microcode,
without changing the instruction in the RAM. We also implemented some basic
authenticated microcode updates. The usual update mechanism is weak — that's how we
got our foot in the door in the first place. So we improved upon it a bit. Also
we found out that microcode actually has some enclave-like features because once
we're executing in Microcode, your kernel can't interupt you, your hypervisor can't
interrupt you and any state you want visible to the outside world. You actually
need to write explicitly. So all these microcode internal registers are not
accessible from the outside world. So any computation you perform in micro code
cannot be interfered with. So you can implement a simple enclave on top of this
one. So our hardware-assisted address sanitizer variant is based on the work by
the original authors and address sanitizer is a software instrumentation that detects
invalid memory access by using a shadow map shadow memory to just say which memory
is valid to be read and written to.
The authors proposed hardware address sanitizer
which is essentially doing the same checks but using a new instruction. And the
instruction should raise a fault if an invalid access is detected. This algorithm
they proposed - The details are not important. What is important is in
essence: It's pretty simple. You load from a certain adress, performs the operations
on it and if there is the shadow after this operations you just report a bug.
Advantages of hardware address sanitizer are for example you get better performance
out of it. Because you only have a single instruction maybe you can do some fancy
tricks inside your CPU that are faster than using x86 instructions, you get more
compact code and you have the possibility of one time configuration which is a bit
hard with software address sanitizer. We implemented hardware address sanitizer our
variant by replacing the bound instruction Bound is an old instruction that is no
longer used by compilers because in fact it is slower to use bound instead of
performing the checks with multiple x86 instructions. We changed the interface.
The first argument is the register which holds the address you want to access. And
the second argument holds the size you want this access to be.
So, 1 byte, 2 byte and so on.
This instruction is a no-op if the check succeeds. So if there is no bug it
just continues on like nothing happened. However if we detect an invalid access we
can take a configurable action, we can for example just raise your normal page fault
or we can raise a bound interrupt, which is a custom interrupt, that only denotes
this one or we can branch to an x86 handler that either performs additional
checking, for example whitelisting, or it generates a pretty error report for you.
Most importantly this is a single instruction. We also do not dirty any x86
registers because they are some intermediate results. You need to store
these somewhere and this you usually do in the x86 registers. So you increase
register pressure. Maybe you cause spilling. So overall your performance gets
worse. We also found out that we are actually faster than doing the checking
using x86 instructions. So just by moving the implementation from x86 level to
microcode, which in some way is still kind of like software, we already improved the
performance. Also on top of this you get better cache utilization because you have
less instructions, there are less bytes in the cache, so we get fuller cache lines.
And also it is really easy to tell which is testing code and which is your actual
program code. Lastly I'm going to show you just a rough overview of our framework
which we used during our development and which you can also find on GitHub. Early
on we found out that we are probably going to need to test a lot of microcode
updates, because in the beginning you just throw everything at the CPU and see how it
behaves and we wanted to do this in parallel. So we developed a small custom
OS called "Angry OS" and deployed it to mainboards. These mainboards are just old
AMD mainboards. All these mainboards were hooked up via serial for communication and
GPIO to a Raspberry Pi. With the GPIO you can reset, support power on, power down
and just have remote control of this mainboard and then you can connect to that
Raspberry Pi from anywhere on earth and just deploy and play around with it.
This was the first version.
In the beginning we didn't really know much about electronics
so we used one Raspberry Pi per mainboard. And it turns out Raspberry Pis are more
expensive than these old mainboards, but we improved upon this and now we're down
to one Raspberry Pi for four / five setups.
For example you only need 3 GPIO ports per
mainboard. You connect each of these to optocouplers just to separate the voltage
levels and then you connect one side of the optocoupler to the GPIO the other side
to your reset pin, to your power pin and for input to know whether your board is up
or down you connect the power LED. And that way you can save a lot of space, a
lot of money. And also if you're really constrained you can just remove the power
LED sensing because usually you know it is in the state your setup is in. As I
already said we wrote our custom operating system and it is intentionally really
really minimal because the major feature we wanted is control over every
instructions that's going to be executed from a certain point on, because we're
playing around with instruction encoding and if we execute an instructions that we
did not intend we might crash the CPU, we might go into an invalid state and we do
not even know which instruction caused it. And Angry OS essentially only listens on
the serial port for something to do. What it can do is apply an update. These
updates are just microcode updates. They are streamed via serial. We can also
stream x86 code which is then run by Angry OS and this is just so that we do not need
to reflash the USB stick every time we want to update our testing code and the
result, all the errors are reported back to the Raspberry Pi and thus they are
forwarded to us. The framework we use most importantly has the microcode assembler
and a pretty verbose disassembler. This disassembler generates the output I showed
you earlier and using this you can just quickly write your own microcode. We also
included an x86 assembler because we wanted to rapidly test different x86
testing codes. Using this framework we were able to disassemble the existing
updates and we also used it to disassemble our ROM after we reordered it and also
during the process when we fed it to our emulator. And we can also create the
proper binary files that can be loaded by the Linux kernel driver. We modified the
stock one to just load any update you give it without checking if it's the correct
CPU ID and all these things just for testing purposes. It's also available. And
also of course the framework can control Angry OS to make your testing easier. And
we implemented a pretty basic remote execution wrapper, so you can work on a
remote Raspberry Pi as if you were using it locally.
And this brings me to the end
of talk. And in conclusion we can say reversing the ROM opened up a lot of new
possibilities. We learned a lot about how microcode works. We learned about how to
actually use it properly instead of just inferring from a really small dataset,
that we have from the updates, or from the random bits things we send to the CPU and
observe what happened. But there's a lot left to do. So if you really want to hack
on it, just get in contact, we were happy to share our findings with you. And as I
said the framework AngryOS, example programs, that we implemented, and some
other stuff like the wiring is available on GitHub. So that's that. And we are
happy to answer any questions you might have.
*applause*
Herald Angel: Thank you very much. So we
have 10 minutes for questions please line up at the microphones. We start with this
one: microphone number 2. M2: Hi. Thanks for a nice talk. A few
questions about your hardware address sanitizer.
Benjamin: Mhm M2: As I understand you don't need the
source code instrumentation because the microcode is responsible for checking the
shadow memory, right? Benjamin: No... The original hardware
sanitizer implementation is also based on a compiler extension, that inserts a new
instruction because it doesn't exist usually. And it also inserts a bootstrap
code that in inits your shadow map and also instruments your allocators to update
the shadow map doing runtime and we essentially need the same component, but
we do not need the software address sanitizer component that essentially
inserts 10 or 20 x86 instructions before every memory access. So yes we still need
a compile time component and we are still source code based in a sense.
Herald: And, so.. M2: And I didn't see, maybe I missed the
numbers. How much it is faster than this initial version?
Benjamin: You mean the initial hardware sanitizer version or the software address
sanitizer. M2: I mean let's say custom kernel address
sanitizer for Linux kernel which is the the usual one and your approach.
Benjamin: We only performed a micro benchmark on Angry OS and we essentially
took the instrumentation as emitted by the compiler for some memory access which is
your standard software address sanitizer and compared it to our version using only
the modified bound instruction. So I really can't talk about how it compares to
KASAN or something or some like real world implementation, because we only have the
prototype and the basic instrumentation. M2: Thank you very much.
Herald Angel: OK. Microphone number 4 please.
M4: Hey thanks for the talk and did you find any weird microcode
implementations. I don't mean security wise, just like you rarely expected to
see it be implemented that way.
Benjamin: The problem is there's a lot of
microcode to begin with. You have f000 triads. Each of which has 3 op-codes. So
you have a lot of ground to cover and also we have read-out errors. Sometimes you are
seeing bit flips, which kind of slows you down because you then need to always
consider: OK, maybe this register is something else, maybe this address is
wrong. And also sometimes you have a dust particles that kind of knocks out an
entire region. So we only looked at the components, we were pretty sure that we
recovered correctly, and we'd only looked at a really tiny subset compared to all of
the microcode ROM. It's just not feasible to do and to go through it and look at
everything. So no we didn't find anything funny but we also wouldn't know what funny
looks like because we don't know what the official spec for microcode is.
M4: Thanks. Herald Angel: Interesting. We have one
question from the Internet, from the
Signal Angel please. Signal Angel: Yes. Which AMD CPU
generations does this apply to? Benjamin: Yeah this is still based on the
work of our first talk and this only works on pretty old ones: K8, K10. So until,
CPUs produced until 2013. Yeah this was the last year AMD produced anything like
that. Newer ones use some public key based cryptography from what we can tell and we
haven't yet managed to break it. Same goes for Intel, they seem to be using public
key cryptography and we haven't gotten a foot in the door yet.
Herald Angel: Thank you. We go one around. On microphone number 3 please.
M3: Yeah. Thank you. I would like to know how complex could the microcode programs
be, that you could write. So what's the complexity of new operations you could
implement. Benjamin: The only limiting factor is the
size of your microcode update RAM. But this one is really really limited.
For example on K8, where we performed the majority of our experiments. We are
limited to 32 triads, which comes down to a sixty nine instructions and you also
have some constraints on these instructions for example the next triad
will always be executed no matter what. Some operations can only go at the second
slot. Some can only go on another slot, so it's really really hard. And you're also
limited from our knowledge to loading 16 bit immediates instead of 32 bit or even
64 bit immediates. So your whole program grows really fast if you're trying to do
something complex. For example our authenticated microcode update mechanism
is the most complex one we wrote it nearly fills out the RAM and we used TEA – Tiny
Encryption Algorithm – because that was the only one we managed to fit mostly due
to S-box and other constants we would need to load. So it's really small.
Herald Angel: Thank you Microphone number 1.
M1: So you said the microcode is used for instruction decoding and it needs to meet
the micro-ops to the scheduler and micro queue in some way. Did you find out how
that works? Bejamin: In essence we are not actually
executing code inside in microcode engine. From what from what we understand, the
microcode engine is just some kind of a software based recipe, that describes how
to decode an instruction, so you don't actually get execution, you just commit
instructions into the pipelines, that do what you want. And because we have some
control flow possibility, that is actually inside the micro code engine, because you
can branch to different addresses, you can conditionally branch and loop. You kind of
get an execution, but in essence to just commit stuff in the pipeline and the CPU
does what you tell it to.
Herald Angel: One more question. Microphone number 2, please.
M2: How did you take the picture of the internal CPU? Did you open it?
Benjamin: Yeah. We worked together with
Chris. He's our hardware guy. He has access to his equipment to delayer it and
to take high resolution optical shots and he also takes shots with a scanning
electron microscope. So I think about five or six CPUs were harmed in the making of
this paper.
Herald Angel: So we have one more last question. Microphone number 2 please.
M2: Are you aware of research done by
Christopher Domas, where he mapped out the instruction set for x86 processors?
B: You mean sandsifter? We actually talked with him and yeah we are
aware, that there's a map essentially of the instruction set and also maybe you can
combine it, because in the beginning we reverse engineered where certain x86
instructions are implemented in microcode. So if you plug these two together you kind
of map out the whole microcode ROM at the same time that you map out a whole
instruction set. However there are some components of the microcode ROM that are
most likely not triggered by instructions. For example it seems like power management
or everything that is behind a write MSR [wrmsr] or read MSR [rdmsr]. wrmsr is a
single instruction, but depending on the arguments you give it it just branches to
totally different triads and the microcode itself is implemented in microcode. And
this one is a huge chunk you wouldn't even find without brute forcing all
combinations for all instructions which is not really feasible.
Herald Angel: Thank you. Thank you Benjamin.
*applause*
*35c3 postroll music*
subtitles created by c3subtitles.de in the years 2019-2020. Join, and help us!