*35c3 prerol music*
Herald: So Trammell Hudson, who is standing here, he's taking things apart.
Don't worry not life on stage, but he will give us a proof of concept and some
details and functionalities about hardware implants. So the same things that we heard
from Bloomberg article talking about Apple and super microcomputers with implants
that, yeah, were implanted into those, into those computers. And I'm really
excited to see this in action. Please give a warm round of applause to Trammel
Hudson!
*applause*
Trammell: Before we begin talking about hardware implants just two quick
disclaimers. The first from my employer Two Sigma investments as it says are
chocolate bars. This is not investment advice. And secondly I don't actually know
what the story is behind the super micro story. No one outside of Bloomberg and
their sources do. But I have spent a lot of time thinking about hardware implants
starting with the thunderstrike firmware attack against mac books as well as the
thunderstrike 2 where we were able to get software to write into the firmware on the
mac books. I've also been thinking a lot about how to defend against hardware
implants with things like the heads firmware for slightly more secure laptops
and also as part of my co-lead on the Linux boot project. We're thinking about
how to protect servers from physical and software attacks. So with all of this
concentrated thinking about firmware and hardware attacks, I was really excited
when I saw the Bloomberg story back in October. But what really intrigued me was
the animated image that they had at the header that highlighted one small part of
the board as where the implant was, but what I found really interesting is that is
exactly where I would install a hardware implant as they described on the SPI bus.
A lot of other people in the hardware and from our security community thought it
sounded plausible. Other people pointed out that supply chain attacks come up
periodically and they are definitely a concern. Some people thought the attack as
described was entirely implausible and in general we sort of had a Whiskey Tango
Foxtrot moment as everybody scrambled to figure out what's going on inside their
machines. So, let's step back very quickly and review what the key claims that
Bloomberg alleged happened. First they said that Amazon's testers found a tiny
microchip that wasn't part of the board's original design that had been disguised to
look like a signaling condition signal condition coupler and that these illicit
chips were connected to the baseboard management controller or the BMC which
gave them access to machines that were turned off. That might sound kind of
extreme, but that's actually what the role of the BMC is, that in most servers the
BMC is running any time the machine is hooked up to power and it's connected to
the power supplies so that it can turn the machine on and turn it off. Frequently you
want to be able to do this over a network so it has its own dedicated LAN port but
it can also share the LAN port with the with the main system. Serial over LAN is a
really useful way to debug the systems so it provides that functionality. It can
also provide fake USB volumes to allow to to do unintended OS installation. A lot of
sites also won't remote KVM so it has VGA but that VGA support means that it's on
the PCIe BUS and because some PCIe it can do DMA into main memory. It also is
typically muxed into the SPI flash for the host firmware, which allows it to
modify it and on some systems it's even connected to the TPM which allows it to
circumvent the corporate of trust. So with all of this capability inside this chip
it's really unfortunate that they are really not well put together. The head of
Azure security says they have no protection against attacks. There's no
ability to detect if an attack has happened and there's no ability to recover
from an attack. So having a hardware implant on the BMC is a really big
concern. The other claim in the article is that it affected 30 different companies
including Apple and Bloomberg alleges that Apple found malicious chips independently
on their super micro boards. Went to the FBI about it and that they then severed
ties with Super Micro. This particular claim was interesting because it
corroborated a story that had shown up back in early 2017 that Apple had removed
Super Micro from their data centers. Apple denied that there was a firmware issue.
But it's interesting that perhaps these two were related. The third set of claims
is that on some of these implants they were actually put between the layers on
the PCB and then the most explosive claim is that this was done by operatives from
China, the Chinese People's Liberation Army. With a story with this you know this
many claims and this significant of allegations we'd hoped that it would be
really well sourced and for a normal story 17 independent sources that Bloomberg
editors agreed to grant anonymity to, including six national security, two
people inside of AWS and three senior insiders at Apple seems like pretty solid
sourcing, except as soon as this article is published everyone denied it. The
Director of National Intelligence said they'd seen no evidence of this. Amazon
said that they've never found any issues of modified hardware nor have they been
engaged with the government over it. Apple was even more blunt. CEO Tim Cook said
this did not happen. There is no truth to this. And Super Micro wrote a fairly
lengthy letter about what they do to protect their supply chain and why they
think this attack did not happen. And it is worth going through to look at some of
the things that they say that they do to protect their supply chain. They point out
that if there's any unauthorized physical alterations during the manufacturing
process other design elements would not match and those things would be detected.
To sort of understand how circuit boards are made, I recently visited a PCB factory
in Guangzhou. This is not a super micro factory. This is just a holiday photos. So
in order to add new vias they would have to modify the drill files which would then
get electroplated. If they had to add new traces, they would have to be able to
subvert the masking and etching process and any changes to either the drills or
the etching on individual layers would be caught by the optical inspection that's
done on these bare circuit boards. Additionally the allegation that things
were inserted between circuit boards would require that the lamination process be
subverted and that the implant somehow aligned into the system. If that implant
changes any of the connectivity the flying protesters would pick it up or the bed of
nails testers which checks all of the connectivity of all the traces to make
sure that there are no shorts and to make sure that everything that is supposed to
be connected is electrically conductive. So it would be very difficult to
circumvent the production process at this stage. And it also would be very difficult
to contain because the PCB factory doesn't know which customers are going to receive
those circuit boards. Super Micro also points out that during the assembly
process when the parts are installed they have their employees on site the whole
time. On my same holiday trip I also visited some PCB assembly companies and
spoke with companies that are using doing contract manufacturing and they said that
they also send their employees to the production line to observe the pick and
place machines and the reflow and the rest of the surface mount assembly. Their big
concern is that if they don't have someone there the parts that are fed in the pick
in place will be replaced with either counterfeits or with salvaged parts. I
visited the electronics market in ??????? bay where there are people desoldering
e-waste and then sorting the parts into bins and selling these salvaged components
by the kilo and for a few extra renminbi they'll put them on rails for you so that
you can save a few pennies on your production process. The other concern that
these companies have, is not just salvaged parts but straight up counterfeits.
Especially for things that cost more than a few dollars each. The Arduino community
was hit a few years ago with a bunch of counterfeit FTDI chips where the internal
construction was entirely different. In this case it caused reliability issues but
you can imagine from a security perspective this is really worrisome that
parts that look identical might have completely different functionality inside
of them. Super Micro also mentions that they X-ray their main boards to look for
anomalies and I wasn't able to take any photos inside the factory there was doing
x-rays. But in this Wikipedia photo we can clearly see active components like this
SOIC chip are different from things like the SMD resistors and capacitors. So if an
attacker were trying to subvert the supply chain by putting a disguise component it
could be detected at this step. Another interesting thing in this photo are these
inductors that are encased in dip packages. This is really common in a lot
of Ethernet boards and occasionally people have thought they had some sort of
hardware implant when they found inductors in their ethernet jacks but it's pretty
it's fairly common and it shows it pretty clearly on the x-ray. Some other security
researchers like Sophia D'Antoine did an extensive teardown of Super Micro boards
including X-ray analysis and her group found a few oddities but nothing.. they
didn't find anything malicious. There were no smoking guns. They just appeared to be
sort of supply chain type things. You can read her blog post for more details about
where they found things that shouldn't have been there. But turned out to be just
actual signal condition components. So super micro in their ???? letter, they
keep reenforcing that the manufacturing process that is the assembly process, it's
during the manufacturing process and I agree with them. It would be very
difficult to circumvent security in a reasonable way in that part of the
process. But that's not the only place this could happen. We know that national
security agencies intercept shipments of computer hardware and then have their
tailored access operations open the computers, install hardware implants,
reseal them and then have them continue on their way in shipment. The NSA even has a
catalog of hardware implants like this JTAG implant Ethernet jacks with embedded
computers in them as well as firmware specific ones that target servers SNM(?)
and then some that can do data exfiltration via RF. So that's sort of
tailored access operations is really ideal for this supply chain attack because it
allows them to contain the exploit to a single customer. It allows them fairly
good concealment as well as good cover that if it's discovered it's really hard
to attribute where things went wrong. Now unlike if you find something inside your
motherboard between the layers you know that had to have happened at the factory.
So Super Micro also claim that this was technically implausible, that it was
highly unlikely that unauthorized hardware would function properly because a third
party with lack of complete knowledge of the design. I think that's inaccurate,
both because we know the NSA does it and also because I have done it.
*laughter*
Really, all that you need to know is that these are common components. These flash
chips show up on all the boards. You can search the internet for the data sheet and
find exactly how it's wired into the rest of the system. And the only thing that we
need to know to communicate to the BMC is the serial output pin from this component,
so the BMC flash is connected over to the BMC CPU via the serial output and it goes
through a small series resistor and that is where my implant goes in. Mine's a
little bit larger than that resistor. It clicks onto the board and it has a small
FPGA that hangs offside but it's completely plausible to fit it into
something that small in fact a modern ARM M0 fits in the space of two transistors
from a 65 002 from a few years ago. The Moore's Law means we can pack an amazing
amount of CPU into a very very small amount of space. So on that 0 6 0 3
resistor could fit around 100 cortex M0 it would be plenty powerful for this system.
The problem is we only have those two pins so ordinarily on the spy flashing you need
at least six pens but we don't have power and ground so we have to passively power
this through the data signal that's passing through it. We don't have the chip
select pin so we have to guess when this chip has been talked to. We don't have the
data input pin so we don't know what addresses are being read or what commands
are being sent. We have to reconstruct it from the data output pin and we also don't
have a clock pin so we have to figure out how to synchronize to that clock. Lastly
we don't have the ability to make arbitrary data changes. All we can do is
disconnect the pin from the BMC so we can only turn 1 bits into 0 bits. We can't go
the other way around. So with these limitations we can still do some pretty
interesting things. Recovering the clock is actually pretty easy. We can look at
the data stream and find the shortest bit transitions from 0 1 0 or 1 0 1 to
estimate what the clock is which allows us to then reconstruct that data stream being
sent to the BMC and if we look at the flash contents we can see that a lot of it
is being fairly random noise but a lot of it is all white which in this case would
mean that it's all one bits. So if we look at the way the flash is organized we can
see there's the u-boot bootloader and that's executable. That's kind of
difficult to make useful changes in, the kernel and the root file system are both
compressed so that they look effectively like random noise but the nvram region is
a jffs2 file system and this file system ??? 3 Megs, it's mostly empty and all that
empty space is F F which is all ones. So this is plenty of ones for us to work on.
Additionally it has fairly nice headers that we can we can match on. So when we
see these magic bit masks we know when we've entered different parts of the file
system. So given that we can now reconstruct the clock we can figure out
where we are in the file system. This hardware implant can start to inject new
data into what was the empty space. So this short file that we put in here is a
small shell script and it is one of the network configuration scripts, so this is
where I'm going to try a live demo and I hope this works. We're running in qemu
since I didn't bring a Super Micro board and what we have on the left is the flash
console excuse me the hardware implant console. And then on the right we have the
serial console from the BMC so we can see it has loaded the kernel and in a second
it's going to we should see a bunch of traffic, okay, so the implant is active.
It has replaced the data when that nvram file system was mounted the BMC is now
continuing on doing its set up. It's going to load a bunch of device drivers for that
video. It pauses here for some reason that I haven't diagnosed because that's that's
not my job.
*laughter*
And eventually it's going to configure the networks and it does that by running that
shell script off of the nvram partition here it starts KVM stuff brings up some
things. Allright. *applause*
OK. So luckily we got to that point without having to fake the demo. In the
hardware it's really flaky. My version works about one in eight times. But it
doesn't typically cause a crash. So that's actually good for concealment because it
becomes now much harder to determine which machines are affected. In qemu because
it's emulating, it's a little more reliable but it's still it's only two out
of three. If we let the BMC boot a little bit further it actually prints out this
message. And if you hit enter it drops you to a shell with no password and you can
then just run commands as root on the BMC and that's a lot easier than all this
stuff with the SPI bus if you wanted to build a hardware implant against it. I
don't know where the serial port is on the on the Super Micro but on a different tier
1 server mainboard I was able to probe around the oscilloscope and locate the
serial console for the BMC. Figure out it's 115 kbaud and it has the same code
that you hit enter and you can run commands there. So that's a much easier
way to do it. A big question a lot of people have is how do we actually detect
this sort of flash implant. A lot of high assurance sites replace all of their roms
with ones that they flash themselves but that doesn't get rid of the implant
because it's outside of the ROM chip. Likewise reading the ROM chip doesn't show
anything because it's not in the ROM itself it's it's outside of it. Even
hooking up a logic analyzer to the bus and watching as the machine boots and seeing
the data stream coming out of the flash won't actually reveal the implant because
you'd have to put the logic probes on the PGA pads on the flat on the BMC itself.
And that's a much harder task. Some people think "oh well we can see the weird
network traffic when the BMC tries to exfiltrate the data" but that would be
that's only one way for the BMC to affect things. There is a great talk a few years
ago at DefCon from Intel ATR where they showed how something that can control the
system firmware can backdoor hypervisors. And then they gave a use case where a
unprivileged guest on a cloud system could read all of the rest of physical memory so
it could see all of the other guests memory. So what do we do? The big problems
is the BMC has way too many privileges. It's connected to pretty much everything
in the system but the BMC is not our only concern. As @whitequark said, our PCs are
just a bunch of embedded devices in a trench coat and they all have firmware. In
fact pretty much everything on your system more complex than a resistor probably has
firmware and if you have one of those Super Micro implants maybe even your
resistors have firmware as well. I've found that the firmware and things like
the power supplies can be used to gain code execution on the BMC. It's really
interesting how tightly connected all of our systems are. And as Joe Fit's pointed
out in his blackhat ???? talk, these are not multimillion dollar attacks these are
five euro bits of hardware that we now have to really be worried about. I really
like the guidelines that NIST has published that suggests that we think
about our systems more in this holistic manner. Although the interpreting pretty
much everything into the TPM is the trusted platform module for doing this
attestation and I think we as a community need to do more to use the TPM. There
actually a really good tool for securing our systems but they are also potentially
subject to their own hardware implants. The NCC Group TPM genie is able to subvert
the core root of trust by interposing on the TPM. So a lot of folks are proposing
we should move to other trusted execution environments like SGX or Trustzone. And I
think these have a lot of promise especially for trusted cloud computing.
There also is a lot of innovation in the hardware roots of trust going on right now
between the Google Titan, which initially was for their servers and is now showing
up on all of their chrome books. The Microsoft Cerberus chip which again is the
Azure system. They're actually publishing their firmware and the ASIC design so that
people can have a little more faith in it and they hope it will become an open
standard. And companies like Apple have also gone their own way. With the T2 and
the T2's are really amazing chip for securing systems. But it does so at the
expense of user freedom and that gets in the way of what I think the real way that
we need to.. we need to solve this problem. We need to get rid of a lot of
these secrets. Counter to what the Super Micro CEO said, having a secret
motherboard design does not make you more secure. Things like the Open Compute
hardware I think is a good vision for how we can move forward that when you buy an
Open Compute server it comes with full schematics and gerber files. So that
motivated customers can verify that the systems that they're buying are the ones
that they think they that they're buying that all of the components are what they
think they should be. I think the firmware also needs more openness. Ronald Minnich,
Google is my co-lead on Linux boot project and we think that Linux in the firmware is
a way forward to get a more secure more flexible and more resilient system. We're
working with a spin off project called micro BMC that is using the Linux boot
tools to build BMC firmware and this is opensource. It's reproducibly built it can
work with roots of trust attestation. It's written in a memory safe language since
it's a Google collaboration and go. And more importantly we've thrown away all of
the legacy features that have been a source of a lot of security
vulnerabilities in these systems. So did it happen? I don't know. Is it technically
possible? I think so. I hope I've convinced all of you that this is
definitely a technical possibility that we need to be concerned about and I hope that
the way forward through hardware roots of trust with attestation and more
importantly with open hardware so that we know that what the machines were running
are running code that we know.. the code that we've built that we understand and
that we can actually have a good chance of being able to take control back of them.
If you're interested in more discussion on this and also on open firmware, there's an
assembly here in this hall that has a bunch folks working on a core boot and
Linux boot and a lot of these projects where you can help contribute and you can
help also pressure vendors to make these this standard and a way forward for a more
secure computing. So thank you all for coming. And I really enjoyed the chance to
show off my modship of the state.
*applause*
Herald: Geat talk, thank you very much Trammel. We have 10 minutes for questions
so please line up at the microphones if you have questions. And we also have a
signal angel probably with questions from the internet. So any questions? Microphone
number three? Mic 3: Yes, I was going to ask, what's
your opinion on the Talos systems? The openPOWER based ones?
Trammell: So the question is about the Talos power 9 based systems power 9 is a
really interesting architecture. The.. it is using a open firmware very similar to
Linux boot called Petitboot that moves Linux into the bootloader. I'm a big
fan. There's a lot of folks in the opensource community who are very excited
about it. I'm hoping that there would be more power nine systems coming out. I'm
also very excited about the RISC-V systems. I think having open source CPUs
use is a real way that we can have more assurance that our systems are what we
think they are. Herald: Thank you, microphone number two
please. Mic 2: Yes, thanks for the talk. I was
wondering if you have just a scope probe over this serial, cause it's just a serial
resistor which we're replacing. If you put just two scope probes on there and measure
the voltage over it, in your situation would the voltage change there once in a
while? Trammell: Yes, yes, yes.
Mic 2: Well okay, in the normal case would it actually be quite consistent current.
Or if you lowered the input impedance of the BMC chip who might already have fixed
a part of the attack because the output sourcing current of your exploit is
probably limited due to the limited supply you only can..
Herald: Your question please? Mic 2: Yes.. but.. do you see a way to get
more power into your setup? Maybe using, well other power sources, other than the
two pins, or maybe somewhere of.. Trammell: Well, so the question is about,
would there be a way to do more arbitrary changes through redesigning the implant.
One of the goals was to fit with only those two pins so that a single piece on
the motherboard could be replaced. With a dual probe soldering iron and you can pop
it out and stick a new one down in a matter of seconds. So, yes, if you have
more pins where you can get more power from you can do much more interesting
things. But that's.. would require a different set of changes to the
motherboard. Herald: Thank you. Microphone 1 please.
Mic 1: So, a lot of the -like- arguments that these implants were not feasible by a
Super Micro where you also show the picture from the fab that you had to
change the etching and the optical inspection and so on and so on. But how
probable would you rate the fact that some acto just intercepted the manufacturing
files and added that component already in the file because then all the optical
inspection and that would all say well that matches what was sent to us. But that
was not necessarily what Super Micro sent to the fab.
Trammell: So the question is, could someone have modified all of the
manufacturing files that went to the factory, and that's absolutely a
possibility. But that's also very likely that that would be detected by Super Micro
itself that in a lot of cases you don't necessarily want to trust the company that
is making the product to also test it. And you probably want to have a separate
company that does random spot checks to verify that the boards are actually being
produced to the specification that you.. that you desire. So it's certainly
possible and I really don't want to speculate as to the accuracy of that part
of the story but yeah it would require quite a bit more changes. And also would
be much more likely to be detected in the spot check.
Herald: Great. Microphone number two please.
Mic 2: Yes, for a lot of motherboards there are also quite a few components not
populated some of which are on which you could consider sensitive myths. Wouldn't
that make it. Yeah exactly. Wouldn't that make it very easy to do just pop something
on there in parallel with one of the components and not have it be detected
because it's like the board is modified. There is a component or you have no way of
telling whether it had to be populated or not?
Trammell: Super Micro puts a lot of extra pads on the board in this one particular
one they have both 8 pin and 16 pin flash chip pads that are just in parallel
together. So depending on which chip is cheaper that day of the week or who knows
what, they will populate one or the other. So that's why in this particular photo
having the position of that circle on the data output pin is very very interesting.
Herald: Question answered? Okay. So one more question on microphone number two
please? Mic 2: How far can signing of firmware be
a solution to this problem? Trammell: Signing firmware solves a lot of
the issues. It does however not all typically not all of the firmware are
signed specifically is probably to be signed in in a modern BMC. The kernel and
maybe the root file system might be signed. But the envy of RAM file system in
this BMC is designed to be user modifiable so it can't be signed by the manufacturer,
so this sort of attack would work against a signed BMC just as well. Also the "Hit
enter to get a serial console" attack circumvents any signing. There are things
on the host firmware on the x86 like boot card that do a really good job of making
it harder to get code execution during the boot process. But there have been several
CVEs where it has been implemented poorly. So even though signature's the firmware is
signed, people have still managed to get code execution during that process.
Herald: Great. Thank you Trammell Hudson again, a warm round of applause, thank you
very much!
*applause*
*35c3 postrol music*
Subtitles created by c3subtitles.de in the year 2021. Join, and help us!