Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2716 Thanks!
[Music]
okay our next speaker Hugo
Landau is specialized in reverse
engineering and a strong component of
open firmware open software and the
right to own your
Hardware um Open Stage for his talk
entitled adventur in the reverse
engineering program n IC
firmware thank
[Applause]
you hi everyone and welcome to my talk
adventures in reverse engineering
broadcom Nick firmware so this is a
story about how I ended up reverse
engineering the firmware to a broadcom
EET controller and why I was motivated
to do that so in order to understand why
I did this crazy thing um I actually
have to take you back to when I was 10
years
old I was into Computing just as much as
I am now and uh I heard that the soon to
be released Windows XP this was in 2000
um would have this feature called
Product
activation and this is a DRM feature
right where they think you might be
trying to Pirate windows so they prevent
windows from working until you literally
get authorization from the Windows
Microsoft Mothership and until then
windows will refuse to work and I was
only 10 years old and I had never heard
of the FSF or open source I have never
even used the internet but I intuitively
knew that this was fundamentally wrong
and boycotted Windows XP as a
10-year-old so
why did I do this because I intuitively
knew that your computer should be on
your side and not anyone else's and that
is something I believe to this day and
that's something I fight for to this
day your computer should be on your side
but is
it
no product activation
DRM locked
bootloaders
talization your smartphone is controlled
by Apple or Google or whoever whenever
or not you want it even if you don't
want it to
be these are all things that we'd call
anti- features right it's a term Sol
invented for features that were designed
to benefit the manufacturer's interests
over
yours and it goes beyond just computers
now anything with a chip might be used
to try and control you because companies
have figured out that they can use
software to control how a product is
used after they sell it
you know it depends on the country
you're in but there's this legal
principle called the first sale
principle which is basically a law that
says after someone sells something to
you you can do what you like with it
they don't have a right to tell you no
but companies have figured out that it
doesn't matter what the law says they
can use software to work around that and
effectively impose their control on you
on this product they've supposedly sold
to you after they've sold it to you you
know printers print cartridges they try
and prevent you from using third party
print cartridges CNC machines that
literally have a GPS chip in them and
they'll shut down if you move them
that's a real example um cars where if
you try and hook into an EET Port you
discover under the Bonnet the company
that made the car literally calls you up
and tells you off tractors that you
can't repair and being given us a talk
today even trains I highly encourage you
to attend that talk being given today
about DRM in Polish trains at 11:00 p.m.
and these are all real examples right
and the best citation I can give you for
this is the coming war on general
purpose computation by kctu who really
sounded the Bell on the start of this
battle so the basic principle that we
have here is one of owner control and in
my view a simple principle must be
defended all hardware and software must
be designed to put the interests of its
owner first over any vendor any third
party and any
government and that's what I
believe and in a way if you want to
think about it like this owner control
is like the diametrical opposite of DRM
the matter and
antimatter so how do we make this
reality so you can make free software
for your operating system you can make
free operating systems but at some point
the rabbit hole start going really deep
and you find anti features in firware
itself where the people that made the
hardware have put that there so I would
argue that if we are to have owner
controll machines where we could know
that what we're using is on our side all
of the software and firmware on your
machine needs to be yours to audit
inspect and change to your
needs all of the firmware on your
machine I would argue therefore must be
open source you know it really goes back
to Richard storman at the very beginning
getting annoyed because you couldn't fix
a bug in a printer
driver um and what's interesting is that
if you have open source firmware on your
machine anti features are sort of
impossible I
mean if you don't like a feature you can
just remove it the ability to impose
anti features is one motivation company
has to keep a feature proprietary so if
a company says you know we don't want to
open up this firmware because because
then you the user could remove this
feature that we put in there I would
argue that by definition if a vendor is
saying that that feature is an
antifeature think about it like this
open-source DRM is an inherently
nonsensical concept it can't
be and it's not just about under control
it's not just about the ethic side of it
it's also about security we need to be
able to audit firmware in order to trust
it how do you know that there isn't a
back door but often it could just be a
zero day in shorty vendor code and the
quality of a lot of vendor firmware is
often astonishingly low I have myself
personally managed to brick a laptop by
installing Linux on it as in Brick as in
not it won't boot as in I can't even get
into the BIOS menu um because companies
they just want to get things working
they ship firm as soon as possible they
never patch it and often when people and
security researchers go in there and
actually look at how things
work they find that actually it's full
of holes and of course there's an
increasing concern about supply chain
attacks and every company shipping
propriety Solutions seems to be saying
well we're just going to make it so the
firmware has to be signed by us and
therefore it's safe and I'm sorry but
that is a terrible security model and
it's one that's fundamentally
antithetical to owner control that by
definition is vendor
control but you also have like
reproducible builds and a lot of people
have done a lot of work over the past
few years to enable reproducible builds
and you need open source in order to be
able to do reproducible builds I don't
know how many people here have read the
famous P of Reflections on trusting
trust where it turns out actually you
can't even trust your compiler even when
it's compiling another
compiler um and with reproducable builds
you can actually start to mitigate some
of those
concerns but with x86 we have hit a dead
end both Intel and AMD have these signed
firmware blobs now which can't be
replaced the hardware a lowly boot code
signed by them you may have heard about
the Intel management engine or the AMD
PSP which is the exact same thing
basically and they've built Investments
on making sure that you can't control
these processes that are hidden on your
x86 system they use these for DRM and so
on and other functions they've invested
in making sure that you actually you
don't have full control over your
machine
and they have gone so far down this road
now that honestly I
think they're never going to come back
from it so it seems like xc6 has become
a dead end and of course you have many
small single board computers that are
fully open phone Weare but they're not
fast so there's this question here where
can we get a desktop or server with open
firmware and it actually look pretty
bleak for a while until it turns out IBM
all this time all these years has been
making very expensive servers with their
own custom CPUs power CPUs and um
amazingly a company in Illinois wrote to
them and said would you open up all your
firmware and to my astonishment they
said
yes so the result of this is we got this
computer known as the tals 2 it's an e8x
motherboard you can literally build it
yourself like any other PC but it's not
x66 is power 9 and it has 99% open
firmware but why is it
99% well it's because there's this
broadcom EET controller that has a
firmware blob so let's reverse the
firmware and get rid of
it and that's how I ended up reverse
engineering this
chip so the BCM 5719 is an Ethernet
controller on one side you have PCI
Express you can get it as an adding card
you plug in your system and on the one
side you have four ethernet ports up to
four ethernet ports it's the 13th
generation of an incredibly long line of
chips that actually descends all the way
from the tyon Nick which was released by
Alon networks in
1997 and they haven't really changed a
lot of this Hardware since then they've
just been tinkering around the edges and
it is literally a direct descent from
this chip to do G EET that somewh
released in
1997 and somehow the IP ended up with
broadcom at some point so this chip the
5719 is particularly oriented to server
applications
and it has management style
functionality so there is something
called NCSI uh which is basically
something that allows the servers BMC to
share the host network connection and
piggyback on the host e for net ports so
if you're not familiar modern servers
have what is called a baseball
management controller and it's basically
an so implementing management features
like ipmi serial over Lan Etc any modern
server has this and yes it usually runs
Linux
this BMC has a network connection so you
can communicate with it and using a
whole separate port on your machine to
access the BMC might be rful so NCSI is
this protocol which allows BMC to be
attached to the network interface
control of the host and actually share
its network connection and piggy back on
the same ethernet ports if you've ever
used a server where you can just plug
into any old ethernet port and get it VI
ipmi this is
NCSI so the basic block diagram of the
5719 is that we have four ethernet ports
and we have four PCI functions on the
the other side and the PCI function is
basically a separate device so to The
Host this looks like four different
devices we also have a flash interface
where it can load some mysterious
firmware from an SPI flash chip and
what's interesting is in the middle here
between each port and each PCI function
these are then connected together but
they also have a MIP CPU in the middle
so for each Port we have this MIP CPU
running some kind of unspecified
firmware where there's kind of a
question of what's going on here but as
mentioned this is a server chip so it's
also intended to have this management
link where you can connect a BMC and
basically that's a parallel ethernet
interface which attaches into the
ethernet ship and it can actually pigger
back on all of these fors so you've got
traffic coming from both The BMC and the
host and then this is on BC's block
diagram there is this mysterious thing
called the ape and takes some kind of
firmware image but it's not clear what
it does presumably is some kind of CPU
but I didn't know it was just this
completely mysterious black box that
nobody seemed to know anything
about so I started by examining the
flash image layout I just took a dump of
the SBI Flash and I looked at what was
in there and I pretty quickly figured
out it's not hard to figure out out the
things that they've got sitting in that
flash you've got a header that points to
the various images you've got Mac
address is configuration that sort of
thing then you've got this firmware that
gets loaded onto those myips cores you
just saw so that's running per port and
then you've got things like option ROMs
so this is actually a code image that is
actually fed to the host where you boot
and that tells host how to do things
like pxe Boot and because this is
oriented for servers it actually
supports also issi boot um what's
interesting here is that we don't
actually care about most of this I mean
the power n system I'm trying to free up
here uh doesn't even use PX option ROMs
which is great because they're not
really great from a security perspective
so the only actual code here that needs
to be reversed is this myips firmware
and this mysterious AP
firmware so I looked into the myips I
started reversing this firmware um it
turns out each Port has an truly ancient
myip core it's roughly my three there's
no Hardware multiply or divide the
multiply and divide instructions don't
exist
um I fully reversed engineered the
firmware for this but it turns out that
these CES are almost vestigial I me it
is amazing how little they do that's
actually important they do do some
things they do things like they load the
Mac addresses from flash into the right
registers and they do a lot of register
setup for the ethernet
filings but that's about all and these
CES are actually left nowadays with
almost nothing to do in the original
days of the tigon back in 1997 I've
believed that these Calles were actually
handling data path so they were actually
spooning data from the network and back
but that's all since been moved into
hardware and these Calles are left with
almost nothing to do but a few very
miscellaneous things so what do they
still do they load Mac addresses they
set up fire registers but even a lot of
this code turns out not to be enabled in
practice and it's just vestigal and
after it does all that initialization it
enters an infinite Loop checking if
certain housekeeping tasks need to be
performed
what housekeeping
tasks well namely this so let me take
you through this this is from my
reversed code I converted all the
assembly into C so you can see these
first two function calls here they don't
actually really do anything basically
99% of the time you have an infinite
Loop and then you have a check to see if
we're on vain which means if we're on
the main power rail where the host is on
and if the host is off it's on standby
power and that's called VX uh now we
just call this function and then we loop
again so we're just calling this
function again and again and again if
the host is on so what does this
function do this function implements a
PCI Express capability this is a
standard in PCI Express that lets you
see if you are querying the host the
host is querying for the device's serial
number I have not making this up if you
have a BCM 5719 after the machine is on
the device spends all its time for each
Port checking if you are querying for
its serial number and that's four nip
CPUs one for each port and each of those
independently checks if you're querying
that specific port for its serial
number these Calles Implement random
drgs of functionality that nobody is
bued to move and elsewh I mean if you're
a broadcom engineer do you really want
to be the person tasked with moving VPD
loads into Hardware probably not um your
is anothera Castle so these myips cores
do some device initialization but they
are otherwise largely unused and there
is this mysterious ape block on the
device so is it a core is it something
else broadcom doesn't say there is this
AP code image on the flash there are
some references to NCSI in the code so
presumably there is some core which
implements NCSI so I needed to reverse
engineer this
code but the image is compressed with an
unknown algorithm and the m for this AP
CPU doesn't seem to be mapped in from
PCI Express or from the my scores so you
can't actually access its memory
directly so how do we get into this
thing and this brings me to the story of
how I ended up reverse engineering x86
real mode decompression
code problem
one I want to disassemble the AP code
but the AP code is compressed and I
can't recognize the compression
algorithm it doesn't seem to be a common
one but it is a really bad
one problem to
I want to gain access to the a memory
space but it's not mapped into PCI
Express or accessible indirectly you can
only access it from the AP core and the
only way to gain execution on the AP
core is by providing a valid firmware
image so this is a conundrum I want
access to the AP address space but it is
not mapped into the PCI Express space
the only way to get access to the
address space is to provide a
well-formed firmware
image formulating such an image would
require me to understand the compression
algorithm which would require me to dump
the ape boot ROM which would require me
to gain access to the ap's address space
which would require me to gain execution
on the ape which would require me to
provide a well formed firmware image
it's a circular
dependency so the problem is
decompression we need to decompress this
AP image in order to study it but we
don't have the decompression code the
boot ROM for the eight must have it but
we can't seem to find any way into the
in memory space so we have a circular
dependency so at this point I decided to
take a break and look somewhere
completely different the SPI flash also
contains things like PCI option ROM
images which I thought were of no
interest and they actually look as
though they're compressed with a
suspiciously similar algorithm and since
they're option ROMs they must be self
decompressing so they must contain the
decompression code so there's two pxe
images there's the Legacy PC bar B mode
which runs in real mode this is like the
original PC bios and there's the UEFI
bios which is modern protective mode
code and obviously I'd rather reverse
the latter but it turns out that the ufi
images are compressed over their own
special
algorithm which is in the standard so
it's compressed with a different
compression algorithm so the
decompression code I was looking for is
only really found in the x86 real mode
image oh no I
said so I had to reverse engineer this
x86 real mode code which to put it
shortly is not fun there's no
decompilers for it you can forget that
um so with X6 mode because it's in 16
bit addresses you actually need segment
registers to address more than 16 bits
and those segment registers are
constantly being changed and make
everything confusing to follow so if you
have a simple load for example you can
load for
um BX which contains a memory address
into the register Al but it's Rel
relative to the seg register es and the
compiler that broadcom used to generate
this code constantly changes the value
of Es as needed so it makes following
the code extremely confusing imagine
debugging like a normal program for
protected mode if the memory mapping has
kept changing and that's pretty much
what you're dealing with so I tried to
reverse engineer this in an ad hoc way
but that didn't really work the code I
ended up with had bugs and that led to
changes in the de decompressed code
where the it L corrupted some of the
code that I was trying to disassemble
which explained a lot of baffling
disassembler output so I realized I
needed a way of reversing this code
which cannot introduce bugs because
otherwise I would just go on a you know
wild goose chase where I'd spend days
trying to figure out why this
disassembly doesn't make any sense and
then I realize that's not the
disassembly you've decompressed this
image wrong it's corrupted
output so then I came up with this idea
to have a a methodical approach that
can't introduce bugs
and this was where I had the idea that I
should construct an x86 real mode
emulator in
C emulator is too big a
word every line of x86 real mode
assembly was turned into a c comment
like
this and uh I then constructed an
emulation environment for real mode code
in
C I say emulation and what I actually
mean is
it's just a global variable I have a
global variable which is the current
value of the es segment selector in x86
mu mode and I have Global variables
through each of the x86 General
registers and then I just have a
function which sets s and then I just
have a function which does an es
relative
load and then beneath each equivalent
line of beneath each line of assembly I
then placed an equivalent line of C
beneath it so you just have it it's
perfectly simple like this we have a
move to the es segment selector so we're
setting es and it just translates to
this line in C targeting my
emulator uh we have a move from one
General register to another and that's
literally just an assignment of This
Global variable we have an increment
operation and then we have an es
relative load where it's saying load
into Al from this memory just BX which
is relative to the current value of the
es segment selector so every single line
of assembly just translates into some
really simple line of
C and the idea is that every conversion
from assembly to C every line has a
trivial and obviously correct
translation into this x86 C dialect
below it and the great thing is you can
keep eyeballing this to check if the
correspondence is correct if something
is wrong and it doesn't match up it
stands out immediately there's no
structured control for here it was all
go-tos no Loops uh and this successfully
compiled as a Linux binary execute
and it fully decompressed the AP
firmware for the first
[Applause]
[Music]
time so at that time I was left with
this really awful C code and I actually
wanted to understand the algorithm so I
recorded the output of this
decompression program and I then
modified it to raise it into more
readable C so I converted goto into if
for while Etc and after the smallest
change I would just rerun it and see if
the output had changed so any mistakes
in doing that raising process to get it
into actually readable see were
immediately caught and those did happen
and eventually with this methodical
approach I actually got a completely
readable clean C description of that
decompression
algorithm so I already had the firmware
decompressed at this point but
interesting point I was at this point I
was actually able to determine the
nature of the decompression algorithm
it's really simple dictionary
compression the only state is a 2K bite
dictionary buffer you just have literal
values and then you have references to
this dictionary
buffer um I was able to determine the
identity of the confession algorithm it
is called
lzss and it has a Wikipedia
article it has a very interesting
statement on it most implementations
stem from a public domain 1989 code by
hariko
Okumura there's a little citation Link
at the end
there sim. net
mirror implementation of
1989 the decompression code was linked
from Wikipedia the entire
time it was posted to a BBS in
1989
and it's public domain code the
decompression code fits on one screen it
is actually a really simple algorithm
the broad conver version does change
some constant a bit but it's basically
the same algorithm your broadcom option
ROM for your network interface
controller is compressed using some dos
compression code someone posted to a BPS
in
1989 if I had known the code was lying
on an internet archive of 1989 BPS
postings of all places the entire time I
could have saved a lot of pain the only
thing better Hur than reverse
engineering Realo code is finding out
you didn't have to
but I couldn't have figured out which
algorithm it was without going through
that process especially since they
changed some of the constants so once
again it's locked doors and circular
dependencies it's a bit like Alice in
Wonderland um so finally I get this code
decompressed so how do I penetrate the
ape and actually break into this
inscrutable Black Box I start
disassembling the code now I've got it
decompressed and it actually turns out
to be an arm cortex M and this was
actually the first time I actually knew
what CPU was I had no idea what it was
until this point where I finally found
out it's just a cordex and Fleet it's a
common Fredy dobit micro microcontroller
like something you'd get on an SDM
fredy2 that sort of thing and the ape
has I peripherals and registers that are
not mapped into pcia these are
completely reserved to the ape and
special I want to get access to
these so because I know how to
decompress this image now I could write
an tool to build an image and Flash it
but the image headers were still a bit
mysterious to me and wanted a quick
weigh in that didn't require me to flash
an image to the SPI flash fently I
wanted to be able to boot from Ram
because that's obviously a more
convenient for development
purposes and at this point I suddenly
discover that the AP firmware image
appears to have an RSA signature at the
end oh no is my reaction oh no no no no
no I suddenly get very depressed um I
only really continued at this point
after some encouragement from others
that really made it clear to me how much
people value this work um I honestly
have them to thank for this um people
tell me you know maybe they aren't
checking the signature maybe they just
completely ignore this signature that
they put there for some reason I'm
skeptical um have I mentioned that
reverse engineering is an emotional
roller coaster more on this
later so I looked for ways to get Shell
Code execution on the using the existing
code um there's lots of shed for
communicating with the host there's like
this IPC mechanism based around shared
memory we have a mailbox style mechanism
where there's this SRAM share of the
host and the AP that is used to send
commands to the
ape one of these commands allowed
scratch Pad read and scratchpad write
well that sounds interesting turns out
it lets you read and write from any area
well not any area in the AP Ram because
it's balance checked except that their
balance checks include an area of the
Apes code what so all I had to do is
send scratch Pad right commands to
upload shell code to a certain code
region that is within the bounds that
are checked and I send a command which
happens to cause the ape to jump into
that code region and I have Shell Code
execution um if anyone is wondering I
wouldn't really consider this runability
because you can only do this from the
host and the host is trusted anyway um
the host can actually reflash the
firmware on the SBI flash anyway so all
this does is save you writing a new
image to flash so it's it's useful to
debugging and if anyone's curious um I
didn't find any remotely exploitable
vulnerabilities in the code during this
entire
project so I then wrote some Shell Code
for the AP um I implemented a shared
memory mailbox IPC mechanism which is
very similar to that used by the AP
firmware itself and it just allows me to
arbitrarily load any memory address
store any memory address or or jump to
any address I like I also added a hard
fault Handler to this so if you try and
load from a mem address that doesn't
exist on this ape um the CPU will crash
so I added a hard hander so that if I
try and access some area of memory that
doesn't exist I just get a arrow back
and my shell code doesn't crash and this
means I can now access all of the AP
private memory space which is great this
was the first time I actually got to
look at the AP memory space and because
of this I was finally able to dump the
AP boot ROM this was the first time I'd
ever managed to get access to it and as
a result of this I got a much nicer arm
copy of the decompression code if only I
found a way to get to this copy first
but again circular
dependencies at this point I was able to
determ the image header format and it
turns out that the boot ROM actually has
a way to boot name image from SRAM
instead of flash which is also quite
useful also the AP boot ROM does not not
check the RSA signature on the AP Co
image why is it
here just just goes to tell you
sometimes they add a signature for no
reason at this point I knew how to get
execution on the AP and form valued
firmware images um so I just started
writing a lot of tooling I wrote build
tooling that allows building these
Custom Image format in C so I could just
write code in C and it would be built to
this custom image format I needed the
compression function as well as the
decompression code I'd already obtained
through that excruciating process so I
just used the original BBS code for the
de compression code I also wrote debug
tooling so that I could probe da
registers using my shell code boot
images from memory
Etc so I comprehensively reverse
engineered the ape code and I developed
an understanding of the ape special
peripherals the registers were
completely unknown there was no
documentation for this broadcom
publishes a register manual for the PCI
he registers so that you can write
drivers for the device so that was a big
help when I was looking at the myip side
of things but for this ape everything
was completely undocumented um vcom does
have a diagnostic tool which did provide
some information on some registers which
was lucky um basically it had code to
read some registers and print diagnostic
output where you had textual output
saying this bit is set so that managed
to serve as a kind of Rosetta Stone
telling me what registers he want and
what the names of different registers
were but others simply had to be guessed
um so the ape has a few special
peripherals it's got peripheral to talk
to The BMC with this RMI interface so
this is a standard interface between
neet Mac and a fi it's a parallel
interface and usually you that talks
between the neet mac and the F but for
NCSI it's also used to actually connect
the BMC to the network controller so it
can pick you back on that network
connection and access the internet um it
also has a peripheral to talk to each
Network ports so it has one to transmit
to each Network Port and one to receive
from each Network
Port but you actually don't want this AP
to receive all the traffic from the
network because it's just a cortex M fee
it is not powerful so the ape does not
receive everything from the network you
have some management filters which are
also implemented in Hardware which
configure what actually gets redirected
to the AP from the network there are
also SM bus and U peripherals on this
chip um they are not really wired up or
used on the TS power 9 systems so I did
not investigate them they are mainly for
quite boring management functionality
which isn't really of interest to us and
of course the ape can also access the
rest of the chip really it can access
all of the main device
registers so previously I showed you
this block diagram where we have this
mysterious
ape and this is what it actually looks
like you can see we've got this cortex
M3 CPU we've got this Mac which allows
us to connect to a BMC and that's a
pheripheral attached to the ape we have
these SM bus and you things which aren
as much interest to us we have these
Network transmission and reception
devices which allow us to communicate
with each Network
Port and we have this allows us to
piggyback on each Network port and for
the reception side this is all
controlled by some RX
filters so what does the ape actually do
we know the purpose of the ape is to let
us attach a BMC to our network
controller and share the network
connection what the a basically spends
all of its time doing is it just reads
ethernet frames from the peripheral that
connects it to The BMC it is a really
simple fiveo interface it's literally
just one register and each time you read
the register you just get 32 bits of an
Ethernet frame so it's a really simple
uart style F interface it's basically M
copying out of a fifo so the performance
of this isn't great you're never going
to get gigabit out of this NCSI link um
so basically all the AP does is it just
reads M copies frames out of this fifo
and then it writes it to a specific Port
specific SRAM which is basically the
actual memory of the port itself it's
actually write only memory so it goes
directly into the uh a memory mapped
area which maps to the actual Network
Port and then you set some registers to
get it sent out and on the reception
side it's the exact same thing in
Reverse there's a special reason of read
only Port
memory which just gives you access to
the frames as they come in you get an
interrupt and you copy them into this
little fifo and they get sent out of The
BMC and the AP basically just does this
over and over again so it's basically a
CPU just doing M copies of ethernet
frames from one device to another and
back and
forth so if the a were malicious if it's
firw to get compromised what could it
do well we can set management filters to
do matches on packet headers to
determine if they get forwarded to the
ape now for simple unicast traffic that
can actually set a MAC address that says
anything for this Mac address just come
to me but there's some traffic that
doesn't match up with that that is
broadcast or multicast and that needs to
be an exception and that's what the
management filters are for so the ape
sets up some management filters that
says if there's a certain field in the
IP header or UDP header or so on Direct
this to the ape and this is used direct
things like broadcasts multicasts DHCP R
IP version 6 nebor discovery that sort
of thing um these management filters
actually let you choose it lets the ape
choose whether the host also gets a copy
or whether the selected traffic only
goes to the
ape so the ape can e drop on traffic
from the network it can send traffic to
the network it can prevent the host from
seeing traffic from the network and it
can man in the middle traffic between
the network and The BMC see and that's
actually quite alarming if you think
about it for the moment
because how many servers with bmc's come
with default passwords if your BMC is
compromised if your network controller
is compromised well it can just log into
your server with a default password and
then it owns the box but you know
vendors are getting better nowadays
they're starting to stop shipping
servers with default passwords but even
if box doesn't have a default password
it could just man in the middle
SSH now you can check the SSH host key
but how many server vendors do you know
ship the SSH host keys for your BMC with
the machine and how many people when
they're connecting to a BMC of a server
for the first time VI a network
connection that is literally coming out
of your laptop and going into a server
right in front of you how many people
are going to shake that s host key if
they even can so in practice Yeah if you
get a server if you buy a server and the
AP on it has been compromised it could
own the box so this is quite
interesting um finally I'd just like to
leave you with some of the most
ridiculous skeletons that I
found in all of this firmware um there's
various things that that were discovered
in this fmw that are just nuts um but
the worst one of all is something that I
like to call the great broadcom bitbang
you will not believe what it is um so I
we were able to build our own
replacement firmware for the ape and get
NCSI
working we had working NCSI with open
firmware but it only worked after the
machine had been turned on once so part
of the point of NCSI is you can access
the BMC when the machine is off and if
you turned the machine on and then we
turned it off again you could do that
but if you literally unplugged it and
plugged it back in you wouldn't be able
to talk to BMC until you turn it off and
on
again so the implication is that there's
some kind of device initialization that
was being done when the machine was
turned on that we were missing and once
you turned the machine on from cold when
you plugged in for the first time that
initialization hadn't been done so you
couldn't actually talk to
it this code was found in the reverse
date firmware and it was pretty weird
now let me just simplify this for you by
converting it into C
code so this is what this code is
doing it's reading the chip ID register
and it's only doing things if that chip
ID register reads
zero well that's really strange because
the chip ID register is literally a
readon register that says this is the
BCM 5719 Hardware revision A1 it's
literally readon it's burnt into silicon
it should never be reading a
zero um yet there's this code here that
is saying well if this reads a zero do
this and then it starts twiddling with
these gpios on the chip which are
reserved to the ape
and then it's waiting for that register
to not read zero so this is really
strange and have't mentioned this is why
it's really nice to have a server board
with
schematics so let's take a look at the
schematics I don't know how many you can
see but on the right there that's our
Network chip that's the BCM 5719 that's
the thing we're trying to reverse
engineer and then on the left we have
this mysterious
chip
so you can see the the AP gpios that are
getting toggled on by this code are on
the left here and then you have some
more pins that run into the BCM
5719 on the right here so what are these
pins on the right they are the PCI
reference clock pins so PCI Express is a
differential interface but it's
multi-lane as you know
so in order to synchronize the clock on
all of those Lanes you have a reference
clock Lane which has a clock signal
which is then used as a timing signal
for all of the other lanes on a PCI
device so if you install a video card in
your PC you've got all these 16 Lanes
but you've also got a reference clock
and you've got these lines running into
the cbcm 5719 EET controller but
ordinarily that would come from the host
CPU and instead it's coming from this
weird chip
here um which is called a cbtl
01023 what is that
it's a differential Channel 2 to1
Multiplex Multiplex switch for PCI
Express gen
she well that's
strange so we have these gpios coming
out of the ape which are controlled by
the ape CPU and going into this
multiplexor chip this analog Marx
chip and then they're coming out of
there and going to the reference clock
Lanes of the same
chip and ordinarily when the host is
operating you'd expect a reference clock
to come from the host and go straight to
the device's reference clock
pins and this is all controlled by
another gpio controlled by the AP gpio 2
so basically this is an analog Marx chip
you know it's like it's literally it's
either connecting these electrically to
one input or another and this is
specifically designed to do this for
pcie
Lanes
so this version register it should never
read
zero if it's reading zero
something is wrong because it's a
hardcoded constant so if it's reading
zero that means that half the chip isn't
even up yet it's not awake so this is
some kind of code which is's trying to
fix this and get the rest of the chip to
come
up it turned out broadcom hardcoded the
chip so that the non-ape parts only come
online after several dozen PCI Express
reference clock Cycles because the host
PCI Express interface will always be
there right unless you someday want to
add this functionality to talk to a BM
MC when the host is
off rather than actually fix this
problem they mandated that every
customer put an anlog PCI Express Mark
chip on their board so that the AP fmer
can manually bitbang clock Cycles on the
reference clock interface of the same
chip to get the rest of it to come up
this is not a joke this is
[Applause]
[Music]
[Applause]
broadcom
I'm not kidding if you have a server
with a 5719 and it's still quite common
if you for some reason order a server
with only gab ethernet it's still
shipped there if you order a server with
a BCM 5719 and NCSI your server vendor
has been forced to add an extra PCI
Express Marks ship to the bill of
materials because Worldcom ship decided
to ship this as a design this is their
reference
design
oh so what was the aftermath of this
project um I ended up calling the
project
AA since all of this is based on reverse
engineering propriety code I was tainted
by the process um from the copyright
perspective this all came from
proprietary firmware that I reverse
engineered and there was a desire to
produce firmware that had a clean
license that we could say this is 100%
legal it is genuinely open source
firmware we hold the copyright we have
licensed this
legitimately um especially since this is
this is shipping an actual commercial
prod works now so all of the gathered
knowledge was used to produce
documentation on how to write
replacement firmware um you may Noti
this is clean when reverse engineering
and the most famous case of clean
reverse engineering is of course when a
compact reverse
engineered the BIOS code to the IBM PC
so that they could make a clone and
literally every machine every laptop you
use now is thanks to the fact that this
clean room reverse engineering technique
is actually legal that's why the PC is
the thing so because I was tainted by
the process I couldn't go and write the
replacement
firware so the replacement open source
firmare was brilliantly written by aan
lusi also known as meot on IC uh he
couldn't be here today but he wrote
replacement firmware in C++ using the
documentation I provided on how to
program this chip all of the registers
Etc and this firmware now ships on all
newly ordered TS 2 and Blackbird power n
systems making these systems finally
100% open- Source firmware mission
[Applause]
[Music]
[Applause]
[Music]
[Applause]
accomplished it is also distributed by
the Linux vendor firmware service you
may know as the firmware update tool and
it's not just for the TS right you can
also install it on a BCM 5719 PCI
Express card for your PC you can get
those on eBay pretty cheaply so if you
just want a PCI Express card that has
100% open firmware for gigabit Ethernet
you can just buy one of those or you can
um if you want to Tinker with it for
example and actually play around with
all of this uh firmware and also please
big give a big thanks to me L for making
open source firmware for the BCM 5719
[Applause]
reality and I mentioned earlier that
engineering is kind of an emotional
roller coaster um it is really
interesting that aspect of it because um
sometimes you think I'm never going to
figure this out and it just seems
hopeless as I mentioned the AP for
example all of the registers for that
were unknown I mean completely unknown
you had a million registers with unknown
names unknown values and all of the code
is unknown and the thing about reverse
engineering is eventually you figure
something out you figure something out
and it lets you figure other things out
and then the discovery snowball and it's
like this Avalanche effect and then the
Avalanche ends and you're stuck again so
there is this constant oscillation in
reverse engineering as a field between
exhilaration and a sense of
impossibility it's the most exciting
thing ever and then you feel it's
completely fugile and again and again
and certainly my experience I think that
successful reverse engineering requires
managing these emotions and persevering
it's weirdly like doing a 100
dimensional crossword with a million
rows and a million columns I mean it is
like that you figure out one and it lets
you figure out other things um I didn't
even consider myself good at reverse
engineering before starting this project
which seems a bit ridiculous statement
to make now I just took a quick look at
this th image to see how insurable it
was and it seemed surprisingly scrutable
surprisingly easy to understand at first
and then I got sucked in and suddenly
had to decompress the eight which was
much harder I you know I never could
have imagined getting to this point and
honestly the fact that I managed to pull
this all pull this off ultimately is um
really amazing to me but I have a lot of
curiosity which I I definitely think is
a really powerful motivator and you know
I would be really interested to know
what motivates others to do reverse
engineering because I think everyone
that gets into reverse engineering
something like this that they have their
own very personal motivation for
it and finally um please do come and
join the community and get involved and
ask questions um we have an Channel
where all of this work was
coordinated um the open source firmware
is available for you to download and use
however you like you can read the
documentation I produced on how to
program this devices based on all my
reverse engineering which is how that
open source fir is produced and you can
absolutely get in touch with me and ask
any questions about this and I'd also
like to say that um you know now or in
years and years time if anyone needs to
know anything about this device um I
prob the best best equipped person to
answer that question outside of broadcom
so I'm always as able to answer
questions now or in the future if anyone
needs to know something about this chip
um finally I'd like to thank IBM for
open sourcing the parine firmware
because honestly without that there
would have been no point to this and I
still think that is an absolutely
stunning act I'd like to thank Timothy
harison at rap and Engineering for
creating the TS 2 the system with 99%
open source firmware and providing
Hardware resources that helped in this
effort
um and I'd like to thank metl EV lki for
creating the open source replacement
firmware and for the tals community also
for the support and encouragement
without which this project could never
have been possible you know I was ready
to give up at various points and you
know it was absolutely because it was
made clear to me how much this mattered
to people and how people were relying on
me for this and people really wanted to
see this happen is why this project was
ultimately able to come through and be
completed and why I was able to
Ely achieve success with this and
actually complete this project which
certainly is an outcome that is stunning
to me and finally thank
[Applause]
[Music]
you the ups and downs of for were
reversing thanks yugu for this talk we
are having some time for questions line
up the at the microphone and there is
already a question from the internet so
please yes correct there is a question
two questions actually from the internet
first one would be without Nic piggyback
mode one should be safe from the manin
the-middle attacks from the ape or
not so the thing to bear in mind here is
that the firmware is all loaded from an
SBI flash on the main board and if you
control that SBI flash you're safe right
so you can audit that you can you could
attach pin you could attach probes to it
if you like and make sure that the
firmware on that is good you can also
audit the contents of that flash from
the host and there's no way for the
firmware on the device to interfere with
that so you can essentially audit this
firmware automatically from the host if
you want but if yes if if you did want
to if you were to be concerned about
what the ape could do if you don't need
that management functionality there's
nothing theoretically stopping you from
running without the ape but um you know
obviously if you're worried that
something on the SBI flash could get
changed um by an attacker maybe someone
with physical access to a machine they
could just reactivate the ape so that's
not a great
mitigation um what I would say is that
during all of this reverse engineering I
did not find anything that would allow
remote exploitation so while e ape does
mem copy data from the network back and
forth it copies these EET frames it
never truly looks at what is in an
Ethernet frame that comes from the
network and the myip CES weren't
involved in the data path at all so you
don't really need to be worried that
there is any exploitable vulnerability
in terms of the AP missp passing some
frame from the network it doesn't really
look at anything that comes from the
network it just copies it blind into the
the the the
BMC
okay there's also a second question yeah
second question would be uh would the
new open- source
firmware um allow the implementation of
some sort of custom protocol off loaders
for VPN for example or is um the ape not
powerful enough to do this in terms of
CPU Cycles second it's um it's a cortex
M3 so if you get like a 32-bit
microcontroller that's the sort of core
we're dealing with here so it's not
really powerful um as I said it does
this num copy in order to connect the AP
The BMC to the network and it's not fast
it's it's far beneath gigabits so it's
not something that you can really do
powerful processing with um it's not
something you can really do performant
encryption with um one thing that was
raised earlier by someone I spoke to was
uh the idea that you could actually foro
the use of a BMC and just do some
management functions on the
AP that that that is probably possible
um so long as they don't need to be
computationally intensive so I mean
there's some possibility for doing novel
things there but any powerful processing
like vpn's encryption anything that
requires line right handling of gigabit
you forget about
it all right thanks yeah microphone
number one please so as an
implementation of the arm v7m
specification V M3 core should be
capable of self-hosted debugging did you
take advantage of that to uh run a debug
step on the m fre cor and the debug
exception that is actually a really
interesting uh Point um I played around
with the envic on the cortex M3 a little
but I did not look at the debugging
registers or whether there is any kind
of JTAG functionality that might be of
used to me because I didn't really have
the hardware set up for playing with jte
or debugging of that kind so I never
really tried using any of the debug
facilities of the cortex M it's all
based on Shell Code and the tooling I
developed that is based purely on
software I don't know whether that
functionality is built into the Silicon
you could probably find out quite easily
because I believe there's like uh
version registers on Armor that tell you
whether those stats Sil is built without
functionality so you could probably find
out pretty quickly if you started
wanting to play with the debug tooling
to Pro some of the registers that we
have in the open source firmware but I
don't
know okay thanks next one on microphone
one um thanks for the talk um I would
like to know do you have any tips or
recommendation on provided a proprietary
firmware um how did you find out or I
guess guess a little the structure of
the firmware like the layout oh um so as
I said the first thing I did was just
looking at the image dump of the entire
flash um so you just do a hex dump and
you just start looking at it and seeing
um what does it look like you I run
tools like binwalk on the the f which
just looks for common signs of different
CPU instruction sets and it said yeah
there's some myips code here it didn't
find any arm code because that was all
compressed but it seemed to match up
with what we knew about this device
which was that they had this um this
myip core on this m firmware I there
were other parts that were pretty
obvious where you run binwalk and it
says oh this is x86 code and you look at
that area of the fode in um just a hex
editor and it's pretty obviously a px
option WR and you just start getting an
understanding of what the different
areas of flash do and as I said you it
doesn't take that long in order to get
to a place
where you get you get the uh the image
layout so if you look
at uh yeah so yeah you've got the image
layout
here and a lot of it is just eyeballing
really where you see little strings
there here and there or you see the
device back address there and it's
becomes pretty obvious what the
different regions do it is it is like
doing a cross word and you do sort of
get better real experience but it's
basically just finding something that
seems to make sense and then that helps
you find other things I hope that
helps okay thanks uh microphone one is
very very popular so just go ahead um
thanks for the talk um so tying back to
an earlier question um on offloading
stuff on to the hardware MH um so now
that we control the firmware on the ape
can um do we gain more uh access to more
features that aren't actually available
to the um to the driver so in terms of
using the peripherals if you can think
of some useful thing to put on the ape
that uses those same peripherals you can
probably do it but because this AP was
designed to implement a feature that
Brom specifically wanted namely NCSI
you're not going to find much
functionality there that is helpful
hardware wise to do things that aren't
required by that feature it is pretty
targeted but if you wanted to do like
some no novel management feature that
reuses the same ferals in principle you
could do that but the ferals are pretty
specific to what you would need to
implement
NCSI
thanks yeah and the next one right so
you mentioned there was a cryptographic
signature but it wasn't checked yes um
would there now be the possibility of
and would there be a point in um
creating signed firmware with uh with a
custom key and insert that key into the
boot loader so what what's going on
there is that the boot loader is a mask
it's hardcoded into silicon and it
happens not to check that so not really
because we can't change the mask like
that is broadcom silicon which is just
hardcoded to load an image in this way
um if you want wanted to get more
control over the firmware image and make
sure that it hasn't been changed um I
think the best way to do that would be
to write some program to audit the
firmware from the host because as I said
the host has the ability to reflect the
firmware and that's all Hardware
registers so the firmware can't actually
interfere with that process so if you
wanted you can just read out the
firmware image from the host and get an
accurate reading of what is on that
flash and that would allow you to audit
what is going on there all right thanks
and one last question from the internet
yeah the internet wants to know whether
you also checked or compared your
findings with other broadcom uh network
interface cards um I haven't um I know
that things have changed a lot with the
faster network interface controllers um
like the net Extreme 2 which does 10g
and above um I am certain that those are
actually enforcing signature checks now
um so I do not believe that this work
would be viable for those um that seems
to be very much the standard for the
network interface controllers for 10g
and fter if they've got some kind of
firm where they're doing some kind of
signature check on that I believe
broadcom does that I believe um Intel
does that and it's not really hard to
check if you just look at like their
onepage marketing brief for a chip and
they say it has secure boot you can
pretty much guess that it's PR checking
signatures in the hardware
um the other thing I would say is that
yeah I mean the 10g chips are very
different I believe that I believe I
suspect that peoplec has stopped using
the the tigon IP that descends from this
chip that began in 1997 with Alon
networks I'm not sure that they took
that for with the 10 gigb parts I think
they may have finally scrapped all of
that Legacy here but um for this 1
gigabit chip which is basically the chip
will program will send you if you're
looking for a one gigabit chip for a
server
this is the last remaining to send I
believe of the tig
online okay thank you everybody for your
questions thank you Hugo Hugo for your
talk um
[Applause]
[Music]
[Applause]
[Music]
please
[Music]