Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1115 Thanks!
[Music]
ladies and gentlemen on this very last
day of this
conference I have the honor and the
pleasure to bring to you two guys here
Jeff Deo and Andy
olon who are going to take care about
the lecture called kernel tracing with e
BPF so for the ones who don't understand
this extended Berkeley pcket filter and
what we're going to do is actually
introduce you to the functionality of it
uh of the Linux kernel huh so they will
cover some uh practical uses of the
technology Beyond merod profiling and
for me most interesting is actually that
um they're going to play a little bit
Gods here on stage is that only only a
little mad with power a little bit okay
okay um I don't know what kind of God
maybe the spaghetti monster will be
around please stage is yours give them a
warm welcome thank
you
welcome uh I'm Jeff this is Andy we're
Security Consultants at NCC group so we
hack things for living and find bugs and
tell people how to fix them and stuff
and one of the things that we've been
playing around with recently is this
ebpf stuff so uh ebf is extended BPF but
what what is BPF so BPF for those of you
who don't know although maybe everyone
does know is Berkeley packet filter
which is a bite code and VM uh designed
for being run in a kernel uh to process
Network pack uh Network packets really
really fast and do
filtering ebpf is this sort of
completely unrelated language that was
designed in the Linux kernel uh as its
own thing and nearly kind of took the
name uh it's designed to be jitted uh
with sort of a direct mapping to x86 64
and other modern uh CPU architectures um
it's sort of designed around transpiling
a sort of restricted C limited line
language subset into this code so it can
run very fast and it's being applied to
anything and everything in the kernel
that might benefit from having
programmatic logic being applied to it
you interact with it through this BPF
sis call and uh basically you send it
some code you send it some other
metadata about what you want it to be
doing and if it likes it it gives you a
file descriptor if it doesn't it gives
you an error so basically the real power
of this stuff is what you can apply it
to um so what what does it look like um
well this is one form of it uh this is a
horrific form of it a macro hell and C
uh that basically no one should ever be
writing this um but basic so what is BPF
ebpf um it's basically a BPF
implementation that just threw every all
the rules aside added a bunch of
registers added a bunch of functionality
to call arbitrary sort of Kernel
functions that are registered into it
and it's just applied to all sorts of
things it has a bite code verifier that
we're going to henceforth refer to as
the validator because it's not really
good at verifying nor validating for
that matter but it it it does um and the
main functionality of ebf really is the
helper functions that are exposed to it
depending on the context of what you're
doing so uh you know your socket
filtering ebpf isn't going to be able to
read arbitrary kernel memory but your KR
ebpf is so why why ebpf high performance
inlane packet processing SA safe sorry
safe high performance inl packet
processing with network tunneling and
custom IP tables rules and S safe high
performance in kernal programmatic
operations with CIS call filtering to
reduce the need for buggy kernel modules
or all kernel modules with a firewall
Subs okay wait what uh so as ebpf has
has sort of gained all of these features
the why keeps changing and people find
new reasons to like it
um so what why really so it depends what
you're doing um we like to hook all of
the things especially the Linux kernel
so this gives us the ability to do that
without risking crashing the kernel
which happens when you start writing
kernel modules in C uh the talk is about
kernel tracing after all so EF is
interesting us because it has a
potential to give dra trce which is sort
of the dynamic instrumentation framework
for Solaris and FreeBSD and Max maos a
good run for its money maybe not in the
power of the things that it provides you
metadata for as sources of events which
Linux is probably not going to get to
that level of unified design anytime
soon um but it's more programmatic and
focused on lower level operations and
taking like C code and compiling it to
this and then using the fact that you've
basically written C code to operate on C
code in the kernel so drace is focused
on like oneoff human-based command line
types of operations but ebpf enables you
to do all sorts of crazy wacky things so
let's talk a little bit about
tracing um tracing is basically very
fancy logging um of program
execution um that doesn't really mean
much for us we don't really care about
the logging so much we care about you
know getting at things and hooking
things and making them do things and
observing things so we care about
Dynamic tracing so what is dynamic
tracing
um there are two main kinds there's the
kind where you enable and disable
existing logging functionality which we
don't really care about and then there's
the kind where you add arbitrary
functionality that wasn't there before
uh we care about the ladder but we also
still don't care about the word logging
we care about Dynamic
instrumentation um so two main kinds of
dynamic instrumentation depending on
your perspective there's things like
function hooking and things like
instruction instrumentation and
depending on what you're doing and what
you're targeting uh for or your hooking
maybe your hooking is actually
implemented with instruction
instrumentation and you know ebpf is
basically like that so let's the the
alternate title of this talk is
instrumenting Linux with ebpf for Fun
and Profit a title that would definitely
not get accepted in 2018 had we
submitted with
this so so let's let's go into history
of Linux just a little bit um on its
tracing Technologies so the most
important thing is K probes which are
over a decade old um but they've gotten
faster recently and basically nowadays
uh they allow you to hook a spec any
instruction in the kernel really but if
you try to uh hook a function at its
entry it'll have extra logic around
seeing it to completion on a return and
so it'll try to analyze it to determine
whether or not it can do the fast path
and if for whatever reason it can't do a
fast path of finding all the exits um it
will essentially just use a breakpoint
in single step through which is slow and
you don't want to do that FR trce is a
basically file system API for adding
things like K probes into the kernel and
and doing all sorts of other stuff it's
not super important um from our
perspective but it is used as part of
EPF per events is a whole bunch of crazy
profiling stuff but one of the key
features is a very fast ring buffer for
copying uh data from the kernel to user
space very quickly uh Trace points are
essentially that former kind of dynamic
uh tracing where the functionality was
already there and U probes are
essentially K probes for user space
memory any ebpf is this fancy combining
robot that plugs all of the things
together so ebpf basically integrates
with all these different kernel
Technologies and but the core concept
when using it for tracing is that you
have your ebpf program and you plug it
in to some sort of data source using one
of two apis
currently um then you use something like
the perents ring buffer or a memory
mapped ebpf map as your output uh to to
user space the ladder actually can take
uh is a wi directional mapping between
user space and and kernel space so the
user space can actually update that and
send input to the
kernel so the sources that we're working
on generally are your K probes your U
probes your Trace points and raw Trace
points which are basically Trace points
um the way that this works is that you
make a whole bunch of crazy CIS calls
and chain them together so you start
with your BPF CIS call to actually make
your program uh then you use the F trce
API in CFS to basically make a k probe
and then get an ID for it
then you get a perf event by calling
perf Event open and you pass it in the
the value that you got out the ID of the
K probe and then after that you've made
that you attach the BPF program to it
and then you enable it uh that's the old
format in more recent kernels there's a
slightly different form where you can
skip the um the F Trace Trace FS
entirely but in practice it still ends
up getting used because this magic
number of six on the slide here is um
actually generally gotten through it but
you don't need that so everything else
basically follows the same apis except
for the raw Trace points which El leads
uh the perf open
entirely so how not to use EPF is to do
is to not is you know how how system D
is apparently using it which is by
assembly by hand um it's really hard
it's basically impossible to do anything
complicated or fancy and it's also
highly errone I'd be very surprised if
that code was correct so what what
should you do you should use this thing
called BCC um it's basically a framework
for compiling C to BPF or ebpf and then
hooking it up to all of those fancy
sources uh this talk I'd like to clarify
is not really about BCC but we basically
end up using it for everything because
it's essentially the only consumer of
the kernel API which is also the case
because the people developing BCC are
also the people writing the kernel code
for the the BPF so essentially they're
writing their own apis for them which
are otherwise essentially completely
undocumented um and where they are
documented it's completely lacking and
none of that multi call multi-is call
stuff is really documented anywhere I
got that from reverse engineering how
this stuff works um so BPF BCC is the
only real option for for doing this
hooking so how do you write a tracer
with BCC well generally you're going to
use the python API uh that it provides
and you're going to unfortunately have
to write it as a single python file so
the bigger this code gets the more
complicated and harder it is to to write
it um and your C code you're generally
going to store in a python string uh and
then the structure the this is a sort of
a big ultimate structure but the really
important thing here is you have your
string you tell BCC to compile it to BPF
and you tell it to register it to events
like K probes and and your maps for Io
and and event uh Handler call back
functions and and everything just works
um the single python file is actually
because of a weird limitation in it um
that probably will get fixed at some
point but right now it's sort of
annoying to get around it so let's write
some
code um this is what a very simple ebpf
K probe looks like when using BCC uh BCC
has a lot of code generation going on
behind the scenes so this fancy K probe
uncore uncore syntax basically lets it
know that there's a cap that you want to
use uh a c Probe on CIS open which is
the kernel side for the open CIS call um
this the name of this changes between
kernel versions at least recently there
was a change it has a slightly different
syntax now um but it works the same way
so we hook this thing and then we just
call a print K which is sort of similar
to your generic print K if you've ever
done kernel module development and then
we run
it and we get nothing why do we get
nothing oh yeah
GBC piece of crap um so GBC has for the
past while uh basically made all of your
attempts in code to call open actually
go to the open at CIS call uh instead so
we need to actually hook open at and so
once we do that we start seeing all of
the events all over the
system but let's generalize the code a
bit because why should we have two k
probes when we really just want one
piece of functionality because as it
turns out both open and open act go to
the same place do CIS open which is the
underlying implementation in the kernel
for them and so we can now hook them at
once and in the print K we now put a
percent s and we'll try and print out
what the path name is and we start
seeing all these random procfs things
and system D Journal D is doing all
sorts of wacky stuff and system D2 it's
really scary stuff but print K is the
trace print K is actually considered
harmful and the reason for this is that
it's a lot like f trce and that there's
one log buffer shared across the whole
system which means that messages from
different active tracers are kind of
running into each other and no one knows
belong what belongs to who um your ebpf
programs also have a race condition in
them uh with this because uh the way
that ebpf is safe uh is that it's
anchored to the executing process that
has created the object um or maintains
control of the file descriptor and so
essentially when your last remaining
process with a handle and that FD dies
um the ebpf gets unloaded and then it
unloads from the the K probes that it
may be traced attached to so there's a
weird race condition ition because you
terminate but if your K probe is still
registered in hits then it runs the code
it tries to log but your K probe then
gets detach afterward but there was also
no client to receive the message the
messages just stick around until someone
decides to read them so the next person
next process to try and trace this stuff
doing print K is immediately going to
cease messages from other people so
that's a problem uh so instead what you
have to do is you have to rewrite the
whole thing to use your own sort of
custom data passing mechanism and so in
this the way that we do this is we use
the perf output um perf events ring
buffer and so we use this fancy macro
that uh basically BCC provides us to our
code and then we also set up a a scratch
space because the ebpf buffer uh stack
is 512 bytes and we're operating on
paths that are being opened and paths
have a max length of 4096 on Linux and
so that's not going to fit so we need to
have some off stack scratch space to put
it in so we use this Pur CPU array that
is entirely safe as long you stay within
the one thread of execution um and so we
do this and then we just have this sort
of annotation for how we access the
field in it we only have one element in
it it's always going to be there this is
never going to fail but we have to put
it in to please the ebpf validator after
this we just copy all of the data in
from the call into the scratch space and
then we perf submit it which copies it
to the fast buffer for for sharing this
is a lot of copies unfortunately there
are ways to optimize this but we're not
getting into them right now so on the
python side if you're familiar with
python C types it's an API that
essentially allows you to interface with
uh native code so we Define this
structure that basically mimics our C
structure and then we register a Handler
function uh to that will get called
every time the perf submit happens and
we register it to the table named output
which is the table that we defined in
the C
code then inside the Handler we do some
casting and then we print the data then
at the end we set up this K probe poll
call on the BPF object that BCC provides
us and the naming of this is actually
changed in one of the more recent
versions of BCC but it's still it's
still there um and it basically pulls
for all of the perf events for us so how
does this all actually
work well so let's just go into a little
bit of cleaned up srace output um so the
main thing to focus on here is this BPF
prog load up at the top with a type K
probe because we want to make a k probe
uh we get a file descriptor of five for
that then all of the sis kernel debug
tracing stuff is the API for f trace the
theis of s um and the way that you
interact the way that you create a k
probe through it is you write something
to it and then you probe at the ID
number that you set uh to get the actual
ID value for it then you open a perf
event onto that ID passing it is the
config it's a very flexible call
sequence so some config may be one of
anything depending on what type of thing
you're actually doing it's very much
like an
ioctl um that then uh we take that and
we call an ioctl actually on that file
descriptor and then we actually use that
to attach the ebpf program to it then we
enable
it that way our ebpf program is actually
associated with the K probe through the
perf event so what does this BPF perf
output thing actually do fancy
macro actually doesn't do anything if
you look at it it just sets up a it
creates a dynamic uh struct type and
then creates an instance of it but it
never actually fills in the fields
that's a bit weird because basically it
doesn't do anything it's fake the whole
thing is just completely fake it's
mocked out code that's going to be
replaced with uh code generation anyway
so it just needs to pass like the
compiler you know quick check and then
it gets replaced this is actually pretty
common way to to do Cod gen based apis
uh so what exactly is going on when we
call this perf submit over here well it
turns out that this gets replaced with a
BPF perf event output helper function
call um and the important thing about
this other than that it passes all the
arguments in uh is that it passes in a
flag called current CPU identifier and
what this does when it's past that is it
attempts to pull a a perent object on
the Kernel side out of a perent array uh
using the current CPU as the index the
reason that there is a perf event array
there is because back in that srace
output we had a couple of these BPF Map
update elm's uh BPF CIS calls which set
index 0o and one with some perent file
descriptors so at the beginning of this
whole thing the first thing we actually
do is we create that perf event array
and then we get a file descriptor of
three back for it then towards the end
we start opening up these perf events
and in green is uh 0 and one these are
actually the CPU uh indexes and then we
get back for it a file descriptor uh
then what we do is we call this update
LM and we use the file descriptor from
the per event array and then we pass in
this key thing and this value thing that
are these weird hex numbers these are
actually pointers so key is actually the
pointer to the zero value uh for the CPU
zero and value is actually a pointer to
the eight and then vice versa at the
bottom where key is uh one and then the
value is nine
then we just start pulling on these file
descriptors and everything's
great so let's switch switch gears a
little bit let's talk about the EPF
validator hell it's a very very painful
place to be
in to make ebpf safe because otherwise
it's doing all sorts of wacky things in
the kernel it potentially has real code
when it calls functions that can read
and write memory it just takes values in
things right it needs to be safe so
Linux kernel attempts to validate all of
the code before actually loading it uh
so you know various things are you know
your simple kind of checks that
everyone's aware of it is not allowed to
Loop or jump backwards to prevent
infinite Loops because this stuff runs
straight in line in the kernel um this
stuff doesn't get like preempted or
broken out of um if this thing manages
to get in an infinite Loop the whole
kernel hangs at least on that one thread
uh but even if your code doesn't have
any Loops the validator May reject it
for a whole host of reasons um you know
you could have call
that aren't static in line and then
you're technically jumping either
forward to it and then when it returns
you're jumping back or maybe the
function is defined before you and then
you're actually jumping back to call and
it doesn't like
that and so then we you know we start
unrolling all of our Loops to you know
get around that but then you have weird
compiler optimizations that kick in so
sometimes your code was actually wrong
to begin with but the compiler uh BCC
uses an optimized compiler pass and so
through clang VM and so sometimes your
wrong code if it uses a small enough
identifier as a Max operation count
it'll just inline the whole thing and
unloop and unroll it uh but depending on
what you do elsewhere in your code and
you start adding features slowly but
surely maybe something about the
optimization changes and it's no longer
able to do that and then you get an
error that doesn't make any sense
because the part of the code that you
changed has nothing to do with where the
error is coming from so the validator
also tries to ensure that all of your
calls to those helper functions are past
safe arguments
um but the logic for this is kind of bad
um there're all sorts of things that
when you're clearly have bounds that
you're not breaking out of everything is
fine the validator just doesn't realize
it doesn't pick up on it the reason for
this is usually that the optimizer has
actually cut them out and the compiler
is smarter than the validator is and
this leads to a lot of problems so the
more code you try to add to make your
code obviously safe with bounds checks
the more the compiler aggressively
optimizes them out and so then the
validator is left with nothing so you
need to play really crazy tricks to
prevent the Optimizer from leading out
the checks so the validator can know
that you're doing them and then
additionally depending on the updates to
either BCC or Linux kernel uh you may
have valid code that becomes invalid or
invalid code that becomes valid anything
can
happen sometimes some of these valid
errors are really creepy like we've had
code that was rejected or accepted based
on whether or not a function return a Bo
or a size T that was either zero or one
it was being stored in a 8
T and and depending on other parts of
the code that we would change or like
comment out one line somewhere else in
the code the bull was accepted or the
size T was accepted we just commented
that one of them would be accepted the
other rejected made no sense at
all so we at a certain point I got
really mad and I wrote this kernel
module that just really hacky hooks into
the the validator to bypass a lot of the
checks so I could just write code when I
need to right damn validator doesn't
know what bounds are so I will tell it
what the bounds are they're all
safe so it turns out the validator is
actually super like tightly coupled to
The Interpreter in fact you can't just
turn it off because as it processes
through the code it sets all of these
fields on parts of the code that are
necessary for it to actually load and
run so you can't you have to put in
these really careful hooks into it to
skip certain checks and then set up fake
bounds on other parts after they've been
set before with bad values so you don't
get over
overruled so we have this hacky kernel
module PC called YOLO
ebpf of course of course um it it
implements its own custom function
hooking implementation uh because it's
useful in weird cases where uh we have
ebpf but we don't have like f trce AP
available because someone modified
something in their kernel config um
caveats x super x864 only like very
specifically uh I have just Shell Code
in there in like strings and and arrays
um and it probably doesn't work with
kernel current kernel versions almost
certainly not 420 which dropped a couple
days ago and you know if if you have
unsafety BPF code it could it could very
well crash your kernel we were going to
make this code available more to prove a
point than anything else no one should
actually use this uh we a link to our
repo at the end of the slides uh we'll
actually be making it public right after
the talk
so if you don't have the luxury of of
such a hacky kernel module what can you
do well there are a couple things that
you need to do to to appease the
validator uh one is to initialize your
stack memory now this is actually a
generic uh sort of vulnerability in
networking kernel service is anything
that crosses a trust boundary where if
you have a struct on your stack or
anywhere else and you haven't
initialized the memory and it has
padding spaces between the fields and
you try to do something like mem copy it
or just write it over to somewhere else
you're going to copy what's in those
padding fields which is maybe not what
you meant to do so ebpf really gets mad
at this for no real reason because like
you're super privileged when you're
doing this and you're already
intentionally leaking kernel memory so
it doesn't really make sense that it
gets mad about you doing this uh but you
basically need to do everything that you
can if you're copying something from the
stack to basically just not have those
padding Fields there are so many things
you can do next you want to eliminate
your Loops so a lot of pragma unroll
through clang lvm API to statically
unroll any static bound Loop loops and
then everything is basically static
inline function calls um any newer
kernels support not having that but also
no one's on those new kernels yet so it
doesn't matter um then a lot of Kernel
code even if it is static inline fun
functions that are in headers that are
aim that are aimed towards helping you
parse various pieces of Kernel data
structures um they don't really work
that well in BCC sometimes because BCC
every time it does a pointer to your
reference to memory that's not in the
BPF memory region it will actually
instrument that um it'll it'll just code
gen it to be this call to a helper
function called BPF Pro read that will
actually pull the data out and so it's
not great at converting kind of chain
calls and anything within weird scoping
contexts in C and so some of that will
just fail to convert and and it just
won't work um so basically there are a
lot of Kernel functions where you're
dealing with things like the task struct
where anything that you want to get out
of that you're going to have to call BPF
prob breed manually and pull it out
piece by piece which is really annoying
um then uh you depending on what you're
doing if you're trying to go for super
high performance uh you may need to sort
of implement your own ring buffer to
conservatively send data down to user
space with as minimal amount of copying
as possible if you try to do this you
need to actually have some you can't
Loop so you need to have a switch that
gets called every time that attempts to
update a synchronized counter variable
and then at the end of it it needs to
roll it back over uh this is actually
very carefully written code that
appeases the validator if uh in this
particular case um we're just doing zero
and one if two happens we're supposed to
cut back over to zero um so if you tried
to actually put a case two in here the
EF validator would just not accept it
for whatever reason so you need to you
need to have the default case actually
flip it over and you can't have another
case actually flip it over um for
whatever
reason so and then a lot of the kernel
data structures are actually very
Dynamic not in the sense that they're
just pointers to pointers to pointers
but in the sense that they're actually
these crammed structs that are just bite
copied into each other uh all hamfist
did and if you want to pull things out
of them you need to essentially walk
Dynamic structures of of pointers to
things and pull them out and do stuff
and ebpf doesn't like Loops so you need
to do that statically with both unrolled
loops and static inline functions and so
you just need to be very careful if you
want to do it it's very tricky but it'll
work usually the most important thing
when doing this though is actually to
know when you don't have enough uh sort
of unrolled Loops to fully count out all
the data and you need to you need to
bail gracefully when you do this um know
that you're you're truncating the data
somehow and then lastly or not quite
lastly but uh one of the more super
important things when you're when you're
doing things in the kernel with ebpf the
reason you do it is because you want it
to be fast otherwise you just use audit
D for most things so you want to
actually be doing a little bit of
pre-processing in the kernel to do
filtering but to do that filtering you
need to actually do things like
comparisons and and other things like
that and if you need to determine that a
PID was within a set how are you going
to do that without recursion or Loops
well what you can do is you can do
something like have a balanced binary
search tree in user space walk through
it and code gen out what the comparison
would be statically for the ebpf code
this works actually quite
well and then lastly one of the things
that EF ebpf validator really hates is
dynamic length bite copies where the
length comes from something that you
read from somewhere else um mostly
because the compiler will optimize
things and it'll be left in the Lurch
without enough information to know
what's going on uh so the way that we
got around this was to basically use a
weird static inline function that would
kind of break up the logic and then
finally it started
passing stuff is
crazy um and then one of the most
helpful things that you can do for BCC
at least is enable the full debug output
that will actually dump out all of the
uh ebpf instructions um but will also
dump out the code uh sort of
pre-processed already that is uh being
implemented with those instructions so
when you get an error you can actually
look back at that and figure out where
in the assembly something went
wrong good
luck that's that's easy
right pass it
[Music]
over so since for Security Consultants
uh we had to ask ourselves can ebpf be
used for defensive uh measures um we
said why not I mean EPF is really fast
and you know if we're successful we can
improve the state of auditing so uh yeah
let's let's do it so what does security
monitoring software do basically it just
watches everything uh program executions
file accesses is Network traffic
administrative
operations and hey efk probes can do all
this so what would make this a good fit
well tracing EF programs they can see
everything and they can hook into any
internal kernel function and we can
observe all kernel and user space
memory so start off let's Implement some
example you know uh monitoring tasks uh
we're going to begin with hooking the
execve call
and uh just keep it nice and simple nice
refresher we've got the K probe uncore
uncore and then CIS exact V uh just a
nice simple uh Trace print K uh all
these examples just going to keep it
that not to be
confusing so now let's do something more
interesting um we can compare the file
path uh
with known uh standard directories where
binaries would be so for example bin or
user local bin um and we could do this
just by
uh checking the file name parameter via
the CIS call so we have it set there um
we have
our uh directory prefect set for this
one we're just going to check bin uh
it's too complicated
EPF to check all that uh all the other
directories and we have our unrolled
Loop there we're just going to check by
by bite uh and see if it's in bin or not
and if not we're just going to let the
user
know so uh another thought that came out
of this was if we have a web app um
would we be able to know if it's
executing stuff it's not supposed to be
so I mean we can imagine that we have a
web app that you know is just the front
end to Ping uh takes an IP address from
user input you know usual stuff um and
we want to know if it's executing
anything other than ping
so uh right here we just have the C and
same thing hooking CIS execve uh the if
dead if def PID is assigned via python
VI a Define using uh
argar and if it's not equal to the pit
of our web application then just don't
continue checking anything so we're
copying the file path to a temporary uh
buffer so we can get the length of it
and then we're doing a quick exit if the
uh if they're not the same length but if
it is the same length we're going to
make sure that it's not ping and then
let the user
know so now we're successfully
monitoring file executions um yeah next
we can watch for file opens from certain
directories okay again nice and simple
straightforward going to hook uh de this
open but uh to make this somewhat useful
we're going to see uh what if something
is accessed under the root directory
and same thing you have the uh path set
unroll the loop and then just check bite
by
bite if it's actually in the
directory but uh we we just have to let
you know
something all these examples are
insecure dangerously
insecure just because EPF makes sure
that it's not going to affect the kernel
doesn't mean that the stuff it's running
is safe and in fact the limitations make
it extremely difficult to write secure
EPF
code one of the things we noticed was a
time of check time of use and
um we notice this when you K prob assis
call so the user Supply data that you
get from the hooked CIS call um can
change from the time when you copy it to
when the kernel copies it and actually
performs sis call and this is relatively
easy to test for so we'll just start
with two threaded program where the
first thread copies two different file
paths into one character ARR array and
then the second one just opens
that and uh we see here some nice output
we got that everything just is all
mixed and how do we avoid this problem
well our recommendation is to hook
internal kernel functions instead of the
Cy calls preferably where uh the the
values already copied into kernel
memory so uh there's some things that we
have to take into account uh and make
sure that we think about when we're
creating these things such as uh how
file names work
so um what if it's not accessed by the
absolute path from inside the directory
from a relative path uh and can we fix
those well I mean we could attempt to by
comparing the value or attempting to
canonicalize it ourselves but Linux
internal structs are just complicated to
parse and it's just way too complicated
for what ebpf was designed
for or we can try to find an internal
function uh later on that has it but for
this example it it didn't get the U it
just gets the same exact value as what
the CIS call
gets so we notice that BCC has example
code uh for network monitoring and we
also noticed that uh it didn't properly
calculate IP header IP header offset
sets specifically it did not account for
the fact that TCP options are variable
length so it was possible to spoof a TCP
header so we sent them a
PC and of course a
patch and here we have the diff of that
patch where uh we're now checking the IP
header
length in general uh make sure you're
requiring strict validation uh because
ebf doesn't have a copy from user so you
basically could be tricked into uh
reading kernel memory that was supplied
by user uh user space
pointer and then uh just manually verify
the pointers so uh we asked the question
again can it be used for defense um not
directly the limitations make it hard to
use it in this fashion um and instead
it's much more useful to uh observe the
data that flows through the system so we
have Unix dump which is basically TCP
dump but for Unix domain sockets um it
demonstrates our successful fight
against the
validator and uh link for that will be
at the end of the
slides so this basically just hooks the
stream send message and datagram send
message and retrieves a message header
uh contents from it and we use Python
like Jeff mentioned before to
dynamically generate C
code so this is used to uh tweak the e
PF program at runtime in quotes um and
also helps us get around the lube
restrictions so here is uh the binary
search tree that Jeff was mentioned
before and
uh the P CPU array which keeps track of
the Ring buffer slot uh because we can't
Loop we have to generate the switch that
Jeff also mentioned before and this is
the code for
that and uh we have to make sure we're
checking the values
uh in kernel space and user space
because there's a race condition where
uh C might might override what python
has because it doesn't know that python
is finished with that value so we uh
make sure that c knows not to use the
value if it's still in
use so if ebpf isn't that good at
defense what else can we use it for
let's talk about
offense so you know let's assume someone
bad get some privileges on a modern
Linux system maybe like capis admin in a
container cuz containers are safe right
not really what could they do with
EPF a lot actually uh all that tracing
stuff can not only see everything it can
also write user space memory I repeat it
can also write user space memory so BPF
probe right user is the the magic uh
call that we can make from all these
tracers um
it allows us to write only to writable
user space memory so unfortunately we
can't write to the text section we can
write to everything else basically
though so what's anything useful in
these in these regions well buffers for
Reading Writing data you know coming
through the CIS calls maybe there's a
you know some sensitive stuff being read
on a sensitive file descriptor maybe
codes being executed from it you know
maybe there's a privileged process doing
that I wonder oh yeah cron cron so we
wrote I wrote this thing called uh con
job which spoofs KRON jobs it is the
KRON Auto poner it hooks all of the stat
family of CIS calls and basically uh
whenever uh you know KRON is looking to
get information to know whether or not
it needs to reload at zon tab it stats
them and looks at the last modified date
so we just update that to make it very
recent so it always loads the file and
then we hook open at and close so
anytime that file is opened we save the
uh FD that's going to be used for it and
we store it somewhere else for later and
if it's closed we stop keeping track of
it and then we hook read for that FD and
if we see a read in there then in the K
rep probe return we actually stomp over
what the the contents of the file was
actually supposed to be with our own
little you know root command as one
does so it's brief
demo um this is actually running
container it's it's slightly tweaked um
because uh Docker happened to update
their app armor profile a little bit uh
before after after we made this and it
it killed one of the the vectors of how
this Ran So we we weakened it a little
bit more but but now we've uh basically
just updated this to you know do the
attack so as we see in here this root uh
you know command is injected right at
the top of cron tab so that's cool what
what you know what's what else is fun
about this so we use some per CPU maps
to have all this stuff communicate
safely with each other then we use some
hash maps to pass them between the pairs
of those things and we also have to use
this Nifty hel function that allows us
to keep incrementing the time forward
without having to go back to user space
so what else can we do with ef we're
going to go For
Broke uh if you recall we can write to
the stack the stack has return
addresses we can also read the stack in
all of user space memory we can scan for
the text section and share libraries I
think I think people see where this is
going so wrote this thing called GB C
pone which is the system D Auto
poner so basically it scans uh PID one
memory for libc it backs up uh the stack
content uh then injects uh R chain into
the process right where it's about to
return to uh the rchain uh basically
runs a deal open uh on uh a path that we
provide uh we load our malicious Library
into PID one and then we clean up after
ourselves all nice and
clean so to talk about what's going on
oh not this one all
right so we have uh this piece of it
called find you Lipsy um this guy sorry
uh basically dumps US out uh return
address and uh and the base of lib C and
then we plug it into the pone libc
function uh that basically goes and does
the rchain which I assembled by hand
based on uh libc GBC which has lots of
fancy gadgets all very useful and so we
run this and then eventually it
returns and it run it both loads our our
shared Library which if you see uh this
C code is actually what was there uh so
it's going to do a CIS log and then it's
going to run ID and output it to Temp
evil temp evil is now filled with with
root and we've also seen a log coming
from PID one systemd hello 35 C3
[Applause]
[Music]
so don't have a lot of time but to
briefly go over how this whole thing
works essentially we hook on timer FD
set time because in all of uh systemd it
basically calls this thing once a minute
guaranteed and so it's very reliable um
we then scan forward from the stack
based struck that it passes forward so
that we can find the return address and
then we part we basically scan for
return addresses back into the CIS call
stub because we know what's in there um
it's going to set you know Rax to the
right the right ID number and then do a
CIS call instruction so we keep parsing
forward until we find it and then once
we find it we use uh we find where the
CIS call stub is we use the offset that
we know from the beginning of libc and
we return both the return address
pointer and this after we made this we
then realized that there was a hacky way
to actually get all of the registers
Associated to the process because ebpf
in BCC doesn't normally provide it to
you you have to scan memory to get it
but so this is still useful for other
things
though um then we hook uh timer FD again
and in the return value for this uh we
actually then go and we back up the
stack for safekeeping then we write the
r chain in uh then we return user space
uh the Tim RFD set time then returns
right into you know right after where
the CIS call instruction was then it
returns into a op chain our rchain sets
up all the all the point all the
variables all the all the arguments it
then calls the DL open um after it's
done with that we we don't want our
module to have to clean up after this
whole thing so we actually do is we have
the rchain clean up after itself so we
actually call close with a magic
negative value that otherwise is not
going to do anything in the kernel the
kernel is not going to care but we
actually in our uh hook on close use
that as a signaling value when that is
hit we actually write back most of the
original stat except for the last
remaining Gadget then we write a new
chain past the end of where the stack
originally was the reason we do it in
two parts is because the DL open itself
could potentially use up a large amount
of Stack space and it would have
clobbered anything that we put there
then we return user space into the last
part of our old R chain the last Gadget
shifts RSP forward to the new R chain
the new rchain then just writes back
over the the the last remaining parts of
the old R chain with the original values
and then it clears Rax so that it looks
like like the timer FD call that we
finally eventually return back to seems
to have
succeeded so gpy is actually super
stable like all of these gadgets work
across multiple versions of lip C across
different Linux dros it's all great like
this is super ortable like you'll have
to regenerate the the specific offsets
but they're all basically the same
gadgets uh so what else can we do with
ebpf use it as
intended so you know once you're once
you're running that K probe you know you
shouldn't let anyone stop you right you
know you can prevent processes from
interacting with the kernel for example
you can prevent processes from listing
ebpf programs and K probes you can stop
them from creating ebpf kres you can
stop them from loading kernel modules
and you can stop them from phoning
home so this is important because the
BPF probe right user actually when it's
loading uh when the BPF program is
compiled and loaded uh this has a bit of
logic in the kernel that sends out a d
message warning that this is in use and
may corrupt things so that's a it's a
pretty good warning sign but you have a
weird race condition because anyone
reading D message has to read from it
and then if they attempt to use any CIS
calls to phone home you can stop those
so they need to basically already have
something like dpdk to do nonis call
Direct memory mapped packet iio to a
network device so that you can't catch
them but even then there are all sorts
of wacky things you could do to stop
them from pH owning home and there's
also this magic BPF return override
return helper that we had problems
getting to work but it's supposed to be
able to just stop assist call from
happening so you know the one downside
of all this is that we need to keep our
ebpf process alive right because once it
once it dies you know the whole thing
goes with it what if we could make our
efk probes basically Immortal you know
we've already taken over pit one you
know so why not just run our like you
know other efk probes from PID one
itself once we're inside this means we
stay alive until the system shuts down
or vice versa if PID one's the system
goes the system goes down with it you
know and which is great for us because
PID one is now systemd for the most part
so everyone you know knows that thing is
just falling apart all over the place
they're not even going to think they're
being attacked by you know Space Age
ebpf
rootkits so ebpf is useful for everyone
except people trying to build IDs on top
of it which makes no sense
you know it needs to get a lot better at
supporting that use case to be honest
and right now it just it isn't there so
a plea to Colonel developers who might
be writing these things you really need
a copy from user so that no every not
everyone has to check the pointers
manually that someone didn't try and
open a path that is actually a kernel
address and you know you also maybe want
to provide helpers for all these
trickier file structures and things like
that so people can actually get full
paths out of
maybe give some direct memory comparison
operations so we don't have to do these
weird things and also mset please memset
I don't like to have to waste my my
limited amount of instruction bites
performing a
memet um so just some greets out to
people the BCC developers have been
super helpful they're building really
cool tooling um all the stuff is just
getting better and better every day um
Julie Evans has great blog posts on how
K probes and all of this stuff works
same with Brendan Greg and Jesse
friselle uh is has been building this
thing called BPF D which is almost
certainly going to be the thing that I
would want to inject into PID one as my
rootkit manager
definitely um because it manages all the
things you can't hide from the future
are there any
questions are there
questions
so instead of going into the garden
Motors we went into the flying uh
spaghetti monster Motors isn't
it questions ladies gentlemen number two
here CH hi there great talk uh can you
just turn off compiler optimizations in
BPF and save yourself a lot of trouble
in the validator um if you want to go
modify the piece of the code that does
it um that whole part of the code is in
C++ because it it's all lovm based and
so you'd have to rebuild um BCC to do do
it I think uh there are there are there
are the ability to set other flags onto
it but I'm not sure if when it where it
does the injection into it if it's
before after and you could then override
the
optimization thanks uh number three um
what kind of version have you tested
with kind what paron which Kel version
have you tried this with because there
is a lot of improvement in every so oh
so uh a lot of this uh we've been basing
on primarily two kernels I think uh 45
from iunu 1804 48 from Ubuntu 1810 and I
want to say 16 or 17 from Cali
Linux thanks okay number
two seeing that it's easy to inject
stuff with EPF uh how can I prevent it
to be one in the first place if I need
it anyway sorry I I see that I need EPF
in a kernel for the base stuff like
firewall stuff and on but how do I stop
as a user
to like scripts I'm installing to inject
more EPF stuff in my
kernel um so sorry can you say that once
more how to make it hard for um unknown
scripts to inject more EPF in my kernel
oh you need you need capsis admin to do
most of this stuff and uh at least on uh
containers uh there are a lot of
operations that are in like app armor
profiles and SE Linux stopping people
from interacting with the CFS that is
used for a lot of this the problem is is
that in the newer kernels they've
implemented another way around this that
just involves using either the direct
BPF CIS call or uh the perf Event open
without going through the CFS so that's
going to completely bypass that
protection
probably right sir number two uh do you
know if any like Cloud providers which
offer like container uh like as a
service are
vulnerable um haven't looked um if
they're giving you capsis admin for real
that isn't username spaced uh they've
got other problems uh going from capsis
admin to full rout is like it's more of
a neat party trick it's not really like
a full privilege escalation but if you
happen to do it in something where
someone is trying to apply all these
other restrictions on your container and
you happen to find a way out that would
be bad but also they probably shouldn't
have given you capsis admin in the first
place great someone out there in the on
the internet with a
question someone
else no no no
no well Jeff uh Jeff and Andy Andy Al
Jeff
Deo we're going to shut it down here uh
give them a warm Applause
please thank you
[Music]
for
[Music]
oh