*music*
*applause*
Raichoo: Yeah sorry about that - beamers or projectors, I don't like them. They
don't like me either. So this is a little heads up - this is going to be the only
slide I'm going to show you today so, "slide", because I think doing stuff like
that in a terminal might be a little bit more interesting for you. But sadly
something is getting cut off so I we have to improvise a little bit. But anyway, so
today I will be able to talk about two of my favorite things right now which are
FreeBSD and DTrace. But this talk has been capped down to 30 minutes so we'll be
focusing a little more on the DTrace part. So there will be a little bit less BSD
than I anticipated. And also adjusted everything a little bit to fit better into
the resilience track so hopefully you'll enjoy that. So before we begin, who here
is actually using DTrace? Okay more than I expected but still not as many as I
would like to see. So hopefully after this talk you will think, "oh, this is a really
awesome tool, I gotta learn it." Because I totally love it - it changed the way I do
a lot of stuff. So for those of you who do not know what DTrace is, first, let me
fill you in on this stuff. So it's open source, it originated on Solaris, and been
developed currently on illumos which is a fork from OpenSolaris. It has been ported
to FreeBSD, NetBSD, OS X, there's also a port for Linux called next called DTrace
for Linux. I think it's done by a person called Paul Fox. It's been ported to QNX
and the OpenBSD folks are currently doing some work to get the technology like
DTrace on their system. And I think there's a port for Windows? I don't know
if this is actually true, but it is it's kind of cool because then that means it's
basically everywhere. So, most of you would probably know static tools like
strace. We have a very similar tool on FreeBSD that is called truss, and what
truss and strace are doing is - you can attach them to a process and look at the
system calls that this process is emitting. So in case something is going
wrong you can well look inside of the program, which can be kind of useful when
you're trying to find a problem. It's kind of handy but it's also pretty
limited. Because first of all it really really slows down the process that you're
currently looking at. So if you want to debug a performance issue, you're pretty
much out of luck there. And also it's kind of like, narrow down - you can just look
at one process. Which is also like bad thing because the system that we currently
have - all these systems are very complex: we have a lot of layers. You have
virtual file systems, you have virtual memory, you have network, you have
databases, processes communicating with each other. And in case you are using a
high-level programming language, you might also have a runtime system. So it's a
little operating system on top of your operating system. So when something goes
wrong in a system that has such large complexity, something happens that we call
the blame game. And the blame game - it's never your fault, it's always someone
else's. So what we want to be able to do is we want to look at the system as a
whole, so we can correlate all the data and come up with some meaningful answers
when something is really going wrong in there. And also, we don't want to
switch out all the processes for debug processes to make that happen,
because as these things are all -- every problem happens in production. It never
happens on the development box. So like, switching out all the processes - that's
totally out of the picture. So to do that in an arbitrary way, to like, instrument
the system in an arbitrary way, we sort of need like a programming language. So, we
need to describe - when that happens, please submit data so I can see what's
going on. So this kind of implies a programming language. And DTrace comes
with such a programming language - it's a little bit reminiscent of awk cross with
C? It's pretty simple to learn - you can pick it up 20 up to pick it up in 20
minutes and you can start churning out your first DTrace scripts. So like awk, if
you know awk, awk can be used to analyze large bodies of text. Dtrace is pretty
much the same, but for system behavior - so a little bit mind boggling, but
probably I can show you what I mean by that. And also, as a bonus we don't want
to slow down the system, so we want to be able to do things like performance
debugging, performance tests like that. So I've prepared this little demo here, and.
So since we had some issues here probably this is not -- I have to play around a
little bit. So what I'm going to do is I'm going to look at a very very naive way
to -- excuse me for a second -- very naive way to -- give me a second -- so very
naive way to authenticate a user. And there's a lot of stuff wrong with this
code, but like what we're going to do is we're going to take a user string as
input, and then we're going to just compare it to another, to a secret. So I
know, the the secret in here is like in plain text I know this is a problem, but
this is a little bit artificial. But I just want to get my point across. So from
an algorithmic perspective, this check function is correct: so we take a string
we take another string and we compare them. So everything's fine and easy. So if
you look at the way string compare works and what it does, it's essentially
taking these two strings and it's comparing every character bit by bit. So
when it finds the first pair of characters that do not match up, it's going to stop.
So we can we can conclude something about from that - so if it takes very short if
if this function this check function takes a very short amount of time, then, what
will happen is it will terminate earlier. And if our password guess is better, it
will take well, it will take longer. And if we can measure that we can basically
extract information from that running algorithm. So I wrote a little driver
program in Haskell that basically just iterates over an alphabet and just feeds
this one letter into that program, And I'm going to use DTrace to get some
timing information. So let me start the driver. So this is now just running in the
background. And you cannot see what I'm typing there, but don't worry - these
scripts will all be; I will push them on my github. So DTrace now produces this
nice little distribution. So if you if you were if you were able to see the entire
alphabet, you would see that everything except "D" behaves differently. So if you
squint a little, what you see there is DTrace the D letter takes a couple of
nanoseconds longer. This is the precision that I'm measuring here - ten to minus
nine seconds. Like really precise. And D takes longer than everything else, so it's
a little bit cut off there, but trust me. I know it sound like Donald Trump I'm
saying that. So yeah, and from that let's just enter a letter. And now the password
and now the script clears everything and it's going to guess the next letter. So
sadly this is cut off, because you would see that this distribution radically
changed. It looks completely different, and so we can play that game a little bit.
So let's just roll with that. And like every three seconds the script is
going to recompute looking at the new distribution. And you can probably see
where this is going. So here you can see, okay, and now it just - it just takes
about like three seconds for me to guess the next letter. So, and this is not a
problem that is only of something that happens when you do string
compares. This can happen with basically everything - so it's especially
in things like cryptographic stuff where you don't want to have some information
leaked out. So this is what we call a timing side channel attack. So I could
essentially use DTrace to analyze the real binary. So I didn't change the
binary - I didn't have some some debug code there. This is like the actual binary
that I would put into production. So what's important about out that, is to
take the actual binary, is some of these these timing side channels might be
introduced by a compiler optimization. And when you insert debug code into that code,
then it might actually go away. So, you want to look at the real code that you're
putting into production. Let me show you the script that I came up with to write
that. So there are three interesting things in this script. So and and don't
worry - this is the more complicated example, I just want to like
inspire your ideas. Because the things that you can do with DTrace that's pretty
much - the sky's the limit. You can come up with the weirdest ideas, and so
this is more complicated example. I'm going to show you simpler ones. So to
demonstrate how we got here. So there are three interesting things in this code. The
first one is something that we call a probe. So a probe is a point of
instrumentation in the system. So whenever a certain event happens in the system this
probe is going to fire. And in this case, the begin probe like marks the state
the moment when the script starts. So the second interesting thing is this clause.
So this clause is basically what this probe is going to execute - what's going
to be executed once that probe fires. So it's a little block of code.
And this probe is a little bit more interesting, because it tells us
something about the structure of how such a probe looks like. Because every
probe is uniquely identified by a four tuple. So it's like four components that
uniquely identify a probe. And the first one is called the first part of this
tuple is called the provider, and I'm going to talk about providers in a couple
of seconds and what they are. The second one is called the module. Third one is
called the function. And the last one is called the name. So these four pieces of
data, like, they identify a probe uniquely. So the third thing that is
interesting here is, sadly something that I don't have time to talk about today,
this is called an aggregation. And this single line that you see here is
essentially responsible for accumulating all this data to print out this
distribution stuff - to generate this distribution. So this is built
into DTrace. You don't have to do that yourself. As it, when you look at this
script, it's like 42 lines of code. And I came up with the first prototype
after five minutes. So it's not a lot of stuff to do to get something out of
that. So it's very useful to have things - if you use DTrace you
will use this a lot for performance debugging so it's kind of neat that we
have that. So yeah, let's talk a little bit about providers, and this will
probably also will be cut off. So I'm going to cheat a little bit here - I'm
just going to double that. So let's talk about providers -- oh that's handy --
so I got 27 providers here and the number of providers vary from operating system to
operating system. But these are the ones that I can see right now. There are
other providers that can be come into existence when you demand them. So I have
these 27 providers, and we're going to look at the syscall provider and the FBT
provider first. So, every provider knows how to instrument a specific part of the
system. So the syscall provider knows how to instrument the syscall table. That's not
very surprising. So if you can look at the syscall provider and here you can see
essentially every system call entry and return that FreeBSD offers. So
here you can see this four tuple, like, the provider syscall, FreeBSD is the
module, and so on. So these are all the system calls that I have in my system. And
the other provider that I want to look at is the so called FBT provider, and that is
pretty astonishing. The FBT provider, FBT stands for "function boundary tracer" and
what it allows us to do, it allows us to trace every single function in the kernel.
So I can look at the entire kernel at functions, as they are being called. So to
illustrate that I wrote a little, very simple DTrace script and this is probably,
look at the upper half please, so this is probably one of the first DTrace scripts
that you will come up with, it's a fairly simple example, so let's break it
down. So I'm going to instrument the mmap system call. For those of you who do not
know what the mmap system call is, what you can do with it is you can so you can
take a file and map that into the address space of your process, so very dumbed down
version. So whenever we enter the mmap system call we are going to set the
variable "follow" to one, and what this "self at" means: this is essentially a
thread local variable and we're going to associate that variable with the thread
that we're currently inspecting. Then I'm going to do something pretty, that sounds
scary but I'm going to instrument the entire kernel. Every function entry and
every function return, I'm going to instrument that and say "please emit data
when you do that". And this is what we call a predicate, so this is where the
awkiness of the DTrace programming language comes in. So this is a predicate
and whenever that evaluates to true then the probe is going to fire, so in
this case when we are in the thread that we're currently tracing we're going to
emit data. And this is just an empty clause, we just want to know "hey we got
here". So when we exit the mmap system call and the predicate is set we're
going to set the variable "follow" to zero, because every uninitialized variable
in DTrace is set to zero, so this pretty much amounts to deallocating that variable
and then we're going to exit cleanly. So let me run that. So it takes a couple of
seconds and boom. So you saw a little pause here, that was when the DTrace guard
reverted the driver, the kernel. So now you can see every function call that
happens inside the mmap system call. And this is a little bit hard on the eyes, so
let me pass this flag here and now you can have nice to read indentation. So
now you might say "I don't like that. You are injecting code into the kernel. That
is, that sounds dangerous" and yeah, but let me show you something that I find
really interesting. So I'm not going too much into depth here, but this
is a byte code, so every DTrace script gets compiled to bytecode and this
bytecode gets sent to the kernel and in the kernel you have a virtual machine that
interprets that bytecode. So in case you write a script that for some reason might
go rogue on your kernel, it like allocates too much memory, takes too much time, this
virtual machine can just say "okay, stop it" and just going to revert all the
changes that happened to your kernel, and that's kinda handy. And it's not a new
idea, so if you're using TCP dump it's basically the same approach. They also
have this kind of bytecode, so that's just a little excursion here. This is called
BPF, Berkeley Packet Filter, so it's not an entirely new idea. So everything I
showed you until now was "hey, I can look when function calls happen". that's not
very much information, so we're going to increase the amount of information that we
get out of the system with every example. So let me look at the actual kernel. So I
had to restart my machine, so my setup is basically gone now. So let's look at this
VM fault function. So this is, this is the source code of the operating system that
I'm running right now. This is FreeBSD current 12 and the VM fault function;
remember the mmap system call that I told you? So the mmap system call
I told you can bring, like map a file into your address space. And it doesn't
necessarily have to load the entire file, so whenever we are touching a page in the
system, like a memory page, this machine is four kilobytes and it's no super pages
here, so whenever it touches a piece of memory that you didn't bring into memory
yet, we're generating something that's called a page fault, and then this
function gets called. So here let's look at the arguments, and I'm going to skip
the zeroeth argument, to look at the first argument. So this is the address that
provoked that page fault, this is the type and these are the flags and I'm going
to show you something to make that a little bit more readable. So what about
this one? So you see it's a pointer and this is a big structure, so we want
to be able to look at that structure. And just probably should do this here, so
let's look at this VM fault script here. So this is, make this a little bit more,
so this is, don't pay too much attention to this code, this this is basically just
boilerplate to make make stuff readable and this is where the actual action is
happening. So this is, so what I'm doing there is I'm instrumenting the VM
fault function and whenever we enter it then we're going to use some information
that DTrace gives us for free. So this is execname, this is the name of the
currently running executable that provoked the page fault, this is the process ID and
here we have a bunch of argument variables. So these arg1, arg2, arg3,
that are essentially just integers, so nothing too fancy there. But we wanna
look, wanna be able to look at that struct. And here I'm going to use this
args array, and this args array is kind of special, because it has typing
information about the arguments. So when you run that, so you're referencing that
pointer there with the star, excuse me, and let's just run that and maybe, that's
a start yeah. So this is an in-kernel data structure that we can now look
at. So DTrace enabled us to look at in- memory data structures as the system runs.
And this is really really powerful. In in the DTrace script I could use all
these fields like I can manipulate this args array, this value in there, just like
just like every other variable; I can pretty much work like I was in C. So
how is it doing that? There is something that's called CTF, that's not capture the
flag, it's, this is the, the Compact C Tracing Format, so you can see that but
there is a man page in FreeBSD, and there's a little segment in the kernel
binary, where all this typing information is stored. I don't know how that compares
to modern DWARF but yeah this is what DTrace is working with. So now you might
ask yourself "Why on earth would I do that? Why on earth would I look at virtual
memory, because, yeah, um, this stuff is safe isn't it? I mean there's no bugs in
there." Except when they are. Anyone remembers remembers "Dirty COW"? So this
was a very nasty vulnerability in the Linux kernel and that that was a problem
in the virtual memory management. So it allowed you to write to a file that you
didn't own as a regular user. So you could essentially just write to a binary that
had "set UID" set. Very unpleasant, but I'm not going to bash the Linux folks
here, this is just, I just want to show you these things are hard. And the first
fix for this problem was in 2005 and then it came back in 2016. So now that's fixed
and then it came back with "Huge Dirty COW" in 2017, so this is, I mean this
was there for way over a decade. These things are hard to debug. And this
is what I like about these systems, so not having, not having tools like DTrace to
figure out what's going on inside of the system somehow, to me, amounts to security
by obscurity. And I've heard that some people who are developing exploits for
systems that have DTrace they say "Oh, I really like developing exploits on these
systems, because the tooling is so great!" Yeah, but, to be honest this is cool,
because an exploit is a proof of concept and coming up with these exploits quickly
is very usable, because you know what's going on you can show "Hey, this is going
wrong". I had situations, where people were telling me "Oh, this is this
is not a problem with our program, this is this weird operating system that you're
using. Like Solaris, weird operating system." And, yeah, and then I churned out
some DTrace scripts and "No, it's actually your problem". "Oh, now I can see
that on my Linux box!" Magic. So, everything I showed you until now was
very, very much related to function calls and we want to have a little bit more
semantics here, because you might want to write a script that inspects protocols,
stuff like TCP, UDP stuff like that. So, you don't want to know which function
inside of the kernel is responsible for handling your TCP/IP stuff, so DTrace
comes with something that's called static providers and I'm just going to show the
apropos here. So these are, so every static provider has a main page which is
kind of handy - documentation whoo - and you can see there is an I/O provider if
you are interested in looking at this guy: Oh, IP for looking at IPv4 and IPv6,
TCP... This one is pretty cool, it's about scheduling behavior. So, "what does my
scheduler do?" And if you look at that, you can see some interesting stuff like length
priority if you ever saw things like priority inversion, stuff like that, now
you can see that happen. I'm a nerd, I find this interesting for some reason, I
don't know. And it's also pretty interesting to figure out what's going on,
"why is this getting de-scheduled all the time?" So, some interesting things going
on there. So, I'm running a little bit short on time here, but I just quickly
want to show you something - this is all kernel stuff right now - can we do that
with userspace? Of course. So, there was one provider that didn't show up when I
had my provider listing, but was in the DTrace script where I did this timing
attack stuff. And that's called the PID provider. And the PID provider generates
probes on demand, because a process might have a lot of probes and you will shortly
see why and this is why I'm going to use a very small program which is called "true",
and true just exits with exit code zero. So, nothing too exciting going on here,
and this dollar target gets substituted in, we get the process ID there. And this
is everything that happens when I'm executing this program you see this is a
little bit more fine-grained than the FBT provider, because now we can trace every
single instruction inside of that function, which is kind of a handy. It's a
scriptable debugger. So, these numbers are the instructional offsets inside of that
function. We can also look at - so this is everything in the true segment - we can
also look at libraries that got linked in and there's a lot of stuff happening in
libc for example when you run true. So, one last thing that I wanted to show
you because it consumed a week of my life: I'm using a lot of Haskell and the Mac OS
people, they also have DTrace and they have GHC Haskell DTrace support - so the
Glasgow Haskell compiler - and glorious... they have probes to analyze what's going
on inside of the runtime system. So, I thought "I want to have that, I have
DTrace, why doesn't it work on FreeBSD?" So, after a week of fighting with make
files and linkers, that works: If you check out the recent GHC repository and
build it on FreeBSD, you get all the nice stuff that I'm going to show you now. So,
this is a very boring program - it just starts 32 green threads and schedules them
all over the place - and now I can do something like this: *phone rings* I can
ring a telephone. *laughter*
No, that would be interesting... So, you can also use
wildcards - and not as name of the probe - and this is what's going on inside, like
GC garbage collection and all this stuff. Now you can look at this and write useful
DTrace scripts that also take my runtime system into account. So, stuff like that
exists for I think Python - I'm not entirely sure because I don't use it -
nodejs same, Postgres - I used it but not with DTrace right now - and what a find
interesting: Firefox. When you run JavaScript in your Firefox, it actually
has a provider, so you can trace JavaScript running in your browser with
DTrace, so after everything I just showed you, there might be some stuff going on
there. So yeah, this is basically everything I wanted to show you and I
think I'm going to wrap out, because otherwise we're not going to have a lot of
time for questions and maybe you have some. So yeah, thanks.
*applause* Herald: Thank you very much Raichoo. We
are actually over time already, but we have two more minutes because we started
three minutes late, so if there are any really quick questions, possibly from the
internet... There is one, the signal angel says, let's hear it.
Question: Yeah, hi, okay. So, the question is, "which changes are actually necessary
to do in the kernel of an operating system to support DTrace?"
Answer: That's a lot of work. So, it's not something like you do in a weekend. This
is... So, the person who started the work on FreeBSD has sadly passed away now, but
I think they took a couple of years to have everything in place, so you have to
have stuff like the CTF thing that I showed you, which is what OpenBSD is
currently working on. And then you need all those those magic gizmos, like kernel
modules and stuff like that. So, it takes a lot of time, but it's been ported to
most operating systems that are available and in use right now. So yeah, hope this
answers the question. Herald: Excellent and there are no more
questions here in the room. I will thank Raichoo and you can find him outside of
the room and also on Twitter at "raichoo" if you have any more further question.
*postroll music*
subtitles created by c3subtitles.de in the year 2020. Join, and help us!