Herald-Angel: The Layman's Guide to Zero- Day Engineering is our next talk by, and
my colleagues out in Austin who run the Pown2Own contest, assure me that our next
speakers are really very much top of their class, and I'm really looking forward to
this talk for that. A capture the flag contest like that requires having done a
lot of your homework upfront so that you have the tools at your disposal, at the
time, so that you can win. And Marcus and Amy are here to tell us something way more
valuable about the actual tools they found, but how they actually arrived at those
tools, and you know, the process of going to that. And I think that is going to be a
very valuable recipe or lesson for us. So, please help me welcome Marcus and Amy to a
very much anticipated talk. *applause*
Marcus: All right. Hi everyone. Thank you for making out to our talk this evening.
So, I'd like to start by thanking the CCC organizers for inviting us out here to
give this talk. This was a unique opportunity for us to share some of our
experience with the community, and we're really happy to be here. So yeah, I hope
you guys enjoy. OK, so who are we? Well, my name is Marcus Gaasedelen. I sometimes
go by the handle @gaasedelen which is my last name. And I'm joined here by my co-
worker Amy who was also a good friend and longtime collaborator. We worked for a
company called Ret2 Systems. Ret2 is best known publicly for its security research
and development. Behind the scenes we do consulting and have been pushing to
improve the availability of security education, and specialized security
training, as well as, raising awareness and sharing information like you're going
to see today. So this talk has been structured roughly to show our approach in
breaking some of the world's most hardened consumer software. In particular, we're
going to talk about one of the Zero-Days that we produce at Ret2 in 2018. And over
the course of the talk, we hope to break some common misconceptions about the
process of Zero-Day Engineering. We're going to highlight some of the
observations that we've gathered and built up about this industry and this trade over
the course of many years now. And we're going to try to offer some advice on how
to get started doing this kind of work as an individual. So, we're calling this talk
a non-technical commentary about the process of Zero-Day Engineering. At times,
it may seem like we're stating the obvious. But the point is to show that
there's less magic behind the curtain than most of the spectators probably realize.
So let's talk about PWN2OWN 2018. For those that don't know, PWN2OWN is an
industry level security competition, organized annually by Trend Micro's Zero-
Day Initiative. PWN2OWN invites the top security researchers from around the world
to showcase Zero-Day exploits against high value software targets such as premier web
browsers, operating systems and virtualization solutions, such as Hyper-V,
VMware, VirtualBox, Xen, whatever. So at Ret2, we thought it would be fun to
play on PWN2OWN this year. Specifically we wanted to target the competition's browser
category. We chose to attack Apple's Safari web browser on MacOS because it was
new, it was mysterious. But also to avoid any prior conflicts of interest. And so
for this competition, we ended up developing a type of Zero-Day, known
typically as a single click RCE or Safari remote, kind of as some industry language.
So what this means is that we could gain remote, root level access to your Macbook,
should you click a single malicious link of ours. That's kind of terrifying. You
know, a lot of you might feel like you're very prone to not clicking malicious
links, or not getting spearfished. But it's so easy. Maybe you're in a coffee shop, maybe
I just man-in-the-middle your connection. It's pretty, yeah, it's a pretty scary
world. So this is actually a picture that you took on stage at PWN2OWN 2018,
directly following our exploit attempt. This is actually Joshua Smith from ZDI
holding the competition machine after our exploit had landed, unfortunately, a
little bit too late. But the payload at the end of our exploit would pop Apple's
calculator app and a root shell on the victim machine. This is usually used to
demonstrate code execution. So, for fun we also made the payload change the desktop's
background to the Ret2 logo. So that's what you're seeing there. So, what makes a
Zero-Day a fun case study, is that we had virtually no prior experience with Safari
or MacOS going into this event. We literally didn't even have a single
Macbook in the office. We actually had to go out and buy one. And, so as a result
you get to see, how we as expert researchers approach new and unknown
software targets. So I promised that this was a non-technical talk which is mostly
true. That's because we actually publish all the nitty gritty details for the
entire exploit chain as a verbose six part blog series on our blog this past summer.
It's hard to make highly tactical talks fun and accessible to all audiences. So
we've reserved much of the truly technical stuff for you to read at your own leisure.
It's not a prerequisite for this talk, so don't feel bad if you haven't read those.
So with that in mind, we're ready to introduce you to the very 1st step of what
we're calling, The Layman's Guide to Zero- Day Engineering. So, at the start of this
talk, I said we'd be attacking some of the most high value and well protected
consumer software. This is no joke, right? This is a high stakes game. So before any
of you even think about looking at code, or searching for vulnerabilities in these
products, you need to set some expectations about what you're going to be
up against. So this is a picture of you. You might be a security expert, a software
engineer, or even just an enthusiast. But there are some odd twists of self-
loathing, you find yourself interested in Zero-Days, and the desire to break some
high impact software, like a web browser. But it's important to recognize that
you're looking to devise some of the largest, most successful organizations of
our generation. These types of companies have every interest in securing their products
and building trust with consumers. These vendors have steadily been growing their
investments in software and device security, and that trend will only
continue. You see cyber security in headlines every day, hacking, you know,
these systems compromised. It's only getting more popular. You know, there's
more money than ever in this space. This is a beautiful mountain peak that
represents your mission of, I want to craft a Zero-Day. But you're cent up this
mountain is not going to be an easy task. As an individual, the odds are not really
in your favor. This game is sort of a free for all, and everyone is at each other's
throats. So, in one corner is the vendor, might as well have infinite money and
infinite experience. In another corner, is the rest of the security research
community, fellow enthusiasts, other threat actors. So, all of you are going to
be fighting over the same terrain, which is the code. This is unforgiving terrain
in and of itself. But the vendor has home field advantage. So these obstacles are
not fun, but it's only going to get worse for you. Newcomers often don't prepare
themselves for understanding what kind of time scale they should expect when working
on these types of projects. So, for those of you who are familiar with the Capture
The Flag circuit. These competitions usually are time boxed from 36 to 48
hours. Normally, they're over a weekend. We came out of that circuit. We love the
sport. We still play. But how long does it take to develop a Zero-Day? Well, it can
vary a lot. Sometimes, you get really lucky. I've seen someone produce a
Chrome-/V8-bug in 2 days. Other times, it's taken two weeks. Sometimes, it takes
a month. But sometimes, it can actually take a lot longer to study and exploit new
targets. You need to be thinking, you know, you need to be looking at time in
these kind of scales. And so it could take 3.5 months. It could take maybe even 6
months for some targets. The fact of the matter is that it's almost impossible to
tell how long the process is going take you. And so unlike a CTF challenge,
there's no upper bound to this process of Zero-Day Engineering. There's no guarantee
that the exploitable bugs you need to make a Zero-Day, even exist in the software
you're targeting. You also don't always know what you're looking for, and you're
working on projects that are many order magnitudes larger than any sort of
educational resource. We're talking millions of lines of code where your
average CTF challenge might be a couple hundred lines to see at most. So I can
already see the tear and self-doubt in some of your eyes. But I really want to
stress that you shouldn't be too hard on yourself about this stuff. As a novice,
you need to keep these caveats in mind and accept that failure is not unlikely in the
journey. All right? So please check this box before watching the rest of the talk.
So having built some psychological foundation for the task at hand, the next
step in the Layman's Guide is what we call reconnaissance. So this is kind of a goofy
slide, but yes, even Metasploit reminds you to start out doing recon. So with
regard to Zero-Day Engineering, discovering vulnerabilities against large
scale software can be an absolutely overwhelming experience. Like that
mountain, it's like, where do I start? What hill do I go up? Like, where do I
even go from there? So to overcome this, it's vital to build foundational knowledge
about the target. It's also one of the least glamorous parts of the Zero-Day
development process. And it's often skipped by many. You don't see any of the
other speakers really talking about this so much. You don't see blog posts where
people are like, I googled for eight hours about Apple Safari before writing a Zero-
Day for it. So you want to aggregate and review all existing research related to
your target. This is super, super important. So how do we do our recon? Well
the simple answer is Google everything. This is literally us Googling something,
and what we do is we go through and we click, and we download, and we bookmark
every single thing for about five pages. And you see all those buttons that you
never click at the bottom of Google. All the things are related searches that you
might want to look at. You should definitely click all of those. You should
also go through at least four or five pages and keep downloading and saving
everything that looks remotely relevant. So you just keep doing this over, and
over, and over again. And you just Google, and Google, and Google everything that you
think could possibly be related. And the idea is, you know, you just want to grab
all this information, you want to understand everything you can about this
target. Even if it's not Apple Safari specific. I mean, look into V8, look into
Chrome, look into Opera, look into Chakra, look into whatever you want. So the goal
is to build up a library of security literature related to your target and its
ecosystem. And then, I want you to read all of it. But I don't want you, don't,
don't force yourself to understand everything in your sack in your
literature. The point of this exercise is to build additional concepts about the
software, its architecture and its security track record. By the end of the
reconnaissance phase, you should aim to be able to answer these kinds of questions
about your target. What is the purpose of the software? How is it architected? Can
anyone describe what WebKit's architecture is to me? What are its major components?
Is there a sandbox around it? How do you debug it? How did the developers debug it?
Are there any tips and tricks, are there special flags? What does its security
track record look like? Does it have historically vulnerable components? Is
there existing writeups, exploits, or research in it? etc. All right, we're
through reconnaissance. Step 2 is going to be target selection. So, there's actually
a few different names that you could maybe call this. Technically, we're targeting
Apple's Safari, but you want to try and narrow your scope. So what we're looking
at here is a TreeMap visualization of the the Web Kit source. So Apple's Safari web
browser is actually built on top of the Web Kit framework which is essentially a
browser engine. This is Open Source. So yeah, this is a TreeMap visualization of
the source directory where files are sorted in by size. So each of those boxes
is essentially a file. While all the grey boxes, the big gray boxes are directories.
All the sub squares are files. In each file is sized, based on its lines of code.
Hue, the blue hues represent approximate maximum cyclomatic complexity detected in
each source file. And you might be getting, anyway, you might be getting
flashbacks back to that picture of that mountain peak. How do you even start to
hunt for security vulnerabilities in a product or codebase of this size?
3 million lines of code. You know, I've maybe written like, I don't know, like a
100,000 lines of C or C++ in my life, let alone read or reviewed 3 million. So the
short answer to breaking this problem down is that you need to reduce your scope of
valuation, and focus on depth over breadth. And this is most critical when
attacking extremely well picked over targets. Maybe you're probing an IoT
device? You can probably just sneeze at that thing and you are going to find
vulnerabilities. But you know, you're fighting on a very different landscape here.
And so you need to be very detailed with your review. So reduce your scope.
Our reconnaissance and past experience with exploiting browsers had lead us to
focus on WebKit's JavaScript engine, highlighted up here in orange. So, bugs in
JS engines, when it comes to browsers, are generally regarded as extremely powerful
bugs. But they're also few and far between, and they're kind of becoming more
rare, as more of you are looking for bugs. More people are colliding, they're dying
quicker. And so, anyway, let's try to reduce our scope. So we reduce our scope
from 3 million down in 350,000 lines of code. Here, we'll zoom into that orange.
So now we're looking at the JavaScript directory, specifically the JavaScript
core directory. So this is a JavaScript engine within WebKit, as used by Safari,
on MacOS. And specifically, to further reduce our scope, we chose to focus on the
highest level interface of the JavaScript core which is the runtime folder. So this
contains code that's almost 1 for 1 mappings to JavaScript objects and methods
in the interpreter. So, for example, Array.reverse, or concat, or whatever.
It's very close to what you JavaScript authors are familiar with. And so this is
what the runtime folder looks like, at approximately 70,000 lines of code. When
we were spinning up for PWN2OWN, we said, okay, we are going to find a bug in this
directory in one of these files, and we're not going to leave it until we have, you
know, walked away with something. So if we take a step back now. This is what we
started with, and this is what we've done. We've reduced our scope. So it helps
illustrate this, you know, whittling process. It was almost a little bit
arbitrary. There's a lot, previously, there's been a lot of bugs in the runtime
directory. But it's really been cleaned up the past few years. So anyway, this is
what we chose for our RCE. So having spent a number of years going back and forth
between attacking and defending, I've come to recognize that bad components do not
get good fast. Usually researchers are able to hammer away at these components
for years before they reach some level of acceptable security. So to escape Safari's
sandbox, we simply looked at the security trends covered during the reconnaissance phase.
So, this observation, historically bad components will often take years to
improve, means that we chose to look at WindowServer. And for those that don't
know, WindowServer is a root level system service that runs on MacOS. Our research
turned up a trail of ugly bugs from a MacOS, from essentially the WindowServer
which is accessible to the Safari sandbox. And in particular, when we're doing our
research, we're looking at ZDI's website, and you can just search all their
advisories that they've disclosed. And in particular in 2016, there is over 10 plus
vulnerabilities report to ZDI that were used as sandbox escapes or privilege
escalation style issues. And so, these are only vulnerabilities that are reported to
ZDI. If you look in 2017, there is 4 all, again, used for the same purpose. I think,
all of these were actually, probably used at PWN2OWN both years. And then in 2018,
there is just one. And so, this is 3 years. Over the span of 3 years where
people were hitting the same exact component, and Apple or researchers around
the world could have been watching, or listening and finding bugs, and fighting
over this land right here. And so, it's pretty interesting. I mean, they give some
perspective. The fact of the matter is that it's hard to write, it's really hard
for bad components to improve quickly. Nobody wants to try and sit down and
rewrite bad code. And vendors are terrified, absolutely terrified of
shipping regressions. Most vendors will only patch or modify old bad code only
when they absolutely must. For example, when a vulnerability is reported to them.
And so, as listed on this slide, there's a number of reasons why a certain module or
component has a terrible security track record. Just try to keep in mind, that's
usually a good place to look for more bugs. So if you see a waterfall of bugs
this year in some component, like, wasm or JIT, maybe you should be looking there,
right? Because that might be good for a few more years. All right.
Step three. So after all this talk, we are finally getting to a point where we can start
probing and exploring the codebase in greater depth. This step is all about bug hunting.
So as an individual researcher or small organization, the hardest part of
the Zero-Day engineering process is usually discovering and exploiting a
vulnerability. That's just kind of from our perspective. This can maybe vary from
person to person. But you know, we don't have a hundred million dollars to spend on
fuzzers, for example. And so we literally have one Macbook, right? So it's kind of
like looking for a needle in a haystack. We're also well versed in the exploitation
process itself. And so those end up being a little bit more formulaic for ourselves.
So there are two core strategies for finding exploitable vulnerabilities.
There's a lot of pros and cons to both of these approaches. But I don't want to
spend too much time talking about their strengths or weaknesses. So they're all
listed here, the short summary is that fuzzing is the main go-to strategy for
many security enthusiasts. Some of the key perks is that it's scalable and almost
always yields result. And so, spoiler alert, but later in the software industry,
we fuzzed both of our bugs. Both the bugs that we use for a full chain. And we know
it's 2018. These things are still falling out with some very trivial means. OK. So,
source review is the other main strategy. Source review is often much harder for
novices, but it can produce some high quality bugs when performed diligently.
You know if you're looking to just get into this stuff, I would say, start real
simple, start with fuzzing and see how fare you get. So, yeah, for the purpose of
this talk, we are mostly going to focus on fuzzing. This is a picture from the
dashboard of a simple, scalable fuzzing harness we built for JavaScript core. This
is when we were ramping up for PWN2OWN and trying to build our chain. It was a
grammar based JavaScript fuzzer, based on Mozilla's Darma. There is nothing fancy
about it. This is a snippet of some of what some of its output looked like. We
had only started building it out when we actually found the exploitable
vulnerability that we ended up using. So we haven't really played with this much
since then, but it's, I mean, it shows kind of how easy it was to get where we
needed to go. So, something like we'd like to stress heavily to the folks who fuzz,
is that it really must be treated as a science for these competitive targets.
Guys, I know code coverage isn't the best metric, but you absolutely must use some
form of introspection to quantify the progress and reach of your fuzzing. Please
don't just fuzz blindly. So our fuzzer would generate web based code coverage
reports of our grammars every 15 minutes, or so. This allowed us to quickly iterate
upon our fuzzer, helping generate more interesting complex test cases. A good
target is 60 percent code coverage. So you can see that in the upper right hand
corner. That's kind of what we were shooting for. Again, it really varies from
target to target. This was also just us focusing on the runtime folder. If you see
in the upper left hand corner. And so, something that we have observed, again
over many targets and exotic, exotic targets, is that bugs almost always fall
out of what we call the hard fought final coverage percentages. And so, what this
means is, you might work for a while, trying to build up your coverage, trying
to, you know, build a good set of test cases, or Grammar's for fuzzing, and then
you'll hit that 60 percent, and you'll be, okay, what am I missing now? Like everyone
gets that 60 percent, let's say. But then, once you start inching a little bit
further is when you start fining a lot of bugs. So, for example, we will pull up
code, and we'll be like, why did we not hit those blocks up there? Why are those
grey box? Why did we never hit those in our millions of test cases? And we'll go
find that that's some weird edge case, or some unoptimized condition, or something
like that, and we will modify our test cases to hit that code. Other times we'll
actually sit down pull it up on our projector and talk through some of that
and we'll be like: What the hell is going on there? This is actually, it's funny,
this is actually a live photo that I took during our pawn2own hunt. You know as
cliche as this picture is of hackers standing in front of like a dark screen in
a dark room, this was absolutely real. You know we were we were just reading some
code. And so it's good to rubber ducky among co-workers and to hash out ideas to
help confirm theories or discard them. And so yeah this kinda leads us to the next
piece of advice is when you're doing source reviews so this applies to both
debugging or assessing those corner cases and whatnot. If you're ever unsure about
the code that you're reading you absolutely should be using debuggers and
dynamic analysis. So as painful as it can maybe be to set up a JavaScript core or
debug this massive C++ application that's dumping these massive call stacks that are
100 [steps] deep you need to learn those tools or you are never going to be able to
understand the amount of context necessary for some of these bugs and complex code.
So for example one of our blog posts makes extensive use of rr to reverse or to root
cause the vulnerability that we end up exploiting. It was a race condition in the
garbage collector - totally wild bug. There's probably, I said there's probably
3 people on earth that could have spotted this bug through source review. It
required immense knowledge of code base in my opinion to be able to recognize this as
a vulnerability. We found it through fuzzing, we had a root cause it using time
travel debugging. Mozilla's rr which is an amazing project. And so, yeah. Absolutely
use debuggers. This is an example of a call stack again, just using a debugger to
dump the callstack from a function that you are auditing can give you an insane
amount of context as to how that function is used, what kind of data it's operating
on. Maybe, you know, what kind of areas of the codebase it's called from. You're not
actually supposed to be able to read the size or read the slide but it's a
backtrace from GDB that is 40 or 50 call stacks deep. All right. So there is this
huge misconception by novices that new code is inherently more secure and that
vulnerabilities are only being removed from code bases, not added. This is almost
patently false and this is something that I've observed over the course of several
years. Countless targets you know, code from all sorts of vendors. And there's
this really great blog post put out by Ivan from GPZ this past fall and in his
blog post he basically ... so one year ago he fuzzed WebKit using his fuzzer
called Domato. He found a bunch of vulnerabilities, he reported them and then
he open sourced the fuzzer. But then this year, this fall, he downloads his fuzzer,
ran it again with little to no changes, just to get things up and running. And
then he found another eight plus exploitable use after free vulnerabilities.
So what's really amazing about this, is when you look at these last
two columns that I have highlighted in red, virtually all the bugs he found had
been introduced or regressed in the past 12 months. So yes, new vulnerabilities get
introduced every single day. The biggest reason new code is considered harmful, is
simply that it's not had years to sit in market. This means it hasn't had time to
mature, it hasn't been tested exhaustively like the rest of the code base. As soon as
that developer pushes it, whenever it hits release, whenever it hits stable that's
when you have a billion users pounding at - it let's say on Chrome. I don't know how
big that user base is but it's massive and that's a thousand users around the world
just using the browser who are effectively fuzzing it just by browsing the web. And
so of course you're going to manifest interesting conditions that will cover
things that are not in your test cases and unit testing. So yeah, it's not uncommon.
The second point down here is that it's not uncommon for new code to break
assumptions made elsewhere in the code base. And this is also actually extremely
common. The complexity of these code bases can be absolutely insane and it can be
extremely hard to tell if let's say some new code that Joe Schmoe, the new
developer, adds breaks some paradigm held by let's say the previous owner of the
codebase. He maybe doesn't understand it as well - you know, maybe it could be an
expert developer who just made a mistake. It's super common. Now a piece of advice.
This should be a no brainer for bug hunting but novices often grow impatient
and start hopping around between code and functions and getting lost or trying to
chase use-after-frees or bug classes without really truly understanding what
they're looking for. So a great starting point is always identify the sources of
user input or the way that you can interface with the program and then just
follow the data, follow it down. You know what functions parse it, what manipulates
your data, what reads it, what writes to it. You know just keep it simple. And so
when we're looking for our sandbox escapes when you're looking at window server and
our research had showed that there's all of these functions we don't know anything
about Mac but we read this blog post from Keine that was like "Oh there's all these
functions that you can send data to in window server" and apparently there's
about six hundred and there are all these functions prefixed with underscore
underscore x. And so the 600 end points will parse and operate upon data that we
send to them. And so to draw a rough diagram, there is essentially this big red
data tube from the safari sandbox to the windows server system service. This tube
can deliver arbitrary data that we control to all those six hundred end points. We
immediately thought let's just try to man- in-the-middle this data pipe, so that we
can see what's going on. And so that's exactly what we did. We just hooked up
FRIDA to it, another open source DBI. It's on GitHub. It's pretty cool. And we're
able to stream all of the messages flowing over this pipe so we can see all this data
just being sent into the window server from all sorts of applications - actually
everything on Mac OS talks to this. The window server is responsible for drawing
all your windows on the desktop, your mouse clicks, your whatever. It's kind of
like explorer.exe on Windows. So you know we see all this data coming through, we
see all these crazy messages, all these unique message formats, all these data
buffers that it's sending in and this is just begging to be fuzzed. And so we said
"OK, let's fuzz it" and we're getting all hyped and I distinctly remember thinking
maybe we can jerry-rig AFL into the window server or let's mutate these
buffers with Radamsa or why don't we just try flipping some bits. So that's what we
did. So I actually had a very timely tweet just a few weeks back that echoed this
exact experience. He said that "Looking at my Security / Vulnerability research
career, my biggest mistakes were almost always trying to be too clever. Success
hides behind what is the dumbest thing that could possibly work." The takeaway
here is that you should always start simple and iterate. So this is our Fuzz
farm. It's a single 13 inch MacBook Pro. I don't know if this is actually going to
work, it's not a big deal if it doesn't. I'm only gonna play a few seconds of it.
This is me literally placing my wallet on the enter key and you can see this box
popping up and we're fuzzing - our fuzzer is running now and flipping bits in the
messages. And the screen is changing colors. You're going to start seeing the
boxes freaking out. It's going all over the place. This is because the bits are
being flipped, it's corrupting stuff, it's changing the messages. Normally, this
little box is supposed to show your password hint. But the thing is by holding
the enter key on the locked screen. All this traffic was being generated to the
window server, and every time the window server crashed - you know where it brings
you? It brings you right back to your lock screen. So we had this awesome fuzzing
setup by just holding the enter key. *Applause*
And we you know we lovingly titled that picture "Advanced Persistent Threat" in
our blog. So this is a crash that we got out of the fuzzer. This occurred very
quickly after ... this was probably within the first 24 hours. So we found a ton of
crashes, we didn't even explore all of them. There is probably a few still
sitting on our server. But there's lots and all the rest ... lots of garbage. But
then this one stands out in particular: Anytime you see this thing up here that says
"EXC_BAD_ACCESS" with a big number up there with address equals blah blah blah.
That's a really bad place to be. And so this is a bug that we end up using at
pwn2own to perform our sandbox escape if you want to read about it again, it's on
the blog, we're not going to go too deep into it here. So maybe some of you have
seen the infosec comic. You know it's all about how people try to do these really
cool clever things. They get ... People can get too caught up trying to inject so
much science and technology into these problems that they often miss the forest
for the trees. And so here we are in the second panel. We just wrote this really
crappy little fuzzer and we found our bug pretty quickly. And this guy's really
upset. Which brings us to the misconception that only expert researchers
with blank tools can find bugs. And so you can fill in the blank with whatever you
want. It can be cutting edge tools, state of the art, state sponsored, magic bullet.
This is not true. There are very few secrets. So the next observation: you
should be very wary of any bugs that you find quickly. A good mantra is that an
easy to find bug is just as easily found by others. And so what this means is that
soon after our blog post went out ... actually at pwn2own 2018 we actually knew
we had collided with fluorescence one of the other competitors. We both struggled
with exploiting this issue ... is a difficult bug to exploit. And we were ...
we had some very creative exploit, it was very strange. But there was some
discussion after the fact on Twitter by nadge - started by Neddy -he's probably
out here, actually speaking tomorrow. You guys should go see this talk about the
Chrome IPC. That should be really good. But there is some discussion on Twitter,
that Ned had started, and Nicholas, who is also here, said "well, at least three
teams found it separately". So at least us, fluorescence and Nicholas had found
this bug. And we were all at pwn2own, so you can think how many people out there
might have also found this. There's probably at least a few. How many people
actually tried to weaponize this thing? Maybe not many. It is kind of a difficult
bug. And so there are probably at least a few other researchers who are aware of
this bug. So yeah, that kinda closes the, you know, if you found a bug very quickly
especially with fuzzing, you can almost guarantee that someone else has found it.
So I want to pass over the next section to Amy to continue.
Amy: So we just talked a bunch about, you know, techniques and expectations when
you're actually looking for the bug. Let me take over here and talk a little bit
about what to expect when trying to exploit whatever bug you end up finding.
Yeah and so we have the exploit development as the next step. So OK, you
found a bug right, you've done the hard part. You were looking at whatever your
target is, maybe it's a browser maybe it's the window server or the kernel or
whatever you're trying to target. But the question is how do you actually do the rest?
How do you go from the bug to actually popping a calculator onto the
screen? The systems that you're working with have such a high level of complexity
that even just like understanding you know enough to know how your bug works it might
not be enough to actually know how to exploit it. So we try to like brute force
our way to an exploit, is that a good idea? Well all right before we try to
tackle your bug let's take a step back and ask a slightly different question. How do
we actually write an exploit like this in general? Now I feel like a lot of people
consider these kind of exploits maybe be in their own league at least when you
compare them to something like maybe what you'd do at a CTF competition or something
simpler like that. And if you were for example to be given a browser exploit
challenge at a CTF competition it may seem like an impossibly daunting task has just
been laid in front of you if you've never done this stuff before. So how can we work
to sort of change that view? And you know it might be kind of cliche but I actually
think the best way to do it is through practice. And I know everyone says "oh how
do you get good", "oh, practice". But I think that this is actually very valuable
for this and the way that practicing actually comes out is that, well, before we
talked a lot about consuming everything you could about your targets, like
searching for everything you could that's public, downloading it, trying to read it
even if you don't understand it, because you'll hopefully gleam something from it
it doesn't hurt but maybe your goal now could be actually trying to understand it
as at least as much as you can. You know it's going to be... it's not going to be
easy. These are very intricate systems that we're attacking here. And so it will
be a lot of work to understand this stuff. But for every old exploit you can work
your way through, the path will become clearer for actually exploiting these
targets. So as because I focused mostly on browser work and I did that browser part
of our chain, at least the exploitation part. I have done a lot of exploits and
read a ton of browser exploits and one thing that I have found is that a lot of
them have very very similar structure. And they'll have similar techniques in them
they'll have similar sort of primitives that are being used to build up the
exploit. And so that's one observation. And to actually illustrate that I have an
example. So alongside us at this at PWN2OWN this spring we had Samuel Groffs
of Phoenix. He's probably here right now. So he was targeting Safari just like we
were. But his bug was in the just in time compiler, the JIT, which converts
JavaScript to the machine code. Our bug was nowhere near that. It was over in the
garbage collector so a completely different kind of bug. But the bug here is
super reliable. It was very very clean. I recommend you go look at it online. It's a
very good resource. And then, a few months later, at PWN2OWN Mobile, so another pwning
event, we have Fluoroacetate, which was an amazing team who managed to pretty much
pwn everything they could get their hands on at that competition, including an
iPhone which of course iPhone uses Safari so they needed a Safari bug. The safari
bug that they had was very similar in structure to the previous bug earlier that
year, at least in terms of how the bug worked and what you could do with it. So
now you could exploit both of these bugs with very similar exploit code almost in
the same way. There were a few tweaks you had to do because Apple added a few things
since then. But the path between bug and code execution was very similar. Then,
even a few months after that, there is a CTF called "Realworld CTF" which took
place in China and as the title suggests they had a lot of realistic challenges
including Safari. So of course my team RPISEC was there and they woke me up in
the middle of the night and tasked me with solving it. And so I was like "Okay, okay
I'll look at this". And I looked at it and it was a JIT bug and I've never actually
before that looked at the Safari JIT. And so, you know, I didn't have much previous
experience doing that, but because I had taken the time to read all the public
exploits. So I read all the other PWN2OWN competitors exploits and I read all the
other things that people were releasing for different CVEs. I had seen a bug like
this before very similar and I knew how to exploit it, so I could... I was able to
quickly build the path from bug to code exec and we actually managed to get first
blood on the challenge which was really really cool.
*Applaus* So... So what does this actually mean?
Well I think not every bug is going to be that easily to swap into an exploit but I
do think that understanding old exploits is extremely valuable if you're trying to
exploit new bugs. A good place to start if you're interested in looking at old bugs
is on places like this with the js-vuln-db, which is a basically a repository of a
whole bunch of JavaScript bugs and proof of concepts and sometimes even exploits
for them. And so if you were to go through all of those, I guarantee by the end you'd
have a great understanding of the types of bugs that are showing up these days and
probably how to exploit most of them. And... But there aren't that many bugs
that get published that are full exploits. There's only a couple a year maybe. So
what do you do from there once you've read all those in and you need to learn more?
Well maybe start trying to exploit other bugs yourself so you can go... For
example, I like Chrome because they have a very nice list of all their
vulnerabilities that they post every time they have an update and they even link you
to the issue, so you can go and see exactly what was wrong and so take some of
these for example, at the very top you have out-of-bounds write in V8. So we
could click on that and go and see what the bug was and then could try to write an
exploit for it. And then by the end we all have had a much better idea of how to
exploit an out-of-bounds write in V8 and we've now done it ourselves too. So this
is a chance to sort of apply what you've learned. But you say OK that's a lot of
work. You know I have to do all kinds of other stuff, I'm still in school or I have
a full time job. Can't I just play CTFs? Well it's a good question. The question is
how much do CTFs actually help you with these kind of exploits. I do think that
you can build a very good mindset for this because you need a very adversarial
mindset to do this sort of work. But a lot of the times the challenges don't really
represent the real world exploitation. There was a good tweet just the other day,
like a few days ago, where we were saying that yeah libc is... like random libc
challenges - Actually I don't think it's... Yes. It's libc here. Yeah. - are
often very artificial and don't carry much value to real world because they're very
specific. Some people love these sort of very specific CTF challenges, but I don't
think that there's as much value as there could be. However a lot of... There's been
a couple CTFs recently and historically as well that have had pretty realistic
challenges in them. In fact right now is a CTF, 35c3 CTF is running and they have 3
browser exploit challenges, they have a full chain safari challenge, they have a
virtual box challenge, It's like it's pretty crazy and it's crazy to see people
solve those challenges in such a short time span too. But I think it's definitely
something that you can look at afterwards even if you don't manage to get through
one of those challenges today. But something to try to work on. And so these
are the sort of new CTFs are actually pretty good for people who want to jump
off to this kind of real estate or a real exploit development work. However it can
be kind of scary for newer newcomers to the CTF scene because suddenly you know
it's your first CTF and they're asking you to exploit Chrome and you're like what...
what is going on here. So there, it is a double edged sword sometimes. All right so
now we found the bug and we have experience, so what do we actually do?
Well you have to kind of have to get lucky though because even if you've had a ton of
experience that doesn't necessarily mean that you can instantly write an exploit
for a bug. Our javascript exploit was kind of like that, it was kind of nice, we knew
what to do very right away but the brows... or our sandbox exploit did not
fit into a nice box of a previous exploit that we had seen. So it took a lot of
effort. Quickly I'll show... So this was the actual bug that we exploited for the
Sandbox. It's a pretty simple bug. It's a integer issue where index is signed which
means it can be negative. So normally it expects a value like 4 but we could give
it a value like negative 3 and that would make it go out of bounds and we could
corrupt memory. So very simple bug not like a crazy complex one like some of the
other ones we've seen. But does that mean that this exploit is going to be really
simple? Well let's see... That's a lot of code. So our exploit for this bug ended up
being about 1300 lines. And so that's pretty crazy. And you're probably
wondering how it gets there but I do want to say just be aware that when you do find
it simple looking bug, it might not be that easy to solve or to exploit. And it
might take a lot of effort but don't get discouraged if it happens to you. It just
means it's time to ride the exploit development rollercoaster. And basically
what that means is there's a lot of ups and downs to an exploit and we have to
basically ride the rollercoaster until hopefully we have it, the exploit,
finished and we had to do that for our sandbox escape. And so to say we found the
bug and we had a bunch of great ideas we'd previously seen a bug exploited like this
by Kiehne and we read their papers and we had a great idea but then we were like OK
OK it's going to work we just have to make sure this one bit is not set and it was
like in a random looking value, so we assumed it would be fine. But turns out
that bit is always set and we have no idea why and no one else knows why, so thank
you Apple for that. And so OK maybe we can work around it, maybe we can figure out a
way to unset it and we're like oh yes we can delete it. It's going to work again!
Everything will be great! Until we realize that actually breaks the rest of the
exploit. So it's this back and forth, it's an up and down. And you know sometimes
when you solve one issue you think you've got what you need and then another issue
shows up. So it's all about making incremental
progress towards removing all the issues that are in your way, getting at least
something that works. Marcus: And so just as a quick aside, this
all happened within like 60 minutes one night.
Amy: Yeah. Marcus: Amy saw me just like I was
walking, out of breath, I was like Are you kidding me? There's two bugs that tripped
us up that made this bug much more difficult to exploit. And there is no good
reason for why those issues were there and that it was a horrible experience.
Amy: But it's still one I'd recommend. And so this rollercoaster it actually applies
to the entire process not just for, you know, the exploit development because
you'll have it when you find crashes that don't actually lead to vulnerabilities or
unexplainable crashes or super unreliable exploits. You just have to keep pushing
your way through until eventually you hopefully you get to the end of the ride
and you've got yourself a nice exploit. And so now. OK. So we assume, OK, we've
written an exploit at this point. Maybe it's not the most reliable thing but it
works like I can get to my code exec every now and then. So we have to start talking
about the payload. So what is the payload exactly? A payload is whatever your
exploits trying to actually do. It could be trying to open up a calculator on the
screen, it could be trying to launch your sandbox escape exploit, it could be trying
to clean up your system after your exploit. And by that, I mean fix the
program that you're actually exploiting. So in CTF we don't get a lot of practice
with this because we're so used to doing 'system', you know, 'cat flag', and then
it doesn't matter if the entire program is crashing down in flames around us because
we got the flag. So in this case yeah you cat the flag and then it crashes right
away because you didn't have anything after your action. But in the real world
it kind of matters all the more, so here's an example like what would happen if your
exploit didn't clean up after itself, and just crashes and goes back to the login
screen. This doesn't look very good. If you're at a conference like Pwn2Own this
won't work, I don't think that they would let you win if this happened. And so, it's
very important to try to go back and fix up any damage that you've done to the
system, before it crashes, right after you finished. Right. And so actually running
your payload, so, a lot of times we see or in the exploits we see that you'll get to
the code exec here which is just CCs which means INT3 which just tells the
program to stop or trap to break point and all the exploits you see most of the time
they just stop here. They don't tell you anymore and to be fair you know they've
gotten a new code exec they're just talking about the exploit but you, you
still have to figure out how to do your payload because unless you want to write
those 1300 lines of code in handwritten assembly and then make it into shellcode,
you're not going to have a good time. And so, we had to figure out a way to actually
take our payload, write it to the file system in the only place that the sandbox
would let us, and then we could run it again, as a library, and then it would go
and actually do our exploit. And so yeah. And so now that you've like assembled
everything you're almost done here. You have your exploit working. You get a
calculator pops up. This was actually our sandbox escape of running and popping
calculator and proving that we had brute code exec, but we're not completely done
yet because we need to do a little bit more, which is exploit reliability. We
need to make sure that our exploit is actually as reliable as we want it to
because if it only works 1 in 100 times that's not going to be very good. For
Pwn2own, we ended up building a harness for our Mac which would let us run the
exploit multiple times, and then collect information about it so we could look
here, and we could see, very easily how often it would fail and how often it would
succeed, and then we could go and get more information, maybe a log, and other stuff
like how long it ran. And this made it very easy to iterate over our exploit, and
try to correct issues and, make it better and more reliable. We found that most of
our failures were coming from our heap groom, which is where we try to line all
your memory in certain ways, but there's not much that you can do there in our
situation, so we tried to make it as best as we could and then accepted the
reliability that we got. You, something else might want to test on is a bunch of
multiple devices. For example our javascript exploit was a race condition,
so that means the number of cpus on the device and the speed of the cpu actually
might matter when you're running your exploit. You might want to try different
operating systems or different operating system versions, because even if they're
all vulnerable, they may have different quirks, or tweaks, that you have to do to
actually make your exploit work reliably on all of them. We had to, we wanted to
test on the MacOS beta as well as the normal MacOS at least, so that we could
make sure it worked in case Apple pushed an update right before the competition. So
we did make sure that some parts of our code, and our exploit, could be
interchanged. So for example, we have addresses here that are specific to the
operating system version, and we could swap those out very easily by changing
what part of the code is done here. Yeah. And then also if you're targeting some
browsers you might be interested in testing them on mobile too even if you're
not targeting the mobile device. Because a lot of times the bugs might also work on a
phone, or at least the initial bug will. And so that's another interesting target
you might be interested in if you weren't thinking about it originally. So in
general what I'm trying to say is try throwing your exploit at everything you
can, and hopefully you will be able to recover some reliability percentages or
figure out things that you overlooked in your initial testing. Alright. I'm gonna
throw it back over, for the final section. Marcus: You're in the final section here.
So I didn't get to spend as much time as I would have liked on this section, but I
think it's an important discussion to have on here. And so the very last step of our
layman's guide is about responsibilities. And so this is critical. And so you've
listened to our talk. You've seen us develop the skeleton keys to computers and
systems and devices. We can create doors into computers and servers and people's
machines, you can invade privacy, you can deal damage to people's lives and
companies and systems and countries and so there is a lot of you have to be very
careful with these. And so everyone in this room, you know if you take any of our
advice going into this stuff, you know, please acknowledge what you're getting
into and what can be done to people. And so, there's at least one example that's
kind of related, that, you know I pulled out quickly that, you know, quickly came
to mind. It was in 2016. I have to say i remember this day actually sitting at
work. And there is this massive ddos, that plagued the Internet, at least in the
U.S., and it took down all your favorite sites:Twitter, Amazon, Netflix, Etsy,
Github, Spotify, Reddit. I remember the whole Internet came to a crawl in the U.S.
This is the L3 outage map. This was absolutely insane. And I remember people
were bouncing off the walls like crazy, you know. After the fact and they were all
referencing Bruce Schneier's blog, and there on Twitter there's all this
discussion popping up that "this was likely a state actor". This was a newly
sophisticated ddos attack. Bruce suggested it was China or Russia, or you know, some
nation state and the blog post was specifically titled someone is learning
how to take down the Internet. But then a few months later we figured out that this
was called the mirai botnet and it was actually just a bunch of kids trying to
ddos each other minecraft servers. And so it's a... it's scary because you know I
have a lot of respect for the young people and how talented they are. And it's a... But
people need to be very conscious about the damage that can be caused by these things.
Mirai, they weren't using 0days per se. Later. Now nowadays they are using 0days
but back then they weren't, they were just using an IoT based botnet. One of the
biggest in the world, our highest throughput. But it was incredibly
damaging. And you know, so when you you, it's hard to recognize the power of an
0day until you are wielding it. And so that's why it's not the first step of the
layman's guide. Once you finish this process you will come to realize the
danger that you can cause, but also the danger that you might be putting yourself
in. And so I kind of want to close on that please be very careful. All right. And so,
that's all we have. This is the conclusion. The layman's guide, that is
the summary. And if you have any questions we'll take them now. Otherwise if we run
out of time you can catch us after the talk, and we'll have some cool stickers too,
so... *Applause*
Herald-Angel: Wow, great talk. Thank you. We have very very little time for
questions. If somebody is very quick they can come up to one of the microphones in
the front, we will handle one but otherwise, will you guys be available
after the talk? Marcus: We will be available after the
talk, if ou want to come up and chat. We might get swarmed but we also have some
cool Rekto stickers, so, come grab them if you want them.
Herald-angel: And where can we find you. Marcus: We'll be over here. We'll try to
head out to the back Herald-angel: Yeah yeah because we have
another talk coming up in a moment or so. Marcus: OK.
Herald-angel: I don't see any questions. So I'm going to wrap it up at this point,
but as I said the speakers will be available. Let's give this great speech
another round of applause.
*Applause*
*Outro*
subtitles created by c3subtitles.de in the year 2020. Join, and help us!