Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1066 Thanks!
[Music]
M said that's the next title of the
title of the next talk why cleaning
memory is hard Ilia fundle is a security
researcher who loves to find out new
things and he found out that it's quite
hard to get rid of sensitive content in
the
memory yes so to today he's going to
give us an overview presentation and
please give a warm Round of Applause to
Ilia
fle okay perfect yeah so as uh as the
Herold just explained my presentation is
called m sad uh why me clearing memory
is
hard um before I dive into that um once
upon a time that was me uh lot more hair
a lot less fat um this is my 17 uh
Congress you're a pop Kettle here
buddy uh this is my 17th Congress in a
row um I've spoken here a number of
times before um I haven't added 35 in
there yet but obviously um that's uh
should be there as well I work for a
company called ioactive I am U the
director of penetration testing um but
that really just means that I lead teams
of pentesters um no uh no uh double
there um obviously we're always looking
for good security people so if you're
interested come talk to me
afterwards um I like looking at lowlevel
stuff Colonels drivers hypervisors that
type of stuff and I enjoy reading code
okay enough enough about me um what
what's the kind of audience that I think
think would enjoy this um pretty all
around security people uh crypto people
if you like code review if you like
compiler stuff and if you're just
generally curious about technology um I
I think you might enjoy this um in terms
of the knowledge required um the first
half of these slides is relatively basic
and so if you have a basic technology
understanding um you should be able to
understand most of the first half um if
you have some cback that would be nice
and then as I move forward past the
first half um it Things become a bit
more advanced um but if you just you
know only grasp the first half I I I
think that will still be
useful right so uh what does this talk
about um basically um it's one very
simple easy to explain crypto
implementation problem um and the reason
I've dedicating entire talk about it is
because while the the problem is easy
the solution is not uh there's a lot of
moving Parts there's a lot of subtlety a
lot of nuance um and I'll get into that
in a little
bit um now I can hear some of you
thinking well Ilia WTF at 2018 why the
hell are you talking about this this is
very very very well known to them I say
well because this stuff is still
everywhere um and I will show that in
the slides I will show this with data I
will I will show this with bugs um but
the driver of this talk the reason why I
start making this presentation is that
um this year alone I did engagements for
three different customers on three
entirely different software projects um
where they all had this exact same type
of bug um and you know you tell them
about the bug and the customer comes
back and says okay well yeah that's
that's great we understand um now tell
us how to fix this in ortable way um and
and that is not very easy um the other
thing is that even though this problem
is sort of known
conceptually like you know people kind
of Blas about it um practically not many
people understand um how how
um how pervasive this problem is how
realistic it is it isn't just like oh
well the compiler might do this no the
compiler will do this and it does it
everywhere um and this this this these
bugs do show up everywhere um even
though because it doesn't it's hard to
tell from the code that it is there if
you look at the binary if you look at
the compiler inits you see that it's
there um and then the third is um given
that one of the teams of the Congress
this year is foundations and to talk
about you know things that aren't
necessarily new uh but to sort of try
and help the bring the um the subject to
the Next Generation Um this I think this
uh fits in perfectly in the concept of
Foundations
right so uh before I dive in um there's
a couple of people or actually a long
list of people that sort of helped me
out um as I said this the problem is
well known it's been well known for at
least 20 or 30 years and so many many
many people have published papers and
presentations about this and um I don't
know all of them personally and and um I
wish I could include them all uh but the
people I've included in here are sort of
the you know one or two away from me
that have had some kind of impact in
these slides
um some of you are sitting in this
audience uh your and your help has been
appreciated okay so let's let's let's
let's actually start now let's say uh
you're going to write some piece of code
right and it's going to be doing
something and it's going to be handling
sensitive data you know be that keys or
decrypted PL text or session tokens or
passwords or or password hashes or
anything that could be considered
sensitive right now if you're a smart
SEC security PR conscious person right
uh the moment you are done with that
sensitive data um you want to dispose of
it right you want to purge it um from
memory right now why do you want to do
this well because otherwise um if if
there's some kind of info leak that is
discovered later on then whatever
Secrets or lingering memory uh could be
used in your info leak and all of a
sudden you know your tokens or your keys
are leaked out right and and I mean that
may sound like a stretch but you know
things like heart bleed you know happen
right so it's this is very practical
this can really happen um and um before
I move on I want to say if if you make
if if you think and make the step where
you say okay I need to dispose of
sensitive material once I'm done with it
that that's really big right most
software that deals with you know
sensitive material does not do this so
if you make this step thinking I need to
perch this um you're ahead of the Curve
right so now concretely it would look
something like this right you would
write a little this is a sample code I
have that basically generates a little
key and it's a function you give it and
you give it a key pointer and this thing
declares um a local variable that's Ser
two bytes goes and reads a bunch of
random bits puts it in K copies K to key
and then before it returns uh because K
is about to go out of scope and it
contains sensitive key material um you
go and say mem set and then you clear
the thing and then you return right
perfect
um you you uh you run this you compile
it you add a main uh and it does exactly
what it's supposed to do you look at the
assembly and it's all
perfect um problem there is what you're
doing is not that's not that's that's
not release-based code right when you uh
sort of make code ready to be released
you sort sort of tell the compiler um
that it should enable the optimize right
you'll you'll give it Os or O2 those are
the most common ones um sometimes you
see o03 when people uh when to live on
the edge but usually O2 or Os um now if
you look at the assembly again you know
you get a whole different picture of of
what's going on um and I want to
illustrate this and there's a website
called compiler
Explorer uh which is beautiful and
integrates a whole bunch of compilers
and it has on the left it shows you the
C code and the right shows you the
assembly and it's very and it it's like
color based and it's easy to make
connections so it let's let's take our
little example and on on the left we see
to generate key and on the right we see
to compiler and sure enough if you
follow the colors in left and right you
can see what CCO translates to which
assembly right to make it a bit a little
bit easier that mem set
um that mem Set uh clearly um gets
translated to assembly right now that is
when you do minus o0 which is the
default which is what you would do if
you're developing code and you're and
and you want to debug this stuff right
now once you uh you're done developing
and you're about to ship this thing you
do minus 01 for example um and this this
this little this kick the optimizer
kicks in and all of a sudden your
assembly looks a whole lot different
you'll notice it's shorter you'll notice
that all of a sudden you know the the
color of your M set changed whereas in
ozer it was the sort of redish and all
of a sudden it became White and it's
nowhere to be found in your
assembly um right it just does not show
up right that's the problem okay well
what happened right um let's let's um
yeah I stole I stole
that um so what happened is a thing
called um dead store optic optimization
or death store elimination um and so
basically that M it at the end um that
what you're doing is you are riding into
a buffer that is never ever going to get
used again um and an optimizing compiler
looks at that and says hey you know I
can just take that M set out and I just
saved you a couple of cycles and you
have a smaller binary huge win um and
because it doesn't really change what
the program does it's fully compliant
with all the the the relevant language
standards
right so that's essenti that is in a
nutshell our problem right um and so one
of the things I want to do is I wanted
to look at all common compilers and see
uh for which of these I can get it to
effectively practically optimize out a m
set like this um and I had to with some
of them it was easy some of it was
harder um I had to fill around with it
for some a straight up meet worked for
others I had to like kind of twiddle and
make a for Loop or you know kind of jump
around a bit um but essentially that
these are list of 10 compilers I tested
um I try to get my hands on the IBM
compiler but um I don't have $220,000 so
I couldn't do that um but these are the
the ones that I did test and then so the
first five or the first four you will
know you know the GCC and clang and the
Intel compiler and the Microsoft
compiler um and they're all also on a
compile Explorer so it was easy to test
those and it was very easy to get those
to um to optimize out mem sets um and
then I moved on downloaded um a bunch of
others you know the the Sun Studio
compiler and the um Embarcadero C++
Builder and the arm compiler and a bunch
of others um and out of these 10 I was
able to get um eight to optimize it out
right 80% of most of the most common
compilers do this in any in a practical
sense so it isn't just like a
theoretical thing this really happens um
uh a funny note um I tried really hard
to get the PGI compiler to do it in fact
it has it has a switch called that's St
elimination um and of course I played
with it and I tried it and I spend over
an hour trying to get it to optimize out
in my mem set goddamn thing wouldn't
move um I don't know what I don't know
what it's doing there I couldn't get it
to do
anything um but basically most compilers
if you ask them to do optimization will
gladly optimize out a lot of M
sets right so the next question is how
common is it to actually see uh uh
projects use um op optimization right um
and this sort of stems from uh a
conversation I had earlier this year
with a couple of colleagues where a
bunch of people said well you know I
don't see O2 or o1 or Os all that often
I don't think optimization is all that
common um and so I started looking
around and I said okay well where can I
get some data and the first thing I
would was so I was okay well I can go to
op. apple.com and that lists about 200
projects and I'll just go through all
their make files and look for O2 or Os
and so on and there's about 100 out of
there and then um I realized that um
they actually don't use make files they
have a really bizarre build system uh
and that build system by default uses OS
um so even though it says 100 out of 200
it's probably closer to 200 out of 200
um and then I I had a whole list of
programs I want to test in like FreeBSD
and you to and a bunch of Linux ISRO um
but that's pretty boring and I ran out
of time so I kind of stopped there um
but the these numbers should be good
enough um in addition if you look at the
common uh idees in particular visual
studio and xcode um when you tell them
to uh build a project in release mode uh
Visual Studio by default doo2 uh xcode
by default Doos right so the fact that
these tools by default give you
optimization um should should make you
confident enough in knowing that yes in
fact optimization is incredibly common
uh in release builds it it it it isn't
everywhere but it is almost
everywhere right so now that we know um
the problem and now that we know it
isn't just theoretical and that we know
it's practical and that in fact it does
occur um very very often um and with
most compilers um in fact basically it's
a real problem how do we fix this right
and this is sort of where things get
difficult
um um there are many sort of solutions
um nothing is portable right it's sort
of the okay well this solution works if
you use this compiler and this solution
works if you use this lipy and this
solution works if you use this OS and
this solution works with this version of
the language spec and this solution
works if you have this particular
executable file format right um so uh
and before I dive in any of those let's
first talk about the elfp in the room
don't just roll your own um I I've seen
people do this where they go like oh
well you know I I I'll just I'll I'll
fight with the compiler I know what I'm
doing and they'll just you know they'll
re they kind of lero Jenkins this and
they'll totally screw it up and they'll
come up with you know some really stupid
um idea one of the ones I heard was like
oh well I'll just you know what I'll
just do IO with the buffer and then it's
and and then it's cool yeah you could do
that but then you're doing IO right for
no reason
um so um don't just roll your own uh
you're going to come up with a solution
that's probably stupid you're going to
look really stupid um and it'll be one
of these things where okay you're you're
sort of your bad solution might work for
this particular version compiler uh but
if you don't understand the the concepts
behind it then chances are the next
version of compiler that is somewhat
slightly smarter um will sort of just
bypass um uh um whatever you implement
it um so so don't roll your or if you
want to roll your own at least listen to
the advice of the next 10 and then base
your um uh your Solution on at least
some of the advice that I that I'll be
giving out in next couple of slides
right so let's uh with that let's move
on to actual Solutions um the first one
is um is is ellip C function um called
explicit uh b0 and this is not part of
any standard as far as I can tell at the
present time um uh but this was um this
was sort of concocted in May 2014 by the
open BSD guys um if you'll note the date
it's pretty close to when heart bed
happened it's a few months later um I
think that may have some relation um
anyway dysfunction basically um uh does
a B zero but explicitly guarantees that
it does not get optimized out and what
that means is that what it does or it
doesn't it's no longer your problem it's
the Li's problem because they've they've
guar they made the guarantee so now it's
on them right so that's really nice um
open it first and then uh npsd said you
know that's a great idea and we're going
to steal it um but we're going to rename
it though um so they changed the name to
explicit memet but it's essentially the
same thing and then about two years and
change ago freeb Z came up uh sort of
did the same thing and then almost two
years ago the gpy guys um came up with
this too and then um diet Lipsy uh
supports this too um OSX however does
not support
it um so uh if if if you're you're if
you're limited those platforms explicit
b0 is a perfect solution uh similarly um
if you are developing for the windows
World um uh there is an API call secures
your memory which basically is Microsoft
saying we guarantee that this thing
doesn't get optimized out and if you
want a securely clear uh sensitive
material just use this API um MS n says
it will ensure that your data will be
overwritten promptly um and this one of
the cases where Microsoft was ahead of
the curve uh by like 15 years uh they've
had this thing since the early 2000s uh
it was in XP and it was in Windows 2003
uh both operating systems are no longer
support it but the API
is um okay so now there's another
function called mem setor s um and it
guarantees it doesn't get optimized out
and it's guaranteed by spec it's
guaranteed by the the language spec it
is standardized it is in C11 it is
wonderful it is great except it's not
great because uh even though it's in the
standard and it's there it's in What's
called the optional NX
k um and if you read the spec and it's a
lot it's like pages and pages of boring
crap but if you end up to the end up
reading the K2 um it it says optional
extension what does optional extension
mean it means you can be entirely C11
compliant and not offer M set s um so
it's kind of this as Reverend L Joy
would say yes with an if no with a bot
um so if it's there great if it's not
it's it doesn't it has the potential of
being this great portable solution and
then it
isn't
um right and then of course um the the
sort of uh uh obvious choice but somehow
a lot of people seem to miss this is is
if you're end up doing something with
sensitive material chances are you're
using a crypto library and if you're
using crypto Library chances are the
crypto Library um offers you an API to
do secure uh memory cleaning um and so I
listed the the the common ones if you're
using openl there's openl cleanse and
open C guarantees they don't get
optimized out if you use new TLS there's
GS memet same thing GS guarantees it
does not get optimized out um if you're
using lip sodium which is one of the
newer ones um they have sodium mzero
um same thing they uh they guarantee
doesn't get optimized out um and uh I'll
um I'll get down in a minute so those
are this is basically up until here um
I've sort of given you a list of okay
well here are specific API functions you
can call if you're using this library or
using this OS use this um the next sort
of solutions are sort of the okay well
what if you can't rely on the apis maybe
we can get something out of the compiler
right um the first solution is um and
this is this isn't portable but most
compilers have this or something like it
where you can go to the compiler and say
hey um don't use the built-in memet just
use the one from lip C and what that
means is you tell the compiler that it
shouldn't assume that it knows what mset
does um and if you do that then sure
enough mimet won't get optimized out um
GCC this is original GCC specific and
then the Intel compile supports it two
and then clang supports it two but up
until and this is true up until clang
3.7 which is maybe two years old it's
not that old uh clang basically you know
supported uh fno built-in mset and then
what they did is they kind of dropped it
on the floor and it got optimized out
anyway um so it's kind of annoying it
kind of re it kind of ruins the whole
use this because it's because it works
because except if you're using an older
version of clang um and also you know
it's not overly portable but but it's a
solution that works um other things
might still get optimized out if you're
if you have some kind of for Loop that
clears memory that might still get
optimized out but at least if you use
memset and you're using no built-in
memset then you have a pretty strong
guarantee that it shouldn't get
optimized
out so uh another sort of solution is um
you know just uh don't use
optimization U that works go you're
guaranteed not to get code to get
optimized if you don't use the optimizer
um uh obviously you know that isn't
perfect uh for one fortified seource
doesn't work if you don't use
optimization so if you want to use
fortifi seource you have to use
optimization um the other one of course
is that yeah you got to change your
build environment okay um it's I mean
it's not overly I guess it's not
portable but then again I most compilers
will have some way to tell it to not
optimize anything uh but obviously the
reason you don't want to use this
particular solution is because you know
you don't get the optimizer so your
product will probably be
slower um sort of a spin-off of this is
um some compilers in particular um the
Microsoft one and then GCC also kind of
supports it um is where you can uh
localize um uh optimizations uh based on
Scopes and functions um and so you can
say oh you know for this function do o z
um it it it seems It's not a commonly
used feature it seems like it might have
some side effects and it doesn't
support all switches that the compiler
generally does um I've seen this
recommended by a few people I played
around with it it it seems to work um
but it doesn't seem to be a um sort of
commonly adopt way of doing things um
the other thing of course is again this
this is this is uh I mean these These
are pragmas this is very very compiler
specific
stuff
um another solution is uh using what's
called weak symbols um anybody familiar
with weak
symbols okay a little bit I'll I'll I'll
try to sort of uh uh very briefly get
into this so the L file format basically
is a format that specifies how to have
an executable that can run on os's that
support this file format um so LS if
it's compiled for example for Linux
it'll be in this particular format um
and uh one of the uh obviously as part
of format you can store symbols for
things like functions and variables and
so on um and uh generally a symbol is is
what's called a strong symbol you can
mark one as weak and what weak means is
that um a symbol may change in
runtime um and what that means is if if
you declare um a function or a symbol of
a function as weak um that means that
compile time the compiler
um would have a very hard time to reason
about what that thing does um because
because of the sheer fact that you've
declared it as weak
um and in fact this this particular
solution is what openbsd uses in their
implementation of explicit
bz um and uh uh what I really like about
this is that um this is the commit
message for the open bz guys and they
they're they're uh they're very
pragmatic about this they say well you
know we think our solution is pretty
clever uh but it's not fullprof there
are still to ways to to defeat this and
they list you know bunch of ways to do
this
uh in particular you know well you know
the compiler could emit runtime code
that checks what this thing is in
runtime right before it's called um and
then you could still optimize it out if
the thing matches or doesn't match um
and but then they go on and say well in
the foreseeable future we don't think
that's going to happen um but it it's
possible that at some point down the
road this might happen um and so I like
the I like their way of reasoning about
this where solu is pretty clever but
it's not foolproof it may at some point
in the future Break um but at the
present time it's a fairly good solution
I
think right
so another solution is to use memory
barriers um does it how many people know
what a memory barrier is or what it
does okay about the same number of hands
okay let me try to very briefly um sort
of explain what a memory barrier is um
and bear with me here because it's and
I'm going to oversimplify because it's
not a particularly simple uh uh concept
if you've never heard of it um but okay
let's say let's say you have a piece of
code two Global variables A and B and
you assign a value you say a equals
something and b equals something right
and there's no relation between A and B
right what that means is both to
compiler into Hardware um because
they're they have no relation um the
both of them are allowed to reorder it
so B can be assigned first then a can be
assigned later because there's no
correlation that's that's perfectly
valid um now let's say you have a second
thread somewhere and your second thread
says okay you know while not B Spin and
then once B is set you use a right um
this is sort of where you're basically
waiting on something to be set and the
idea is that you you wrote your code so
that b is set after a is set um and and
that seems logical and that would work
except um if the compil and heartware
don't know there's a relation between
your loop on one and and the assignment
on the other and either the heartware or
the compiler reorders it and sets B
before it sets a really really nasty
things happen um this is this is and
this has been the source of numerous
security bugs very very subtle stuff um
the k s and tsan kind of stuff we've
seen Linux kernel last couple years
bunch of that is related to these kind
of bugs um and so um the way the way
that you fix this is you introduce
what's called a memory barrier and that
is basically when you write your code
and you say a equals something and then
before you say b equals something you
basically and it'll say memory barrier
and then you say b equals something and
what that means is uh it it gives a
signal to the heartware into the
compiler and it says whatever happens
before this and after this you are not
allowed to reorder this there is a
correlation there that I know that
you're not aware of so don't reorder it
um and I hope I explained it well
it this usually takes a lot longer to
explain um I hope I got the message
across well enough to sort of uh um give
give you an idea of what a memory
barrier is and now the cool thing about
memory barriers is that um it's a way
for a programmer to tell the heartware
or the compiler um I know something
about this memory you don't stay away
don't touch um and because of I mean it
works for reordering but it also works
really well to not get something up
optimized out right um the idea is you
could basically um just do your mem set
and then uh on the the thing you m set
it you basically do a memory barrier um
and uh that tells the compiler not to
optimize it out um it uh and I know the
concept sounds complicated uh but it's
it's uh pretty clever and there's I've
oversimplified this because it it's
relatively complicated subject um but uh
this works really well and this is used
by Dipsy and it's used by gpsy and um
engine X recently had a fix where they
have their own explicit m zero uh and it
also uses a memory barrier um so this is
this is a tried and tested concept and
it
works um so those are kind of the
solutions that are known and that work
or and have been tried and tested by by
various fairly well-known pieces of
software um if if if none of those
things if somehow you're in an
environment somewhere and none of this
is available to you or it's not portable
enough and you're looking for a solution
that works everywhere um the best you
can do is fall back on constructs that
are known in the sea language and this
is basically used of the volatile
keyword um this I call this a a fallback
people often go well just use volatile
and then that solves the problem and it
turns out um optimizers can be very
clever and very tricky and even when you
use volatile there are case is uh
there's cases that can be made where the
if the optimizer is clever enough um
your data may still get optimized out um
so the default ey Solutions I are sort
of best effort fallback
Solutions um and they're sort of two
variants of this one is a volatile
pointer right that's one of the that's
the fallback solution lip sodium and the
other one is um a volatile memet
function pointer which is what open cell
uses um and here's what that looks like
this is the lip sod fall back um and
this looks like you know volatile you
know you tell the compiler hey you know
I know something about this you don't um
except and this is where it gets very
langu
loyy language loyer um if you look at
the spec it says something along the
lines of the axis object something
something um and that that they're
describing the actual memory volatile
not just the pointer L value and so if
the compiler looks at this code and it
can trace and it can prove that wherever
pnt came from and if that isn't actually
volatile then this volatile doesn't
really mean all that much um and it can
still optimize it out um that sounds
very theoretical and I don't know if
that actually happens but um a number of
people smarter than me or that know more
about the sort of this nitty-gritty
little SE language things have told me
that yes in fact uh you would be allowed
to do that if you're a very smart
optimizing compiler um fact that it's a
fallback solution for uh sodium and a
few others leads me to believe that it
probably doesn't but it
could right and so this this is the um
this is the solution that open the cell
uses which also uses volatile doesn't do
a point or right but instead it creates
a volatile function pointer that points
to mem set and it sort of gives you you
get the same concept more or less as
with uh the weak symbols the idea being
is that your volatile function pointer
can change as any time without the
compiler about it um and that seems like
a pretty good solution um except um you
you when you uh one way of of in theory
getting around this is um if the
compiler um emits runtime code that
right before the function Point gets
called it looks and goes well like it
captures it and then goes is it memset
or is it something else if it's
something else then we call it if it's
memset we just return and then you
optimize out the runtime and save a few
Cycles in theory the compiler is allowed
to do that and a mid code like that I
don't know if that actually happens
anywhere um but it's a
possibility um so think of these last
two solutions as fall back they may not
work in theory in in reality they
probably
do right so this is sort of uh the SEC
the first half of my presentation and
I'm perfectly on time which is great um
so now there isn't one portable solution
and this is why this is why clearing
memory is hard right the problem is well
understood um but uh if you're looking
for like an all around solution that
works everywhere regardless of compilers
and operating systems and so on um it
it's very hard to have a good solution
um and this is what where customers have
come back to me and said give me a
portable solution I need something
better than this or that
um and so the best solution I have for
this is sort of you know apply all of
the above as best as possible um and my
initial idea was I'll just write a
little function that does this and put
it on GitHub and people can use it um
but um if you look at lip sodium and
then
um yeah you see I mean I'm not going to
click on it now but that's a link and if
if you download the slides later you'll
see it it points to GitHub and it shows
you the actual implementation uh lip
sodium's uh um m zero is is
really well written and it's beautiful
and it sort of has this fairly elegant
you know if you know this this and this
or if def this then do this solution um
L if you know this particular setup then
do this solution and it does has this
for six or seven of the cases I've
covered um it's really nice it's really
elegant uh if you're looking for
inspiration um Point people to lip
sodium I I think it's it's a good
portable is way of solving this
problem right okay so that now we've
discovered we we've talked
about uh the problem we've talked some
solutions okay well um I want detection
um when does this really happen I want
to see this right and I want compilers
to tell me this like why doesn't GCC
tell me it's not oping something outse
like if it has security consequences it
should tell me I want why are they not
doing this I don't understand so um I
set out and I modified GCC I looked at a
death elimination and I came up with
this patch and instead of um the this is
this is uh their Tre SSA death
elimination pass and when it calls uh
Delete dead call I sort of take that out
and say uh if you know uh it's a
built-in mem set um before you call
delete call admit this warning uh tell
me the file and tell me the line
number
um and then do still optimize it out and
what this means is every time an M gets
optimized out I GCC now tells me um and
this sort of uh this is very interesting
because I not only um um get get you
know um detection for my own code um
this is a great way to get to get really
cheap fast zero day um and in fact
that's what I did
um I downloaded a whole bunch of very
well-known open source uh um uh projects
and I ran them through my modified
version of um
GCC um and I came up with a list of
things oh awesome thank you
um so I know of this particular problem
in in like practically affecting op SSL
MIT curb Heim doll Curt Matrix SSL PHP
DHCP bind uh squid cash and the list
goes on uh I have our sync as well and
there's more um so we know this problem
is very widespread right if the stuff we
all rely on the stuff that is built on
has these problems that means your code
probably has this as well um and of
course I'm just giving names out
here but let's give you guys some zero
day
uh that's MIT curb that MIM sets
optimized
out uh that's PHP that M set's optimized
out uh this I think is Matrix SSL that
uh uh decrypted plane text gets mimed it
that gets optimized out that's Ling
around in memory um this is open SSL um
that uh crypto extended data that
doesn't get optimized out
um this is engine X um that password uh
that mzero gets optimized
out um this is uh bind and
DHCP uh that mem set of uh private key
data gets optimized
out um this is squid um that it goes to
L app and gets creds and uh basically it
tries to clear the creds and then that
gets optimized
out um yeah well I had to play around
with I play around with PowerPoint a
little bit um so uh and then same thing
uh this is a a a key that gets optimized
out and then this is rsync um and these
are uh stored credentials in a file and
that has random memory and those M sets
get optimized out so that's nine bugs
right there um and all it took was uh
five lines of code change in GC GC and I
it just GC just told just gave me all
these
bugs um the other thing about seeing
exactly
what thank
you the other thing that was really nice
about uh getting the data back from Gus
isn't just that it gave me bugs um it
also um showed me things that optimized
out that I thought it wouldn't um
obviously what I what I was expecting is
you know a variable that's about to go
out of scope and that you m set that
would get optimized out obviously um but
what I also noticed was that um you know
when you uh there's a common code
pattern when you uh just what you just
maloc something or you declare something
on a stack and the first thing you do is
mem set to clear the whole thing and
then you move on um it turns out that in
a number of cases that also gets
optimized out um the idea is that if the
compiler it only gets optimized out if
the compiler can prove that every
element in the struct or that the whole
field gets filled in um I'm not sure if
that's entirely true um and we were
talking about this earlier um but um
what about you know things like
structure padding or maybe enums or you
know something like that
um or you know uh unions I mean um that
I'm not quite sure how that works um I
suspect I haven't dug into it because
this is I I ran I found this yet I mean
I wrote this patch yesterday right these
these bugs are like they're fresh um so
I I don't know exactly how much
potential there is here but it smells
like there's there's room for bugs here
um so I I was I was surprised to see
this and um some more research reses
need it so if anybody uh wants to feel
free um and then the other thing I I
sort of noticed is that you obviously um
the common case what I was looking for
in terms of bugs was something that was
sensitive material um and then obviously
um I saw a whole bunch of things where
nonsensitive material um was being mems
seted and then freed and then that memet
would get optimized out as well and that
struck me as odd at the beginning but
then obviously um there's a common sort
of coding pattern uh that I've seen
where you know anytime somebody does a a
malok and then uh before using does m
set null or um when a free right before
they do a mem set zero uh not because
they want a clear sensitive material but
because they always they want to have a
guarantee they always start from a clean
slate um and so they end up building
code that ends up working for something
that always has a clean slate right um
and if that if that MSA gets optimized
out then those guarantees no longer hold
um and so um code that works around the
sort of well we always have a clean
slate when we you know when we get a
fresh piece of memory um that is no
longer true and so I I I think that
coding pattern doesn't shive well with
uh um with compiler optimization again
this is the sort of realization I made
yesterday I don't have all the facts on
this yet but it seemed interesting and I
think there's some room for research
here
um the other thing I noticed is that um
sort of close to uh the memets that were
optimized out I noticed like other bugs
um and this is kind of like you
know it made me think it's like well bad
code attracts other bad code um so one
of them was n refs and the other was the
use after free where you know instead of
doing uh memet free the code was doing
free memet right and that obviously is a
use after free um so
um this is the case of the N dra where
basically um a Malo happens and then the
memet null happens and then there's a
check to see if the variable that was
allocated is null right um obviously
that mem set um if the value is n that
meet that memet would cause a n d um
except that meet gets optimized out so
it never generates the N D um kind of a
cat 22 there um but uh um that that
construct is clearly
broken so
now I mostly spoke about memset but
really um there's a thousand variations
that clear memory uh and sort of comes
down to the same thing right obviously
you can do a for Loop or you can use
other apis or you can like roll your own
and do something very exotic um and then
there's C++ and which there's like a
zillion ways of doing it um you can have
like weird classes with like a
Constructor and like you know
inheritance multiple objects and like
virtual it gets really crazy uh once C
C++ comes in mix um and so basically it
all kind of looks different but it all
does the same thing it has the same root
causes the same problem um and the thing
is uh when you look at this from just a
code perspective is that um sometimes
the Optimizer is smart enough to see it
um sometimes the optimizer is not but it
could in the future be smart enough to
see it um so it's one of these things
where if you're looking at a piece of
code and you're doing some kind of um uh
security assessment and and you see this
and like should you should you report to
bug should you not um I think you should
because even if the compiler doesn't
optimize it out today it may very well
optimize it out
tomorrow right okay so um now oldest
talk has been about C and then when you
write it in C well what if you're not
writing C code what if you're using
other languages non- native
languages uh um you know um and go rust
Objective C CP Java flavor of the month
um and really I wanted spent more time
on this uh but then this my slides would
have gotten so long I couldn't um so I
only have one slide on nonc but I I I uh
spend a little bit of time on this um in
C there's something called secure string
which is supposed to hold a string in a
safe way um and uh the problem I have
with secure string is is isn't the
implementation it's how do I get
something securely into secure string
and how do I get something securely out
of it um and then um in terms of java
there's a Java crypto guide which
basically says um it recommends not
using strings to hold sensitive material
but to use a b instead um and this I
can't there was some reason behind I
don't remember um but the idea is
basically this um most managed languages
uh don't really offer
any decent way um to clear memory uh or
to hold sensitive material in memory uh
without it leaking and it will leak and
it'll kind of happen behind your back
because you wouldn't know it
leaked um especially you know uh when
you're dealing with like garbage
collection where something can get
reallocated without you ever knowing and
all of a sudden before you know what
there's like five different copies of
your key sprayed all over memory um most
of these languages as far as I can tell
they don't have the infrastructure in
place uh to deal with sensitive material
uh it seems to be entirely Miss in a lot
of places um and other places it's kind
of like you know shoehorned on or bolted
on with some varying degrees of success
um I remember seeing there was an uh
some impation for go but it had been in
revision three or four CU there was
always something wrong with it um and so
um again I wish I had more time that I
could elaborate on this um from what I
saw is that um it's a pretty sad State
of Affairs in notzi
um that uh most people haven't tried and
those who have have not tried hard
enough
so now that I've sort of run
through all of the sort of you know
related ISS uh sort of you know memet
problems and how to clear up um memory
um I want to sort of talk about some
related
issues um
first of all um what I said initially is
that you know when people make this step
and go oh well I should clear this uh
memory because it's sensitive um I said
that's huge because it really is uh most
code doesn't even try uh there is an
unbelievable amount of code that just uh
keeps keys in memory and sensitive
material and it goes out of scope and it
never gets cleared and it just ends up
lingering on the stack or the Heap um I
mean often it would get overwritten fast
but sometimes it could linger around and
sit there for a very very long time um
problem with this is that it's it's hard
to find in any kind of automated fashion
because it's the you're looking for the
absence of something right so that means
the only way you can really find these
kinds of bugs is to manually look at it
and go oh this is sensitive material you
didn't no no effort was made to clear
this um a a second related issue um and
this is this is really cute actually so
when you call memet the the way it's
done there uh that that that's wrong um
the length and the the bite you want a
memet with are transposed the zero
should be the second argument and a len
should be the third argument so what
that really does is a noop it basically
says
memet uh and then use Len as the pattern
and then the pattern is used as the
length in this guys in this case the
pattern is zero so a m set of zero which
becomes a no op um and what's really
cool is that um GCC actually had you can
tell TC to made a warning for this um
and what I I one of the things I wanted
to do is I wanted to run through all the
same code I had tested before and
enabled a warning and then sort of show
another list of bugs but I kind of ran
out of time uh but I strongly suspect if
you use this warning and you go download
a whole bunch of like well-known open
source code and you're run it through
you'll end up with a very similar list
of
bugs
um so um another list of sort of related
bugs um and related in the sense that no
longer clearing Secrets but related in
the sense that um optimization was
involved that is to say if the optimizer
wasn't turned on a security bug wouldn't
have occurred right or would have been
less severe right and there sort of
three cases of bugs that sort of ran
into that I I I think is somewhat
relevant um that I sort of want to talk
about so the first one is what's called
pointer overflow uh it turns out um if
you let's say let's say you have this
code PD and then and then there's a len
and the Len is untrusted right the idea
is that you want to validate that PDR
plus Len isn't beyond the end of the
your buffer but before you do that you
also want to make sure that PTR plus Len
doesn't overflow right and so you would
do code like PTR plus L is smaller than
PTR if that's the case then the pointer
overflowed and you bail out right
problem is according to the C standard
pointer overflow can't happen and so
that is undefined Behavior and the
optimizer sees that and goes oh um the
find Behavior optimized out gone so your
bound check just got optimized out um
that is a relatively common bug to see
if you don't know it's undefined
Behavior if you don't know the compiler
can optimize it out um you would just
read over it but once you know like you
start reading code and and you'll you'll
see it
everywhere um also the way to fix this
is basically to cast your pointer to an
integer type that's big enough to hold a
pointer and all of a sudden uh the
optimizer can no longer optimize it out
and it'll it'll it'll be it'll be in
your code um so that's the first one the
second one is is a lot more subtle um
this has to do with uh uh switch case
optimization um so so uh you know the
when when you have a switch in C and you
translate to assembly um you know you
could just do you get one one um sort of
translation um but if you have if you
use the optimizer um one of two things
will happen either you know it'll
generate a binary tree uh that's
generally observed in the Microsoft
compiler um if you look at uh um GCC and
clang what they do is they'll create a
jump table um and what that means is um
they'll look at the value to compare and
then if they compare is a certain number
they get the value again and they use
that value as an offset in a jump table
um and um and that usually isn't the
problem and that's an abstraction and
most people don't care about it in most
situations except uh if you're dealing
with a a shared memory trust boundary
because all of a sudden there is a
subtle double fetch that was emitted by
the compiler behind your back that
doesn't show up in the actual C code
um and and these are situations like you
know a hypervisor trust boundary right
these are very very strong trust
boundaries um and um and that thing is
actually a link if you look and if you
and if once I publish the slides if you
click on it it links to a blog post that
shows just that um it's it's a virtual
box uh guest to host prestation because
of a switch case um uh uh jump table
optimization um and and so the the the
there basically is that um when you
fetch the first time you do the compare
it's fine and then two instructions
later you fetch again to do the jump
table between the first and second fetch
um the guy on the other end of the Shar
memory could change it and all of a
sudden um you can jump outside of your
jump table and basically cause an
arbitrary jump um which obviously is bad
um yeah not going to cover this okay I I
got to wrap it up I got three more
slides and then we can get to questions
um this this is actually this is very
important so now after everything I've
covered we're good we know what the
problem is we know what the solutions
are we know there's real world problems
um but but we have a good grasp of it
now
right okay turns out I kind of lied to
you um this is not the whole problem if
it was the whole problem that'd be great
we know how to fix that more or
less um it turns out that compil
optimization is really really clever and
it does many many clever things and
they're all very subtle and a lot of
them are arure specific um and here's a
scenario of things that can occur right
um you'll have uh the optimizer will do
things like oh well you're handling this
string of a certain kind um you know
what we'll do uh I'll just shove it into
a bunch of Rister it'll be faster and
then you know it passes something and
then old son Optimizer goes oh you know
what you don't have enough registers
it's okay what I'll do is I'll take
whatever's in the register and I'll dump
it on the stack and then we'll we'll go
from there um and all of a sudden what
happened is um you leak key material
registers and then you leak them on the
stack um and this stuff happens
throughout um and um Secrets leak out it
just happens um and it is because of
optimization um even if you try really
really hard um by trying to do the
really hard to do the right thing and
the compil and the optimizer just it
just they just screw you right um and
it's problem is echoed in a blog post by
Colin pille who uh used to be the
security guy for freeb is the um now has
I think a cloud storage company very
very smart security guy um uh who had
and I would recommend reading that blog
post um that this is this problem is Al
echoed in the Linux map page on exposive
b0 um and the map page basically says
well yes this is a fundamental problem
um we still recommend use b0 and our uh
our hope our thought process is that
towards the future compi we will have a
way to get the compilers to not do this
and we can move on um but at the present
time there's no good fix for this um and
this is my my the first statement I have
um that I want to make is that at the
present time optimizing compilers in
cryptography are mutually exclusive you
can have one or the other you cannot
have both it does not work at the
present
time okay before I get to my
conclusion H um I want to rant a bit
about optimization as if I haven't
already um but basically I mean I get
that optimization is great and it gives
you all these things and uh things get
faster but I have a real problem with
optimization uh because look if you're a
developer and you write
code and you're pretty smart you can
reason about your code because you wrote
it and you know what it does um but if
if you didn't compile it and it's gone
through an optimization pass you can no
longer reason about it because you don't
know what the optimizer did right and
that that is I that is I think a
fundamental problem um and um sort of
what I want people to sort of think
about is whenever they do- o is the and
I'm not saying you shouldn't use the
optimiz because it has many many uh uh
uh uh uh many Pros um but don't be like
Blas about it right before you type- o
really think what it means do what it
means to do that um because it will
introduce all sorts of things like that
you weren't sure about or that suddenly
changes the meaning of something um so
uh and what I really want um for the
compiler people and maybe the language
people to implement it so the compiler
people can do it is I want strong
accountability and control of the
optimizer right that is if I compile
something I want to be able to compiler
I want to be able to go to compiler and
say hey before you do anything I want
you to give me a detailed list of
everything you're about to do in terms
of optimization um so that I can take it
that take that list and look at my code
and then with that list in my code I can
now reason again about what the binary
is going to do um without having that
kind of accountability you just you
can't reason about your
binary um and the other thing is control
right I want to have fine fine grain
control over optimization uh and what I
mean by that is you know like the
localized stuff that I mentioned before
that the Microsoft compiler for example
has where I want to be able to go and
say this particular scope don't optimize
or this particular scope don't do this
particular
optimization um I would like to see
something like that um I'll I'll just
skip this um so okay here's my
conclusion right we know what the
problem is the original problem um and
then uh I have some solutions and then
in retrospect um they're just partial
Solutions but there are still kind of
solutions okay um and then um I I also
have a call to action of things I think
should happen and hopefully will happen
at some point in the
future
um the problem as I've Illustrated I
think is rampant
uh perfect um uh and so basically um I I
would like people to use that GC patch
that I've shown or or or create a better
one and you know go find some bugs or
better yet go fix some
bugs um in terms of compilers as I just
mentioned what I want is optimization
accountability and control right um it's
kind of the Wild Wild West in terms of
optimization where the compilers just go
and do all these things and um I mean we
have we have some Flags but there there
there's not enough control and there's
not enough accountability there's not
enough transparency you just don't know
what's going on um and and some comps
have have like a dumb functionality
where it's like it's like a needle in a
hay stack um you want something that's
easy to work with easy to read or easy
to parse and that tells you exactly all
the optimization steps you're doing um
and ideally I I we would I'd like the
the the language people to get involved
and the standardize on this because if
they do we can now the Mantis of all the
compilers um and then lastly sort of the
you know coming back to my Nazi is sort
of the uh what about the Nazi uh Ruby
python Pearl go rust and so on um it
smells bad it looks bad uh especially
when a runtime is involved um this is
probably worthy of a presentation of it
so on or multiple
presentations
um but I I I wish I had done more there
um that is essentially it um I uh hope
you enjoyed
[Applause]
[Music]
it
[Music]
yeah