Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/321 Thanks!
Warm round of applause for our speakers.
Thank you.
My name is Dr. Drew.
I'm a graduate student at the University
of Michigan, where my research probably
focuses on what can we learn about
security from these large data
analysis. Since last
year, you've heard about some of my
research from Alex Halderman, including
work on ZENAB and how do we scan the
entire Internet?
Some work on we cryptographic
keys, the ecosystem.
I'm here today to talk about Heartbleed.
What happened when Heartbleed actually
hit the Internet?
What was the aftermath
in April 2014?
Open SSL disclose a catastrophic bug in
the implementation of the Tea Party
extension, and the vulnerability
essentially affected every piece of
software that used open SSL to
facilitate close connections.
This included everything from Web
servers. We heard a lot about Engine X
and Apache, but also affected everything
such as Tor.
It affected the Bitcoin client, a lot of
other pieces of software,
and in the end it allowed attackers
to essentially remotely steal any piece
of information that potentially sat in
memory of the SSL client.
And this might include logins.
It might include your credit card number
that you had just transferred to Amazon,
maybe included your cryptographic private
key that you're using to facilitate
connections.
And it affected a large number of
sites, not only just a large number of
software products, but in the end, a huge
percentage of the Internet was actually
impacted in our estimates.
Somewhere between twenty four to fifty
five percent of all Web websites
were initially vulnerable.
Could have had this information stolen
when the bug was first reported.
So the T-West HAPI extension,
I hadn't heard of it. I bet a lot of you
hadn't heard of it prior to the
Heartbleed exploit.
The extension was released in 2012
and essentially allows either end point
of a connection to ask its partner,
are you still there?
And the partner essentially replies, Yes,
I am still around, keep the connection
open. And for the most part, we never use
it heals heartbeat extension because
we used Helus over TCP
and TCV takes care of this on a lower
network layer and make sure that our
connection is still alive.
But US is still designed to work with
other lower level protocols such as UDP,
where you might not have the session
management and you just want to be able
to say, are you still present without
having to renegotiate an entire session
in the protocol? It's actually very
simple. Either end of the end point,
either endpoint the connection can
essentially say to his partner, are you
there? And it does this by essentially
sending some piece of data and some
ran the padding and then asking its
partner to repeat that data back.
And if partner repeats the data back, it
knows it's still there and it doesn't
terminate the connection.
So very simple.
The bug existed in that both
a length header was sent and
then a set amount of data was set and
an open SSL.
We trusted the length header
and essentially would echo back that
amount of information and in the memory,
regardless of how much data had actually
been specified.
So if I sent you for bytes of data but I
told you I sent you eight bytes of data,
you'd echo back eight bytes of data.
And if those extra for bytes of data
happened to be my username and password,
you would happily echo those back to the
client.
So the fix is very simple.
We check the length field and we check
the amount of data. And if the length
field is longer than the amount of data,
then we throw away the request.
But the same time, the effect is rather
catastrophic.
Any piece of information up to
two to six bytes after
that data could be echoed back.
And depending on how your memory is laid
out, this could be your login
credentials. This could be a key.
It could be anything else.
And this affected a very large number of
software. A lot of pieces of software
actually end up using open SSL to
facilitate connections.
In the case of Ingenix, in case of
Pache, this is used to serve as sheet
sites in terms of the Bitcoin client.
If you connected to a vault to
a malicious server,
they could potentially read back your
private key in the case of Tory
might feel to look at a relay and look at
other information that is currently being
passed across that relay or one of its
short term keys.
So it's affected a large number of
of different services.
We're going to focus on Websters
primarily because that's where we saw a
lot going on.
But it did affect everything from mail
servers to database servers to everything
else.
So when this when this vulnerability
first came out, we decided to try to
track it's essentially global
effect on the entire Internet, and the
way we did this is by scanning the
entire Internet to look for the
vulnerability, by altering the ZENAB
scanner and scanning the IPV for address
space on a regular basis.
Scandi Alexa top one million websites,
scanning four different mail servers in
four different database servers and just
looking at who is vulnerable.
But we do this fairly special way.
We didn't want to take advantage of the
actual vulnerability itself.
We didn't want to leak any private data
from other people's servers.
And so instead of actually exploiting
vulnerability, we took advantage
of a rather lucky phenomenon that was
associated with a specific vulnerable
version of open SSL.
And according to the RC, in
which the extension is defined,
essentially says, if I send you a zero
byte request that doesn't have any
information to be echoed back to me,
you should drop that request.
And there's really nothing that's just
not current staff. See, it's not quite
a spec, just drop it.
But in the specific vulnerable version of
open SSL, instead of dropping that
request to actually respond back with
some random patting, it would respond
back with any data, but you'd get back
or a response back.
And only the one specific
vulnerable version of open SSL
acted like this. The patched versions of
open SSL actually fix this bug in
different libraries.
Can you tell us, for example, all file
RACC and didn't have this specific
phenomenon? So essentially what this let
us do is that let's check whether a
server was vulnerable without ever
exploiting the actual Heartbleed
vulnerability.
So probably what's the most interesting
people are the top 100 websites
in the top 100 websites actually did
fairly well at patching
all of the top 500 websites from
within the first forty eight hours when
we started to do our scanning.
When we can go back and look.
However, though, because these are very
large organizations are kind of in the
press and media spotlight, most
of them put out press releases.
They answered questions from reporters.
And by going through interrogating these
pieces of data, we found that about forty
four of the top 100 websites were
initially vulnerable.
So a fairly large percentage.
And this is a lower bound.
There are some organizations that never
responded who didn't really
didn't put out any information, but these
were the numbers that did information
that they were vulnerable and a handful
of websites were so vulnerable.
Twenty four hours, this included
Yahoo! Imja, Stack Overflow.
And these were the sites that we saw
active attacks going on.
People were stealing Yahoo!
Credentials for Yahoo!
Mail accounts for essentially in
the wild 24 hours after
the vulnerability was released.
Within forty eight hours, all of these
had been patched.
Over the top 100 websites are fairly
special, these are websites that
probably not only have a dedicated
sysadmin, they might have dedicated teams
of sysadmins who are looking after these
sites. What about the other websites?
In our first scan, we've found our first
scan approximately two days after the
vulnerability was first released in
about 48 hours after the
first thing we found is that about 45
percent of all Web sites for EDPS.
And that's a bit discouraging
to start.
But of those approximately 60 percent of
sites, the heartbeat.
But this doesn't mean that they were
initially vulnerable.
Other pieces of software such as Tlas
or Microsoft is supported
the extension, but they didn't have this
bug, so they weren't necessarily
vulnerable like open SSL was.
Well, if we split this out even further,
we start to look that approximately
91 percent of the sites used open
SSL to facilitate connections.
So we were able to look at this based on
the server agent strings that are
reported back by Web servers.
So nine percent are using these other
software vendors, such as Microsoft.
But this lets us estimate that
approximately 55 percent of
Web sites at most were likely
vulnerable when the heartbeat
vulnerability was first announced.
So that's provides an upper bound on the
lower bound, we're able to estimate as
well, because two extensions are
two pieces of functionality were also
introduced along with the heartbeat
extension and open SSL one, that
one, 2012.
And these are tell us one point one, two,
one point two. And so while no one is
really thinking about the heartbeat
extension, no one was really testing for
it before the Heartbleed vulnerability
happened, people were checking for
less one point one at one point two.
These are major features that our people
are interested in, the deployment of
an even Ristic, who runs the SSL
pulse and the Quarless SSL
lab site where everyone goes to check
their their website to see if it's
correctly configured, have actually been
scanning for this and for looking for the
deployment of these two new protocols.
And what we found is that about thirty
two percent of websites supported
us one point one or one point two prior
to the Heartbleed vulnerability.
And that lets us estimate that
approximately twenty four percent at a
minimum of the web of websites
that use GPS were initially vulnerable.
So it lets us ballpark between twenty
four to fifty five percent.
In the beginning, a lot of the news
reports said that somewhere between sixty
sixty five percent were vulnerable
just based on how many sites to use and
Gen X or Apache.
It might have been a little bit of a high
guess, but not by much.
A very large percent of the Internet was
actually vulnerable.
When we go back out, one more layer and
look at the full IPV for address space,
we see yet another story and we see
that only approximately 11 percent of
hosts support heartbeat.
Six percent of hosts are vulnerable.
And the main reason here is because most
of the hosts on the IPV for address
space are websites.
They're not the Googles or the Yahoos.
What they are of these embedded devices
on these are home routers, their video
cameras or these devices we'd hoped
would never see the light of day and
everything in between.
And for the most part, these tend to use
much slimmed down pieces of
software. They never support the
heartbeat extension to start with, but
about six percent of hosts were
vulnerable in.
The sad thing about these embedded
devices is almost all of the embedded
devices still remain vulnerable today.
We know that embedded devices aren't
really patched, even if patches that are
available in most cases they aren't.
Users don't call and apply these
in these range from everything.
We really did see scattered devices that
were vulnerable to Heartbleed.
We even saw a thing with the strangest
case was a pizza point of sale
software that we saw repeatedly
being vulnerable.
When we start to look at the patching
behavior, we actually did fairly well in
terms of the top 100 Web sites or top
1000000 websites, if we expected that
between twenty four to fifty five percent
of Web sites for vulnerable.
When the vulnerability was first
released, we were down to approximately
11 to 12 percent within the first 48
hours. And that's fairly impressive.
We did pretty good as an operator
community of getting our websites
patched.
What's a little bit satyrs when we look
at the IP for address space and also look
where the Aleksa websites kind of
stagnated.
Somewhere around four percent of Web
sites remain vulnerable for quite a
while. We're down to somewhere around
below four percent. And unfortunately,
the number remains there today.
These are big websites.
These are the top one million websites
of approximately four percent are still
vulnerable today.
The one thing that jumps out right away
is what happened here.
There's this large jump for you almost
have the number of vulnerable
devices on the entire Internet.
What happened here is that there are a
couple of very large Azis that are very
large shared hosting providers.
Every single one of their websites was
vulnerable on the vulnerability came out.
They patched every one of their virtual
machines and the number drops drastically
in a couple of days. And these are
actually you can kind of see there are
three or four specific hosting providers
all patched in this time period.
But as you subtract that one, drop
the IP before address pace itself
is actually fairly stagnant.
We haven't really patched it.
So is this fast enough,
we said twenty four to fifty five percent
or initially vulnerable and this looks
pretty good for the top one million
websites. And when we said, are we
fast enough to look at when did attacks
start to first happen
and sadly tech
start to happen?
Before we started to do our scans,
we saw our first Internet wide scan
for the Heartbleed vulnerability
approximately twenty two hours
after the vulnerability was released.
So, yes, we did well, but we did not
do quite good enough, and this is an
Internet wide scan, if we consider
targeted scans, these are happening much
before. If you looked at who is attacking
Yahoo! This is going to happen before
these large, indiscriminate scans of the
full IP for address space.
So looking at the entire attack scene,
what we did is we looked at a honeypot on
the Amazon. Easy to
just see an unused IP address that was
watching all traffic that came in.
We also looked at packet traces from
the Lawrence Berkeley National Lab and
from EXI in Berkeley
to look at what techs were coming into
these three disparate networks.
And we find no evidence of attacks taking
place prior to disclosure.
And this doesn't mean that no one knew
about it, but it means that no one was
doing large, indiscriminate Internet wide
scans of it. There may have been targeted
attacks, but we weren't seeing large
exploitations in the wild.
The first piece of scanner traffic we saw
was approximately two hours after
disclosure and originated from the
University of Latvia.
And in total, we observed approximately
six thousand different Heartbleed exploit
attempts from almost 700
hosts within that first couple of
weeks. And there are two major outliers
that jump out right away, which are
Philippos site that lets
you test whether your website was
vulnerable to Heartbleed.
And the second was SSL Labs, which had a
similar app that let you test for the
vulnerability.
And what we find is actually the number
of people doing large Internet wide scans
was actually fairly small.
Only 11 hosken both the Amazon,
easy to honeypot and our
network at Berkeley and
approximately only six percent total
actually scanned more than one hundred
hosts. So there actually appeared to be
very little Internet wide scanning.
Only six hosts.
And several of these are academic
institutions, Michigan and to
Berlin, that were looking
for the vulnerability.
There were PSol, however, for other
hosts to in
Chinese As-Is that were completely
unidentifiable.
We really don't know what they were
looking for. They could have been
academic institutions.
They could have been attackers.
We really no way of telling.
We know that they actually exploited the
vulnerability, but that's about it.
On the other hand, there are a very large
number of hosts that targeted our
honeypot. And this wasn't a well-known
website, but it does show
that with almost 200 hosts scanning that
honeypot, people, attackers are going
after this other address space.
They are looking after these possibly
their cloud providers.
They could possibly just be more densely
populated IP address space.
Two weeks after disclosure, approximately
six hundred thousand hosts on the
Internet remain vulnerable, and at this
time we decided to go ahead and to
notify everyone on the Internet who
had a vulnerable host.
And this is one of the first times
one of these large kind of Internet
wide vulnerability disclosures had been
done, so we didn't necessarily know how
this would go.
And so we decided to run an
experiment and essentially split these
hosts into two groups and to see what was
the effect of actually doing an Internet
wide notification.
And so we contacted everyone eventually
that had a vulnerable host, but we split
them into two groups and contacted them
approximately two weeks apart.
And what we find is actually quite
surprising for the research community.
When we went into this, we expected that
global notifications would have actually
very little impact on actually who
patched.
And for a long time we've believed this.
And what we found was that actually the
opposite. We found that by doing these
global notifications, we could actually
increase patching by almost 50 percent.
And this is actually fairly encouraging
to us as researchers.
I think a lot of times when we find
vulnerabilities, we say there's nothing
we can do.
We found these, but no one will pay
attention.
And what we found is that actually people
are paying attention.
Even if you don't get responses to
emails, even if we don't get a positive
response back, people are actually
reading these and actually are
catching even in cases where we received
bounce notifications back and we thought
no one actually received this email on.
A lot of times the email made it through
to a second person that we didn't know
about. And even though it looks like
a wasted effort to us to actually
increase the trace, just something to
keep in mind as you're doing these large
Internet wide scams.
But unfortunately, patching wasn't
enough, cryptographic keys were also at
risk.
So partly let you steal cryptographic
keys to talk about this a little bit
in the second talk, but essentially if
you are using GPS or Apache, overengineer
someone could go and they could
potentially pull down your cryptographic
key. And so not only did you need to
patch your server, you needed to change
your cryptographic key.
And while we did fairly well, patching
the sad story is the cryptographic keys
didn't go nearly as well.
What we find is that only 10 percent
of the sites we found vulnerable actually
replace their certificate's.
Even more discouraging, 14 percent
of the websites that did replace replacer
certificates just reuse the same
vulnerable private key.
So no protection
for percent of websites that were
vulnerable revoke their certificates.
These are pretty dismal numbers.
There's a lot of improvement here.
So what do we learn out of all of this?
I think the first thing that jumps out at
us is that we have these large
open source projects that we are all
depending on. We have open SSL.
We have 60 percent of websites
that are depending on this for a
lack of attention. Support to these
critical projects is likely what led to
these types of massive vulnerabilities.
These projects deserve more attention
from us. They deserve more support from
us.
The ecosystem is incredibly
fragile, fragile.
I wish we could say a Heartbleed was the
only major vulnerability we had last
year. It was one of several.
If we look at poodle, if we look
at its channel, I know pools a little bit
less severe than than
as a channel or Heartbleed, but we
affected almost every site between
those two vulnerabilities.
One was a remote code exploitation.
The other one allowed us to steal
credentials and cryptographic keys.
These are bad, right?
These are pretty severe.
These continue to come.
We just saw Puddle to a matter of
weeks ago.
Remains incredibly fragile.
Even today, the supporting
Pecky is not equipped to handle the
massive revocation.
I'm Nick. We'll talk about this a little
bit more. But essentially the current
what we have in place, which is
certificate revocation, does not work
right now, no matter the fact
that only four percent of websites tried
to revoke their certificates, the
ecosystem barely handled that.
We could say actually didn't.
Web browsers were not aware of most of
the revocations that took place.
As we start to discuss and we have then
as a community discussing how do we
change revocation or what we do going
forward, we need to consider the fact
that massive fraud vacation is something
that we have to take account for.
We need to take account for the fact that
we might need to revoke every certificate
on the Internet.
That's what happened here.
We're not talking about trickle here.
They're one person lost their private
key. We need to be able to handle large
correlated revocation.
The recent advances in scanning let us
quickly respond and measure the impact
here from one of the first times you
could actually see what was the impact of
one of these large vulnerabilities in
in a positive light.
Let us actually not only understand what
happened, but impact patching.
We're able to increase patching for the
entire Internet by almost 50 percent.
And that's incredibly encouraging.
And we have this fragile ecosystem, but
we're able to spur on patch and we're
able to help out.
But at the same time, we think we do have
to take a look back and say
there's a lot of room to grow still.
And with that, and we'll switch over to
Nick.
OK, this seems to be working.
Hello, everybody, I'm Nick.
I'm the security engineering lead
at CloudFlare.
And I'm going to talk to you today about
Heartbleed if you guys aren't sick of it
already after this long
year. So specifically,
I going to talk about my personal
experience or my experience at CloudFlare
with what happened during
during Heartbleed, when it came out
and what happened after the fact.
So has anybody read this
article?
This is this is a snippet from
The Verge and it
is kind of semi fictional account
of what happened during the disclosure
of of Heartbleed.
I can kind of paraphrase a little bit
how that conversation went, but
it kind of kind of got a little bit like
this. Like so do you do you
use details?
No, not that I know of.
Does anybody use details?
And that's for websites.
Nobody use data.
Grumped details.
How about topics?
Well, while my answer was
what's Attila's heartbeat?
I mean, at this point, barely anybody
knew what this was. It was very obscure
feature.
And the answer
here was, was this the stupid
and and there's a bug.
You should turn them off, right?
So recompile open SSL
with openness.
Lascelles No heartbeat's.
And that was, that was kind of that
was that. So heartbeat's were off
for CloudFlare. We have a pretty simple
architecture so deployed it really
quickly and the answer was
OK, looks good.
Public disclosure should be around April
9th.
So I guess a lot of people had this
question. But why tell CloudFlare.
Well, let me just go real quick
and describe what CloudFlare does.
So it's a reverse proxy.
So if you have a website,
CloudFlare can sit in front of it and
block malicious traffic.
That's the sort of red X up there,
as well as send cached content,
static content. That's the bright orange.
So it reduces anybody getting
to your website that's malicious
and reduces the load on your website.
And for this to work, this cloud
has to be closer to the visitor than it
does have to be to your website.
So we have this global network.
So are our nodes are closer to
the visitors and that's that's how it
works. And there's over a million
sites on CloudFlare, including banks,
government websites, Bitcoin exchanges,
almost every Bitcoin exchanges on
CloudFlare, the ETFs
website, Reddit.
I can go on and on and on.
So lots of sites.
But what CloudFlare does is very simple.
It's essentially three
services or two DNS,
HTP and
which is powered by open SSL
and Engine X.
So the architecture
is very simple in that every machine that
we have can serve every site.
So thinking back to
Heartbleed, this is actually
you can see why this would be a really,
really bad situation for Heartbleed
to hit anyways.
So it happened early
April 7th.
Ten, twenty seven open SSL publish
their advisory and that hit
Hacker News really quickly after that.
Within half an hour it hit the front page
and about an hour
later we posted our standard
CloudFlare customer sites are patched.
You don't have to worry about it sort of
post. And this is this
was you know, it was
it was a thing. It was it was a bug.
And it was starting to gain some steam.
And then about an hour after that,
there was a tweet from Cottenham Con.
And I think everyone can
know what this is.
This is that's the next site.
This Heartbleed itself, it was branded
and it came out to mass media.
So this became a really
big deal.
Heartbleed dotcom had a logo
hit the mainstream press Heartbleed
virus. I don't know if you guys remember
that, but people were saying, oh, there's
a Heartbleed virus out there.
And I knew I knew
it got really bad when my my
mother called me and said, what?
What's going on?
So this was kind of a big deal and,
well, we were finished patching so
well, we had some time to kill.
What are we going to do
at this point?
There is we decided on three things.
And one was to help
keep this scanner that Flippo had from
falling over.
The second was to turn our network into a
large honeypot to see what type
of attacks or what type of scans are
happening.
And then the third was to decide
or to figure out what we're going to do
about our certificate's.
We have a quite a few websites that
use our service and many of them use SSL
and we had about hundred thousand
certificates and revoking them was not
really at this point.
The day after disclosure, it wasn't
absolutely clear that this was something
that you had to do.
So first, let's talk a little bit about
this Heartbleed scanner.
So Filipa, who's now a CloudFlare
engineer, wrote this server and go
and you type in your hostname and it
scans it for her for
whether or not it answers Heartbleed
heartbeat requests that are malformed.
So these are small ones around one
hundred bytes so that it doesn't leak
anything beyond your standard
frame. It shouldn't leak any any
information. You put it on us and
then put it behind CloudFlare and
shout out to Kyle Eisen from our team for
helping keep this up.
But this this is kind of what from
Philippos server, what it looked like
there was up April
8th, up to two thousand requests per
minute. So this was a
very highly used tool.
And that's nothing because
this is the next two weeks.
Two thousand is the the bottom tick right
there. So up up to ten,
ten to twenty thousand a day,
a ten to twenty thousand
scans per minute for the next
two weeks. And we held it up to
two hundred million tests in the first
two weeks.
So with the scanner up and up and
running. Yeah.
And thank you to Filipa, wherever he is,
he's somewhere in the room stand up.
This is thanks for the tool.
I don't I don't see I'm somewhere.
But in any case, this is this is what
what he found in terms
of Domain's, it
was really bad at first.
This is the knife.
This is two days after Heartbleed was
originally announced.
Up to 30 percent of the sites
that he scanned were vulnerable.
And this luckily cut down.
And a lot of people use this in their
automated testing to validate
sites. But, yeah, it got down to a low,
low number pretty pretty quickly.
So now that that was up and running back
at CloudFlare, we decide,
well, what can we do?
Well, we can log every heartbeat
that we see that comes in with with a bad
size. And, well, we can
put that data on the shelf until now.
And that's that's kind of what we did.
So here are logs from
the 9th and sixty
nine percent of them had a message size
of sixteen three, eighty four,
which you might
know as the largest power of two,
you can fit into a signed sixteen bit
integer.
But you might also recognize it as
the hard coded value in the SSL
test python tool that came out the first
day, 20 percent were one
twenty one and that was actually from
Flippo site and University of Michigan
maybe where you guys scanning on April
9th or
in any case there are a lot of requests
that used the
zero length packet, which is another way
to just check to see if your site is
vulnerable. But
about a week later it was around the
same. So if there were people
who are mass exploiting this against
sites, they were probably
using just the basic SSL
test Python script and
around 20 percent were still Philippos
tests. So if you can,
you can do the math here.
It's it seems that a lot of people
were just scanning with Philippos test.
It wasn't a lot of mass exploitation.
And flipping the
numbers back, you see that one percent
of flippo scans were actually against
sites on CloudFlare.
So this is what the map looks like
for where the attacks were coming from.
I know IP maps are not really that
interesting. They don't tell you that
much. But this is
this is where it is. There's some some
strange spikes like in Iceland, but
don't read too much into it.
Now, the question is, and this is
this is what we were thinking when it
first came out was why was it
really so dangerous?
Why was Heartbleed so bad?
Well, it's a kind of a layer six
request that doesn't necessarily
get logged. People don't log
parts of handshake's very often unless
you have a specific ID's rule
or something like that. And it's really
bad in that. Sixty four K of server
memory can be infiltrated
by one request.
And this has login, login info, session
cookies and perhaps
two private keys.
We didn't know.
And if you kind of look
at the diagram here, this is that's the
heap right there.
Everything that's above the request that
if you have a new request comes in, it
gets put on the heap and anything that
was previously removed from the heap
is still sitting there.
So we know that there are passwords,
cookies. People were finding this right
away. And the question
was, would the key be there?
What is the key going to set above one of
these requests?
So we looked at the code.
Right. And what did the code
say? Well, it said this can't happen,
at least not in Engine X.
The key gets loaded right away and
therefore gets at the bottom of the
stack. And any
time that you do applications or
for requests coming in, they're going to
be higher up on the stack and they're not
going be able to read the original key.
Right.
And Engine X itself was single threaded.
So if you have this, it's not going to be
able to catch something halfway in the
middle of another thread doing
an operation.
So open SSL
has a big number library that they use
and they clear the memory when they're
done. So if you're doing a handshake for
TLS, all the cryptographic
material is going to be cleared by open
SSL.
At least that's what we thought.
Right. And we
we weren't sure. I mean, I just looked at
some code and what do I know?
So we launched the CloudFlare
Heartbleed challenge.
So it was a this was something that we
did to crowdsource an answer.
So we set up a standard Engine X,
which was outside of CloudFlare.
It was on a third party VPs, and
it had the vulnerable version of open
SSL. And we said to
all of you guys, come and find
it, see what you can do and to
to show proof, give us
a message signed with that private key.
What did we find?
Well, for the first couple of hours,
there was there was trolling and
as you can see, somebody clapping
because they did this.
But basically
anything that you post onto the page is
going to be put into memory and the
engine looks. So people were
posting private keys in there and they
were posting what looked like
you can see my name there, Nick, what
looked like a password's file.
So everyone was getting really confused
getting all this. There's a private key
in my Heartbleed question, but nobody was
actually getting the key until we saw
this tweet from a fettah
and.
We took a look, this is
our office. It's me pointing out a
television screen and
yeah, he solved it.
So congrats to Fettah.
And so he wasn't the only one.
This was in the first twenty four hours
or so, there were 12 people in the first
48. About twenty five people had had
solved it and got the real key and
sent proof.
So can you still
private keys?
Absolutely, yes.
And it was solved in under 10 hours
at private keys can definitely be
vulnerable. But another thing we did was
we logged where in memory the
Heartbleed requests were coming in and
we compared that relative to where the
private he was initially allocated
and they they never overlapped.
So how did this how did it
actually get solved?
Well, there
was yeah, there was a second bug and open
SSL who would have known
looking through that code.
If you dump the memory of of the request,
all the places in red are where
key private keys did exist at one
point and it turns out some temporary
variables were not wiped.
This is the this is the code to clean up
the mess. There's a big name,
free versus big, dumb, clear, free.
This is just yeah.
It was just in certain cases in the
Montgomery multiplication, they didn't
clear up. They didn't clean the partial
pieces.
We can do a little bit of math just to
show how people actually
solved it with this.
But RSA, you have a
couple of different things. You have a
public exponent e
you have two primes multiplied
together. They make the public key and
the private exponent D and
any one of the if you get any one
of P, Q or or D,
you get the whole private key.
So what people did was they took
every one hundred, twenty eight bit block
that they saw in X
filtrated sharply data and they just
tried to divide it into the modulus.
And if it divided, there
you go, it's factored.
And this is how nine of the ten people
solved it turns out that one of the prime
factors is just sitting there on the heap
after tens of thousands of requests,
you might look upon it.
But one enterprising
gentleman, Rubenstein, who's
at University of Cambridge, he
used a much clever method, which is
coppersmiths attack, which is a Latisse
reduction attack, where you
only really need about 60 percent
of one of the private keys to to
to find it out. And this depends on the
fact that the public exponent
is small. So
for performance reasons, the public
exponent in RSA is small so that
any public operation is is
really fast.
But he solved it in only 50
requests. So this was
that was actually really interesting.
So private keys are gone, right?
What does that mean,
revocation time?
You know, the Internet was built for
this, right? I mean, people who designed
the peak, they said, yeah, you know,
people are going to revoke hundred
thousand certificates in twenty four
hours. This is just this
this is how we designed the system.
Right. And this is
what sounds kind of reported.
That's that's mostly us, actually.
That's that's all the Internet.
But that's mostly CloudFlare.
And this is this is what it look like.
The blue line is revoke certificates.
And as you can see, that's April 7th.
This is after Heartbleed.
This is when everybody was revoking and
then that green spike is is CloudFlare
revoking. But once we
revoked all these certificates, we found
out that, well, it didn't really
mean that browsers wouldn't accept these
certificates anymore.
And I can go into that
really quickly. There are three methods
to do this that were
built for, you know, handling
revocation and certificates and five or
nine. And first is the certificate
revocation list.
And this is just a
flat file with a list of certificates
that are revoked and
did as revoking one hundred thousand
certificates, breaks the URLs.
Heck, yeah, I did.
So the the CRL from Global
Sign grew from twenty two
to five megs.
Yeah.
This basically does the CRL
server and
lucky for this, lucky for Global on
CloudFlare was in front of their CRL
server, but unlucky for us.
What we're used to that kind of traffic,
but yeah, you can see here every three
hours there are waves and this had to do
with the cycles in which scrolls were
updated and Microsoft Internet Explorer
downloading them.
And that was pretty rough anyways.
Scrolls broken.
How about XP?
XP? Well, this is the
online certificate status protocol.
This is a kind of question answer.
Is this certificate revoked, yes or
no? And well,
it's really broken.
And Chrome has kind of known this
for a while and stopped checking.
But if you have hard fails, it does
disallow you from using certain networks,
especially with captive portals.
And if you have a soft fail,
somebody on the network can just kind of
drop the okapi response and voila,
the site is no longer revoked.
So it really doesn't work
or scale to the
degree that we'd want it to.
In plain Hoxby being requested from the
browser in that sense.
How about zero sets?
This is Google's proprietary method
that they basically collect
all the calls that they can from
different certificate authorities and
put them together and install them in the
browser. So you get updates with the
browser of these sets of scrolls.
I shouldn't say all the scrolls, but
it's specific certificates.
And this is kind of what we found out.
They only do EV certs and special EV
certs, and
if the browser doesn't get updated, then
you're not going to get an update on your
URL set, which is it's
kind of bad.
So CloudFlare challenge dot com.
Once it was solved, we revoked it and the
way it was, it was
not an easy search. But Chrome did market
as revoked because, well, they added it
manually into adjacent file.
So none of the hundred thousand
CLOUSER certificates were
being marked as revoked by
Google Chrome.
So we we basically
made a hack and I dunno if you can read
this, but it's the most efficient for
line of C++ revocation.
This isn't chromium.
It revokes all of our certificates.
This is a hack.
This is not scalable.
You shouldn't do it this way.
But it was it was how
it had to get done because there were no
valid ways of of doing revocation.
So, yeah, revocation is pretty
much broken.
And what can we do?
Well, there's shorter certificate
expiration periods that could help that
would at least help with the cartels
because you don't have to be holding on
to old certificates for very long.
You can sort of shrink the size of the
sets pretty quickly.
Okapi must staple is
an extension that requires you
to send the the server to send the
okapi response in the handshake
that can help to certificate
transparency, something else that's
thrown out there to solve this.
But none of these are
widespread and none of these have been
implemented. So something has to work
for revocation.
So I guess in summary, there are three
things that we did after after
Heartbleed.
We kept the scanner from falling over,
turned our site into a honeypot.
And while we definitively
answered that, yes, you have to revoke
your certificates and there's no excuse.
So there's a lot of takeaways from
Heartbleed, right?
I don't want to be the one to sort
of tell everybody how to take away what
to learn from this. But open source
disclosure is hard.
And this this really was the first
one of the year of
what turned out to many be many open
source disclosures.
And we did learn a lot
of lessons of how to do that correctly.
Other things that pointed out, which seem
obvious to people, but they weren't
obvious in or weren't in open SSL,
which is features should be disabled by
default. Nobody who installed open SSL
one oh one wanted
heartbeat's necessarily so
turn-off features by default.
Another thing is, while expect
the unexpected, that's sort of obvious
in computer security, that we didn't
really learn that from Heartbleed, but it
was it was definitely a shock when it
came
to other things.
These attacks, a lot of the attacks on
real sites like CloudFlare saw we saw
quite a few attacks as
as I mentioned, but a lot of them were
still were just scans from people just
trying to see if their sites were alive.
So that's that's a reassuring sign.
Crowdsourcing was effective for the
challenge. I couldn't find the private
keys.
And luckily, there are
very smart people out there who were able
to find it. Is anybody here in the in
the auditorium winner of the
CloudFlare challenge?
Is anybody here? I don't see any hands,
but anyways.
I don't have my glasses, so
congratulations wherever you are,
and the last thing is, is revocation
needs a solution.
And the last
conclusion from this is, is really,
you know, support open SSL.
It's it's.
I messed up my microphone there, but
thanks.
No, really, I mean, this is this is part
of critical infrastructure for
the world and for not only websites,
but as here was telling
embedded devices and
these guys need support. So please
support open SSL and I'm done.
Thank you.
OK, quick announcement before we start
the Q&A, if you are going to leave
the room, please get up now, go
out quietly without talking so we can do
the Q&A nice and quickly.
Also, you will be only able to leave
the room at this point.
Yes. If you have any questions, please
line up at the microphones.
We'll start with microphone number one.
Thanks. Not really a question.
Thanks. Thanks. HERTOG We really, really
followed the progress
even before the talk with your paper
published. And we just want to contribute
a piece of missing information.
So the first
attacks, as you call them, 24 hours
after the disclosure that left for
us, we weren't really attacking.
We started working on the scanner
about 20 minutes after the public
information, and it was a top
scanner in the region.
So it was a scanner and
used multiple different methods.
We just as we went, we
came to the conclusion that there
might be firewalls to be deployed
in between which
which may stop some
of the packets. So we use many
different packets in one in
every request. So that's also a scanner.
There
is a remote control, no.
A quick interruption, if you're
leaving on the ground
floor, please only use the front left
and the front door.
If you're up there, you can use any
door you want. But if you're down here,
please only use this door and
this door to leave the room.
Thank you.
Does my work at FRAGO.
Thanks for the comment. I don't think I
saw anything specific to you to
to your your account, but, yeah, it's
great to know.
So, I mean, it's one of those things is
really hard to tell from our perspective
about the attack, whether someone's
malicious or whether someone just
scanning a lot of the
the host that we see don't have any
information that identify and
what the intent is.
Microphone number two, please.
Hello, I have a question to CloudFlare,
could you stop blocking for
yourselves, please?
You make the Internet more central every
day, all the fancy home, but just switch
there and come on, the network
is so small you could handle live with
that thing is all of the bandwidth and
you're blocking all the notes all the
time.
We're looking into solutions for
the main problem is that
there is a lot of spam on
tour for Web sites.
And right now filtering
through the good and the bad is
something that we're working on and we
are focusing on that this year.
So look for something coming up this year
for Tor users like.
Next up, we have a question from our
signal Angel asked on IFC.
Thank you.
So given that maximum tearless record
length is 16 kilobyte, how
is it possible to even get 64
kilobytes
or it's
you can split you can split a request
over multiple records?
I believe I think I'm pretty sure that's
how it works.
Microphone number one, please.
What is your opinion on Libros?
Go for it.
Um,
I'm not sure for the most qualified
person to answer that.
To be honest.
I think there's well intentioned I think
the community acknowledges that there are
a lot of problems in open self
in terms of maintainability.
It's one solution.
I'm not sure it's the correct one.
Yeah, I think I think Libros itself works
for its goal,
which is OpenBSD.
They have a portable version by now
as well as they did with
all of our other folks.
And it's
much cleaner code.
And still, open SSL tries
to support way
old systems like Windows, 16
feet and bullshit. You really don't need
so down maintainability, you
will remain for a while.
Yeah, I agree that open
SSL does support quite a lot of
different platforms and some people need
that.
But the different forms of openness as
well that have come recently, Libras and
sell boring SSL are
taking patches from each other.
So I think the more people
looking at this project, the better.
We have time for one more question.
Microphone number three, please.
Hello. And both your talks, you showed
numbers of vulnerable sites
decreasing and increasing again
afterwards. Can you explain that?
So I think there are small bumps
that were just people coming and going.
I don't think we saw a lot of
websites that became vulnerable.
I think a lot of it is just pieces
of measurement websites that came and go
between different scans.
I don't think there are any large jumps
that we saw
when it comes to the data that I was
showing.
This is from Philippos scanner and
not everybody was scanning the same
domains every day. So that's just
standard variance.
Thank you very much.
Psychotronic, please give our
speakers a warm round of applause.