Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/90 Thanks!
Well, good morning, everyone,
so thank you for getting up early
to to come to my talk today.
I barely made it out of bed myself.
So I hope we have a good time, it's a
really exciting time
in the history of
networking when it comes to
visibility into the internet, and
that's what I'm going to be talking about
today is new
advances, new techniques
that are coming out of the evolution of
technology that can enable
all of us, each and every one of us,
you and I, to individually
get visibility into
what is happening throughout the
internet, like has
never really been possible before.
Now I am
just
a little bit about myself.
I'm a professor at the University of
Michigan, where I work in
security and privacy with
an emphasis on problems that affect the
real world.
I am,
however, an
armchair professor, and it's always
a really great privilege to be speaking
to people who are actually in the world
doing things instead of up in the ivory
tower or somewhere sitting on our
asses.
So please,
please bear with me if anything
is is not clear, is
is not at the level of detail you'd like.
If you'd like more detail, please do
ask in Q&A or even interrupt
me, as is the style in talks
where I come from.
So just before I begin, what I'm going to
be talking about
today is not only
my work,
but is collaborative work with
a great many people
who I
will list up here so you can go and
take a look at our papers.
They're all available on my website.
I've taken an open access pledge, so
everything that I write
is available for free.
In particular, I need to
acknowledge my student
as a queer drummer.
Edge this.
Most of this work I'm going to be talking
about today is going to form a large part
of his dissertation, and he's been
working on this extremely well
over the past couple of years.
So the internet, right, this is where we
live,
and it
looks something like this according
to the kinds of maps that
that people have been making since the
late 90s. This is
a visualization of
routing interconnections on the internet,
sort of the the topology of the network.
But just seeing the shape of the land
in a picture like this doesn't really
tell
us that much about what's there.
Are there trees or are there people?
Are there flowers or are there mountains?
We need to know a different
level
of detail and a different kind of
information in order to make sense
of the devices,
the information that makes up the
internet.
And one place we can
get some insight into that is from search
engines like Google.
But Google indexes the web.
And so when their crawlers are
navigating the internet, they're
concentrating on web sites that are
making up this interconnected
graph of links.
That is where most of us spend most of
our time.
But if you think about the internet in a
different way in terms of what's plugged
into it, what makes up
the public address space?
You end up getting a very, very different
picture from what you see in a search
engine.
And what I'm going to be talking about
today is methodology
techniques that
we can use to get that kind of view
of the internet, a kind of view of the
internet that is about the devices,
the computers, the embedded systems that
are plugged into it.
So internet wide scanning
the topic of
my talk, I just find this
so exciting, I've been fascinated by
the idea that it's possible
it's tractable to connect to every
computer
on the internet and have a conversation
with
it
for for many, many years.
And some of the first inspiration for me
came out of this project pioneered by the
EFA
called the SSL Observatory.
What the SSL observatory
did is they had a small cluster
of machines that they used over a period
of
months to try to make a connection
to every
HTTPS web site.
And they did this not with a list of web
sites, but by exhaustively
enumerating the IP address space.
So if you think about it, there
are
IPv4 addresses are 32 bits.
There are about four billion of them.
Maybe I don't know, 15 20
percent of them
are not publicly reportable.
So you
have
a
substantial fraction of four billion
addresses.
That seems like a lot.
On one hand, but on the other hand,
networks and computers
have gotten so much faster in recent
years
that suddenly just making
a disappear connection to all
of these machines is something that that
you can do that we can write
programs to do.
It's not easy, but it's tractable.
Further inspiration
came about a year ago
from
this research from the the Qana
botnet, the Qana botnet
was
an experiment done by
some anonymous people who may or may
not be here
who
who discovered
that if they did this kind
of internet wide probing,
if they if they probed embedded
systems, they found many,
many systems that
would allow them root access
with just a short list of default
passwords. That's my
understanding.
And using this
technique, they compromised about
400 20000
boxes.
Things like I presume that
they were of the form of home
routers and so forth.
Many of them
and
they used these boxes to do
probing of
the rest of the internet on a variety
of really interesting parts.
So this is a map of the
hosts that were
compromised as part of the Karnit botnet
to do this study.
So they got
a huge volume of really, really
interesting data from that, as the ATF
did when they were probing HTTPS.
But what I think is true about both
of these
studies is that they're both
really impressive in terms of what they
got, but also were a tremendous
amount
of effort of time for the people
who did
them.
I don't
think every one of us
is able or willing to
create a botnet, at least I hope not
for purposes of internet wide research.
So there have been a couple of other
studies I should mention, too.
And these are, I think, at
least from my perspective, these are some
of the more interesting
works that have come out over the past
five years. The first
one on this list census and census and
survey of the visible internet in
2008 was
a study of
basically
what
what addresses were connected
and ratable in the IPv4 space.
Not a security
study,
but it claimed to be the first exhaustive
survey
of IP
addresses plugged into the internet
for
20 years.
So since the time the internet was very
small until just five years ago,
has been a dark age.
If you think about the evolution of the
universe, we have these these periods
when we have more visibility and less
less visibility.
Well, that was a period of very little
visibility.
Since then, we've had studies like the
SSL observatory, like
when I was involved in two years ago that
I'll tell you a little bit about
where we were able to compromise a
substantial fraction of
the private
keys used for internet cryptography
using internet wide scanning techniques
and then the Khanum botnet just last
year.
All exciting work.
But what unifies these is they
take so much effort to do so.
We're talking about thousands of CPU
hours, months of waiting.
The first three studies on this list
all involved
the use of off the shelf network
probing tools, things like end map
and
end map. Although a wonderful
security tool,
something I use all the time, I have
great admiration for Theodore, and his
work
in Map is not a tool that's
designed or optimized for internet wide
scanning, and that turns
out to be a large part of the problem.
So in each of these cases,
either tremendous effort by the
researchers, tremendous time
or a large number of hosts was required.
So the problem, from my perspective, is
we've been using tools like this
to try to get visibility into the
the devices plugged into the internet
where we really want something like this,
we
want something that's designed, optimized
for the purpose of internet
wide scanning.
So in my research group last year,
we asked, well,
what is
this kind
of work didn't require heroic effort for
researchers? What if we could democratize
internet wide measurement?
What if
we could do a scan like the SSL
observatory that E.F.
did
every day?
How much more detail?
What new things could we learned with
that degree of visibility?
And in particular, what kinds of
optimization could
we do if we brought a scanner from
scratch to do internet wide measurement?
So to answer these questions, we went
and built that scanner and we built a
tool called ZMapp that
that you can download right now.
ZMapp is an open source
tool that will allow you
to do a horizontal
scan of the whole internet.
So looking at one port across
every route, able IPv4
address
from just one machine in
about 45
minutes. If you have enough upstream
bandwidth
and to be able to
get
to be able to see almost everything
that you would see by using more
precise and thorough methods.
So this is something we call shotgun
scanning.
Just stay in the same way as shotgun DNA
sequencing.
Rapid, relatively dirty but
very effective technique that gives
you that
drastically reduces the cost
of doing scanning.
So if you have, say, a gigabit
upstream
like you could get if you can plug into a
wall
here,
I hear that they
have much more bandwidth available.
But my understanding
and maybe someone involved in the network
here can correct me if I'm wrong.
But my understanding is that the
distribution layer that takes it down to
the the ports you can get access to is
only running at a gigabit.
So you're going to get performance like
this instead of 10 gigabit or or more
performance.
But you can complete a scan
in less than an hour and contact every
redoubtable IP address.
Now it really
is as
simple as this.
I'm going to give you a demo in just a
second.
However, I recommend not doing it
from the Wi-Fi, at least during not
during my talk.
Because that might interfere
with the ability to
to demonstrate what I have to show
you. So let me quickly start
something up here so that it
finishes before our hour
is up. So I'm going to do this in
a VM and I'm going to try to
make it a little bit bigger.
So what I'm going to do?
Let me show you
ZMapp, an operation.
So
if this
is still communicating
with the internet.
Come on to sleepy.
Let's try that again.
Boo, boo, boo.
Well, maybe the internet is no longer
working.
Internet, are you still there?
Someone broke the internet
if anyone in this room is running an
internet wide scan from the Wi-Fi.
Can you please? Ah,
can you please pause it?
All right, so I
should be
able to get a few packets through
here. This is one of those cases where
it works before
the audience is all on their laptops.
But as soon as you start speaking and
people get bored and start reading email,
it stops.
OK. This is not being.
Terribly responsive.
OK, well, maybe I can run something
locally. Um.
Let me see if this works, and I haven't
tested this from the local machine yet.
Not working from the local machine,
either, because all the packets are
dropping right for the same reason that
it doesn't work remotely.
All right. Well, if I was able to give
you a demo, which I promise I will start
a demo as
soon as this internet
comes back, I could demonstrate a couple
of things.
One thing is ZMapp
is good for some fun tricks.
Let's say you want to find some open
proxy, some machines that are listening
on ADHD.
Well, you can type a command
ZMapp,
Dash eight, Dash Capital
and say you want a thousand addresses
that are listening on that port.
And in substantially
less than a second, you're going to get
back a list of 1000 IP addresses that
are responsive on that port.
A large fraction of them are going to be
open proxies.
You want to find some random web servers,
the same thing.
So that's if you want to just do a small
sample.
If you want to scan everything,
you
could type a command like the one I
was planning to run.
Still not working like the
one I was planning to run,
which would be something
like.
This.
So what I'm going to do in the hour
after this talk, assuming that I can get
on the internet and talk to a machine at
Michigan that has a fast uplink
is I'm going
to do a SIM scan
on Port
zero x thirty three.
So let's just let the world
know that we're here today
by
making a connection to every public IP
address on on that port.
So if you are on your machine anywhere
in the world
over the hour after the talk, want to
say run tcpdump and look for incoming
send packets on
that port?
I think you'll get one from us in the.
The Michigan subnet is one for one
two one two six.
So look for an incoming packet from
there. If you get one, you'll know
it's part of the scan and
I'd appreciate if people who are looking
for that.
Let me know whether they've gotten one or
not, because that'll give us a nice
informal experiments, some scientific
purpose, an informal experiment
about
reachability in coverage for the scanner.
So I'm
sorry I was planning to do this live
during the talk. It'll only take 45
minutes, but it involved sending about
100 bytes to
another machine in order to get it
started, which we don't have the ability
to do, apparently.
All right. So I am not here, however,
just to push ZMapp.
And there are other tools that have been
developed in about the same time frame
that I'd like to also lend my endorsement
to.
In particular, there's a very excellent
tool
called Mass Scan that's
been developed
by Robert Graham,
which is aimed at very similar
purposes. And these are parallel efforts.
They're both open source,
actively developed open source projects.
They're both high performance scanning
tools. You should try both of them and
see which
one better suits your needs.
I don't
think either of
us knew about the other effort when we
were building our tools, but now
you have two very good options to choose
from.
Mass scan
if you are willing to install, some
commercial
drivers can use this
thing called PEMF Ring
DNA to
drive a 10 gignac at line speed.
ZMapp is entirely in user space
right now. We can drive from userspace
a one gignac at line speed.
We can drive a 10
gig nick at some multiple of one gig.
We're not quite up to 10 gig yet, but
we're planning to port over offering
DNA support very soon, so you'll be
able to use your haul very fat pipe.
And we do have a 10 gig uplink waiting.
All right, so
let me talk a little bit about how high
speed scanning works.
Some of the engineering tricks we've used
to do it. I'll give you a little bit of
evidence that this actually works because
I too was
skeptical when I until we ran
the experiments and then we'll talk about
some
fun things that you that we
all can do with it.
So in developing ZMapp,
we realized we needed to do a bunch
of things differently from existing
general purpose scanners like in Map
in order
to try to to get really high
bandwidth performance out of it.
So one thing we decided to
do was
attempt to eliminate as much state
as
possible from each of the connections
that connection attempts were trying to
make.
So
the way and map
works, it maintains some
state to track each connection it's
trying to open. It has time outs for each
of those connections.
It uses batching
in order to do rate limiting and
avoid over saturating the network.
We can't do any of these things really
efficiently because in order to scan
at even one gigabit per second
line speed, you have to send
about 1.5 million packets per
second.
So managing state for that many
connections
is not something that we
that we, we felt was a good idea.
So we're able to do it the way we
architected ZMapp.
We have separate processes for sending
separate threads, for sending and
receiving packets, the sending processes
or sending
in, say, a SIM
scan. They're sending signals as quickly
as they possibly can.
They're not communicating with the
receiving threads.
The receiving threads are looking at
what's coming in
and
processing it.
Asynchronously from descending.
Another thing we did differently
was the way that
we
are tracking which hosts are responding.
Let me get to that in one more slide.
In order to avoid flooding the network,
and this is a really important thing,
too, because if we send, if
we overwhelm distant networks,
they're
just going to drop our probes and
we're not going to get accurate data.
And one passed
approach to this has been just to to
severely
limit the rate at which you scan.
But the thing
that makes more sense
is to do randomization to
use basically get statistical
guarantees that you're unlikely to
overwhelm any distant network with
traffic, even if you're scanning as fast
as you are, upstream will allow.
And I'll talk about how we do that to.
Finally, a lot of the credit
for the ability to do scanning
at Gigabit 10 Gigabit Ethernet
line speed ought to go to
the operating systems and to the
underlying architecture because there
have been vast improvements to
things like the Linux kernel,
to the way
processor, to network card
communication happens in Intel
architecture
that underlie all of this.
But still, there are inefficiencies there
things we don't need in the kernel
like the TCP stack.
So one thing we do is we
write out Ethernet frames directly.
We're skipping everything that the kernel
is trying to do to help us.
This lets us just change the
parts of the packet that need to be
changed for each additional probe we're
sending.
It helps us get much, much better
efficiency.
So a couple of the more interesting
challenges we face.
One is how to do addressing.
So the challenge here is
that we want to be able to address.
We want to enumerate the
IPv4 address space
in a random order
every time we scan so that
we're not overwhelming distant networks
so that if we want to scan less than the
full internet, we can get a statistically
relevant sample and so forth.
The way we chose to
do this
is
to and we want to do this without
storing a lot of state.
We don't want to have to make up this
list of all the addresses in advance
and
then
and then use it because that's going to
be for four billion addresses.
This is going to be a multiple gigabyte
list.
So what we do instead is we use this
trick from from
the world of math.
Now this trick that the crypto people
would think is trivial
where
we iterate over a multiplicative group
of integers modulo p for
some p, that's about the size
of the address space.
So what this allows us to do is
with just a negligible amount,
a constant amount of space
that is the the primitive route
that defines
this multiplicative group, our current
location within the group and the
starting address.
We can enumerate the IP address
space in random order.
And if we want a different permutation,
a different ordering,
we can just vary
the generator that we're using
to to define this.
So with just a very, very
small amount of state, we
can get a nice random ordering of the IP
address space.
Another thing that we
do is
the way we validate responses
in an almost totally
stateless
way
is by taking
some secret and basically
encoding it in fields of the outgoing
probe such that we can
recognize whether a packet coming back
in is a result of our scanning.
Or is it background traffic?
Is it just someone trying
to attack us?
Whatever we need to differentiate
responses that
are a
result of our probes from any other
traffic we might be receiving, including
sort of the noise that happens
after we run a big internet wide scan?
Some people are going to scan us back.
Some people are going to be trying to to
dice us.
Some people will be connecting just to
see why they got a probe packet and
so forth.
So the technique we use for this is very
much
like the idea behind sin cookies.
So the idea is if we were
to look
into the headers for, say, a sin
probe here,
there
are a bunch of
fields that we can use to encode
distinguishable data.
So based on some per scan
random
secret,
we
encode bits into the
sending IP address if we have a range
of addresses to send from into the
sending port and into the
the initial sequence number.
And so when a response comes back,
the receiving IP address received report
and acknowledgment number are going to
reflect
that information we put into those fields
so we can then look at them and get
and learn with high statistical
certainty. Whether this probe was a
response to whether this
response was coming back as a result
of a probe we sent or not.
All right, finally, the overall
architecture of the tool is
optimized for
extended
ability for the ability to build new
things on top of it as easily as
possible.
So we tried to do the hard work in the
main body of the tool.
This is all written in C.
We provide some some probe modules
and response interpretation modules
that you can take and modify and build
on to do all sorts
of different scans. So we have we ship
with things for doing since scans, we
ship with things, for doing
certain kinds of UDP probing.
If you want to extend this to do other
kinds of probing, it's a simple matter of
modifying these modules.
We also try to because because we wanted
to use this for research, we tried to
make the output
system very flexible so that we could
take data coming back from the scan or
throw it into a database.
Throw it into Redis.
We have a variety of output handling
modules that you can use for different
purposes.
We also have because
initially what
this is providing is just basically
a port scan.
You probably want to do more.
You probably want to find out hosts that
have that port open and make a real TCP
connection to them.
So to allow you to do that, there are a
couple of ways to plug this together.
We have we ship the tool with a
banner grab program that can very
efficiently make many thousands of
connections in parallel to
download, say, the root of
a website or the the banner from
a Telnet server.
Put that in a database for you,
or if that's going to be too much work
to use a separate program
for that, you can integrate that into
your output handler and ZMapp, and we
do provide a kernel module.
If you want to do this
with
the TCP connection, you've half
opened with that initial syn scan.
So we have a TCP
kernel module that will complete the
handshake and give you back a socket.
That's not something that Linux can do
out of the box.
All right, so really quickly, if
this is how it works, is it does this
actually work?
So you might think that if you're doing a
scan even at a gigabit, packets
are going to get dropped.
There's no way that
if you're doing what I'm proposing, which
is just send one
sim packet to everyone and see
what comes back.
There's no way you're going to get very
good coverage. Surely enough will be
dropped that you have to send multiple
since
you have to retry, right?
Well, the data says says different
ways, so this is one experiment we ran
where we tried doing scanning.
This is, I think, on forward four four
three multiple trials
of scanning for.
I think this is for
one percent samples of the IP address
space
at different rates of
of
outgoing probe packets.
And what we're really looking for here is
we're looking for how fast is too fast.
We're looking for the point when we start
seeing fewer responsive hosts
because we're overwhelming the network.
Well, in the experiments we did up to
this is with a gigabit uplink
up to a gigabit.
We were not able on our network to
observe any meaningful drop off.
So the there's some
very there's some
fluctuation based on other things I'll
talk about in a second.
But basically, we cannot find the
place where we start seeing drop
off because we're overwhelming the
network.
So based on this,
we
conclude that that gigabit scanning
at least
up to a gigabit is not enough to
overwhelm
upstream.
However, your mileage may vary.
This depends a lot on what the network
topology
around you looks like.
We happen to have a very, very well
provisioned uplink
that connects in Chicago to
to several different
major backbone providers.
Your mileage may vary if your your
network upstream is not
well provisioned.
Part of the reason for that variation you
see, though, is that the number
of devices
that are responsive on the internet
varies quite
a bit over the course of a day
over the course of a week, just
as there is a result of people
in aggregate. Turning things on and off
is our our main theory.
You turn off your, your phone
or your cable modem at night.
We see a variation of about plus or minus
three percent on a typical day.
So this is based on doing
continuous scans of a one percent
sample of hosts on fought for four
three over a 24 hour period.
And you can see that very lovely diurnal
pattern.
Another question is,
even if we're not overwhelming
the network.
Are we getting back responses
from from everything that
should be responding, even though we're
not sending more than one cent
packet?
So one way we can see this is
by
trying to send multiple
SIM packets to to each address.
And what we'd expect to see is
that as we send more and more, eventually
we reach one.
Some, some plateau, some
rate, some number of SIM packets at which
only a vanishingly small number
of
hosts will have
lost them
all. So basically, we're trying to reach
ground truth here, some some denominator
against which to
compare the numerator of how many results
we've gotten.
Well, what we do see what we find indeed,
is that there is a plateau
and really it starts to level off
at about 8000 and probes
per host. But the difference for fewer
than that is actually quite small.
You can see the scale on the left.
So based on this, if we use this point,
this plateau as a kind of ground truth,
this
is where we get at this number of about
97 to 98 percent of
responsive hosts are going to be reached
with just one cent packet.
All right, so it's a little bit unfair
to compare scanners like ZMapp to end
map, which was not designed
for internet wide scanning.
But it's also a natural question to ask.
So just really quickly, what do we get
if we do this comparison?
So let me give you some average at times
for using ZMapp and Nmap to
scan a random sample
of a million hosts, and this is all
scanning the same sample.
So with end maps, this is using
maps. Most aggressive default
insane is what it's called,
plus a
few flags that make it a bit faster than
that. This is basically the configuration
we used in the
the wikis work I mentioned.
So with this configuration in end
maps default
configuration here, what I'm talking
about, it sends it
tries to send up to two send packets.
Perhaps it waits for a timeout, then it
sends another one.
It will take about 116
days to probe the
entire internet using
that configuration.
If we reduce and map to just sending
one probe to just waiting one timeout
interval, it
will only take about 62 days.
ZMapp in its default
configuration in this experimental
setup would have taken about an hour
10 minutes.
We we normally do faster, but this
well, these tests were done with a neck
that was not quite as good as the one we
used for the 45 minute trials.
ZMapp, in a maybe more
comparable to probe configuration
takes, would take about
about twice that about two hours and 12
minutes.
So quite a bit faster.
In fact, this is
about 3500 times faster
than in Maps most
aggressive default configuration.
And I'm particularly proud of that,
because it's not often that you get three
orders of magnitude through aggressive
optimization, but when you do, oh man.
Surprisingly, though, ZMapp
also finds more hosts that are
responsive than any map does.
And we were really scratching our heads
about that because even with two probes
and map finds fewer responsive hosts
than ZMapp does with one probe in these
tests.
So why is that?
Well, that turns out to have to do with
response time.
So if we tried
to do an experiment where we measure how
long it takes for an
act to come back after we send a sin?
So with a huge number of hosts, we can
plot this as a cumulative
fraction here.
What we find is that if you just wait two
hundred and fifty milliseconds
for an act to come back,
you're only going to see about 85
percent of responsive hosts.
If you wait 500 milliseconds, you're
going to see about 98 percent.
And it takes almost
almost eight seconds before things
a little more than eight seconds,
actually before things level out and you
see the last
responses trickling in.
So things rattle around in the network
for a while and then come back.
It turns out that in map, because it has
to maintain state for each connection,
including a timeout in its aggressive
modes, has relatively
conservative, relatively short timeouts.
Actually, the time outs are something
like
250 milliseconds and 500 milliseconds.
So Nmap is going to just give
up and move on after 200 milliseconds.
ZMapp because we're using this
asynchronous design where we
don't have any per connection time out,
we'll just accept the response when
it comes back. Eventually, if it comes
back before the entire scan run
terminates, that allows us
to see these very long delayed
responses.
This is why we're able to get even higher
coverage, even though we're using
what's probably a much more aggressive
means of doing an internet
wide scan.
All right, so let me talk
about a
few applications, a few of the fun things
that we've done.
So let me start with a really fun one.
So one thing that you can use
internet wide scanning for is finding
things that are are vulnerable and
at really, really big scale, at internet
wide scale.
So this is important from a white
hat perspective because we want to know.
We want to
know the impact of a
problem that's been announced.
Or maybe we want to know before we do
disclosure. Before we do we
go public
with a vulnerability how big the impact
is going to be.
Maybe this will help make a case to the
manufacturer of whatever technology it is
that they need to get their patch out
now.
In any case, we experimented with this a
few times.
The first time had to do with
these vulnerabilities in UPnP
that Eddie Moore announced
about
a year ago, where several common
UPnP
implementations had just terrible,
exploitable problems they were used in
in many millions of devices and
could give a remote
attacker root
in
in just a single UDP packet.
Well, it took less than six hours
for for
one of my Ph.D.
students
to
code
using ZMapp, a scanner
that would perform a UPnP
discovery probe across all of
the internet and also to run the probe.
So,
and by doing that, we found about
six
million
responsive UPnP devices.
About three million of them would
have been vulnerable.
Still, even when we did this, this was
maybe a month after the vulnerabilities
were disclosed.
We're still vulnerable
to the vulnerability is that that
HD anymore went public
with. So in the same
amount of time, if you had an exploit
ready, you could have exploited all of
these machines.
So it's both really valuable from a
defensive standpoint and really, really
scary
from
a vulnerability standpoint
that this kind of very high
speed probe probing is possible.
Of course, you always could have done
this with a botnet, right?
So in one sense, it doesn't make anything
possible for the attacker.
That wasn't possible before.
But this does democratize it.
This puts this same technique the same
capability into all of our
hands that previously, maybe just
a few of us who had
botnets, a few of maybe you who had
botnets, I don't have a botnet
had the ability to do,
Oh, I should add, we did another study
over the summer looking at Ipman
devices.
So I PMI is this remote
management interface that's built into
almost every server motherboard
where the motherboard actually has.
In addition to the CPU,
it has an embedded arm
system on the
motherboard
that's on even when the server is powered
off.
And you can get
into through a web interface to power
cycle the server, monitor itself, get
a remote KVM, you know, basically
anything you could do if you had physical
access to the box?
Well, you'd think that they would have
gotten it PMI
security, right?
Since this is protected, really valuable
assets and giving you an incredible
amount of power, it's basically a
backdoor that comes with your your
server.
But we looked into a
leading
IP, my manufacturers implementation,
and it was just riddled with
vulnerabilities.
I mean, I'm talking about
a
root buffer overflow in
the web interfaces, username and password
fields.
Anyway, using ZMapp, we
were able to do a scan and figure out how
many systems could be exploited
immediately using
this problem, and we were able to find,
I forget the exact number.
I don't have it on the slide 50 to 100
thousand servers
that were plugged in to public
IP addresses, even though these are
server grade machines
that had this
remotely exploitable IP am-I.
So they're sort of more depressing things
you can
do with this,
like
enumerating hidden services.
So if anyone here has built a service
where you rely on the
internet being large
and your things being hard to find,
you might want to rethink your threat
model.
We can easily uncover things
using
tools like ZMapp and
high speed scanning.
So one experiment we did was looking for
Tor Bridges, and these are
hidden Tor nodes that aren't publicly
advertised that are used to let people in
countries that practice censorship.
Access Tor without that, that sensor
just blocking access to these IP
addresses
and
using a very simple application
of
ZMapp, we identified
more than 86
percent of the tour
bridges that were live and allocated
at the time in the Tour Directory
service.
This is with
a single scan, just looking at
two ports.
Now, this is not the state of the art
tour. This is luckily tour is
already moved on.
They've developed other techniques
that are much more resistant to this
attack, including OP's proxy,
which randomize is the port it listens
on, which makes it thousands
of times harder to exhaustively enumerate
everything and also has maybe
a more difficult to spot signature.
However, this is something
that other services ought to be worried
about.
Here's one more thing that's kind of fun
we can do with this.
So detecting service disruptions
so let's say there's a
hurricane that happens like what
happened in my country last
year. We had this this thing called
Superstorm Sandy
that hit the Northeast and flooded New
Jersey,
and sandy was a terrific natural
disaster.
But one thing that we could
do using internet wide scanning
is track exactly what
was going off and when.
So looking at power outages
using internet wide scanning.
So what we did is I is
my student secure and I just use the map
continuously to scan as fast as
we could on.
I think this was Port
443 again https
while the storm was going on and
we were able to look for locations
for areas in the network where there was
a greater than normal decrease
in the number of listening house.
So places where the power went out and
suddenly people's Netgear routers,
for instance, were no longer exposing
their or their web interface to the
world.
So by looking at the whole internet,
we could spot evidence of this localized
disaster and we can do
something like this on a continuous basis
all the time if we want to to spot for
disruptions
in in critical infrastructure.
All right, so let me tell you about one
other thing, that's a pretty
and very interesting to me anyway,
security
related application of this.
And this has to do with cryptography,
and some of you have probably heard of
this work. This is work that we did
using
basically the prototype to ZMapp
in
2012, where
we did a scan of everything
that was using HTTPS and everything
that was using SSA in order to
collect the public
keys that are
used
for
for crypto on the internet, for the most
important protocols.
So we found some very interesting things.
Basically, we wanted to give to
do a health check
for the state of crypto online.
And this was one of the
the
easiest ways to get broad visibility
into the global use of cryptography.
One really interesting thing we found
is that even though we found
on the order of four HD types, almost 13
million listening house for SSA to
about 10 million hosts, the number of
actual distinct different
public keys is much, much lower
than that.
So we have about we have less than
half the number of distinct public
keys for HTTPS
as we have hosts.
This means there is a very large amount
of duplication
of keys, the same he used
on multiple systems.
Now there's some legitimate reasons why
hosts might share keys may be there.
These are all the different IP addresses
that belong to Google,
but there are also reasons that we
detected, which are distinct security
vulnerabilities,
things like default certificates
that are in use across many,
many different devices, and house
also evidence that many systems
have problems with their random
number generators,
and that this resulted in distinct
systems, completely distinct
opposite sides of the world, different
owners
generating the same keys.
Now this is a problem, of course, because
if you and I share the same public key,
then we can each impersonate the other.
And when you talk about this happening at
global scale with almost
with having only half the distinct number
of keys as hosts, it's evidence
of a really huge problem.
But there is.
Let me see here.
But there's another
problem that we could spot using this
kind of global data from scans,
which is that some keys
are not only duplicated in ways
that would lead
you, and I
impersonate each other,
but are duplicated in ways that would let
me with only the data we collected
with scanning. Actually, figure out what
your private key is.
So one example of this
has to do with RSA, where in RSA
and you should come to the crypto talk
tonight if you
want to to learn much more detail
about how these crypto systems actually
work from real cryptographers.
But in RSA, you have this modulus
and that is a product of two large
primes pacu.
Normally, we don't know
any efficient way to go backwards
to get P and Q your private key
from any which is part of your public
key.
However, if your key wasn't quite
properly generated
because,
say, there was a problem with your random
number
generator and you
and I end up having different keys, but
we share one of those two
primes, let's say we share P and
have
different cues
or public keys will look completely
different. But if, say, we examined
them together, we
can very trivially factor both
of them just by taking the greatest
common divisor, which takes milliseconds.
So we came up with a way
based on a technique by
Dan Burnstein, Dan Bernstein,
that lets us take the jack
of
all pairs of keys on the internet.
And what this let us do is
get the private keys
to about half a percent of all.
The TLS
hosts all
the
HTP as hosts on the internet
because they have improperly generated
keys of that form.
Similarly, with.
Similarly, with DSA, which is
another public key
algorithm that's used
even more so for S.H.
than for
TLC, we got more than
one percent of everyone's of all the
hosts private keys,
and this is all based on
visibility that we can get with internet
wide scanning.
But there's a question of why are these
systems vulnerable?
Well, the what?
What are they? Why are they vulnerable?
Well, if we actually look at the
certificates say that are
served
for
HD types hosts with these
vulnerabilities, here's what we
see.
These are not web servers.
They're not normal web servers anyway.
This is not.
This is not
Google or or PayPal.
Let's see. The systems say things like,
Oh, system generated
TP-Link technologies from TP-Link
software. You can learn that
this is some company that makes some
kind of embedded system gateway
device.
What's this Xerox office printer
at the University of Michigan C.S.
Department?
And I swear I didn't pick this
specifically for that.
Turns out that almost all of these
devices are different kinds of embedded
systems.
The Internet of Things is already here.
This is the world we're living in.
In fact, if you look at just random
samples of device of IP addresses,
almost everything you're going to be
seeing is some kind of
embedded or headless device.
It's not a normal public server.
So these are things like home routers,
professional routers, server management
devices, printers, et cetera.
So in the end, in this week, key
research we identified devices from more
than 40 different manufacturers got
in touch with those manufacturers to try
to fix the problem.
And we also traced the problem back
to a to a fundamental problem in the
Linux kernel.
So this is one of the things that many of
these devices
shared was that they were running
Linux while on Linux.
Where do you get random numbers for key
generation?
You probably get them in practice
from dev you random.
They say you should be using Dev Random,
but nobody does trust me.
So the way Linux generates
random numbers is, it looks at things
like the time of
boot, like keyboard and mouse movement,
like disk access timing, and it
matches all mixes.
All of these
together in something called the input
entropy pool.
Then periodically, when things read from
dev urandom, it mixes in
the time the system turned on
and
mixes mixes in stuff from the input pool
and gives you back
some some bits.
Well, on a headless device, let's think
about your Netgear router at home.
You probably don't have a keyboard and
mouse plugged into it.
It doesn't have a traditional desk.
It might not even have a clock.
Hopes.
And there's another problem, which is
that because of another security
vulnerability that might happen if they
didn't do this detente only gets copied
from that input pool into the non
blocking pool after you've accumulated
a
certain amount of data, 192
bits in the kernels we looked at.
That means dev urandom can take a long
time to warm up, and before
it gets this much data, it may be
entirely predictable.
And this creates a period of time between
when the system first boots and
when it's warmed up, when
it's a potential window of vulnerability.
So in one test system, here is when
Dev Random warmed up and actually became
randomized.
Here is when open SSA ceded its
internal random state.
This is what we call a boot time entropy
hall. And our hypothesis
is that many of these vulnerable devices
generated on first boot
their private
key
during this entropy hall, when
they had not
yet collected any form of randomness.
So there's a fix in the kernel now.
As a result of all of this, it may or may
not have rolled out to your distribution
yet.
Linux has now added other sources
of entropy and is trying at least to
make everyone's system distinct by
rolling in
the the Mac address.
If you
want to learn more about this part of the
work or to check your own keys online,
we have a key check service at factor
rebel dot net that will
tell you whether we have your private
key.
So just one more
thing I want to tell you
about, which is more, we learned
about HD tapes, we have another
paper about this actually looking at the
ecosystem
surrounding HD types.
So we all know there are too many
certificate
authorities that aren't secure or
trustworthy.
But what we don't all know is who those
certificate authorities are.
There is no single list of
all of the entities that your browser
will trust to
sign a certificate
vouching for the identity of a web site
you're connecting to.
The reason for this is that those root
certificate authorities that are listed
in your browser have the ability to
delegate their authority to other
parties to what are called intermediate
certificate authorities.
And those intermediate certificate
authorities aren't on any list anywhere.
You don't know about them until you see a
certificate from them.
So what we did was we use ZMapp
to crawl the to
crawl the whole public address space.
Find out what certificate chains it
saw and compile a list of all
of those intermediate certificate
authorities.
We compiled a lot of information about
the way those servers were connected to.
So this is an example of how you might
change ZMapp together with other kinds
of connection and processing
libraries in order to do a more
sophisticated kind of scan like this.
In the end, we did about 200
internet wide
probes to get this data.
More than a
internet wide scans each of them
more comprehensive
than the FSL
Observatory.
More than a trillion probe packets sent
over the
course of a bit more than a year.
So in addition to finding,
I think thirteen hundred
entities that can sign a
certificate for any
website you visit because there's no kind
of limitation to that scope
of authority and trust that's
put into SSL certificate
authorities.
We find interesting things like this.
There are
still 50 percent of certificates are
eventually routed in a 10 24
bit key.
This is something on the verge of being
factor rebel.
More than 70 percent of these routes will
expire more than
more than two years from now.
So this is bad.
On the other hand, there's some good
things. During the life of this scan, we
see SSA deployment.
I mean, SSA TLS deployment
growing dramatically.
We see a 10 percent uptick in the number
of servers, a 23 percent
uptick among Alexa top one million.
We need to encrypt the web.
Progress is happening.
I hope it will happen more quickly.
All right. So just a couple of concluding
remarks.
One.
One of my goals from all of this is to
encourage everyone here to be doing
more
work
involving internet wide scanning
involving measurement involving trying
to understand the internet
from this perspective of global
measurement.
And we've shown with work like the week
key work that this can give you really
new and different kinds of
security insight
in order to try to make this possible,
since not everyone has bandwidth to do
these scans themselves.
We're taking the data that we collect in
our scans and throwing it up on the web,
and we'd be happy to host data from other
people too. We're getting data from
Rapid7 now already.
So if you go to scans Dot Io, you
can download the data sets,
do your own data
mining on them yourselves
and and
ask probing questions.
ZMapp is also available
online. You can get it.
Here are demo is not
actually running, but I'll run it soon.
Finally, if you do decide to
do scanning yourself, just
a few reminders about
ethics, so it is possible
to cause damage by overwhelming
other people's networks.
And we don't want to to be dosing people.
We want to be measuring what's publicly
visible,
but there's no way to request
permission from everyone involved because
there's no equivalent to robots.txt
for for IP.
So for now, in order
to avoid alarming
administrators in order to avoid
knocking networks offline inadvertently,
there are bunch of steps you
can take to try to reduce impact.
One.
Just using the random
ordering that things like ZMapp provide
should help with avoiding
dosing people.
It's also a good idea to try to signal
that you're not maliciously.
You don't have malicious intent.
We do things like put up a web page
on Port 80 or on the scan source
IP addresses to let people know
and, of course, honor request to be
removed.
So the bottom line is, be a good
neighbor,
don't be a jerk, and everyone can get
along.
So over 200 internet wide
scans, we don't.
He got about 145
requests from people who wanted to be
blocked from further scanning.
Certainly some people blocked us without
requesting,
but very little.
Just one more story.
One thing went wrong, which
is that at one point during our scanning,
this article appeared in a certain
publication link to From the Drudge
Report about
how Iranian attackers had taken over
a large part or all of the University of
Michigan College of Engineering
Network in order to attack U.S.
banks.
There are several things wrong with this
story.
It wasn't Iranian hackers.
It was Zakia, my student.
So we got them to run a little bit of a
statement from us saying, no, this isn't
accurate, but they say their expert
stands behind their assertion that this
was Iranian attackers.
All right, so there's much more to be
done, but I hope everyone will use this
technique to to do great research.
We're living really in a golden age now.
Networks and computers are fast enough to
exhaustively scan IPv4
and IPv6, which has a much larger
address. Space that's going to be hard to
scan isn't here yet.
So this is an opportunity
for all of us who are interested
in what's on the internet
to go ahead
and find out.
So again, here are some new URLs for
things you can try out.
Thank you very much.
Thank you, Alex, certainly a very
interesting talk.
Now we've got time for about one or two
questions maximum because we're close to
wrapping the session up, so if there's
any questions, please just quickly line
up a bit of behind one of those eight
microphones.
OK, I've got three questions and then I
think we'll have to finish up, go ahead
at number four place.
Yeah, thanks for all the talk.
What kind of hardware do you need
for the scans?
You mentioned one geek uplink
pipe, but the server doing
the scanning, what do you need?
So for most of these experiments
we used either I think about
a five year old Dell workstation
or a really cheap
$800
hp one. You so low end,
low end but solid
grade hardware, I think, is probably
the
the ideal.
You can do pretty well just from
cheap commodity stuff.
OK. Thanks.
All right. And no place.
Yes. Things that you mentioned
IPV six at the end was about
to ask about that.
The other point.
Are you aware of the
RIBE Atlas project concerning
scanning the whole internet in a
distributed way?
So I have heard of, but I'm not
well-versed in that in that project,
and I'd love to know more if anyone
involved in it is here.
As far as IPv6, IPv6
raises a bunch of interesting new
challenges. Of course, the address space
is much too big to exhaustively
enumerate.
My hypothesis is that
the main
way that Wills will scan that
is based on
lists of addresses that are in use
that are collected
through passive measurement, probably by
by people in industry.
And one question from the internet,
please.
OK, so how do you think
that sysadmins should be dealing with
full scans in the future?
Do you think it's really worthwhile to
send abuse emails and phone calls for
each and every port scan?
I think this admins have to.
Well, it's difficult for sysadmin
sysadmins. I think should recognize
that not all probe traffic they're
receiving is malicious and
that if it's malicious,
people aren't going to stop it just
because you ask them to
that if it's malicious, the
the bad actors are probably just going to
be probing you from somewhere else.
They're probably probing you from a
botnet.
So I think the gains from
spending time
trying to stop probe traffic
are relatively small compared
to what my advice to sysadmins
would be, which is to make
sure you're scanning your network
yourself and locking it down
so
that there are there isn't anything that
a bad actor can gain by
getting visibility into what's publicly
addressable in your network.
OK. And the very last question, because
we're running out of time for microphone
number one place.
All right, thanks. Yeah.
Just going back to that thing about the
when you use the time of day
to explain the difference in responses,
were you using?
Does that mean that that scan sample used
in that test was based on the same time
zone? Or how do you explain that?
I didn't quite get that because
it shouldn't.
It shouldn't be explainable that way.
So, no, we picked a random sample of IP
addresses about, I think, a million
addresses in those
tests and
just probe them every few minutes
throughout a day.
And what we're seeing, yeah, is that
there were actually fewer responsive
hosts at certain times of day.
So this is their reasons you might expect
that to
happen, that if some
of these devices are things that people
are powering off at home while
they're asleep, they're things that
offices that are going to
that are going into hibernation mode,
their phones.
So I think it's a reflection of just the
aggregate diurnal pattern
of everyone
who's plugged into the address space.
I'd be happy to
talk about it more, but that's the
working hypothesis
because we can't actually find any
difference in.
We can't find it's not at all correlated
with any
local variation with
the cycle of local variation in network
traffic. In fact, we're seeing more
responsive stuff at times when
we'd expect our network to be more
heavily loaded.
You know, OK, before we're wrapping up,
I would like to remind everyone to please
provide us with some feedback because
that helps the speakers and that
presentations, and it also helps to have
a contrast. If you want to do that, you
hop online at the top line.
There is some feedback forms would be
really great if you could fill them out.
Last thing, don't for get to collect
your rubbish would be really great if you
take it out and we do recycle the bottle,
so there should be crates outside to put
them in and lasting.
One more round of applause for Alex.
Thank you very much.