Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1096 Thanks!
[Music]
okay our next talk is sneaking in
network
security um our speaker Max has is going
to tell us how to scale up the uh uh
defense for computer networks and in
particular how to um integrate that in
existing networks
okay um max here is a former pentester
and now a blue team
member please welcome him with a huge
round of
[Music]
applause thank
you hi everyone my my name is Max
burkart I'm here to tell you today about
sneaking in network security how I and a
small team of other security Engineers
managed to implement a strong Network
segmentation model in an already running
high-scale large Network I'm a security
engineer at Airbnb and so the sort of
practical experience of this project uh
occurred in that Network however I think
that the techniques will go over here
today uh will apply to many other
networks and so I'll spend some time
talking about the technical theory
behind this approach as well as what
happened when we rolled it out uh in an
attempt to give you some uh good
evidence and experience to run this in
your own
environment so uh let's talk about
network security in 2018 uh segmentation
continues to be a really good idea
because we all know that compromises are
going to happen uh those boxes are going
to be popped whether it's a zero day or
something uh less fancy like you know
somebody forgot to patch a server and
network segmentation gives you the
controls to be able to keep those
compromises contained to make sure that
low security systems can't pivot into
higher security zones and uh help your
instant response teams keep incidents
localized however uh if you've ever been
involved in network pen testing you'll
know that a well segmented network is a
thing to see and I think we know why
this
happens as networks grow quickly uh
small security teams especially ones
that something like a startup like
Airbnb was uh find themselves having to
prioritize their work where it is the
most impactful and that usually ends up
being the perimeter the internet facing
hosts and so as a network grows quickly
you end up with a large Network that has
this sort of hard shell soft Center
architecture where the external
perimeter may be hardened but once an
attacker is able to compromise that they
may have relatively free reign inside
the rest of it and um this obviously
isn't something that we want and ask any
blue team member and they'll know that
this is a bad place to be but change is
hard and uh especially with a pretty
large Network so to give you an idea of
uh the scale that this project dealt
with uh earlier this year when we're
implementing this uh airbnb's production
Network had about 2,500 Services about
20,000 nodes and I Define a node to be
something that's sort of like a host
whether it's a uh instance running in
ec2 or kubernetes
pod and over a thousand Engineers who
were doing hundreds of production
deploys per day so things are moving
really fast and uh it's hard to go in
and you know build in these large
architectural changes like adding
segmentation furthermore because of this
sort of Highly service ified
architecture there was a lot of complex
interconnectivity between these things
uh so determining where the zones should
be uh was difficult in
itself finally uh developer productivity
is a really big concern for us and
especially to uh my managers and their
managers you know if you have over a
thousand Engineers writing code every
day if you slow them Ma down by 5% or
10% that's actually a really expensive
thing to do and it's not something
that's going to fly so the question
became how do we go from a soft Center
Network to something that has good
segmentation and has the security
properties we want and we're not allowed
to stop development we can't start over
we've got to be able to build a security
in as the network is running so uh we
hear a lot especially in the pentesting
offensive Community about like trying to
be like a ninja right get into the
network do stuff without anyone noticing
uh I'll argue that it's also as just as
important in defensive security we need
to be defensive security ninjas and be
able to sneak in put in the defenses and
have nobody know we were
there so what's the theory that we're
going to be applying uh in in this
approach uh we need to stop thinking
about security as this uh like layer
around development as another step in
the waterfall model you know this uh is
maybe what we were thinking about 30
years ago that you'd build an
application and then you do security
testing and then you'd ship it to
production but it just hasn't really
held up any more so there's been a lot
of smart people talking about sort of
the new way to do things agile security
you know Dev SEC Ops Z devops people
can't decide this whole concept of
really unifying security operations and
uh software engineering so that you're
building a secure thing all the way
through and this certainly isn't
something that we invented um many
people have been working on it but I've
found that most of the time people think
about this concept when in the terms of
application development and I think it's
time that we integrate this with network
security as
well uh there's I I think the important
thing here is scale right the we need to
build a security solution that scales
with development um there's this saying
that it's good to hire lazy engineers
and developers because they're going to
build things that s sort of scale up and
don't require a lot of manual work
that's even more important for security
Engineers you're never going to outwork
the attackers and so you need to build
something that's going to scale along
with your Engineering
Group so um we're good project managers
so we're going to lay out the
requirements for this uh solution before
we jump into how it actually
works uh whatever we build needs to stay
out of the way of Engineers uh it may be
something they're aware of uh but the
farther we can keep it out of their
scope the better so they can just keep
writing applications that make the
company money or accomplish your
organization's goals and uh their stuff
ends up being
secure
security by default is of course
something that we have been chasing for
a uh a long time but I think that we can
also go further than that say um Beyond
being secure by default it should
actually be hard to have an insecure
configuration with this system so we'll
try and design things in that
manner and finally we want to build
something that as much as possible is
flexible to whatever sort of network or
uh protocols you were using you don't
really ever know what's going to be
coming 6 months down the road um when
this uh was being worked on every views
mostly Linux on Amazon shop but I don't
know what's going to happen in the next
six months we might acquire a um you
know hasell on Azure company and try and
integrate that or we'll you know start
you know going to on-prem Data Centers I
have no idea what's going to be in the
future so we want to build a solution
that's going to be as agnostic as
possible to those sorts of
decisions so uh my next slide is uh
basically the whole solution tried to
condense into two sentences uh we're
going to use Mutual TLS built into the
service Discovery system for
authentication and confidentiality
across all service Communications and
we're going to discover those access
lists totally automatically for security
with zero to almost zero configuration
uh this is a lot of jargon on a single
slide so I don't expect you to like kind
of visualize it yet we'll dive into each
of these parts and I'll show you how
they fuse together to build a system
that is uh invisible and
secure so to start off um I I've sort of
isolated three pillars of this approach
uh the first is uh TLS in service
Discovery so we love TLS it's um one of
the really powerful protocols that the
security industry has uh managed to
build and it's gives us great security
properties if we can use it everywhere
so the first pillar is get everything to
be using TLS and by building into
service Discovery make sure that it runs
everywhere without a lot of per app
configuration pillar two is binding
identity to nodes so uh in a more
traditional Network segmentation model
you might Define subnets or restrict
things by IP address we're going to be
able to be a little more flexible with
how we refer to individual nodes in this
network because we're using TLS as an
authenticator and therefore can sort of
Define our own concepts of identity and
I'll get into that see
finally we're going to generate an
authorization map so by automatically
determining what services need to talk
to what and figuring out how data flows
through this network we can attempt to
update akles automatically to stay out
of engineers's way while still ensuring
that uh the connections between services
are trusted and uh can be
verified so this is a diagram that we'll
be diving into individual pieces of um
but basically this is a very simplified
view of a network we've got three nodes
those nodes each have a certificate sort
of defining who they are and they can
use those certificates to communicate
with each other through TLS tunnels they
have authorization logic that runs on
them that is fed by this sort of
centralized map um of what nodes and the
network should talk to each
other let's jump into the first pillar
here which is uh the implementation of
TLS
so specifically here we're looking at
these uh
tunnels um before I start though uh it's
important to cover some basic um uh
Concepts here uh just to get all on the
same page we're using Mutual TLS here um
which is uh you know you've certainly
heard of traditional TLS that's what
your web browser uses all the time where
you have a client that is verifying the
identity of a server normally it will um
get the CT make sure that the subject
alternative name or the CN matches the
domain name and if so um tell you it's
verified but TLS is really awesome and
it actually out of the box supports
verification in both directions so you
can have the client also present a
certificate in that initial handshake
and the server can check who is talking
to it using an equally strong
authenticator this is pretty hard to
deploy on the public web because uh
users can't really manage certificates
but in your own uh production Network
this works really well because you can
distribute CS to
everyone so this is really great because
this means we can have two-way strong
authentication uh in with key material
that security Engineers understand right
we know how to deal with these sort of
systems and so not only can we make sure
that uh clients of services know they're
talking to a legitimate service but that
service can look at who's talking to
them and make sure that that's a caller
that um seems
appropriate so that's Mutual TS Service
Discovery um this was a term that uh I
hadn't heard about a lot before I
started working in companies that used a
lot of um Cloud environments in SOA but
um at its core service Discovery is this
concept that you have some node in a
network and it needs to find other nodes
to provide services to it so uh if you
think about it DNS is like a very old um
sort of basic service Discovery uh
system you want to perform a Google
search so you do go to ww.google.com um
and DNS finds you a server that can
provide Google services to you um so
these have gotten a lot more uh complex
and varied as people move to these
environments where hosts are very
flexible and stuff moves around a lot
and they're um pretty ubiquitous in in
modern um uh service oriented
architectures and service Discovery can
actually be kind of problematic for
security if you uh do it wrong cuz
fundamentally it's trying to be a map of
the network and be really helpful about
like oh hey find this service here find
this service here uh but I'll argue that
we can actually use this to great effect
um in achieving security so um Airbnb
uses a framework called smart stack um
and so that was what was there when we
started this project and we built this
security extension on top of the smart
stack framework um so that's what I'll
uh sort of be talking about but I
believe that these Concepts can be
applied to most service Discovery
systems as a brief aside into how smart
stack works this is an open source
system that A&B uh created and open
sourced a few years ago but the basic
idea is that it uses uh two other uh
publicly available projects zookeeper
and ha proxy in order to make it easy
for services to talk to each other if
you look at this uh example above uh
node 2 is hosting a service service B
and so uh service B is going to report
into a zookeeper cluster hello I'm a
service B instance and you can find me
at node
2 node one wants to talk to service B uh
and so it will load the uh relevant
addresses for service B from zookeeper
and it will put them into its local ha
proxy instance ha proxy is a reverse
proxy that just um kind of forwards
traffic along service a if it wants to
uh make a call to service B simply then
just sends a request to Local Host and
leaves it to ha proxy to find a suitable
host to fulfill that request so an
important thing to note here is that
this system was not designed for
security um anything can write into
zookeeper uh it is like the most prone
to impersonation thing possible because
uh you just ask for a list of Nos and
you get them and it's not really
authenticated but uh I'll show you how
in the next uh few slides we can build
security into this
system so the old way that we connected
to Services before uh any security
upgrades is that service a wants to talk
to service B it sends request to its
local outbound proxy and that sends it
along so it's going to make an HTP
request to Local Host that gets sent
through the reverse proxy goes across
the internet um to service B not a lot
of security going on here what we added
is a uh secure shim so we added a new
reverse proxy that runs on the receiving
node in front of service B and we
reconfigured the proxies to communicate
to each other with mutual TLS so now all
the traffic that's going over the
Internet uh is in a TLS tunnel but
crucially service a and service B did
not change at all service a is still
sending HTP Traffic Service B is still
receiving HTP traffic so we were able to
uh pretty radically change the security
model of this uh cross-host
communication without touching a single
line of an engineer's
code so this is where we're getting our
invisibility
from there's some other really big
benefits to to this because there are
these two service discovery proxies that
are doing the TLs setup and they are the
things that can do authentication and
verification of this TLS
tunnel um security was able to build
these controls once and distribute them
basically across the entire fleet the
same proxies can run no matter what the
what language the service is written in
um what sort of protocol that service
uses and so uh instead of having to
verify authentication authorization code
in dozens of different uh Frameworks and
languages we're able to do it just about
once the other thing uh that ended up
being really helpful is that having
these proxies on either side of your uh
service Communications is actually
really helpful for non-security reasons
um so things like uh consistent metrics
better tracing um better ability to do
load testing we got all of those for
free by adding in more proxies and thus
we got to really get the um support of
other infrastructure teams at the
company who uh maybe didn't have Direct
Security goals um but they wanted to
help us do
this so basically what we've done with
this whole proxy thing is sort of the
opposite of what the NSA wants you may
remember this slide um from a leaked NSA
presentation uh where they were
discovering with Glee that inside
Google's uh Cloud Network at the time uh
there was a lot of plain text HTTP going
on and SL was added and removed we are
just adding SSL and keeping it there um
all of the arrows on the right need to
be TLS in the Modern
Age one important uh caveat about this
particular approach is this concept of
proxy
exclusivity which is that basically we
are relying on this inbound proxy to
provide the security benefits of TLS
confidentiality and authenticity and
thus it is crucial that going through
the inbound proxy is the only way to
talk to a given service if that service
is reachable by going around the inbound
proxy you would still be able to talk
plain text HTP to it and possibly evade
authentication mechanisms and so it's
important that this is impossible um
I'll talk a little bit about how we
solve this particular issue um it's just
something that's important to be
thinking about uh if you're going to
implement this
approach so that's TLS by implementing a
new proxy um into an existing service
Discovery framework we can switch all
the traffic to be going over TLS um
without radically changing um the code
of services running next up though is
that really what we wanted out of all
this is segmentation right we want to
make sure that only legitimate things
can connect to a given service and so we
need to build a sense of identity that
can be used to do this uh
verification so in this next p I'm going
to be talking about how we put these
certificates there and more importantly
how we decide what that certificate is
going to
say
so segmentation you know we're trying to
make sure that a node in the network can
only talk to the things that it should
be allowed to talk to um you know uh if
a node needs to talk to the payments
backend service it's going to need to do
that for business reasons but we can
maybe make sure that only nodes that
have to talk to a given service
can but a lot of previous thought about
segmentation tends to happen on this
subnet level um you make a zone of hosts
and um things in that zone can talk to
each other and then maybe they can get
out to other zones via certain
predefined channels but in a
microservice network or um something
that has a lot of like Dynamic
communication going on it may make more
sense to think about this on a service
level as opposed to a host
level so um we'll say things like we
want the uh payment config page service
to be able to to talk to the payment
backend service that seems like a
reasonable thing to do but in our
Network we've also got a slack but
running that makes memes for engineers
and that thing should definitely not be
able to talk to the payment backend
service so um we can start representing
these sorts of decisions as uh instead
of you know these sort of static tiers
of hosts um we have a bunch of these
services and each service sort of keeps
a list of identities that's going to
allow to connect to
it and we just did all this work to uh
build up these proxies on either side of
a service communication that understand
TLS and are using TLS and TLS is
fantastic at verifying
identities so we can now start to build
the segmentation by saying for a given
service listener here are the following
identities which are allowed to connect
to it and thus you can end up in a state
where uh only the right things can talk
to a given service based on business
need we do have to identify all the
nodes in our Network though and this is
something that's going to vary a bit
depending on how your network is set
up
so uh you need to sort of find a concept
of an identity that fills a few uh key
few key um
attributes so this identity that you
decide for a node needs to be pretty
varied you know if you have one identity
for everything you're back at soft
Center Network again because uh you
won't be able to do any
distinguishing you need an identity that
a node can't change about itself
otherwise an attacker would be able to
compromise a particular host change its
identity and then move into zones the
network it shouldn't be allowed
to it should really be able to be
something that you can detect
automatically so that you can sort of
automate the distribution of these
certificates uh if you end up having to
go through to an Excel spreadsheet and
figure out what each host is and then
like men those CTS yourself it's not
really going to work and finally we do
need to represent this concept of an
identity in a uh TLS certificate so um
in our case we wanted something that
could fit into a subject alternative
name so most modern networks have some
concept of a role that works pretty well
for this um when you have a config
management system or a cloud permission
system you almost always are giving
things identities based on their
function and this tends to work pretty
well for this so in our Network we used
uh Amazon I am roles which is a sort of
designation given to an instance uh that
in gives it some level of permissions in
AWS
and this worked really well because um
most different Serv Services had them uh
they can't be changed unless you have
very high level administrative
permissions native us uh and it is can
represent it as a string so it fits well
in a
certificate so to kind of look at uh
what we're going to do here now uh we
need to give everything identity and we
need to make certificates that allow
nodes to prove their
identity uh in these TLS communic
we can then build this map of what
identities should be allowed to access
what services this is what is going to
give us our segmentation because we're
going to be able to distribute that map
saying for the payments backend service
you allow the following identities and
no
others and thus um you can get to this
place where uh only a very select set of
nodes in your network can access the
sensitive
stuff but how do we make that map that's
pillar
three which is the final segment of this
diagram so how do we figure out what
needs to talk to what and distribute
that so a big question here is really
all about uh
trust
how do you figure out what needs to talk
to what and do it with a minimum of
human involved uh computation a lot of
what I was talking about earlier in the
very beginning of this presentation was
about the sort of human cost of
segmentation um if you have people who
are spending all day trying to make
firewall configurations um that's going
to be rather expensive difficult to keep
safe Etc we want to try and get away
from the configurable list uh style of
security engineering where you hire a
ton of security Engineers to try and
figure out what is supposed to talk to
one so we wondered could we just infer
this from existing code can we look at
how the network currently works at how
our configurations are defined and use
that to build the sense of uh how
communication should happen so this is
getting to an interesting point because
the decisions you make here really
depend on how you think about uh threats
at your organization
so we decided that if you are somebody
who can merge peer-reviewed CIP pass
code into our config management system
that means you're reasonably authorized
to make
changes and um this is something that
may vary based on your organization's uh
setup and I'll kind of dig more into
those questions in a bit but for our
case we realized we have this Chef repo
the chef repo is a you know Chef is a
config management system um that can
distribute information to all of the
nodes running in our Network and it
already in a nice machine Parable way
was saying what the dependencies of
every service for so in this
hypothetical example we have a service
one service one has dependencies on a
production database a cache and a
monitoring service and this is already
set up in a repository that is rather
heavily controlled you have to be you
know an engineer that gets peer review
Etc to commit to this so what we can do
is we can take this uh determine that
service one is an authorized caller of
these uh services and then sort of build
that into this map say for production DB
service one is
authorized to do this we built this
service called
Arachne uh Arachne named after a Greek
spider goddess uh is kind of computing
the web of services and nodes at uh the
airnb network and so basically it's
continuously pulling our Chef repository
and deployed kubernetes artifacts to
figure out what connections have been
defined by trusted people and building a
sort of reverse uh map of for a given
service what identity should be allowed
to connect it can then push these into
S3 I'll talk about why we did that in a
little bit um and then those can be sent
to all of the nodes that are actually
doing um this comput this uh
allowance so the barriers that you're
going to be putting into place uh about
how this map is generated really depends
on how you think about uh Insider
threats at your company so in our case
um we've made the conscious decision to
trust our engineers and rely on things
like Ci checks and peer review in order
to make sure that uh legitimate things
are committed um but depending on how
you approach this you may want to have
more controls in place and this system
is rather flexible to do that all you
need is something that can automatically
config discover as much as it can and
then under some conditions publish a new
authorization map um to some location so
you could certainly imagine if you
wanted more controls than this um making
it so that when a new connection is
discovered it prompts the security team
for a quick manual review and an
acknowledgement before that actually
gets distributed so this does give
security a single point of control where
they can do any sort of monitoring or
additional approvals if they wish while
still taking away a lot of that
boilerplate work of trying to figure out
what actually connects to
what we can actually go further with
authorization instead of just telling
all of these um
uh service Discovery proxies to allow
these identities and ban these others um
because we're just using vanilla TLS uh
we can rely on the heavy support for
these uh sorts of protocols in many
things so uh the reverse proxy that we
use is the inbound proxy has the feature
to inject information about the client
certificate used into hdp streams that
went through it most of our apis are HTP
based ones so this applies to most
things and it means that whenever a
service gets a call over TLS it can just
parse this very simple header and know
exactly what sort of identity is calling
it making it trivial to implement uh
various like permission levels depending
on your service caller so this is
something um that sort of authorization
control would have been really tough to
implement before this system because
you'd have to set up maybe your own TLS
system or maybe a system of uh tokens or
keys or passwords uh but this lets us uh
uh you know leave all of the tricky
crypto stuff to the security own
components and let app developers uh
just parse a very simple header and uh
make decisions based on
that so those are the three pillars of
this solution we set up TLS in between
everything to give us the security
properties and communication that we
need we uh give everything an identity
in order to make sure that they can
authenticate to each other and enforce
segmentation by having uh specific allow
lists for every service and then we
automatically discover this map uh by
parsing configuration that's already
there but uh I'm not here just to uh
sell you on this solution because I like
it um there are some downsides and uh to
be perfectly honest I want you to be to
know about them before you consider uh
implementing some something like this
and so these are just some of the things
that we thought about and decided to
accept first um you are going to need to
constantly synchronize out this map app
of uh allow lists so or some subs subset
of those allow lists instead of having
centralized allowance of various network
communications like you might have if
you have a central firewall um you're
sort of doing in a distributed way every
node is determining whether or not a
connection is allowed and so that means
that you have a reasonably um strong
need for a lot of bandwidth to
synchronize this
out uh you can use caching which will
make some things a lot easier and I'll
talk about what we did that but that is
going to uh cost you some in terms of
update latency if the web changes and
you need to allow a new identity for a
service that may be slower if you're
using a
cache second if TLS has a problem partly
um you have way more problems than you
used to uh because you're now relying as
one of the core security elements of
your
system um so this is something that we
know uh but the reasoning here is
basically if heart bleed happens again
if we find some sort of major core issue
in TLS um already security is going to
be you know working nights until we can
get that patched on our front-end web
servers and so if we're going to be
massively deploying new opell versions
as quickly as we possibly can um that's
going to end up patching up all these as
well uh so basically we are relying on
the fact that major sell issues are
going to get a quick Community response
and be something that we can move
quickly
on uh adding more reverse proxies in
your traffic flow uh is turns out to be
kind of complicated um this introduced a
lot of interesting behavior in some
Services um and I'll talk a little bit
more about the specific things we ran
into um but it's just something to note
that um the actual addition of TLS to
things broke very little um but the
additional hop in the network had
surprising
effects you do need to be able to run
software wherever you're receiving
traffic through the the system because
you need to install that secure listener
and uh something that can download the
allow lists if you manage all of your
own infrastructure this is relatively
easy but if you have things like vendor
devices or hosted Services where you
cannot install arbitrary software that
gets a little harder um when we have
some services that are in this state uh
we basically put proxy boxes in front of
them and uh use those to handle the
authentication finally you are going to
want some sort of certificate revocation
uh because if a node does get
compromised you'll need to you know kick
out its permissions um and this I say is
usually tricky um there are certainly
ways to do it uh but this is something
to be thinking about and scoping as
you're considering um doing a deployment
like
this so rolling it out um my hope is
that you know I've described this
solution but it's not just thetical this
is something we did um and uh so I hope
that I can share as much as I can about
what we learned throughout this whole
process so to start with sort of the
technical details uh we built this
mostly out of components that are
available and open source
so uh for the inbound proxy we used
Envoy which is uh a project open sourced
out of lft that is really growing in
popularity in the service mesh world and
for good reason it's really designed for
this sort of
thing it's modern it's fast it has great
support for TLS has a ton of metrics
which are really
useful and uh generally served us very
well uh the one thing we ran into with
Envoy that is quite the stickler about
the HTP 1.1 standard and um that led to
some uh funny behavior in certain other
applications that were not so uh uh
strict about it uh but overall Envoy was
a great choice and we're actually
migrating to use that on our outbound
side as
well as I sort of alluded to earlier we
gave every node an identity based on its
AWS IM role and uh this was just sort of
a natural choice for us because this is
already how we were thinking about
permissions for nodes uh now nodes got
their permissions by their IM Ro and
that also uh kind of controlled what
services they were allowed to talk
to the Arachne service I mentioned is
basically just a continually running
Ruby
script that loads a chef repo and some
kubernetes artifacts and pares
them these authorization Maps the quote
unquote web files uh are uploaded to and
downloaded from S3 so we're using S3 as
the source where this all gets actually
pulled
from all in all um it was it's about 4
minutes to uh fully compute the web of
services and generate one of these web
files meaning that it's about a 4-minute
delay in between a change in topology
that is a service adding or removing a
dependency and when that gets reflected
and allow lists in our experience this
is far shorter than the time it takes to
actually deploy such a change to
production um so we haven't really run
into uh race conditions where a new
dependency gets added before um it's
allowed we had some pretty specific
availability considerations um mainly on
caching the output of these uh of
Arachne these web files so um we wanted
to make sure that if Arachne went down
and we stopped being able to generate
these uh authorization maps that all the
traffic kept working um we didn't want
to be owners of a service that uh if it
went down would would ban all traffic
and really if you think about it by
decentralizing all of the authentication
of service calls you want to be able to
rely on decentralization benefits so um
by putting everything in S3 and letting
nodes download it from there we can make
sure that if Arachne has some sort of
critical problem if it stops running the
worst thing that happens is that new
topology stops being
reflected so
um uh this means that uh traffic keeps
flowing even if S3 goes down as it uh
famously did last year I think that was
a fun day um uh things basically still
work uh nodes won't be able to download
new topology changes but they'll still
have local cache ones on disk and all
the traffic will keep flowing as normal
so this was a choice who made early on
and has just very well because uh when
there are new and interesting things
that happen with Arachne uh no one
really notices generally security is
able to fix it before someone changes
the
topology So the plan for a roll out uh
was basically this these six
steps we started by Computing this
authorization map since this all kind of
works on offline data uh we were able to
spend some time writing the software to
do this and getting it to work nicely
before we had to actually touch any um
Production Services
so we could build that map and verify
its
correctness next we wanted to give
everything an identifying
certificate uh so the idea of doing this
first is that this is a pretty small
change and something that we could
pretty safely roll out uh we're simply
dropping a certificate on a bunch of
nodes and it's also relatively easy to
verify that this worked before moving on
to the next step we can check for the
existence of these files in a um large
scale way and make sure they look
good third we installed this receiving
proxy everywhere and started listening
and setting up this traffic routing at
this point no traffic is actually
flowing through these TLS tunnels we're
simply giving it the path
to this also lets us uh configure or
verify the step before moving
on next we can start actually doing the
testing and uh building the confidence
in this system so we can start routing
some traffic through these new secure
listeners and we configured our uh sort
of configuration in a way that we could
turn it on or turn it off per service so
we picked a bunch of services that
seemed representative High QPS ones low
QPS ones ones that use HP ones that were
plain TCP um just a a great variety of
things that that seem like they would
sort of stress test the system and we
turned these on one by one and built
confidence that this is going to
work step five is sort of the radical
one and that is this switching
everything over at
once uh this is not always how you want
to run operations but we chose this for
a very good reason which is that there
were two people working on this project
and there were you know 1,98 other
Engineers Building Services as fast as
they could and we were reasonably
confident that if we tried to go one by
one uh we would we would never catch up
we had to be able to build a system that
we could switch on all at once and
confidently move into a post plain text
future our final step is uh rebinding
services to Local Host so that these
security guarantees um were were
enforced so we did this last um and this
was sort of painful from a security
perspective because you really have to
wait till step six is complete before
you really get the security benefits
here but we had to give ourselves the
ability to roll back if things turned
out to have problems we wanted to make
sure that if switching to TLS for a
service caused some unintended effect we
could roll back fix that and then roll
forward again once that was dealt
with so to sort of visualize this um we
start with the nodes we've always had
those we built um the authorization map
and made sure that was available first
moved on to adding the certificates to
everything we installed the reverse
proxies with their authorization
logic we turned on TLS for some things
to make sure that it worked and then on
PL Tex deprecation day everything went
to
TLS so uh we did this in April of this
year and there's a lot of things that
went well we went from about 15%
internal TLS usage to 70% in one evening
which was really awesome um and
something that I don't think would have
been possible with any other
scheme we made sure that there were a
lot of security or not Security benefits
to this system and this let us get wider
organizational support for such a change
you know these sorts of massive sweeping
infrastructure changes uh because they
affect everything can make other
Engineers nervous uh especially people
who were primarily concerned about
uptime and so we wanted to make sure
there was plenty of stuff in there for
them too uh some of the chief benefits
we provided included much easier
configuration because we were
automatically uh assigned identities to
everything and preconfiguring
certificates Engineers no longer had to
think about setting up a custom mtls
connection if they needed Security
benefits uh performance I will talk
about that in a sec but it uh the
numbers are
good and then there were a ton more
metrics available so people could have
greater observability in their services
and realize what was going on and that
was operationally very
helpful the other thing that we did that
uh was a really good choice was making
sure that we had the right configuration
we could disable TLS routing for
Individual Services on a one-off basis
so that if we determined that a certain
service was having a problem it uh we
didn't have to roll the whole thing back
we could keep the winds we'd gotten and
uh roll certain Services back in order
to fix them before moving forward
again of course um you know I'm here to
be honest with you there are some things
that were uh hiccups during the whole
process
as I mentioned earlier uh running
everything through an inbound proxy uh
sounds good on paper but leads to some
weird stuff in uh practice so of the
2500 Services most of them took this
fine um there was just a small
percentage that did weird things um
there are some things that change if
you're using a reverse proxy like all of
your traffic is suddenly coming from
Local Host even small things like
changing the case of HTP headers which
is fully Allowed by the spec um can lead
to weird behavior in some
applications reverse proxies also um can
mess with particularly stateful things
like websockets uh we didn't think about
the websockets case and uh did not have
support for that on Day Zero uh that was
a quick day one patch to teach our
reverse proxy that websocket connections
are special needs to be handled
specially so all of these things are
generally surmountable uh but you are
going to run into some weird Behavior
the thing that I thought was funny about
all this is that really um the biggest
problems we had uh had nothing to do
with the security properties um even if
we' had a plain HTP reverse proxy we
would have had the same same
problems our testing process because of
how we turned this on was very good at
testing the case where suddenly all of
your traffic starts coming in over this
TLS channel so you enable uh TLS for
service B and suddenly all the service B
nodes get all the traffic over TLS we
tested that well what we didn't have
great testing coverage on was what
happens if all of the services that your
box depends on suddenly start requ iring
TLS and so we ran into some interesting
um issues with this most uh
particularly uh ha proxy what we were
using for the outbound proxy was a bit
of an older version it handled TLS
certific certificates very poorly and so
for certain roles that had thousands of
dependencies uh it would load the same
certificate into memory over and over
again for every connection it was making
uh and that led to some pretty crazy uh
memory issues so that was something that
we could have tested a little
better the final thing to mention is
that uh binding these services to Local
Host again this took longer than
expected we expected to be able to use
Easy service config templates that were
built into our uh configuration
management to say okay everything that
used to be binding to 0.0.0.0 You're Now
binding to Local Host uh this ended up
taking a few weeks longer than uh we
expected because there was more drift in
how we did configuration than we
expected this is just one of those
things that um I wish we could have
allocated a little more time to in the
beginning I mentioned I'd talk about
performance because this always comes up
whenever you introduce a TLS project
someone is like but what if it's really
slow uh and fortunately I can sort of
confirm the security industry's
assertion about it which is that things
often actually got faster which was
really I actually I didn't expect it um
whenever somebody says this you at least
I had this sort of like kind of
disbelief like and did it really and
yeah um so for a number of our services
uh we improved 95th per of latency by as
much as
80% uh so what was happening here is
that we had a bunch of these services
that had sort of hand implemented Mutual
TLS for security reasons particularly
High sensitivity things like password
services or things like that did
Implement MTS because they wanted to be
secure but they were implementing
entirely at an app layer and so
application to application was
uh communicating with mutual TLS and
these applications tended to restart
reasonably frequently Whenever there
were deploys new boxes spun up
Etc and so they were unable to take
particularly good advantage of uh TLS
session caching and session resumption
meaning they had to use the TLs
handshake all the time making them quite
slow service Discovery proxies restart
very very infrequently uh they kind of
come up when a box comes up and often
can last for weeks or months and thus
their TLS session caches are very well
wared and thus we're able to keep a
session resumption rate of near 100% uh
meaning that we were basically just
paying the AES encryption cost which was
just happening in hardware and added
very little so that was a really great
uh benefit and we were able to pretty
much squash the concern of this will be
too slow for our
Network so doing this in your
infrastructure uh I imagine some of you
may be involved with networks uh that
are not as segmented as you like and I
think this provides a good approach to
uh implementing segmentation on a large
scale in a way that is actually
shippable there are some questions you
should ask yourself um when thinking
about this that might help you assess
whether or not this is a good solution
for
you
first how effective is it for you to be
able to distribute these proxies in your
service Communications so we had a lot
of benefit
um in that we had a configuration
management system that could deploy
software and configuration and we
already had these outbound proxies uh
that were in place because of the
service Discovery system we used so this
is something that came pretty naturally
for us uh but it's something to think
about in your own
environment how would you assign
identities this is really important
because an identity is a Zone it's a
segment in our Network and so if you
have a um highly specific way to refer
to things that you can turn into
certificates this may work really well
for you if you don't have this you may
need to do some work to get there in our
case uh you know IM rolls what we went
with but at the beginning of the project
not every instance had an I am role uh
we had to do a little leg work in the
beginning to get that to be an
enforcement for our entire
infrastructure uh will you need to sort
of manually configure uh these Access
Control lists or will you be able to
automatically generate them uh if you
can automatically generate that's where
you're going to get these huge uh
efficiency wins so that's something that
uh you really want to push for if you
can the other good news is that uh
there's some available options on the
market right now that can help uh do
this for you we sort of hand implemented
the whole thing but uh IO and console
which are both uh Solutions being pushed
as um sort of the new way to do service
mesh especially in kubernetes um
implement this sort of security system
already so um to be clear this is not
something that we we totally invented um
this is an idea that's been going around
for a while and sto and console
implement it for you in a uh easily
packageable way they do less on the
automatic generation side uh but you
could easily kind of build this sort of
system using these tools but if you
don't want to make such a huge leap and
switch to a whole new service mesh
system you can certainly implement the
security benefits here um with your
existing Service uh discovery stack as
we
did
so to kind of sum up here I'm here to
tell you that you can switch to a deeply
authenticated Network and the reason you
can do that is because you can make the
changes here invisible and you can make
the system
fast because of these generated
authorization maps and the automatic TLS
an engineer who is working on a
microservice um before the system and
after the system has basically the exact
same experience they still use the same
hdb calls they always did they still add
a new dependency get that changed
approved and merge to master and then
their service talks to it no problem
their flow Remains the Same as it always
has been but now when an attacker
compromises that slack meme bot with
some sort of you know Meme injection or
whatever it is um they find themselves
in a network Zone where they can talk to
basically nothing and all of the
services that were wide open to them
beforehand um simply reject their
connection out of hand when they ever
try and um go past a layer 4 connection
so um this is something that uh I
believe is possible we've done it now um
and I think it's a great strategy uh as
you try and build into the security what
you weren't able to do when your network
first
started so uh thank you very much for
listening uh if you want to stay
connected or ask me more questions about
the details of this something that's not
as easy to do in the Q&A section um you
can hit me up at maxb on Twitter max.
bout airbnb.com or um if you just want
to see what we're up to at Airbnb
engineering on
.io thank you very
[Music]
much thank you
Max thank you
Max uh if you do have a question please
line up on the microphones try to limit
your question to a single sentence if
you'd like to leave at this point please
do that as quietly as
possible uh signal Angel your first
question from the internet
please
hello uh question from internet why open
SSL and not Li
SSL uh so I I guess I said open cell
just sort of a random example um you can
use whatever
SSL uh stack works best for you um I
believe that the way our um packages
were being built uh used op sell but
switching to something like boring or uh
Libra sell would uh probably be a good
idea for further
hardening thank you microphone number
two your question hi great talk uh what
are you currently doing to mitigate the
increased risk of Local Host bound
ssrf uh so local yeah uh srf is
something that is uh deeply troubling to
me uh as somebody working on appsec in a
company that works uh almost exclusively
HB API calls um our approach honestly is
uh very dedicated static analysis um we
are watching engineer written code very
vigilantly for anything that might make
HTTP calls outbound and trying to ensure
that um uh it doesn't hit uh internal
stuff uh that's an area that uh my team
is trying to do a lot of work to improve
and perhaps that could be a future talk
cool thanks for number one please uh
very interesting idea uh are you going
to include workstations too uh
workstations um it's an interesting
thought uh at the moment we don't have
our workstations plugged into the same
service Discovery um and so they don't
have the sort of core proxies that could
work for this um but I think that if
that's an architecture you wanted to go
to this would actually lend itself
pretty well uh because you if you're
managing your workstations you could
hand them identities just as well you
need to think of probably a a slightly
different approach to identities because
you can't give a physical machine an AWS
IM role at least for us uh but if that's
something that your uh network has then
I I think it's a a very reasonable way
to
go thank you signal in your next
question okay um did they AIT the proxy
code before placing it in the front of
all
services uh yes we took a close look at
it all right microphone number two what
are the cost implications for
implementation and uh and on your
operations in
general uh costs are pretty low because
the uh reverse proxy is pretty efficient
um it doesn't use a ton of extra compute
so we didn't have to scale up anything
uh in order to support this verification
um again being able to just do AES is is
pretty cheap uh the generation of the
map is very cheap um it's running on a
single kubernetes pod um just running a
ruby script so that's fine probably the
greatest cost is simply an S3 transfer
of that um uh authorization map um and
that's something that we think we're
going to be able to continue to reduce
by uh sort of being a little smarter
about how often we check um in certain
areas the network that evolve very
infrequently and don't have lot of
topology changes uh we'd be able to sync
that a lot less frequently uh so that's
something I think we can improve in but
overall the cost is pretty
low right signal in do you have more
question from the net okay then
microphone number two
please uh in terms of uh certification
Authority how um how are you managing
the lifetime of the certificate and
which kind of consideration did you do
on that side so like uh certificate
experation renewal also ocsp if it's
already implemented or not yeah so so uh
we want to get to a point where our
certificates expire a lot faster um
there are some companies that have done
a really great job with CTS that only um
you know refresh that only last about a
month or even two weeks something like
that and um unfortunately I think our
infrastructure isn't a point where we
can reliably reduce it that much um our
current approach is that we can uh
currently ban things U by basically
introducing them into a deny list in the
uh allow list generation stage um and
that will result in something being
banned within about four minutes um so
that's how we can deal with active
compromises U but there's just sort of
an over uh a longer running effort um to
be able to uh increase our
infrastructure refresh rate so that we
can have really short live certificates
to deal with those sorts of um uh like
stolen C
attacks thanks your
question um since you do all of this on
a flat uh layer three Network um and you
already mentioned
payment information what does this mean
for your PCI DSS scope and how does it
affect the certification if you handle
payment data and the systems are
connected to other systems in your
network and not separate like by like
firewalls or something um it's so uh our
PCI network is a little interesting it
actually is a totally separate thing
from the most of the Airbnb production
Network um so that specific
certification didn't um affect us but I
think we've all set a pretty effective
time of um convincing uh Auditors that
this is an effective way to do access
control even though it's traditionally
happening at a layer that is um not as
uh standard so for PCI DSS specifically
um our uh card holder environment
actually is just a web page that syncs
to brain tree so um uh we don't have to
deal with that one specifically but uh
it's something that has been received
pretty favorably by our compliance folks
thank you signal
Angel could you elaborate on how you got
the management and application engineers
byy in for the changes described in your
talk what the objection did did they
eras and how did you address them um
thank you for asking this something that
I uh like talking about I think that a
lot of security is actually um being a
good salesman for your Solutions um
whenever you are presenting something
like this that has such a wide scope
it's crucial to make sure that there's
something in it for the stakeholders
Beyond just Security's goals and so a
lot of those things for us were around
um developer ease in productivity um
reducing the pain that Engineers were
feeling in trying to set up their own
TLS implementations or their own
authentication Stacks um uh better uh
performance benefits like I I discussed
um these were all things that other
infrastructure and product teams heard
about and um Wanted and so they were
very open to our original request uh and
then from there on it was all about uh
being a good Steward of an operation um
you know having really good operational
plans showing that we've done our
homework in terms of testing and um
really thinking like a uh infrastructure
engineer or an SRE instead of just a
security engineer you know security is
our ultimate goal uh but we need to make
sure that we are not you know burning
our um credibility with the rest of the
organization when when going for that so
uh there was a lot of time spent
thinking like you know forgetting about
all the security benefits right now how
am I going to make sure this isn't going
to take everything
down thank you uh microphone one your
question do all nodes uh have the whole
web files and what technology te do you
use to appli them to
NY um so yeah uh everything uh gets a
web file um it's yeah technology stack
is Json so basically there's a very
small shim that downloads this file from
S3 and then uh puts the relevant uh list
of allowed identities into the onvoy um
into an Envoy uh configuration file and
then uh Envoy is using it sort of
automatic updating uh SDS configuration
um to load that every few seconds uh so
that's how that synchronization works
okay thanks microphone two last question
have you considered using a pub sub push
of just the relevant metadata based on
the x509 identity of the the clients
that you're not also giving them all the
information about the entire map for the
entire network yeah um so you can rather
easily segment um what information
you're providing um and it's really just
a matter of sort of engineering time um
at the moment we have uh pretty wide
availability of that through other
service Discovery mechanisms so it
wasn't a priority for us um but it would
be relatively easy to have customized um
web file availability in particular
since everything is an IM Ro and uh
everything has its own IM Ro you can
simply make IM IM roll specific web
files in S3 and set up the permissions
to allow just those to access it so um
that actually wouldn't be that hard to
implement all right thank you um thank
you for answering all the questions
thank you uh yeah please give us some
Applause for his
[Applause]
[Music]
patience