Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2733 Thanks!
[Music]
okay so the next talk is going to be by
Teo longin also known as teimo login
he's a security consultant and a
researcher he will be talking about a
new technique called SMTP smuggling for
spoofing emails and exploit one of the
most widely used services on the
internet thank you give it up for
[Applause]
Timo thank you thank you for the
introduction so first of all a big fat
sorry for me personally and from Sault
for this absolute dumpster fire of a
vulnerability disclosure so specifically
sorry to Vita and Victor for having to
well fix post fix and to all the CIS
admins Worldwide having to apply these
fixes during Christmas
holidays and also
however yeah also however big thanks to
V and Victor for their commitment and
also big thanks to the community for
pushing this issue for releasing tools
and so on and for
okay
[Applause]
and for everyone that has no idea what
is going on let me catch you up to speed
so about a year ago I was just finish
finishing up some DNS research and I was
looking for a new research Target when I
probably found the easiest way to hack a
company and all that just with one
simple Google
search
and it might sound stupid but this led
me into the direction of something that
I already knew but I never really
realized and that is that fishing is
still the number one initial access
Vector into a company and then it hit me
like why don't I look at SMTP the simple
mail transfer protocol which is used to
send like billions of emails each day
across the globe for like the past 40
years so my journey went on from DNS to
MTP and today I'm introducing SMTP
smuggling a novel technique for spoing
emails who am I my name is Timo longin I
work as a security consultant at se SE
consult and a day I do penetration tests
and at night I do vulnerability
research and for like the past three
years I've been looking at lots of DNS
vulnerabilities and I've published lots
of blog posts and
tools and yeah I had to move on and the
last time time someone from Sault had a
talk at the
CCC it was about
dildos and I know that I have to
disappoint at least some of you but this
talk is not about penetrating the human
life
form however it is about penetrating
SMTP protocol barriers so to see how
this works well we must first understand
how sending emails in general works so
here we have a pretty simple email
infrastructure we have a mail user agent
for example Thunderbird and Thunderbird
like wants to send an email via user
outlook.com for example so now if we
want to send an email that way we must
first authenticate against the Outlook
mail transfer agent or outbound SMTP
server and once we've authenticated
we're now allowed to send an email as
user
outlook.com and and only as user
outlook.com and this email then gets
transferred over to the receiver inbound
SMTP server and this inbound SMTP server
is now checking the email for
authenticity and the most popular way of
doing this is via SPF so the receiver
inbound server is basically getting the
SPF record via DNS for outlook.com and
then it sees okay these IP addresses and
IP ranges are allowed to send emails for
outlook.com and since in this case the
legitimate Outlook SMTP server or
outbound SMTP server sent the email well
the inbound receiver accepts it and now
of course there is well a very
interesting question and the question is
well is is it possible for an attacker
to send an email for example from admin
at outlook.com or from a spoofed email
address and this is what we're going to
look at today and that's more more or
less the goal of This
research and I don't know if you've
realized but the colors of these servers
and I cannot unced but they really
remind me of SpongeBob and
Patrick and therefore we're going to
refer to them as
such so the general goal of This
research and the premise of this re
research was like finding a way to spoof
emails and I was like okay why not take
vulnerabilities from other text based
product OLS like
HTTP and put them into
SMTP and there was just one HTTP
vulnerability that really fit the bill
and that is HTTP request
smuggling and the thing is we again see
we have SpongeBob and Patrick but in
this case in the HTTP world so what
happens here is SpongBob is getting a
request a post request over the
internet and the interesting thing about
this post request is that it has two
headers
specifying how to handle the post
request data it has a Content length
header specifying 43 bytes and it also
has a transfer encoding header and now
SpongeBob has to decide okay well how do
I handle the post data now and SpongeBob
decides to use the content length and
since the content length is 43 bytes
well all the red highlighted data is
forwarded to p Patrick and Patrick is
not like okay I have no idea what to do
so am I interpreting the content length
or the transfer encoding but now Patrick
is interpreting the transfer encoding
and now we have an interpretation
difference SpongeBob uses the content
length Patrick is using the transfer
encoding and since the transfer encoding
specifies chunk and the first chunk is
zero well the rest of the data that
SpongeBob transferred is now interpreted
as a second request which which now
basically means that SpongeBob sees one
request and Patrick sees two requests
and the second request can have like or
can Target an arbitrary path like for
example admin which in this example is
only accessible
internally which is of course a problem
and I was like okay why not take these
interpretation differences and put them
into SMTP and to understand or to get
close to understanding how this works at
least we must first look at the SMTP
protocol itself so SMTP basically looks
like this we have two main components we
have in red SMTP commands and we have in
blue the message data and to send a
message now we must first send the SMTP
commands so we must first introduce
ourselves we must specify a sender
address one or more recipient addresses
and then we send a data command to tell
the receiving SMTP server to essentially
yeah now receive the message data and
then we send message data we specify a
from address again we specify a two
address we specify a subject and then
comes the message body and at some point
when we now want to stop transferring
the message data we must send something
that is called an end of data sequence
which is a carage return line feed.
carriage return line
feed and now was like well maybe we can
use that to somehow confuse SMTP server
implementations and the idea was
basically that we have SpongeBob again
and we submit an email to
SpongeBob and this email contains
something very weird it is something
that looks like an end of data sequence
but it's not an end of data sequence
because it's not RFC conform but like
someone could mistake it for an end of
data sequence so SpongeBob looks at this
and it's like okay this is not the thing
from the RFC I'm not going to interpret
this as n of data sequence so the the
next or the message data that follows to
this is still like message data until
the actual end of data sequence comes
and now SpongeBob sends this over to
Patrick and now Patrick is like uh yeah
I don't really care about rfc's so what
I do is well I interpret this fake end
of data sequence as an actual end of
data sequence
and the problem here is that everything
following this end of data sequence is
now interpreted as SMTP
commands and what we can do now as an
attacker is we can specify SMTP commands
that send another email so even though
SpongeBob only saw one big email Patrick
is now seeing two smaller emails and the
problem is that the second email can
have like arbitrary SMTP commands
meaning it could come from admin at
Outlook look.com or
whatever so yeah that's the theory at
least and to see if this really
works I first of all checked out some
SMTP server implementations by simply
connecting to some servers via tnet or
via
netcat and when I did that and I sent
the data command first of all it looked
like very RFC conform so basically when
you send the data command the server
asks you to stop this data stream via
carryer turn line feed. carer turn Line
Feed but then I also found some servers
that just say well just send me a DOT on
Al line by itself and the thing with
this do on Al line by itself is that
it's very much dependent on the
operating system that you're on meaning
on Windows this could be a carriage
return line feed. carriage return line
feed on Linux this could be a line feed
doline
feed and at that point I I thought I was
on to something so
I checked something out and the first
thing I tried is like using these kinds
of line feeds Carriage returns dots and
trying to like stuff them through
SpongeBob so I wrote an SMTP analysis
client which basically sends emails
containing fake end of data sequences
for example line feed. linefeed and I
sent them through various kinds of SMTP
software this includes email services
like Gmail like Outlook like GMX but Al
email software like send mail like
postfix XM Microsoft uh Exchange Server
and so on and now on the inbound side on
the inbound receiver side I checked
which fake sequences actually go through
so can I send a line feed. line feed
through an outbound server and in most
cases it doesn't work like a line feed.
line feed very often just gets filtered
or removed from the initial email
but in some
cases it just goes through without a
change and this was the case for GMX so
basically I send an email from GMX to my
analysis server and the line feed.
carriage return linefeed sequence did
not get filtered whatsoever so now what
I did is I created a proof of concept
here where first of all we s an email
then we have this fake end of data
sequence and then we have SMTP commands
and data for a second email so now if
this proof of concept works we should
receive two emails on our receiver end
and I sent all of this to Gmail and
Gmail was like yeah this is not an end
of data sequence and you can see that
because everything after this end of
data sequence is still interpreted as
message
data but in some cases it actually works
and this was the first case of
successful SMTP smuggling so from GMX to
FastMail and we can see that it works
because well we have two emails we have
first of all user at gmx.net and
secondly an email from admin gmx.net
that passes SPF and Demar checks as well
because it comes from the actual server
from
[Applause]
GMX and I was like okay this is super
sick but I was thinking so we have this
second email and everything can be in
there like the sender address can be
anything so why not try other domains
that point to the server of
GMX and then I analyzed the SPF record
and I realized well there we have the
GMX SPF record but this is very similar
to the web de SP PF record which is also
very similar to the one of
yonos and the thing is that we can now
spoof like 1.35 million domains with
this so if you don't know yonos they're
like a super big hosting provider and
they also provide emailing services and
yeah they have like all these 1.35
million domains pointing here so yeah I
had of course had to try this so here's
an email from admin at
webd and now of course there's a
question of does this work everywhere
because as I said this only works with
servers that interpret well non RFC
conform end of data sequences like line
feed. carry to return
linefeed so we can smuggle or spoof
these 1.35 million domains to well some
servers so essentially we have 1.4
million postfix instances and like 150k
sand mail instances and this works well
because they interpret car interpret
Line feed. Car linefeed confusing myself
here as n of data
sequence and I was like okay this is
pretty severe but there is more so I
looked at outlook.com as well and
Outlook gave me a very weird uh error
message because they were like bare line
feeds are illegal and I was like oh damn
they they know what I'm doing
here
and therefore I analyzed it like okay
you're not going to scare me with that
one so then I found out well line feed.
carriage Line Feed is also possible
here yeah so let's have some
fun the thing is however at that point I
wasn't really sure if I'm going insane
or if this actually works so I needed
some kind of Sanity check and
therefore I sent an email from
admin@outlook.com to some
co-workers and the idea behind this is
if they react to this it works otherwise
I'm insane
so
yeah and maybe now for a more
International example because this is
very Austrian I guess I don't know so
yeah
I guess you also understand this if you
don't speak
German and now you might be like okay we
can send text based spoofed emails but
you're not going to like fool anyone
with this but thing is like we can
smuggle basically anything HTML included
so here I sent an email a fishing email
an an unusual signin activity email from
no reply at outlook.com from the actual
outlook.com server was to myself and it
just went
through yeah but again what's what's
possible with this and I was wondering
so we can spoof outlook.com but what
what else can we do with this so I
looked at the SPF record and it was well
weird in terms of weirdly familiar
because this is not only the SPF record
of outlook.com but also the SPF record
of like millions of other
domains because this is actually like
the SPF record from exchange online
because outlook.com sends its emails
well Across The Exchange online
infrastructure so yeah now we can
basically spoof them all so I again had
to test this
one
but the thing is here this only worked
on a technical level so I didn't get the
race so what's the impact essentially we
can spoof from millions of domains but
who accepts these emails again post fix
and send mail however only postfix and
sand mail instances they do not support
the bat command and the bat command is
like an alternative to the data command
and if that is supported it doesn't
work so that was outbound SMTP smuggling
so that is like one part of it but then
there is also inbound SMTP smuggling and
with outbound SMTP smuggling the issue
was basically at the outbound server
because the outbound server basically
failed to filter or insufficiently
filtered like fake end of data sequences
like line feed. Carri feed so now with
Microsoft and GMX and then there's also
inbound SMTP smuggling and this happens
when there is just an inbound server
that interprets such a wild end of data
sequence that the outbound server would
never even get the idea of filtering
that and I found exactly that with Cisco
secure of course
email Cloud
Gateway so because what they do is they
clean the message and every bare Line
Feed and every bare Carri turn character
now gets replaced with a carriage return
line feed so essentially a carriage
return. carriage return gets interpreted
as end of data
[Applause]
sequence and the issue of course is now
that this Carri return. carar return
does not get filtered at all basically
so there are of course some service that
filter this but we have again exchange
online iCloud sent mail postfix and so
many others that just let this through
and I guess it kind of makes sense to
let this through because I wouldn't get
the idea of filtering that
either so I had to test this out
again and unfortunately or well
technically it worked again but I did
not receive any Apple devices not that I
wanted that anyways but
yeah what's the impact here so
essentially we can spoof from even more
domains as before like we have exchange
online we have iCloud we have all post
fix and
sendmail and we can spoof to companies
that can actually afford this stuff so
like pretty big companies here this also
includes like 40 other uh 40K other
domains and that that's only the ones
that are hosted in a cloud so they're
also on Prime but I wasn't able to
enumerate
them yeah and now to the part that's
probably interesting to you or to the
most at least so the responsible
disclosure so essentially on a purely
technical level regarding inbound and
outbound servers who's at fault here so
essentially we have inbound servers that
are never supposed or not supposed to
interpret anything else as end of data
sequence then a carriage return line
feed. carriage return line feed at least
by the RFC and then there are outbound
SMTP servers that are not supposed to
send Carriage returns and line feeds
independent of each other so yeah
basically everyone's at fault so we sent
that to GMX and they were like super
thankful and they were like okay we're
going to fix this right away 10 days
later they fixed this they said okay can
we recheck this we recheck this they
actually fixed this they paid us a small
Bounty they even put us in their back
Bounty Hall of Fame and all all on like
that was a 10 out of 10
[Applause]
experience I'm going to send them this
clip of you clapping thank you and yeah
it really felt like we should have paid
them the back bound you because yeah but
the thing is from here it just goes
downhill
so we have Microsoft and Microsoft was
like yeah this is like a moderate risk
vulnerability
because I don't know I guess they have
bigger fish to fry but yeah
anyways so we sent it to them and like 3
months later they closed the case and
said yeah all right it's fixed no bnie
whatsoever so okay but the thing is at
that point well I already got what I
want wanted and that's an email from
admin
microsoft.com and now to Cisco well
Cisco was like okay this is not a
vulnerability this is a documented and
configurable
[Applause]
feature and we were like okay that's a
weird feature because
like we can spoof some emails with that
so yeah we said okay at least like tell
your customers about this feature that
it's maybe not the best default feature
but yeah
anyways yeah they said
no
yeah so then we were like okay we got
this whole big SMTP smuggling issue
Cisco doesn't want to fix anything
Microsoft at that point was was
vulnerable and we take this case to C CC
and C CC for all of you that don't know
that is the C coordination Center and
they basically handle all these big
vulnerabilities so that could have a
worldwide impact and we sent this to
them and they actually accepted the case
and now you're like in this Vince portal
this which is basically a big glorified
chat room with 14 vendors we have like
sent mail Microsoft Cisco uh Google in
there and yeah so now you can start
talking about the vulnerability and the
first conversation was basically with
Cisco still saying that they're not
vulnerable you know yeah it's not a bug
it's a feature and yeah essentially they
were still like it's not a vulnerability
and then sis uh CC was like yeah this is
not a vulnerability for some reason okay
so what about the general SMTP smuggling
problem so what about postfix and sent
mail interpreting non RFC conform and of
data
sequences so first of all sent mail was
in this Vince portal to begin with so
from the start they got all the puc's
all the messages all yeah everything and
they were like they didn't say anything
and then what about postfix because
they're 10 times larger essentially C CC
was like first of all I don't know why
they didn't add them to the case
directly but they sent them an email or
contacted them VI email but forgot to
mention SMTP smuggling so now we're here
thinking that they told po so we thought
that they told postfix sent mail
apparently didn't care C CC sides with
Cisco regarding this feature so now
we're like okay CIS uh Cisco yeah is
definitely vulnerable we've confirmed
that so we would like to publish this
any blog post and that's where all the
problem
starts so we said okay okay we would
like to publish this any blog post and
we would warn Cisco customers about the
feature and CC is like go ahead so just
send us the link to the blog post once
this is
published and we were like yeah we go
ahead with that and now basically we
opened Pandora's Box because now 1.6
million post M uh post fix and send mail
instances on the internet are vulnerable
meaning now if there is another case of
outbound SMTP smuggling you can smuggle
to post fix and send mail
still does that mean second consult is
not at fault at all of course second
consult is at fault because we published
this blog post thinking that the impact
of all of this is way lower than it
actually is based on our great
conversation with CTC and Vince in
general and also if we would have just
double checked that with postfix that
this is really not not an issue none of
this would have happened because postfix
is very adamant about the fact that this
is actually an
issue so yeah in conclusion what does
this mean please fix your postfix
service you can find more information
about that on the postfix website also
please fix your Cisco service you can
find more information about that on the
second consult blog
post
[Applause]
in the end however there is some kind of
Sil Silver Lining at least because now
postfix is directly in the Vince case
and they've already made some great
contributions so we're not putting our
head in the sand and we're actively
somehow trying to close Pandora's Box
again and what does that mean
essentially more research on like SMTP
smuggling I mean as straty before to
like look at this more but yeah this
didn't really end well and more blog
posts more research first first and
foremost um more vendors in the Vince
case so we're trying to get everything
fixed right now and again we are very
sorry that this happened in the first
place and
finally something that probably all of
you know anyways do not trust emails
blindly especially not
now thank you
[Applause]
nice thank you let's go with the Q&A um
please go ahead hello I one small
question the C CC stuff was this
communicated via email and could you not
have sent the mail at Cisco
CDO um yeah I mean we could have done
that and created some kind of love
triangle between Bill Gates Elon Musk
and someone else I guess
but yeah no we didn't see this email
unfortunately like we saw it after that
yeah but then it was kind of too late
unfortunately uh hi thank you appreciate
the work um can you maybe tell us a
little bit about the timeline like when
did you first discover this like how
fresh this was first discovered so I
started the research in June this year
so basically we I had a research project
at Sault for like 10 days and I was like
okay these 10 days they're not going to
be enough so I started 10 days earlier
and then I like found SMTP smuggling on
day six and yeah then it just goes on
from telling this GMX Outlook or
Microsoft and Cisco because they had
these severe cases of SMTP smuggling
meaning outbound and inbound and then we
went ahead and told this to cery because
there seems to be a general issue with
SMTP smuggling worldwide and timeline
wise I don't have it here right now but
it is in the second Sal blog post and
yeah but is is there a specific date
that you want to know no no just I'll
check the blog post
thanks
one hello um thank you for the good talk
here um I wanted to ask is there any
evidence or any suggestions that this
has been used already um because this
seems like a well pretty easy exploit to
use so I might imagine it has been used
already yeah I've mean I mean I received
some comments on on my Reddit post and
it was like yeah this is super old but
thing
is medium severity yeah probably no I I
don't know I don't know really I I
looked at lots of research but nothing
really said otherwise so that's why I
called it the novel vulnerability so I
don't
know
[Music]
la