Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1005 Thanks!
our next speaker is sge GIC sge has been
doing security research products and
services for the past 15 years more than
15 years since 2011 he's director and
script writer at positive hack days
Forum the largest cyber security event
in Eastern Europe SE has for instance
been working at Capers lab and positive
Technologies he's also a wisting
professor at Harbor space University in
Barcelona and leader of the scada
Strange Love industrial cyber security
research team today sge will talk about
how to hack software defined networks
and and keeping your senery while doing
it let's give a warm Round of Applause
for SEI
GIC hello hello good night uh let's
start to refresh on memories uh this is
a big honor for me to speak on the 35 35
C3 because my first talk here was on 29
C3 with SC strange G team and I think I
can skip this introduction thanks for
our host uh because
everything is here and um what I want to
say about me still I am very Russian uh
living in Abu Dhabi and do all this
stuff because I saw this uh album in the
uh airplane when I flight here so except
Bitcoin only so let's start uh to talk
about software defined networks what is
software defined networks in general and
is divine um in particular case is a
magic so according Gartner it will kill
imps it will replace soio Cisco and
Juniper device or Huawei if you prefer
Chinese but it's bad you know it's
according last new and it will solve all
your network problem because it's have
AI inside uh and it will magically
optimize network operations and do
everything including security uh so
because it's perfectly safe to implement
in quiet area networks efficiently and
security so okay sounds good what is
actually software defined
networks uh it's uh so
simple uh if you familiar with the
routers and switches like it's just
accept packets and send it somewhere
according routing tables or Mark uh
tables but s defined networks is quite
different and um when we Tred with uh
our team to understand how it's work our
first impression was like this uh we are
hackers we don't want to deal with this
[ __ ] but uh the only challenge we met
before you hug something you need to
actually activate it and make it works
which is why we uh start to understand
how software defined networks and sd1
works so what the main difference
between traditional landan and the sd1
in traditional L you have different
device which sof very uh specific papers
for instance switches or routing or
firewalling or uh Network Lo balancing
in case of sd1 you have just just a
server which runs operation system in
most cases is like Linux and on the top
of this uh operation system you have a
very specific uh models like CP which uh
do specific Network functions it can be
firewalling it can be rooting it can be
switching it can be uh Network load
balancing so you replace specific
devices which with one big server uh
which magically do everything with AI
and in the cloud sure so
uh uh in the sd1 uh we have several
layers uh so lower level is uh data plan
uh when actually you process packets and
decide how to go it in which way how to
firewall or drop it uh we have control
plan which uh manage different uh
routers different uh devices we have
management plan which can help to uh
apply
policies and orchestration plan because
it's serious things that should have
something which called
orchestration uh but on the technical
plan again uh we have
uh Hardware with operation system and
the layer which called Network function
virtualization what's Network function
virtualization the way to apply
different uh Network functions sorry uh
to the specific device it's very useful
for instance to
the network operators who will provide
you with the specific box and if you
want to activate any functions this
could be a web application firewall or
the sand boxes we just upload very
specific uh virtual machine it can be
Docker uh it can be uh
qvm image uh to your hardware and you
can start to use it because inside of
this uh box you already have all the
system infrastructure which process
packets and pass passes from one virtual
Network function to the
hour uh this help to organize things
like the service chaining uh so you can
distribute different network function
on the branch level in the cloud level
or on the uh HQ level for instance uh
things like the content filtering which
can be very heavy from performance
perspective uh can be distributed uh as
example on the branch level you can use
uh simple things like the antivirus to
process content on the HQ level you can
use more heavy things like sandboxing
and uh if antivirus or specific rules
see that this content is suspicious it's
forward it to HQ through the uh MP or
other uh VP VPN and next process is in
the HQ or uh you can also analyze it in
the cloud for the simple things like the
cloud threat intelligence where actually
your sd1 box will send m 5 hash to the
cloud and check is it good contact or
bad or send all files to the cloud to
double check it uh not bad and uh I
think that is why sd1 become more more
popular and you can see what the
military guys uh in US way decide to
switch to sd1 because of security cost
saving and all these benefits okay
security sounds very familiar for us and
uh we decide to obviously hack it uh I
think most of you have experience in
hacking of different network appliances
and you know that sometimes uh you need
to have complex things like a AR solder
kit or debugger gagul
Etc uh but not in this case because sd1
actually is not Appliance is a virtual
Appliance and to start hack it uh all
you need is to go to
the uh AWS or Azure uh and just activate
this virtual Appliance for I don't know
10 bucks per month and next step is get
root uh get rot on it uh it's a very
good talk uh presented on different
conferences including zero kns uh how to
haug Virtual Appliance and we use this
like a checklist for our research uh
because if you hack in Virtual Appliance
uh you already have access to this
system you can mount file system to the
uh other virtual ples you can uh grab uh
Etc Shadow uh you can find a lot of
different back doors uh just through the
static analysis uh but all the good
things starts from the
Google uh for instance uh to find uh
admin password for one of the sd1
appliance we uh just Google for GitHub
and found what uh most of the scripts
which use it to automate Visa plans use
uh username administrator and password
Versa one to three and actually we found
that this username and password is
hardcoded there because there is no way
to change it uh next step to
uh rooted is just to Google for old
vulnerabilities uh for instance in uh
Silver Peak we found that guys had
reported vulnerability in September 2015
and it's still working in March
2018 so Google works because Google f is
strong with his
one
um next things is grab it's always work
as a strings you know uh and uh using
grab and code code word password you can
find a lot of interesting thing like uh
hardcoded password in different location
in the um configuration files in the uh
database connection string and the
system locks because again it's virtual
Appliance and someone had deploy it
before you start to use it so in in the
logs where are a lot of Life interesting
information uh in the shadow file like
in the one of Cisco Appliance Etc Shadow
file which use Des encryption in
2018 uh you can do some
forensic uh because again if you get
virtual appls someone had deployed and
sometimes we're trying to hide with this
and you can see that uh cutting bash
history uh you can find that sumon Ram
uh scrub a WS shell script which
actually set up different password Etc
so if you somehow can recover with
script you can find a lot of interesting
information this kind uh um uh password
of admin users and uh you can see that
from this password you can find the hash
and next try to BR Force it's just was
my guess that uh maybe because uh versa
have uh password Versa one to3 maybe
other network Appliance like silver Peck
have similar password silver Pi one to
three and this guess was successful and
it's good because you cannot stop the
progress if you have experience with red
teaming in the Enterprise Network you
know that Cisco Cisco or Huawei with the
administrator and Huawei 123 it's quite
common password in this case it's more
complex thing sometimes very lead like
this tattery network stuff you know uh
but still if you uh did not get the
route uh with these simple steps again
it's virtual Appliance and you can
always PCH it so you can change for
instance uh hashes in the uh Etc Shadow
you can change boot script you can uh
change Remote Management configuration
the password and next boot in this
configuration and get root password to
uh do next step security
assessment uh so uh security assessment
U uh at the beginning we did in very you
know not scientific way we just hack all
the things uh but uh after all we did uh
some let's say uh scientific research
and we have an article I will give you a
link like sd1 thre landscape with the
stepbystep assess assessment what you ex
should hack to get maximum
results uh but uh let's like mix this
thing with funny hugs and the uh
scientific approach so from the system
engineer point of view sd1 have uh
Hardware which uh of the sh Hardware uh
operation system in most cases again of
the Shelf Linux
and uh different uh virtual Services uh
let's start from the operation system
because again again uh everything uh you
saw in the recent talk about uh uh BMC
and Remote Management interfaces related
to uh Hardware it still works here
unless it disabled by the vendor but
it's highly unlike it uh on operation
system uh we did very simple research we
just check the patch level of the all
components installed on this box and U
you can see that patch
level
uh
ridiculously old uh for instance the
oldest things we found it was a open SSL
Library which was released in May 2006
it's for network devices with security
functions but uh our guess was what but
they choose this library because this
Library too old to be vulnerable to a
heart bled
attack the um and sdan wins because uh
oldest uh Library open Library we found
in commercial product uh was in the S
before was in SE semantic wicc uh which
was released in 2007 so as the one is
like old school really next thing
related to uh operation system
configuration is sudu and it's actually
everywhere and actually everywhere for
management interfaces including web
services shell Etc and it's implemented
in in terrible way as you can see uh
triple W data have all uh ability to
execute all command and some scripts uh
just execute uh
uh with execute any command through the
sudo that is why if you have any small
uh vulnerabilities in the web interfaces
like in this case uh it's a common
injection you can execute command with
sudo so it's again
1990s uh next uh point it's a software
uh not system but software design point
of view uh in from software design there
are a lot of Open Source component which
Implement uh IPC um uh routing uh but
from management perspective uh we use uh
mostly HTTP and things around it
so uh let's analyze uh HTTP and web
management interfaces so in this case
it's not so old school like a system
site uh everything in based on nodejs
and JavaScript it's like very cool but
under the hood you can find hardcore mix
from the Pearl Java PHP whatever uh
which like I don't know looks like the
guys develop it last 10 years uh with
all this modern node GS stuff developers
confuse the client and the server uh
because you know javascripts on the both
sides and it's hard to understand where
is server client side I will show you
examples and uh there are a lot of U
simple things like a slow HTTP dos attx
which should be fixed for a long time
ago but still you can uh stop uh web
interface uh with few HTTP requests so
few examples about uh client
site uh Json csrf is everywhere uh so
almost no web interface
Implement protection from Cross site
request
forgery uh in proper way xss is
everywhere and this not a problem so as
a response from the product manager of
one vendor they told me that xss from uh
cross script info whereare application
is not issue because Chrome blocks
it uh it's uh just an example
uh of uh using accss on S such
appliances
um uh in uh this Appliance we can use a
combination of the xss and crite request
forgery to download and upload uh
certificate which use it to authenic
with the uh server which like control
plan with management server it's just uh
one HTTP uh request and uh obviously
there is no uh a response from the
vendor we just silently fix it so we
decide to publish it through the full
disclosure uh one uh example of the
perfect
authentication uh so in this case you
can see that it's client side JavaScript
which just send request to the server to
the uh login status function and if
uh user is lged is go to the uh request
page in our case go to the uh username
and password page so this 100% client
side no how it checks on the server side
just if you can change it second uh uh
example is just perfect so guy uh I
think he tried to Port the uh
authentication from the server side to
client side understand what is
Javascript which see still JavaScript
but it doesn't work in browser and he
just like commented and say if a
username is uh wh and password is this
when go home so authentication is passed
so this thing
so highlighted box so this all
authentication on this
box uh next funny things which are
related to is the one uh is
uh uh um about uh different uh privilege
escalation if you're already able to get
access to any of virtual appliances site
uh you can try to establish
connection uh from uh this Appliance to
other Appliance through the and uh Local
Host function so why can be interesting
uh because uh there are a lot of Open
Source component for the Remote
Management for instance like shell in
box which provide you like Shell through
the web interface or moon or solar which
like uh system management boxes we just
uh bend it to Local Host so you cannot
establish connection from outside
because it's listen to the Local Host
but if you're already on this box uh and
uh uh you can connect from this box to
uh Local Host
uh when this establish uh this
connection works and this works because
on each virtual on which appliance you
have a lot of virtual appliance which
still um uh listen to One IP addresses
if you have experience with the docker
for instance so all Dockers container
have own IP address but from the all
from all computer is actually connection
from Local Host so if you own any of the
virtual Appliance you can own uh next
all virtual machines installed here this
give out uh different interesting uh
ways to escalate privilege inside the
box for instance if you able to let me
switch to
the laser pointer can see it okay uh if
you can uh get access to traffic
processor for instance for the some uh
TCP stack vulnerabilities uh next you
can uh get access to management
application and this management
application in most cases uh have no any
traffic filtering and Trust to
management application of all virtual
Network functions which run on this
Appliance so you can uh do horizontal uh
privilege escalation or next jump down
to operation system level
and and next go to the management plan
upstairs to the um management
applies uh but uh it's it's really
boring to find uh web application
vulnerabilities in uh such uh big amount
of code which is why we just download
this code from the different network
appliers and drop it into the uh
interactive code analysis uh system in
this case we use uh positive technology
applic inspector and this help us to
find a lot of
vulnerabilities uh including uh uh such
funny things like um for instance uh
pulley patchet vulnerability in the the
Citrix sd1 which uh was PED in
2017 but it still works if you use not
get HTTP rquest but post HTP request so
we we patched it once again
uh in this in this case better uh also
it's Yoda style uh Yoda lesson style uh
vulnerability uh so it's uh obviously
patch traversal but it's just reminded
that attachments lead to the jealousy
and the shadow of G that is so if you
send attachment Shadow you can get
Shadow
file uh so this a full list of sorry a
full list of vulnerabilities we found uh
during just uh source code analysis
without you know brand
interaction uh next step is crypto
because it's security Appliance it
should Implement
cryptography uh and there are two things
is SSL TS and
typ uh in most cases uh sslt is used to
protect management interfaces between
the different appliances and
uh because it's automatic uh is use uh
different kind of automatic setup we
found that there are a lot of things
related to the unsecure configuration
for instance we can use
uh uh sslt without forward secrecy so if
you have access to the uh certificate
you can sniff off traffic in uh without
full man in the middle uh uh different
uh things related to the old uh cyers
suit like uh uh tripod death
orc4 for uh IPC we found that in most
cases we used uh very strange way to
select uh uh certificate or the uh preed
Keys uh which in most cases just
hardcoded so one example from real life
uh our example we will publish soon
uh from again Citrix net scaler uh these
appliances use uh Master Control uh note
protocol uh to communicate between the
orchestration plan control plan and the
uh data plan uh it's run runs on the TCP
uh
2156 and use a TS without uh forward
perfect secrecy and what interesting
certificate like at in the
home T user certificates and
account uh ww data have full access to
this certificate for some reason I don't
know okay it should maybe read it but
why to write and uh what interesting all
sd1 appliances we able to find during
our security assessment we use same key
uh pair which located in apply liance AG
PM key so all sd1 appliances in the
world used to protect communication
between and management and J uh same key
pair so if you know this key pair and
it's obviously you can just uh uh cut it
from the file system you can uh
passively or actively Snee traffic do
manage the medal uh spoof uh management
Appliance and if you this device have
any web application vulnerability you
can overwrite it I don't know why but
maybe if in next turn way will change
this certificate you can download all
and do man in the middle
again uh interesting uh stuff uh we
found we uh run some test from the uh uh
DS Dosa tax and found that uh uh
suata uh which US it in uh sd1 Appliance
are the IDS uh is vulnerable to Reg do
so it's uh old story uh some regular
expression can uh spend a lot of CPU
time if you send specific
uh uh queries uh it was fix it in the
default suata kit but still working some
Modern sd1 solution and uh for sure if
you do some fuzing uh it's always work
and give you some fun unfortunately we
cannot present reverse engineering part
because uh most of such uh sd1 solution
we have restriction uh in the licensing
agreement to the uh reverse engineering
but just for fun uh I think that some of
Engineers way uh also love Star Wars and
have Marvel that is why in calization
functions CS Marvel
sucks so uh just an overview of detected
vulnerabilities uh so green is good or
bad so good for vender bad for us we are
unable to detect it uh but you can see
that most of classes of vulnerabilities
like hard codes broken Access Control uh
old uh product or Linux components or
third party components where in most of
uh such system you can find is most of
such system so just select any sd1 and
make
shot interested thing in sd1 is zero
touch deployment so the Fe it's a very
cool feature uh for instance let's
imagine that you have a branch office
you need to deploy branch office uh with
sd1 it's absolutely not necessary to go
where or establish tet or SSH connection
and try to upload configuration all you
need is just to ship this
device uh note with unique ID set up it
through the cloud console and ship this
device to remote location uh this device
will automatically connect to any
internet it see around connect to the
Cloud Server the node configuration and
start
operation
uh so for uh for example how it works in
the Citrix system uh we have V applies
which uh ship it to the office it's
first tried to establish connection with
the surrounded Appliance if no it's Tred
to go to the uh zero touch deployment
Service uh present uh own ID and next
this service will provide all
configuration which you upload through
the uh your sd1 Center uh so from
security perspective this scheme uh
looks terrible why because this sd1
Cloud deployment server should be
friendly but any attacker not if you no
ID if you can Brute Force this ID you
can pretend to be this device uh if you
have any any weaknesses in
implementation of this management
servers you can own all devices which
deployed from the service and as you can
see uh even Cisco which is like the best
uh device from security perspective we
found on this
uh uh product line let's say I have
enough uh vulnerabilities uh it's and
this you can see that zero touch uh Pro
uh provision in common injection
vulnerability so it's Cloud Server which
uh to which all Network device should
connect sometimes but also uh we found
very funny things related to the
distribution of this device uh because
as I told you uh in the beginning uh
most of uh such device can be uh
activated as a cloud Appliance through
the aw us or the uh other cloud services
and we found that most of default
default images use old version uh with
nonn vulnerabilities so you go to AWS
you trust the vendor you activate this
system and you receive attack uh because
where there is no vulnerability it's uh
remind me and those story I'm really old
man sorry uh you know Cod red nimda this
kind of Kido this kind of worms and this
real disaster when uh you just install
fresh uh Windows 2000 which have uh
internet information server by default
and just connect it into internet to
download patches you receive new
infection and need to reinstall it from
the beginning so these things look
similar for me but is
much worse because in this case is a
security network device installed on the
perimeter of your
network uh this overview of upto-date
Statistics you can see that um uh very
few vendors actually no one of vendors
uh uh have upto-date uh version in the
deployed on AWS or Internet so I think
it's a use of the
force but uh it's also a very
interesting part of story as a security
researchers we always work in the
responsible disclosure way and uh as you
can see according this article uh some
vendors they also understand that uh
responsible disclosure is very important
to communicate with Community to fix uh
uh issues and way even have product uh
security incident response team in place
great uh but when we try to uh submit
vulnerability report to this vendor we
unable to find the email of this uh
product security incident response team
so there is no
pool uh we tried to Google for it a
different way no luck but we found that
uh guys who did similar research before
when they found a great way sent email
to CEO of this company uh unfortunately
my googly f is not good enough but I am
unable to find the mail but I found this
guy and the link it
in and uh he he he answered it in few
minutes actually and uh put me in
contact so if you trying to deal with
sdan vulnerability uh reporting just
do his
way so uh uh we uh prepared this table
about different vendors uh how they
communicate with researchers uh and you
can see that um actually Cisco Citrix
and uh V Cloud which actually whenw were
no not bad but all the rest it's it's
just a beginning uh this is my favorite
mail uh from the one vendor uh when we
sent notification we start to ask me why
we send this email from the
Gmail do we have official ID what they
mean I need to present my passport or
whatever to submit
vulnerabilities uh but the uh funnest
fun uh funnest things here uh this
vender wrote that where devices not
generic web service which have full
access uh full internet explosure so uh
after reading this Emil I go sleep and
during the night someone told
me
uh so uh to understand Fred Escape of
sd1 uh we built the uh bunch of script
which works on the on the top of the
different search engines like cus Shodan
Google and uh use also nmap L scripting
platform to fingerprint uh different sd1
solution uh we have
a article published it uh as sd1
Internet uh census and uh which uh and
also some tools which can generate
beautiful Maps if you want to present on
the house communication Congress because
in our case it's useless
um and uh what we found that there are
not so many sd1 devices uh yet in the
internet about 3,000 management
interface which uh contains no
vulnerabilities and you can know it in a
few minutes uh and uh uh also we uh
built some kind of vulnerability
assessment Tool uh which uh help you to
find no vulnerabilities in this sd1
device deves this like example of for
open SSH patch level as you can see that
like a uh some CVS uh from 2010 from
2014
Etc uh this open source you can find it
on the GitHub uh we have two version uh
one sd1 Harvester which uh use uh Google
uh shown and uh Tus To collect
information in all the all the all
internets and also sd1 infiltrated
infiltrator sorry which is bunch of the
uh network uh nmap script engine uh
scripts and you can use it during
penetration testing so it's not
necessary to be connected to the
internet you can just uh uh use it uh
inside the
network uh when we uh did this research
uh we also found uh in interesting
article from Silence uh about dark web
Market when the bad guys uh
sell usernames and password to different
network appliances uh let's say
Enterprise govern Network appliances and
we found that there is there are some IP
addresses which we found during our
assessment internet harvesting in this
list uh and in our experience there is
no such things as a lck we tried to find
why uh such appliances can be so easily
hacked and obviously the uh default
password which hardcoded sometimes but
uh never change it uh is us it was us it
on this appliances uh we try to reach
vendors and say guys uh maybe it's bad
idea to uh use
uh for instance hardcoded SNMP not
hardcoded by default Community uh like
uh public and uh again public for read
right uh but they told us that SNMP is
off by default but still simple shown
search show that more than 200 uh users
of this sd1 we enable this uh SNMP
service and still use default password
so uh uh we have a lot of uh tools which
publish it in our GitHub uh and uh
please contribute uh there are a lot of
things to do with the new fingerprints
for sd1 Harvester and infiltrator with
sd1 fret landscape
description uh with new vulnerabilities
and also it's like special uh
uh CCC release we start to publish metas
sploit models for the sd1 so
uh uh we have public vulnerability
description you can create own models
for it uh and
conclusions
so from my uh in perspective how sd1 uh
development like cycle works so someone
come with brilliant okay let's build sd1
because Garner told us that it's like
brilliant I have in AI in the cloud uh
so what we have to do we can download
bunch of open sources I put off together
set up default routing things and uh
after uh use it at the sdan so sd1 is a
bunch of Open Source uh which not bad
but still uh you need care about it and
unit install patches configure in the
proper way and
maintain
uh these complex products uh have
problems with patch management uh have a
lot of management interface like uh
machine to machine and uh also uh human
to machion interfaces have a lot of weak
defaults like password hardcoded
certificate uh P keys for the IPS
uh and many vendors unfortunately have
uh issues with the patching responsible
disclosure uh and this in the cloud so
if you decide to switch your network to
the sd1 hack it before byy all you will
fail so uh thank you for your attention
uh I want to ask you to give big hand to
the sd1 New Hope team to Dennis Maxim
Nikolai Nikita Alec and Antony uh who
did most of the things here just a like
a frontman of this group thank you so
much
guys Applause for SE and his
team we now have about like 50 minutes
uh for questions and you know the rules
please please move to the microphones
over
there any
questions yep aside from kind of generic
things that any random Linux that we can
have on web interface and SSH have you
looked at any specific uh sd1 security
problems like with the encapsulation of
of tunnels or some stuff like
that uh so uh
uh sd1 you know there is no like
technology like sd1 so for for SDM where
is kind of protocol which more or less
reuse it in different vendor uh in
different solution and sd1 every vendor
Implement things in own way uh as an
example uh is like uh uh citric
management protocol Etc uh one thing we
did here
uh but we we didn't publish it is for
the
uh uh virtualization because again this
vnf story uh very interesting because if
you have have vulnerability in any
virtual function next you can get uh
access to our uh virtual machine but
again problem here uh that there is no
standard and different vendors call V
and NF uh so Implement vnf in different
way it can be qm it can be just a script
which we upload to where Appliance so
you're saying because everyone is
writing their own code there are a lot
of bugs to find for people who put the
work in yeah okay okay and you can just
uh like uh it's not necessary to try to
buy things through the a to H to hack it
you can just go to AWS and activate it
for free okay Mike one please um my will
be simple there was a lot of vendors
that you were look in um what about
Juniper is the
one
um we don't know
yet looking forward for contrail
investigation thank you good time and
yes please okay say thanks for the talk
uh you mentioned a lot of of these um
virtual machines were like running in
Harden Linux or something how do you
what do are they running like G and uh
what hasch levels are they usually on
these kernels it's like always 2.6 or
something like
that uh not not always 2.6 it was like
the worst example so
uh uh some some of us like you know new
so
it's not necessary uh again for Network
function virtualization uh some vendors
uh they call vnf just a bunch of script
we download and uh change configuration
of default sensor and this case we can
use uh very old to uh kernel uh are way
more use more recent
version is
is sorry yeah yeah that's okay um and
the hardening how is it usually the
hard like uh in most cases no way so no
upor armor or things like this no J
security or selino nothing okay thanks
all right um your
question um have you looked into Cisco
maraki uh no
the um for Cisco we sorry it's it's
night
let me find the
list D for Cisco we did our exercise
with
the VIP
Tower
okay all right last question over there
please uh just advice maybe you drop the
slide in with uh the 9s call they want
the hardcore password back sorry um
because
there there were so many uh hardcoded
passwords everywhere maybe you should
just drop a slide in the '90s called
they want the upod passwords or lock
back I don't know so it it's public
things so was that a
question
um just
recommendation all right we have a few
minutes left but if there are no
questions left then I would call it a
day um so another warm Round of Applause
um
for and
team
oh