Hello and good evening on day two of the chaos communication Camp 2023 Translated by {Yang}{Li} (ITKST56 course assignment at JYU.FI)]
it's late in the evening this is meleeway stage in case you're wondering
and the next talk is going to be about incident report responses
so if you're curious about how to even get there to have an incident response how you could prepare for an incident response and how you could support a new organization
uh, the incident response team in doing the job and trying to fix whatever broke
let's put it that way um we have the right talk for you
this is stories from the life of an incident from incident responders Harry and Chris
please a very warm Round of Applause [Applause]
so, good evening and thank you for joining us today um we will tell you a little bit of our
life as incident responders and I'm Chris I did my computer science
studies at the University of alang and Nuremberg I do this security stuff for
over 10 years now so my CV is a little bit longer at the moment I'm a detection
engineer before that I was a long time working in dfir so digital forensic incident
response in different organizations and
yeah I'm Harryr I studied electrical and computer engineering at RWTH
University and I played a lot of CTF and did some hacking stuff at chaos computer club RWTH
during my masters I worked at x41 dsac doing pen testing patch analysis
so I also have some kind of offensive security background on for around one year now I'm working at G data Advanced
analytics doing digital forensics and incident handling
first Christian will give you a short introduction and then he will tell you how a classical ransomware attack looks
like and in the second part of the talk I will tell you how the incident
responders work and what you can do in advance to make it go as smooth as possible and support the incident
response team so as Harryr told you I will probably
we'll talk about ransomware because the customers we usually have are small and
medium-sized businesses universities and hospitals and those are regularly
unfortunately regularly hit by um um
ransomware gangs the main reason for this and that's if you heard the last
talk um why they maybe not that responsive
and are not so interested in they just lack the resources so the manpower to do
uh proper security measurements to secure their systems especially in in erm
situations where you are for example in a hospital have medical devices
um which where you cannot simply install an AV on or even patch the system
because you lose the certification as a medical device then but also in in
companies manufacturing companies on the shop floor we're talking about systems
that have run times of 25 plus years so if you look back now 2023
we're talking about XP and older systems fun fact I was in a ransomware case and
Wannacry in 2017 when I got a call from from a person from the shop floor
asking me if we have a nt4 expert, um
that can tell us if WannaCry is affecting nt4 of course you don't need
to be a expert for NT-4 this one requires of course not affecting nt4
systems so due to the time uh slot we thought
memes are the best way to to tell you those stories and we have a lot of them
so in the first uh um section I tell you a little bit of how an attack Works
um there are a lot of different possibilities how you can describe and how to structure the how an attack works
there's the miter attack framework for example there was for example a talk Yesterday by Maker Salko
um here on the stage there's the original cyber kill chain from from Lockheed Martin you have
stuff from from companies like Mandy and their targeted the tech life cycle but
that's all in my opinion two two fine-grained it's that's the reason I
just take three simple steps yeah get a foothold in the door
look move play around and cash out those three uh I will just go over
so start with uh get a foot in the door so normally we
see three ways how attackers can can get into the environment in the ransomware
cases you have vulnerabilities in uh remote uh internet facing systems you
have the remote Services itself and you have malware
starting with the with the the vulnerabilities and um I just looked uh up the last four
years and maybe somebody remembers netscaler the the so-called Citrix
vulnerability in December 2019 um it was released mid of uh 2019 uh
December 2019 the first POC publicly available POC was in beginning of
January and the patch was available in middle of January so there was a round one week to one and a half weeks between
a public proof of concept for the vulnerability and uh patch for the vulnerability and what we saw
during 2020 a lot of companies patched but the patch didn't remove the the
compromise so they were already compromised and um yeah with it with the patch they
didn't remove the compromise so what we found what we could provable
see or proof evidence for uh was nine
month uh customer was breached after nine months using this this vulnerability
and we had other customers where we could see that the netscaler was affected after two years but we couldn't
prove that this this compromise was the reason for the actual ransomware case
and of course such vulnerabilities happen not that often
yeah so 2021 gave us uh hafnium exchange
vulnerability also a similar situation the patch
appeared as an out-of-band patch from Microsoft on a Tuesday evening 10 o'clock in German time
we saw during our uh incidents or the the assessments we did that
um the first exploit exploitation attempts were seen on Wednesday in the morning at
5:00 am so around seven eight hours later um I know one guy who could patch
because he was online when the patch was released otherwise Germany was unable to patch in
time and of course we can go on with 2021 proxy shell also
exchange vulnerability proxy nutshell also exchange vulnerability
we have uh in 2022 VMware Horizon the the virtual desktop infrastructure
from VMware just to name also open source stuff Zimbra a collaboration platform
including an email server uh has had a vulnerability actually the vulnerability
was in cpio from 2015 I think which led
to a compromise using via email so you send an email
with a cpio with a specially crafted archive file and you could drop a web
shell in one of the directories yeah you have of course 40 OS which is a
40 gate VPN and firewall operating system
and if you read the news we start at the beginning again
netscaler had some issues several weeks ago according to foxIT we have 1900
still unpatched net scalers worldwide how many patched
was netscale has exists that um have not been checked for compromise we
don't know of course so that will be a nice year probably
um so what can you can you do against this kind of of attack vector patch your systems is one thing as you
see this that doesn't lead to the the um or what you need to do afterwards in
such cases you need to check your systems for possible compromise
that is important to reduce this I highly suggest put your
uh Services behind some VPN so that only people who already have
connection to the VPN um can access your services or the services
they need and that would reduce the attack surface
at least to the VPN server so but I
of course we can also think about remote services without vulnerabilities
um there can be configuration mistakes so the admin does something wrong there can
be insecure default configurations like this um I don't know if you know it but the
local admins or the administrators on the Windows system are are
automatically in the remote desktop users group you know and so
we had several cases especially in the beginning of the pandemic when everybody moved from uh to the home offices and
they needed to put people fast in the position to to access their the assist
the internal systems again they just put a RDP server on the internet and hope for the best
um additionally if you put services on the internet of course brute forcing and
credential uh stuffing are attacks that are possible so brute forcing just trying the the
username and password combinations uh credential stuffing using already leaked
passwords or credentials from leaks you find on the internet
what you can do about this kind of of attack Vector is uh just as I said use
multi-factor Authentication and reduce the attack surface as in the
point with the vulnerabilities before by moving the services behind a VPN and
then use multi-factor authentication on VPN of course
the last Vector that we see normally that the attackers can get in the
network is malware we all know this about
those funny emails you get with the attachments
um include that have either Word documents
attached either zip files with with Visual Basic scripts javascripts and
what you can get isos you see a lot these days
um or what you can also have that you can have just a link inside the email and
you download the respective file from some some shady file sharing website
um what we saw over the last year was uh USB sticks again funnily
um I'm not sure if you have heard about raspberry Robin which is a malware that
warms via USB sticks um but I haven't seen it as a vector for
ransomware yet on my own but there are people who said that it's
an initial access broker for some of the ransomware gangs
so what can you do about this if you think the
you can of course ban simply some file extensions in your mail server or you
change the file Association types in your operating system meaning that you
don't open the JavaScript and Visual Basic script files using for example the
windows scripting host but open it with notepad and that will
of course some people will be
uh some people will think about what this this is then and ask the IT guys
but it's better than running the the script itself
one thing I I I don't like to to say it but keep your AV updated
um uh this is one thing keep it updated and read the logs
we see a lot of incidents where we see that the already
days or weeks before we you can could have seen that there's something going
on in your network yeah and if you see malware in your AV logs
then react to it just check it you don't know how long this AV this malware has
been on your system the thing is that
just because you're AV detected it now it might have been get received an
update for its signatures and the malware was active for days or weeks
before so when they are inside
then they usually look move and play around a little bit
so when they look around what they do
is they they enumerate AD they do Ports scan the you they search for
vulnerabilities they check uh what they how they can escalate
their privileges they try to find credentials
um Kerber roasting we heard in the talk before for example is this one thing
um they try to identify accounts you around
have running on your systems they can use they can get the credentials from and you reuse
and for that reason one of the most important things I think is that you
have a principle of least privileges in your environment so only what a account needs
you should be able to do you should use dedicated service
accounts for your services of course and just for your information
um service account is not an account that has a SVC underscore in front of it
and is otherwise a normal user account um there I exist educated service
accounts in Windows environments so use them use strong passwords I still
I today I can I know companies who still use eight
character long passwords um I think that's 20 minutes on a decent
graphics card today so use strong passwords length matters
12 plus characters is the minimum in my opinion
and don't reuse passwords especially on your
systems the local administrator especially in small and medium-sized businesses you see a lot that people use
the same password for all local administrators so if I have it on one
system I have the whole company don't reuse it
um yeah [Audience LoL]
so when they they move around in your network using either a password hashes they found valid credentials they they
discovered somewhere vulnerabilities are also used to to move around in your
environment they try to to establish persistence mechanisms here we heard
also in the talk before about the C2 channels command and control channels they use
um they install in some cases directly any desk team viewer and or other remote
control software or sometimes they also use tunneling softwares like ngrok or recently they
started using cloudflare G um and this is what you need to do is
prop have a proper Network segmentation and with the proper Network segmentation I don't talk about subnetting
subnetting means you just have different subnets you need to have a firewall between them
and you have need to have rules between them that
restrict access between your your subnets and one thing is especially important
please keep or use Network segmentation to
restrict the access to your backup and your Management Systems as far as
possible we see a lot especially as I said in the small and medium-sized businesses or in
the other organizations we we have as customers that
yeah they have they tell us in in in in in in workshops yeah we have
a network segmentation every building is one segment and on the question yeah you
can move between in a segment you can access everything yes you can and
between the the buildings you can have also a firewall and you cannot access
anything no you can access everything and also your your uh VMware Management
console oh yes yes we can so everybody can access it yes of course and that
doesn't work so
um when they play around they normally try
to to gain more privileges so privilege escalation is assisting local privilege escalations normally but also using
vulnerabilities um or misconfigurations insecure default
configurations um my personal favorites are Group
Policy preferences or passwords in group policy preferences
this is no longer possible since I think 2014 to put passwords in group policy
preferences however if you had your password stored in those preferences
before the patch in 2015 2014 then they're still there and yeah there
are AES encrypted but the encryption key is on the Microsoft website
so you can just download it and just take it and decrypt the keys
um then during that phase they also try to disable your security measures
the thing you can do is of course patch your system so you know try to get your
availabilities out of of this equation can try to configure your systems in a
secure way this is not always possible due to some shitty uh third-party software
and keep your AV um updated and please please as I said
already check the locks and act accordingly
so in the last phase they cash out that's when when they
start using a uh being your backup service so they copy
data from your your environment using um
file sharing platforms for example yeah Mega and set was was once the thing we
transfer we had already uh every every other file sharing platform you you can
think about is a possible way to exfiltrate data they also use their
their C2 communication channels so sometimes you they also they just use the the possibilities in any desk or in
in RDP clients or they use uh file transfer protocols
like um SS SFTP um
we saw for example in one case that they try to install filezilla on every machine they had access to
um because on the first one it didn't work on the second it didn't work on the third it didn't work yet because SFTP
was blocked uh outgoing and that is one of the things you can do to to prevent
exfiltration block at least
protocols you know that you don't need in your environment and proper Network segmentation of
course is a general thing so in the last step that's when they
start the encryption um they're running the ransomware or
normally they are have domain admins at that point so they can run it on all
domain connected systems they can also disable of course when they are domain admin they can disable
the AV before they they start to run somewhere ransomware's today disable services like
databases and such things so that they have the full power of the machine for
the uh for the encryption
um if you get lucky not how everything works perfectly because they use
group names and windows is especially picky when you have a non-english uh
windows installed for example in Germany the the group everybody is called yida
and we had cases where the ransomware didn't really work that well because they couldn't
change the permissions of the files first um
they use different encryption schemas normally they they come with the
asymmetric and the symmetric encryption type the asymmetrics or public key cryptography the public key comes with
the ransomware and is used to encrypt the symmetric keys they generate on in
your environment depending on the ransomware they they generate one key for each system or even one key for each
file it depends a little bit on the on the ransomware how it works but that's
the usual thing they use um I would never count on the the fact
that there are possibly maybe there could be
decryptable uh things um in in my opinion in my uh in my world
The ransomware Gangs have learned and used the standard Microsoft Windows or
some other publicly available libraries to to do the encryption
they executed by a remote tools like PSX Powershell or some use
gpos group policies to execute the ransomware on every
machines they they connected to the domain and what can you do about this no it's
it's hard but the the most important thing is have online backups offline
backup sorry thanks you you see you see off online backups
are not that are great but not that great offline backups is the most important this is the most important
thing so um don't have it connected to your environment
the the the USB disk on the system is not offline backup
um in my opinion if you see that something is is still encrypting
I I'm I'm always hesitant to say shut down the system because you can break
the encryption and maybe the file that is currently in under encryption or the files will never be decryptable if you
want to buy a decrypter um or gather the cryptos through some
discussions with the with the ransomware guys um if it's a VM just suspend it and
that's it and if everything is already encrypted
keep cool and call your incident responder
so now let's talk about incident response what happens when it's already too late and what can you do to support
your incident response team at first the things I'll say in this chapter are
for our company and how we work so other companies might work a little bit different than that
first for some reason incidents always come on Friday afternoon
so some customers think it is a good idea to try to solve a case by
themselves maybe until the end of the week and if they didn't solve it until the end of the week they call the
incident response team please don't do that it doesn't help your company and it
doesn't make your incident Response Team happy to have to work on the weekend and in addition the longer you wait with
calling the incident Response Team the longer the incident response will take and the more complicated forensics will
be because you have lock retention times while trying to do stuff by yourself
maybe you modify some of the systems and it becomes much more harder to do
precise forensics so what happens on our site when such a
new incident ticket arrives the first thing we do is team internal coordination
so we discussed do we have enough people do we have a person for each role in our
team we have three roads incident handling forensics analyst and Mayweather analyst
so first let's talk about incident handling incident Handler is responsible for all
the tasks that our customer facing and the first point is always get the customer out of that headless chicken
mode like we call it because when an incident comes at our customer site
everyone is like running around in so-called like headless chicken doing something but not doing anything helpful
so this is always the first task for the incident Handler Handler structure the customer do meetings and then do all the
relevant decisions leading to a secure emergency operation mode that means in
this case that you have working core infrastructure so working domain controller maybe a working email server
and whatever you need or whatever you define as very business critical systems
let's go a little bit more in detail probably the first measure will be to cut off the internet connection because
you just buy you a lot of time with doing that no matter how many back doors
the attackers placed in the network if you cut off the internet connection the attackers can't access their back doors
anymore and then you will start to rebuild your network you will Define everything in
your current infrastructure as red Network and then start building up a
green network with clean systems maybe you will start with some admin workstations so that the
administrators can work properly and then we go through a prioritized system list and build up the most
important systems this can be of course like I said domain controllers or email
servers but also whatever is important for your current company and for the
business this is of course in a hospital something very different from a small
business or maybe a university and please have such a prioritized list before your first incident because when
you start discussing in an incident which system is the most important then everyone in the company will tell you
something else and everyone will tell you that their system is the most important and has to be migrated and
checked and analyzed at first and this isn't really helpful
so during incident handling there might also be obstacles for example if you
have backups and you encrypted your backups this is great but you should think about where to
store that encryption key because when you store it in your password manager and the password
manager database is in a VM which then gets encrypted by the ransomware well
you have backups but you don't have your key to decrypt the backup so this
doesn't help you very much and this is not only two for password
this is also true for like everything else which you could need during an incident like contact lists or network
lists and so on so
working with third parties is also a task for the incident Handler this could be a data Protection Agency
or the customers customers because they also often yeah have some panic and have
questions which the customer maybe can't answer so this is also a talk a task for
the incident Handler and of course also working with law enforcement but there's
one thing to keep in mind Law Enforcement wants to do criminal investigation incident response wants to
bring you back to business as soon as possible this is not necessarily the same goal but it finally makes sense to
allow your incident responders to share all information with law enforcement because this is beneficial for everyone
um so why do I do forensics at all one could ask well the less precise the
forensic results are the more conservative the rebuild needs to be so if you don't know anything about the
attack you have to rebuild everything from scratch so let's say you have maybe two week or
backups and you know the initial access of the attackers was only one week ago
then you could use that two weeks old backups for rebuilding which is really
great but if you don't know how long the attackers were in the
network then it's really hard to to say
a second Point why you should do forensics is to estimate the impact
especially in an early um phase of an attack it is important to know that the attackers only compromise
some user devices or did they already gain domain administrative privileges
because then the incident handling is completely different and of course it is also relevant to
know which attack path the attack is used so you can Harden your network and
improve in the future how should incident how should forensics
be it should of course be correct it should be non-disruptive for the customer so they can concentrate on
rebuilding their infrastructure and it should be fast what means correct you want to have a
dedicated forensics analyst so you want to have incident response team which is
doing incident response in their day-to-day business and you want in your forensics teams one
person who can really concentrate on forensics and is not also doing the incident handling and the malware
analysis and so on because then you can't really concentrate on doing the technical work
it also should be non-disruptive as I said that means for our company we do remote triage collection we use the tool
videos adapter for it this is an official logo of velociraptor it's a really nice tool because we just send
our customers an exe file the customer just has to execute the exe file and
then it collects all the logs we need all the um yeah information we need
and absolute uploads it to our um to our infrastructure so we don't
need to drive to the customer and like start copying hard disks for a week for weeks because this doesn't help anyone
I should also said it should be fast so we have a lot of Automation and tooling
to make the work really fast automatically pushing our the uploaded files we want
to lose this infrastructure automatically doing reporting and so on I won't go in detail here because this
is enough for a single talk about it maybe maybe on Congress let's see
so what happens when we do manual forensic analysis
there are some questions you normally want to answer one is of course about the vectors you want to find all vectors
the attack is placed in the network of course you want to have General IUC
so indicators of compromise so maybe which IP addresses did the attackers use so you can block them in the firewall or
which tools did they use so you can scan over the whole it infrastructure and
find further compromised systems and if you have the iocs you can also cross-correlate between cases or show
them share them with law enforcement another important thing is of course
lateral movement because you can use lateral movement to build a timeline of the attack so the mythology here is to
look where the attackers came from and where the attackers went so if you have a system a and you see the attackers
came from system B then you analyze system B and go back in time and the same you could do forward in time until
you found yeah all relevant systems which were compromised and have a proper timeline of the case
and other common thing in the life of an incident responder is to fall in a rabbit hole
because it can be really really interesting to do forensics but yeah you need to
check what the customer really wants to know or what your incident Handler really wants to know
and not just analyze something and fall into the rabbit hole because this yeah
really doesn't help anyone here also the 80 20 rule applies so I'd say you get like 80 percent of
the relevant forensics results in 20 of the time and often this is enough for the incident Handler to do the right
decisions to rebuild the customer Network customer's networks as fast as possible
so let's go into reporting nobody really likes reporting
but producing fancy forensics results doing fancy technical stuff is worthless
if you can can't explain it to a manager nobody pays us to do fancy technical
stuff nobody understands but we are paid to help our customers and explain the
situation to them and to do the right decisions so yeah nobody likes reporting but it is a really really important
thing so after learning the report the
incident is over and all work is done right no not really because such a
report normally contains a lot of recommendations what to do
for the customers and as I already said incident response more or less stops
when the customer is in an emergency operation mode with a core infrastructure and the most critical
systems are working but there's still a lot to do and of course security is a process so
there's always something to improve to stay ahead of the attackers
so now let's go into a quick recap
how to protect against those kind of attacks we had this uh the topics just
as a recap patch your systems it's important keep them updated
vulnerabilities come and go that's one thing have a sane privilege
account management so use the correct user accounts for the correct for the
the right task use the second two-factor authentication
for remote services have a proper Network segmentation so
put firewalls and access rules between your networks check your AV logs regularly and act
accordingly and not really protecting your you against an attack
but keeping the the symptoms a little bit um not that that devastating have
offline backups so how to make incident response easier
for you for yourself and for the incident Response Team first thing is don't try to solve this stuff by
yourself just call the incident response team of your trust as soon as you know
you have an incident the second thing is logging policy because if you have a very short lock
retention for example and you only have locks for the last like two days then it
might be hard to find out what happened and if you don't lock stuff at all
especially in the cloud environments this is a thing where you need to look at then it might also be hard to get
proper forensic results then as I said have offline contact
lists because your main server is down your accounts are compromised so you can't use like your Microsoft teams or
whatever you use in your company you need an alternative way to communicate to your colleagues and or
employees then as I said prioritized asset list and network plans
and a disaster recovery plan in best case and please all have all that in a
secure place in best case on paper well nobody likes doing stuff with their
trees I know but that trees don't get ransomware right
so that's it from our site more or less normally this is a point where
you can ask us questions but we want to spin that around and give you some
questions maybe someone can guess the right answer
what do you think is the shortest time to domain admin we saw an incident between initial compromise and the
attackers gaining domain admin
yeah it's not it somebody said 17 seconds two minutes well it's not that
bad six minutes is is what we have seen in incident hmm
what do you think think is the shortest lock retention on a domain controller so how long
zero 10 hours a year well it is 2.5 minutes
what what do you think is the highest number of domain administrator accounts
we saw in an incident 200 nah 50.
64 with 120 people working in IT
what do you think is the longest dwell time so the longest the longest time between being initially compromised and
realizing that you are compromised 60 now it's it's not that bad I hit here
a lot of years well it is like around two years
so well that's really it from our side no time for questions thank you
if you
if you have questions we will be there outside waiting come and join us
thank you thank you thank you Harryr and Kris warmer applause thank you
End of subtitles:[Translated by {Yang}{Li} (ITKST56 course assignment at JYU.FI)]