Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2627 Thanks!
[Music]
ladies and gentlemen welcome to the talk
at the Millway stage about how to use AI
in penetration testing Jacob and Richard
are going to tell you all about this
interesting
topic
you good morning and Welcome to our
presentation we have mastering the maze
um there is like a small company icon in
some of the corners um we got like a
training deal um so please don't mind
that um yeah let's first talk about
motivation and our agenda so how did we
came up with this topic um both Richard
and I wanted to learn a bit about Ai and
security is a nice topic so um we
thought about like um combining those
two and yeah what did we want to do
we wanted a bit learning by doing and
tried to create a neural network that um
could find an optimal action in a pen
testing
environment um yeah so um today in this
talk you will hear me talking a bit
about um deep Q learning how it works
and yeah how to um how the um best solu
solution is found and then Richard will
explain how this deep Q Network can be
used in penetration
testing so um first uh we started using
some tutorials and there are actually a
lot of tutorials for deep Q agents that
are related to gaming so um we have had
a look at these and yeah started with
some of
the um tutorials
and yeah
these sorry um so the uh
Games Camp tutorials kind of Mak sense
because in the video game you have like
a state and there are a few actions you
can take but there are a lot of states
and these actions and States we will
have a look at them later are perfect uh
for Q learning or deep Q learning U we
also try to make an own agent for a maze
and the title is from that but that yeah
that way went okay so um how do does um
Q learning actually work
so I already mentioned States and there
are some elements that that are required
for Q learning um first we have an
environment the environment is like the
thing the agent interacts with so that
can be a game and uh agent can interact
with that or the environment is like a
real environment and that's also
possible um in this environment the
action has uh the agent has some actions
um that's like these steps it can
perform so um for example in
the example game environment Mario can
jump or go right go left that are
possible actions and uh it's like the
things you have on the controller the
buttons you can press that's like the
actions the agent can do
um there also is the agent itself it's
represented as Maro here Mar Rio here
but um the action is the agent is like
all of the things combined kind of
because it's like the the main part of
the
AI um the one of the most important
things is the reward so there is like
the score in the upper left corner of
the environment that's not really the
reward but um that's what you might
associate with the reward also maybe the
time both of the things
are um in the dotted
boxes but um the actual reward we are
aiming for are the um game failed or the
game one state kind of so if the level
is completed then there is a big
positive reward and if the level failed
there is a huge negative reward to um
influence
training um and also the kind of most
important element in the learning
algorithm is the state that is some kind
of res representation of the environment
and um the state is kind of what the
agenes uh it can be for example the
position of an object or or its velocity
or things like that so in Q learning we
have some tables um um these tables have
a state and actions
Associated and for every state action
pair there is a q value and the Q value
inside the table is the expected reward
over the time so if I
have uh a state X
and I look in the table and if I go
right now then I expect to the reward in
the end to be one and if I go left then
I expect the reward to be minus 10 or
some something like that um the rewards
over the time so looking into the future
kind of the rewards always are a bit
discounted so the reward in the action
itself is a bit higher than future
rewards um and also using U q-learning
table um you can if it's fully trained
it kind of memorized the per perfect way
and you can find a perfect path but um
that's quite problematic if you have
like a lot of states or infinite States
so therefore we used uh deep Q learning
Network it's a neuron net and as first
layer you input um State parameters it's
not like the state itself but uh for
example a position and velocity and
things like that so um you input a state
and after some hidden layer magic then
an action pop falls out so instead of
memorizing the table the know notes um
in the noral
network um use U an approximated optimal
Q value and therefore find like the
perfect action if there are a lot of
different states and then there can be
infinite States instead of a list that
is in the table like before I hope um
you understand the basic um concept of Q
learning
and Richard will now show what you can
do with that thank you Jacob um so to
use this kind of algorithm for our real
world example um we have a agent um this
um handy person sitting in the middle
and the goal is to have a network of
systems so um the landscape of of all
the systems with um vulnerabilities
accounted to one of them or or even um
more than one and the goal is to find
the optimal path through the mework so
to attack the first system gain access
to the
second um which vulnerabilities get the
highest reward which are most prominent
to attack um so instead of the game
example we heard earlier we want to
train an algorithm um that decides on
the given Network which is the optimal
path the optimal attack path through the
network
and what actions to take to gain the
highest
rewards um therefore um we have um some
systems with low hanging fruits for
example so very easy to exploit exp um
vulnerabilities are very hard to exploit
vulnerabilities so um the low hang
fruits would be optimal to attack first
gain access to the system or some
systems um have higher rewards because
some critical data is um cter to it more
traffic runs through the system or
something like that so there could be
many parameters um to account for this
and the agent itself knows um in the
beginning a list of systems in our case
so um we have
um a list with for example 100 systems
and then the algorithm would go and find
vulnerabilities at the binning system it
searches is it worth to attack a found
vulnerability is it
um giving us an reward that is
acceptable for the algorithm if for
example um an SSH service is configured
um in a way that it's easy to exploit
this will give us access to the system
maybe root access so it will generate a
higher reward and um the algorithm
decides to um use an attack vector or it
has the option to select a new system um
do some reconnaissance States and find
new vulnerabilities on the new selected
system and then jump from system to
system deciding is it worth to attack a
vulnerability on the system or even more
than one or is it worth to go to the
next system so in order to have it um do
that um we give the reward as explained
depending on how great of impact the
vulnerability is but we also give a
penalty for actions taken so um we don't
want to create a lot of noise scan all
the system at once and so on so um for
example this the selecting of a new
Target would be a penalty to encourage
the algorithm to do some exploitation of
the vulnerability pounds on the
servers um and depending on um what
reward will be get back from each action
we have the landscape and um in the best
case a algorithm that decides go to
system a then go to system C attack um
there and have at the end a big um path
through the network that we can use
afterwards to do our exploitation so the
goal of this algorithm is not to exploit
the vulnerability itself but is is to
decide if it's worth to EXP explo it or
not so um the final exploitation and
more um in-depth research would um still
be done by um a pen tester or maybe um
in some stage or L to the algorith
itself but we come to that
later um in this research we done for
this presentation we have set up a small
project um with a very simple
environment and very simple landscape um
to train the Deep Q
Network um you can see we got a result
of minus 29 so uh we averaged more
penalties than we got results back um at
first s um when we compare to the um
catric um optimal result it's not the
best score you would hope for so um at
first side our algorithm may not have
taken the best route through the network
and um we could explain this by the very
um few data points we've given and the
Very rudimental uh Network res set up so
to answer the questions we come into
with the to this talk is it possible to
master the maze to find the way through
our system landscape with the DEQ
learning algorithm um first of all as we
seen in the last slide there are some
problems um
for the algorithm to work properly we
need um more diverse and um also more
data overall to train the algorithm and
it's not very easy to obtain for a wide
a bright variety of systems um to train
the algorithm and um without this data
it is also possible to say that the Deep
ke learning algorithm is maybe not the
best way to achieve this goal um um so
you have uh you need a lot of um
extensive knowledge about the server
about the vulnerabilities um to train
the um
algorithm but if you have enough data if
you have enough knowledge um the outlook
for this algorithm is that it can
produce a sophisticated attack Vector on
um our system landscape and it's very
useful if you have a very large system
landscape um very complex system
landscape maybe one with um that grown
over time nobody knows exactly what is
running where um what is going on in the
network and you can use it to do um
automated pentesting ex um stages in
this landscape use the data recovered by
the algorithm to do further
investigation or even as said train um
or um extend the algorithm to do the
exual exploit in um the steps where we
have the tech vector and um then it
could be generating um automated um
results of the network we can use it to
harden our Network without the need of
um extensive human resource um for as
set very large Networks so to summarize
it um the lessons we've learned so um
our goal was to know something about the
um deep Q network uh deep Q learning and
um use it in pentesting we have Che that
we have grained a lot of knowledge about
both of this and um we have a good
understanding of how to develop it
further to be useful in our showcased
example
[Music]
[Applause]
thank you for your interesting talk we
still have time for a few questions
anyone wants to ask a question we have a
microphone over here please go to the
microphone to ask your question signal
Angel does the internet have a question
the internet does not have a question
yet they're still asleep but we have one
in the audience how does this compare to
um things like um
pagi sorry I hav pagi which is used for
Wi-Fi attack and uses an actra critic
model um yes so our goal was
um first of all to gain knowledge and um
in this case it is possible to design
the algorithm um for your own use case
so
um to
um um provide it for the network you
have and um optimize
[Music]
it um in terms of the exploits uh does
it basically check service version and
then has some local database with
exploits or can we also hope for U
misconfigurations or uh keys in bash
history or something like that um yeah
so um in the current state it um does
not um use very sophisticated um
databases and everything but our goal is
to extend it to have a more advanced
data set um the algorithm used to
evaluate what vulnerability is useful or
not and um then the more data it has to
evaluate all those things the more
um um the higher impact the result is of
the systems what is vulnerable what is
not and what could be exploited or
not so um how is the training of the new
network actually done because you don't
have a direct um feedback what action
will result in the desired uh state or
final state so how how do you train it
supervised so in the training we used
um like uh kind of a test environment so
um we did not really had good data
because uh we realized that companies
are not so willing to um share kind of
network scans and things like that um so
we
had kind of
a not really mock but uh quite mock
environment so um there was not
really a good
um
um I'm not quite sure how I phrase it so
they're kind of like we kind of use the
mock environment so uh we gave
the
um the answer if it's uh vulnerable or
not a vulnerable
service and that rounds up our talks
please give another round of applause to
Jacob and Richard thank you for your
[Music]
talk