Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2529 Thanks!
[Music]
the next talk is from Jaden and she will
tell us about the magic behind one click
deployments like I think it's wonderful
to just when you do server
infrastructure you do one click and the
server is just there and you don't have
the hassle of managing it by yourself as
normal um we also have a signal agent
for your questions later you can post um
with the hash CCC cam23 notx then we
will see your questions and now have fun
for the talk okay let's just head back
to the first
slide and then we can
start well it's
buggy okay welcome to my talk um the
magic behind Auto deployments um as
Leona already said Auto deployments is
something that most of us have
interacted with before um yeah but let's
start first thing I'm Jaden uh my
pronouns are she her I'm 22 I work in
cyber security and I do lots of things
with computers and Hardwares and stuff
um you can find me on K social Jaden at
car social and uh on my website if
you've already seen um where you can
also find the slide
what is an autod deployment and what is
a
oneclick deployment the we will have
about four parts in this talk first what
even is it then how does it work then we
will try to demonstrate it right here
with real hardware and real data
centers and afterwards there will be a
Q&A session uh where I can hopefully
answer most of your questions and and if
I can't just hit me up uh after the talk
or on social media or via email or
whatever okay oneclick deployments are
fully automated deployments what does it
mean it means that we can spawn anything
like VM or container IP addresses DNS
entries and much much more like user
management firewalls software
configuration monitoring and anything
you can
imagine the whole
thing is pretty easy but you have to get
some key ingredients first to allow
access for an automated system on all
your um devices and access to apis you
usually need some kind of root access um
usually done with at that age and you
need your deployment sources which are
usually from just a git repository where
you have your config files and your
scripts and then there's your source of
Truth which is in my case
netbox and then you have the big issue
of secret storage you have to generate
secrets on demand for root passwords
user
passwords and a lot of other things
databases everything and that is most
easily done with a utility called pass
which is just using a gpg to store your
credentials um and enrp them so you can
just put them in a git repository and
everything is
fine the easiest way to get that is just
get a dedicated host um it doesn't have
to be in a data center in my case it is
um and you just plug in a small Ubbi key
most of you uh probably know what that
is it's a small device uh you plug in
via USB into your computer which can
store your encryption Keys your private
keys in a fashion that doesn't allow
them uh to get off there they are on
that small USB stick and nobody can get
them well you could but it's really
really really
hard okay in a workings um you need to
gather some data before we can even
start with your deployment this data is
usually provided by some kind of source
of Truth netbox is my favorite way of
doing that because handling config files
is to be honest quite a pain um to get
them everywhere and writing in config
files uh isn't that easy either and as I
said we want that to work with one click
via a web UI so writing config files is
not that easy
and in your s of Truth you usually s
store things like your host name network
interfaces IP addresses gateways DNS
servers your hypervisor config like
memory or if you get your uh machine
from some provider like AWS or header or
something um you need to know uh how
many cores you need how much memory and
such uh things and you also want to know
which Services you want to install
the second step is gathering the secrets
um I've implemented a very easy way
which is based on the password to.org
utility um as mentioned before it just
uses
gpg and I've written an answerable
module uh to dynamically generate uh
Secrets uh and store them in an
encrypted way um this is just an example
you have your anible uh secret storage
which contains your host
Secrets which are individually for each
host and also your secrets like
databases super user passwords and the
host root password of course and your
static Secrets like API tokens DNS
access uh and things that you don't want
to generate on demand because the
secrets will stay the same in your
environment
second step is to configure your
hypervisor or
your VM of some sort for that I'm using
proxmox but you can use AWS or head snow
or whatever it's pretty much just an
implementation
detail um you need to spawn and
configure the hardware which usually is
done with an simple API request to your
favorite provider or your own
Services um then you already get the
first issue um network configuration you
need to probably set things from your IP
addresses to your Mac addresses and all
of that um that is usually handled by
your provider or by your hypervisor um
and netbox just or your configuration
management just tells you which IPS do I
want which IPS are free which Mach
addresses are free and you just generate
that data on demand
and you already set your default
password usually uh you either get a
default password from your API request
or you send one with it or you send your
um SSH public key after that your device
is spawned and the first thing is we ssh
in using our root user that we just
created then we run things the usual
setup everyone does it probably like
you're installing your packages and your
default packages you're upgrading
installing the firewalls making firewall
rules like allowing
SSH and allowing your services that you
want to install for example a web server
which would need Port 80 and uh SSL
ports um and then you ensure that your
root password is matching the one that
you have in your database in your
password database and then you configure
stuff like zudo like that's the easiest
and basic things you could stop at this
point and then you just have a machine
that is running somewhere in the
internet or on your own network and you
have a root password your SSH keys are
deployed and you can start from there
doing your stuff via
SSH but then we have things like DNS so
we gather for example our SSH uh
fingerprints and put them in DNA which
is a need security feature uh for those
to haven't heard of it you can put your
SSH fingerprint um onto your DNS and SSH
will contact your DNS server and
validate that the host matches those
keys uh which will just ensure that your
DNA isn't doing something weird or your
connecting to the wrong host or
something um then of course we set our a
and quad a records our C names if we
need them and everything
else um for
example we we saw that for example you
can also uh Set uh the prome uh DNS
records which are just sov records that
um allowed your monitoring to learn
about new hosts um we will see that in
action
soon yeah mon recurs and then we get
into
the into the uh nitty-gritty of the
whole thing configuring and installing
services in my example I just want to
install a netbox instance um so we need
to ensure that we have a postris
database so that your service can store
all its data after the database is
created we have to install all our
dependencies and we just do that via
anible and after that we configure our
service and then we run some other small
stuff and then you uh bring it up the
whole thing is running you connect um
via your admin Choice uh admin software
of choice and then you create your super
user and in the end your password
database will just have admin and your
randomly generated pseud user password
or admin password for the service you
need and you never have to touch that
host with SSH yourself all everything is
just in your password database which you
can integrate into your password manager
and it's done you don't have to connect
to the host
manually afterwards we also want
monitoring monitoring is a very
important but often
underlooked um thing in in it security
and also in it in general I don't only
need the service to be up and running I
also need to know if it's well um and if
it's having issues or the drives are
getting full and stuff like that so you
usually have something using prometh and
grafana or some variation of that and
I've we seen before um we just add the
uh SRV record to our prometh Zone and
promes will automatically discover our
new host and we don't have to do
anything on that part
either then even more Overlook part is
what do you want to do when you get rid
of the whole thing there are DNS records
and monitoring things you have to remove
and also your secret storage has to be
cleaned
up so we will unregister the whole thing
from monitoring delete the DNS entries
and then remove the VM or container from
our hervor and everything is removed and
it's
done I sped through this part quite a
bit because now we're going to see it
happen in real time hopefully so we
start with our netbox
instance that is uh the way I have it
configured okay it's really loud over
there so I've created a new virtual
machine with name CCC cam23 point.
jupiter. cafenated labs.net uh Jupiter
is my hypervisor and this is the host
name of the uh virtual machine we want
to
create um as you can see here we have
our network interface
already very
nice do do we have broken
IPv6 maybe yeah no IPv6 over here very
nice let's check yeah well no IPv6 for
me
um it's a modern standard like it's way
too new to be supported everywhere I uh
I shouldn't have expected it to work I'm
sorry
okay that is really distracting over
there
well do we H have IPv6
now
H well that's uh
embarrassing I'm I'm on the Wi-Fi right
now so I'm
trying uh to get a connection let me
just test
this yeah I've unplugged
it and I'm on the Wi-Fi but Wi-Fi
doesn't have IPv6
either
thanks I I I I'll try let me let me just
try let's
reconnect
H it's working it's working again trying
get off and on again has
worked so let's
see
well Internet is real fast
today okay we we have
our thing yeah it it has worked very
good and the SSH uh connection is also
up okay um we can see our uh AET device
with our public IP address and our
preconfigured Mech address we can also
randomly be generated on theand but I've
preconfigured it um so that it doesn't
change quite as often and the op tables
don't get full as much when
we yeah I
can yes give me a second is there light
theme light mode okay can you see it now
yes okay perfect um let's go back to the
interfaces Tab and let me just close
this we have our Mac address and uh our
ethernet device and our IP
address then we can go over here we can
see this is our um host name um the
device is currently offline because we
want to PR configure it then we have
selected our debn 12 um as a base
image down here are some of the proxmox
features we want to configure it should
start at the hypervisor boot it doesn't
have a proxmox VM ID yet but we will see
that this field will get populated once
we have run our
script and we have configured on which
cluster it's running on which node and
all of that and we've also configured
our virtual CPUs our G our two GB of RAM
and the 8 GB of
storage there's nothing else configured
yet so everything you see here is the
config now we will go over to the SSH
session and run the whole thing um I'm
currently connected to the host I've
described before which is um has the UB
key PL into it so I could even unplug
this one if I wanted
to so let's go to our redeployment this
will happen automatically with a VAP
hook but um for demonstration I will
show it to you this way and also the
script will pause um sometime so I can
explain what's
happened um the here we have a deploy
and then our
environment and then we can just run
deploy
deploy okay let's deploy this
machine uh first thing is we are
decrypting our secrets then we are
installing our python dependencies for
the anible
controller then our anible script will
talk to netbox and figure out which
machine we want to deploy and which
which with which settings then we are
cloning our secret
repository and
generating our secrets as we can see we
have generated our root password for
example our root password has been uh
generated
and where is
it yeah our TC key for uh DNS challenges
has been and DNS access has been
generated yeah up here we can see that
our uh Secrets _ hostor root password
has been generated
um yeah let's continue the
deployment um the next thing is we will
ask netbox about the gateways that are
uh in this network which DNS servers to
use and records and out DNS
configuration then we can also look uh
some tasks are skipped that is a quite
normal thing in anible for um we are now
trying to configure a physical device
but as uh the netbox has told us it's a
virtual machine so we don't need to run
physical device
configurations okay now we
would create an update book yeah because
we have selected that it's offline now
it will just tell me yeah it's offline
and it won't deploy
anything if we change
that uh in our
netbox change it over to um
online let me just show
you if we change that over here to um
provisioning uh stage or let's just say
this to active and save
it our anable will
now uh get the new data new information
and and we'll try to deploy the
machine let's switch back to the
terminal and let's run our script
again okay same thing as before we are
collecting our initial secrets for the
netbox access we are installing local
dependencies we are now talking to our
netbox figuring out what to
do
loading our secret repository and as you
can see now we don't generate
it the secret has been just
loaded um it was generated before as we
saw in the change log and now it's just
loaded because it was already stored in
our secret
database okay now more stuff is
happening um we are now creating
our
container
um right
here um as you know this part does take
quite some while to spawn a new
container after that is finished we will
see that
our yeah
exactly now we update the
netbox which I can show
you
um okay netbox will
now show me
the new proxo ID down here it has been
created as container
104 and now our deployment
continues and we are now upgrading
the default
packages um and
installing uh the basic packages we need
to have our Debian
configured
well Dean is a bit slow
today as I said before it's real
hardware and real data centers so
probably it's the usual effect of when
you show it to people it doesn't work
but I can tell you what's going to
happen next and what this means um
because we first tried to connect with
our user to check if the device has been
provisioned before which it hasn't so we
switch over to our root user um because
the access was denied with our
unprivileged
user
yeah we have updated uh our Debian and
then we want to install our default
packages which are in my case Vim curl
zudo and Tre I think um just the usel
stuff you want to have on every
machine
yeah
okay now we are configuring our zudu now
we are configuring our firewall
um in my case I just use
ufv um it's the in my opinion it's the
easiest way to have a simple firewall on
your device um it will install the
package and then automatically configure
um our trit Network so everything within
my IP range is considered trusted and
everything outside it is considered
untrusted um that is important for
monitoring access as I don't want my uh
monitoring data to be available from the
internet so it's only available from the
internal Network and now we are
rebooting our host because we have
configured our firewall we have
configured our root user and we have
configured our default packages after
the reboot we will
continue and as you can see all of this
happen happening
automatically um the device is rebooting
the device is being configured and now
we yeah we have have reached the point
where we can start our custom
deployments so first of all we are
collecting our fingerprints for the
DNS which we can see over here we have
added our quad a record our three SSH
fingerprint records and our monitoring
record um which just tells the
monitoring to uh look at this host
now after that we will configure our
users
with their SSH Keys default passwords
and their permissions you can configure
a many users you could also
configure um P elab for example so you
don't have to configure the users
yourself and your Linux is just talking
to your Lup um for
validation then we want to apply SSR
hardening to our instance just the
default asso options you want to enable
and disable and we are also installing F
to ban um which is just a need security
measure and helps on getting rid of all
the people trying to connect to your
host with bad
passwords especially if your host is
connected to the internet uh steps like
these are highly
recommended
okay now we have configured our
SSH um now we don't want to uh install
proxmox so we skip that uh we also uh
skip uh dis
encryption and those are all the
services you could Deploy on a machine
um and only the the services we have
selected um in the netbox are actually
deployed everything else is
skipped
okay now we are installing node exporter
and the creating our SSL certificates so
that we can connect uh to the device
with uh encrypted uh and valid
certificate
um I can now show you as we have
configured our SSH uh I'm sorry our
monitoring records for
Prometheus um I can now show you the
promethus uh side of things let's zoom
in a bit and switch to light
mode
um as you can see our host has a
appeared uh it is currently considered
down because we haven't uh finished our
deployment and once we have finished
deploying which we can see in the
terminal right
here as you can see um our monitoring
stuff has been installed and
configured and now we are installing a
final thing I quite like um I have a c
message of the day package which will
just display the last successful anible
run it's get commit and stuff like that
so you know that um when you s this
agent to machine when the last uh
deployment has happened because anvill
doesn't deploy continuously it uh gets
run by your web hook or by yourself and
after that um it just stays like
this okay we have finished our
deployment if we check back at
Pras we can
see that our host is now considered up
because everything has been
configured and we
can SSH into our host as
well let's just um
SSH
into this
host
well it's it's not a good day
today let's just uh remove that really
quick
yes and now we can see our custom
message of the day uh which will just
tell us
um oh I'm sorry
okay now we can see the custom mage of
the day which will just show
our very good very good Johanna very
good we can uh we can see our custom
message of the day um which will uh show
you the OS version anable commits and
everything else
um well that happens if you have two it
administrators I
guess um okay now let's
just decommission the whole
thing
um and I'll try to let the
decommissioning happen in the background
and then we can switch over to our
Q&A
um yeah the slides are on my
website and the code can be found on can
can you you can find the code on cafet
lab.org
um and in the next coming days uh I'll
try to open source as much of this as
possible um so you can take a look at
for
yourself um yeah but I'm now open for
for Q&A
questions yeah
we have about 12 minutes left for Q&A um
I will go
around okay so how much of this is
standard anable or at least readily
available anible scripts how much is
custom anible scripts and how much is
custom glue
code um depends on what you consider
gluc
code um I
have um reimplemented um some modules
the biggest of those are the secret
management um which will hopefully be
open sourced soon um and the proximo API
um I written a small wrapper uh so you
can directly interact because Community
General uh proxmox anible modules try to
um abstract very much so you can uh use
them
as uh closely to you you
can configure things directly in some
cases so I've just written a small
rapper around the python Library prox
moer um which just passes through the
arguments to proxmox directly without uh
any logic in
between um just for as a reminder we can
also have a signal Angel so if you watch
in the internet you can also ask a
question with the hasht CCC cam23
notx uh now the next
question uh how do you handle backups
and restoring them if something get
broken or something do doesn't work
anymore okay um
I uh run a backup before every anible
deployment um
automatically which means that I can
simply restore the container to the
state it was before the
deployment um those backups are CT uh
for five times or something like that so
you have your standard backup uh
rotation there um and anible just
creates a backup uh um in case something
goes wrong you can just restore the
previous state and continue from
there uh thank you for the presentation
um you said you are going to
decommission the system now in the
background uh I'm quite curious uh is it
like extra roles you wrote for uh
unregistering the VM on proxmox and the
DNS entries or I have written um most
playbooks which deal with deep with
provisioning and depr provisioning to do
it themselves you have a variable which
will tell um it's the netbox uh status
uh variable uh which will just tell you
it is is it active is it de-provisioning
is it offline and the playbooks um
decide on those uh variables what to do
so you have a complete list of all your
VMS and if one is not in the list but on
proxo it get deleted no uh um the I I
I'll just show you um when we switch
over uh to our
netbox uh we just change the status uh
to decommissioning which is
just this and now we save
it and afterwards we can switch back to
our
deployment
and now the system will uh detect the
change from netbox and uh Delete the
container um we will see also that it
will create a backup before deletion um
just just as it would before any other
inle run um I'll let you continue that
is there other other questions
yeah sorry um just do you have some
experience on um Secrets Management in a
more Enterprise
environment uh what do you mean by by
more enterpris um I mean with one yub
key you just have one but you have
personal several people needing to
access the management a part of called
Bal
um um just out of curiosity yeah you you
can uh scale the way I've done it to uh
a small team um but the encryption is a
serious Factor when deploying uh
hundreds of hosts um because every
operation with the UB Keys just takes
some time um there are are ways around
that to use uh hashing for secret
generation for example um I am planning
to implement something like that in the
future um but I haven't implemented it
yet do we have any questions
left hello uh with regards to the scope
of the secrets is it just for stuff
that's is it stored encrypted outside
the container and inside or is it stored
encrypted outside then decrypted inside
what's the kind of scope for that yeah
it
is stored uh encrypted outside the
container and unencrypted inside because
usually the config files don't allow for
encryption um if you have a config file
that allows for encryption or hashing um
you should do that and I I'll do that um
but sometimes if it's not possible to do
it it will be stored un encrypted inside
the container and encrypted
outside uh
um um how much time does it take to set
up a whole new service service which
contains about three or four new
processes which need to be started a
database and so on how much time do you
need to set up all these anible scripts
and all this stuff if you know how to
deploy
everything so you mean like um if I for
example have a service that I don't have
a script for how much time it takes to
write an anable Playbook to run
it service okay so yeah um as most
services are usually the same structure
you have your system D units and you
have your code which has to be installed
somewhere um that is usually um a copy
paste job of just taking another another
similar um process and just just uh
changing a few things around so um a
basic rooll can be done in about an hour
and more advanced one in yeah like uh 10
hours to a few weeks depending on the
size of the
project okay as as we can see before we
have now changed our state the proximus
ID has been given back to the pool the
device is offline and the container has
been removed
um final step here is to remove our s uh
our DNS
records which will just remove the old
records and then the rest won't run
because it knows that the device is Now
offline um and one final thing is to
remove the um entries in your SSH known
hosts file um yeah those will be removed
and now you can redeploy your host under
the same IP address without SSH um yeah
[ __ ] its pants to be honest um okay
are there any more
questions um you use um containers this
works also with uh normal proc flux VMS
yes um the configuration is a bit
different because um in VMS you usually
have to configure the IP addresses
inside the VM and not inside proxmox but
um that's just a change to your um
deployment playbooks uh and you can make
that uh have that work as well you could
even uh go
to uh go as far as to do do that with a
host you just screww it into your rack
so you just install your base system um
and then from there you can deploy just
like any other
machine any further
questions
okay then thank you for your
talk like it was interesting for I think
for all of us and a huge Applause for
[Applause]
Jaden