Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2625 Thanks!
[Music]
uh good morning to the mway stage to the
opening session on day
four it's nice of you to uh be so early
and uh attend this session we will look
into the past for present
dangers welcome Richard for peeking over
the tape mode wonderful thank you and
thank you all for being here indeed this
bright and
earlier enjoy the shade while it lasts
so yes for the next uh 20 or so minutes
I will be speaking about Magnetic Tape
storage the title of this talk is
peeking over the tape mode so the idea
is we'll be spending half the time
looking at how modern large scale
Magnetic Tape storage systems work and
then the second half we'll be looking at
what this implies for the security the
motivation behind this talk is that well
we're storing more now than ever before
and still in this sort of ecosystem of
various storage media tape is still a
very a very Central pillar at the same
time tape is taking on a a dual purpose
besides being a cheap and efficient way
of storage it's also becoming a sort of
Safeguard against many cyber attacks
that that are happening against uh
organizations for instance ransomware
which where where tape is is an
important protector
nowadays so before we proceed into the
details why should you listen to me I
don't have this you know the long Gray
beard usually associated with your local
tape wizard um I instead I've worked
with Magnetic Tape for only two years
but I've had the pleasure of doing so at
CERN the European Center for nuclear
research where I've been well in the it
group specializing in
it um this of course warrants a small
disclaimer this talk is in no way shape
or form organized by uh approved by or
or associated with the CERN the
organization this talk does not
represent cern's moral values this talk
is my
own
so
no yes why are we speaking about
Magnetic Tape Magnetic Tape is still
very important for data storage uh and
the data on Magnetic Tape is often times
very interesting there's a bit more of
an overhead usually to to using these
tapes compared to your dis which you
just plug into your PC and so people
tend to be you know curating the data
that is on the tape tape storages may
may include you know important data that
you you want to keep for future
Generations you want to keep it for the
next decade or so it may also include
personal information you know odds are
your government is using tape archives
somewhere in their storage
ecosystem um as do you know your
friendly hyperscaler Cloud providers
either through Services which are
directly exposed to the customer or in
the back where they keep their you know
the the personal data they've
accumulated over the
decades there's also backup data you
know if you want to do a backup we're
better to put it than magnetot tape
storage which lasts for longer than than
the life expectancy of a hard drive and
where it is less accessible to mean
hackers tape is also becoming popular
again because it is less bad for the
environment it has a lower
impact um both energy wise and for the
whole life cycle of the the tape and the
related infrastructure than spinning
discs or other tape storage systems or
other storage systems so so there is a
sort of push to use more tape in various
institutions because of its
environmental
friendliness and then as I mentioned
tape is becoming this sort of last line
of defense against ransomware attacks um
data written onto a magnetic tape
cartridge is in in this state right now
inaccessible to an attacker so so if
your ransomware attack is going on and
you make this thing hard to get to then
you have data that is safe from the
whole thing and well it's not just that
this is a nice thing for organizations
to have as a protection it's also
becoming mandated so so here for
instance for this um uh for the European
digital digital operation resilience act
the door act it it makes it a
requirement for financial institutions
to have the appropriate safeguards uh in
place in their system and then for
magnetic tape will be one of
these finally Magnetic Tape is still
storing a lot of data at the end of the
day um just to hammer this home here I
have
some there we are just an estimate from
from back home atern how much data will
be needed so here we see
the for the atlas experiment alone you
just need to look at like the y- AIS you
see that we are looking at the EXA bite
you know units for the coming you know
decade times this by more experiments by
more institutions which will have even
higher requirements we'll be in you know
setes of of data storage and so we need
to kind of we need to be efficient and
and scrape all the bits or all the put
all the
bits on on media where we can
so what does a magnetic tape system look
like at scale or at medium is scale it
looks something like
this um there is data input on the left
side where the data first hits what we
call a dis buffer this is because well
tape can be fiddly to interact with and
most external systems know how to deal
with discs instead of tape so so a disc
buffer is a nice interface for those you
know then the dis buffer servers will be
speaking the protocol the transfer
protocol that you want to use um they'll
be an intermediate buffer for both data
going in and out and then uh that makes
the whole operation smoother it's
generally a bad idea to write to tape
systems directly through the internet uh
so so this is a nice buffer that that
Smooths things out however this is a
filthy dis system so we won't be
speaking about it anymore so for the
rest of this talk uh we won't dedicate
more time to it after the dis buffer the
data goes to the the tape server uh and
the tape drive the tape server is sort
of the
intermediate uh for for connecting a
drive to your your local it system so
it'll be connected to the internet or
the local network at least it will have
the appropriate drivers it will uh have
some disk storage as well to also buffer
the files going in and out
locally and then it will have the tape
drive connected somehow usually as a scy
device uh and by way of for example like
fiber channel for the dat
transfer the tape drive is responsible
for writing to these things so so as you
may note where in a in a a normal hard
drive you have both your your storage
media and the reading uh infrastructure
in one unit for tape this is not the
case so so the drive is the separate IO
device uh tape drive is housed within
the tape library and as are the
cartridges so so the tape library is a
container container um it can be it can
have many shapes it can be round it can
be you know container shaped and inside
of there you have shelves of these
Magnetic Tape cartridges and the the
drives for reading in addition to that
you have some robotics for moving the
cartridge to the drive and getting them
into
contact and yes so now we can look at
these components a bit more in detail
individually the
media
the medium or tape quadrate is what in
the end stores your data uh it is if you
open one up a well it's a plastic
container and inside you have this
wonderful
uh um this wonderful tape all coiled up
which is composed of magnetic media and
some polymers to keep it all
together the media comes in two flavors
Enterprise and lto Enterprise is the
more expensive option usually a bit more
feature Rich and more more capacity rich
and then lto or linear tape open which
is um a more open standard by the lto
Consortium it has is us it's usually
more focused on on cost Effectiveness so
a bit cheaper but you get more bang for
your
buck around this time the time of
writing or speaking a tape cartridge
holds of well a tape cartridge of the
present generation holds about 18 to 20
terabyt of data each uh and it has this
Nifty property that this doubles every
couple of years so whereas your disc
storage at this time is sort of
struggling to scale because it has
reached its param paramagnetic limit
where the bits are becoming so small
that it's really hard to keep the
magnetized and to have the the
technology around it to actually
magnetize more stable media tape driv
still have or tape media still has lots
of easy victories ahead the bits are
relatively big in contrast to dis or
optical storage but it makes up for that
by virtue of having a big surface area
so the the band inside the the the tape
cartridge is 1 cm wide but 1 kilm long
which gives it a lot of surface area to
work
with this however introduces a cost and
that it may take time to seek to your
files so so we'll get back to that so so
once you actually have the drive or the
the drive mounting the the tape and are
reading or writing it it takes some time
to get to where you want to
go the media may be formatted it's you
know similar to to how you would format
a disk file
system the the Baseline sort of is the
ltfs um or uh yeah file system where you
still have you know your tape is sort of
like a a disk uh file system with all
the metadata attached but at scale you
will find lots of systems which don't
use this so this means that the metadata
such as as the file name will not be on
the tape itself which also has
implications moving on to the tape drive
as mentioned it's the input output
device used for reading and writing
tapes these also come in the flavors of
Enterprise and lto Enterprise being more
feature and lto being more bank for your
buck and also they come into shapes they
come in full height and half height full
height is the sort of standard size that
you would put into your tape Li
and half height is the sort of smaller
desktop or desk sized uh
Edition it can go up to 400 megabytes
per second about there about at present
um but this is under ideal conditions uh
if you write your data stupidly and you
need to seek back and forth you will of
course uh incur uh a great cost to your
uh read
times the drive is connected to the tape
server as mentioned this is also known
as the data mover in the lingo uh and
yes connected by scuzzy the drive is
capable of Performing encryption and
compression which it makes it nice so
then you can you don't need all that
Hardware in the tape server you can
offload that to the
drive what does writing to tape look
like it's something like this this won't
be on the test don't worry when you
mount a tape drive onto your tape server
you get these St uh st0 ST1 uh NST Z
nst1 and so forth Dev deves showing up
these are your your destinations uh or
your your ways to interact with the
drive uh for bigger systems you would
then have some sort of command uh to to
put to get the the correct cartridge
from your library to your
drive and then you can operate it on it
using actual tools or Linux tools which
may or may not already be present on
your machine uh fun fact tar was used to
interact with tape AR or with tape
storage
uh the Mt command is maybe not installed
on your laptop but it's easy to install
on most disos this allows you to
interact with the drive to read its
status to load tapes into an active
State and so forth um yes do notes for
the two devices one of them rewinds one
of them doesn't so so once you have
written a file to your tape
STX will go back to the start to the of
the tape so if you go I'll write my file
number one thank you and then later on
go I write file number two it will
overwrite the first one so
beware
for the oh yes and then there are some
more examples on redhead site and and on
IBM's uh Pages for more
details um the library very quickly done
it's theis
container the library maintains the air
gap between the reading and writing
device this is what gives you this nice
property of defense against for example
ransomware attacks the attacker has a
more Troublesome time getting to your
storage than the discs which have to be
connected all the
time there may be robots inside usually
one there may be two for redundancy uh
and these together with the rest of the
internals all communicate by way of
canbas or ethernet depending on the
model of Library very important a
library may be partitioned into logical
libraries similar how to would you would
like partition your dis so a logical
Library would show up to your software
as a as a separate Library uh to to
interact with so then you can split your
physical one into multiple logical ones
which is useful as we will
see all right finally security the part
you've been waiting for what does this
all mean for the security of the tape
system overwriting takes time as we've
mentioned seeking takes time uh getting
to the right place also or also more
just getting the tapee from the library
to the drive takes time there's usually
a queuing system because you have not so
many drives in contast in contrast to
how many tapes you
have so it's a system of patience you
may have to wait a day to get to your
data conversely the the deletion process
is not as you would expect it perhaps
you may know that on a dis you may
delete a file and then it's still it's
still like someone scraping the data
from the dis and after the fact May
recover it depending on how how you've
actively been writing it's similar to uh
it's similar on tape usually the
deletion process is just a metadata
operation so so wherever you keep your
metadata is where you just do the
deletion and you don't actually touch
the tape that means the data is still on
the tape so if someone steals the tape
after the fact after you thought you
deleted everything data might still be
FAL um to remedy that it's important use
encryption by default on your tapes
so that if someone runs off with them
they still can't read them also use this
uh key management system called Lether I
don't have the time to go into the
details but it's basically it's a a neat
way to organize encryption keys so that
you can on a more fine grained level
delete files by just deleting encryption
Keys which is then compliant with for
example gdpr and other data um data
management
systems also be sure to wipe and Destroy
used tap don't just throw them in the
trash someone might be able to recover
them in
addition if you override the start of
the tape your data may be completely
gone um I I mentioned this this um NST Z
and NST these devices which may rewind
and not rewind um if you by accident go
oh I'll just you know I write my 18
terabytes of data and now I'll just add
another file and whoops I wrote them to
the wrong device it's it's now at the
beginning you've overridden the tape the
data is no longer accessible to you
because these tracks on the uh tape
media don't line up they will no longer
line up and so without special equipment
you will not be able to read any uh
successive data on the tape without
special equipment which you can get to
you by your local three-letter agency or
which you will have to ask your your
um uh your uh tape manufacturer or or uh
distributor for which is usually a
costly
operation an attacker May exploit this
by very quickly just overriding as many
tapes as they can so be aware of that it
just takes the time to queue in the
system the T time to mount the tape into
the drive and then just to write a few
bits remember to protect yourself
against that by restricting access of
course
uh as much as possible monitor access
patterns so log and monitor the log
files for suspicious activity is someone
mounting 2,000 tapes in my library and
just writing the first bits that's
odd yeah be aware of that um also put in
logical libraries with readon access so
I mentioned these partitionings these
logical libraries you can um put for
instance replicas of your tapes so if
you have data on one tape and you want
to be really sure that's safe you
created a double copy put the replica
into one of these logical libraries
which in the tape Library does not have
a drive Associated to it and then you
suddenly have a a separate partition
where the data cannot be easily read
from or
overwritten uh tapes can be moved in
between usually by way of special
administrator commands or command line
tools uh I believe rest apis are on
their way as well so so this will be the
way to to protect against sudden
overrides the patient attackers
encryption key switch attack tape is
often used for backups where you will
have to encrypt for your users a patient
attacker can compromise a host and then
just wait observe then they may um
switch the encryption keys if they find
them maybe you have put the encryption
Keys onto a file on dis or you have put
the encryption key into a database the
attacker can switch it just you know
they look at the Chone job they see the
backup starts at at 1:45 each day we
switch the key just before and just
after so that the backup takes place
with the wrong encryption key do that
for the duration of the data retention
policy and then suddenly if an incident
happens if an attacker pulls the switch
on the ransomware attack then the
backups are inaccessible to the horror
of the system
administrator so to remedy that
um try to Outsource Key Management to
your
users it's nice to not centralize
everything in your tape service to the
best you know of your abilities uh if
the software you use for the backups uh
allows the user to manage the keys
that's great also monitor of course um
make your users take samples of their
backups you know um read them back and
do the same yourself for the case where
you manage the keys and then finally
distribute your your architecture it's
if you don't make everything Central
then it is uh certainly easier to avoid
attacks on these files um finally one
more um as we mentioned you may have
tape which is not self-describing so
your metadata may not be on the tape
what does this mean it may mean that the
metadata is on a central database
somewhere the central database is a very
juicy Target for an
attacker um so remember to back it up
and make sure you are not keeping the
backup of the database
if you are you know if you're speaking
to your your local database guy um you
and ask him what's your what's your
database policy well don't worry about
it we put it on tape will they will say
so so and if your tape is then not
self-describing you will not be able to
recover in the time of an incident so
that puts me at the end uh to conclude
it's good to understand tape systems
because they're still part of our mix of
data storage media
um attackers May understand these quite
you know by just observing so you should
understand them too and then I move on
to questions um you can ask me questions
here or later on I'll be at the cold
North Area most likely and I have about
10 of these souvenir CERN LHC tapes for
the interested people to to claim thank
you for listening thank you
Richard
okay there's a microphone one or two
questions
yeah uh thank you for your tape and uh
for a blast from the past which I
understand is also the future uh about
you talk about an act European act that
requires using tapes which is a nice
thing because tape is a great media that
has been
forgotten however do you know if
companies large companies have actually
uh started implementing it and going
back to
tape I
think I think they never left tape
actually so so just did oh
oh sorry to hear that uh you should you
should uh tell your management about the
security
benefits then but but tape is for sure
used still at scale so so I know that
CERN used to be big players in the
market back in the day now we have been
sort of eclipsed by the hyperscalers so
now before we were sort of the the big
buyers for the the vendors of tape um
you know IBM the producers are Sony and
Fujifilm nowadays
um the yeah but but the the use of tape
has probably never been greater it's
just that well as more people move to
the cloud I suppose they they implicitly
use it through the the cloud services
more and more it's yeah of of course
maintaining maintaining a tape system by
yourself is hard there's lots of
proprietary software around
um breaking exactly exactly and you you
have to take responsibility for the
breakage um we're trying to reduce the
the the the cost somewhat by well we
develop our own open source software for
the data management on tape so this this
reduces the the sort of uh require or
the the cost of of running it
yourself uh but yeah yeah I
for for mediumsized companies can be
tough Thank you Rich oh thank you
for soltions one more time no oh one
more no all right um speak to me after
thank you
[Music]
[Applause]
[Music]
all