Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1381 Thanks!
[Music]
foreign
virtual machines this is a hardware talk
and it's in English it skipped later
for now
um virtual machines are all the rage
for many reasons for security reasons
for practicability reasons
but the fact is that just because you're
running a VM
and you're just a provider you can still
your hypervisor can still read and write
into it
and that may not be such a good thing
so here are Janos raise your hand
and Claudio Claudia is from Italy but
he's been in Germany since I think 10
years something like that
and they come from
chaos West in a further sense I think
they're based in Stuttgart
and they're going to tell you the
challenges of protected virtualization
not only VM but protected VM so that he
can sort of claim SRI just say this is a
black box and I have nothing to do with
it let's have a big hand please for
Claudio and Janos
[Applause]
all right let's start up with
um a big greeting from our corporate
lawyers
so
what are protected VMS I mean he already
told us that we'll have a look into
what's the basic definition of it and
then
um what we did how we implemented
protected VMS on our platform s390 or um
you may know it's better from the name
Mainframe
um Claudia here is KVM and qmu developer
for s390 and he actually made another
talk about the topic which you can see
on the YouTube of KVM Forum 2019
and this is Janos he is the KVM s390
commentator and also camera developer he
also made other dogs to the KVM Forum
but not about this topic
okay
let's step in
so protected virtualization or sometimes
also secure virtualization allows for
virtual machines
whose state is not observable and
alterable by the hypervisor so the
hypervisor can't reach into memory it
can't get the registers
um and that brings us a lot of benefits
one is we can protect against malicious
operators
so if you are hosting a cloud
environment
you might fear that one of your
operators extracts customer data from
VMS and sells it to somebody else
we can protect against malicious
hypervises if the operator himself was
not able to get access to the VM but
maybe he was able to infect the
hypervisor and then get access to the VM
um
protected VMS kinda
also
give some protection against bugs
because we provide protection against
memory overrides into the VM
and then there's the big topic mostly
for banks which is compliance
when we speak about
secure entities secure VMS we mostly
currently speak about secular secure
enclaves
and enclaves are pretty small you need
to adapt your code for it for them so
they can run
but they provide one of the most
important things memory integrity and
confidence confidentiality
they don't provide denial of service
protection but well
basically nobody provides that because
you can just unblock the machine
protected VMS on the other hand can take
basically any workload it's just a VM
it requires
moderate kernel support but user space
can basically stay the same we currently
do not have any changes to our user
space
and they also provide integrity and
confidentiality for the memory
um as with everything
they are still vulnerable to those
attacks
as an example
Intel and arm have sgx and Trust Zone as
secure enclaves
and the hot new thing protected VMS is
provided by AMD who stepped forward at
first with SUV and they're also working
currently on add-ons for for that
ibmc the Mainframe we currently work on
protected virtualization
which is working title so
don't put a pin in that and IBM power
has protected execution
so the typical parts of a server are a
CPU memory some form of boot image
persistent storage other devices like
Network IO
and for VMS also a big part is host
memory management swapping in swapping
migration and all of that
how do we protect it well first of all
for the state for the memory
and um for the CPU we can give
encryption protection that's actually
what AMD Sev does they have different
keys
for memory encryption for the hypervisor
and the VM so if the hypervisor reads
from VM memory it will read
encrypted data with the hypervisor key
but not with the VM key so it will
basically read garbage
same thing applies if it writes to the
VM
then we have Hardware assisted access
protection
well
that's basically one of the key things
to have in our opinion because that
allows us to also protect against right
accesses then the hypervisor
cannot only not read from the VM or
reach just garbage but it can also not
corrupt the VM
we also need mapping protection so that
if we have a page the hypervisor cannot
just remap it at a different address if
the hypervisor knows the contents of
that page it may be all the execution
within the VM
and then we also
can give Integrity protection of the
data
which is not completely the same as
access protection because if we think
about swapping then you also need to
make the data readable
but if you give it back to the VM and
swap it in again it should be Integrity
protected so that what the VM gets back
at the end is still the same that was
swapped out beforehand
all of these can and basically should be
combined
to give maximum protection
foreign
well
for the boot image and for booting in
general we used to have the TPM
and
the the current implementation of boot
image protections are basically a
successor of that
we can have encryption which allows us
to precede a VM with secrets
um SSH Keys
looks passwords and so on
and we can have remote attestation where
a VM comes up boots and at some point in
time it will be measured so a hash will
be generated that hash will be sent to
the customer server it will be validated
there if it is a trusted boot image
and then the VM continues running
at the station
it's really nice to have but having the
ability to just encrypt the boot image
have and also have it Integrity
protected just by a hash
is great because of
the ability to precede it
and just reducing complexity of the
whole attestation stack
IO is
a bit of a problem because
for Io you need to make VM memory
visible to the hypervisor
if you have a look at video
that's all done through VM memory
so
everything that the VM
sends and receives needs to be encrypted
you need to have full disk encryption
you need to use https SSH encrypted
protocols
except for well
if you want to make something really
readable
which is maybe boot messages or just an
error output via standard console
the VM
cannot and should not make assumptions
about the behavior of a device
but well in
today's Times how often is that the case
anyway so that's just
normal behavior currently
in the future
we might have
um dedicated Hardware which actually
provides a protected Channel between a
VM and the hardware and then you could
communicate with that device without
encryption
but I'm currently not aware of any
hardware devices who support such things
Swap and live migration is a complete
other Beast
for protected VMS because you normally
can't read the memory you can't write
the memory
so normally you get it in an encrypted
form transfer it to the other host for
migration and then you need to bring it
back but when you bring it back it needs
to be Integrity protected
and integrity needs to be checked and
you need to check if everything has been
transferred to the new host
so that's a whole other challenge of
complexity
which a lot of people are currently
working on
so
cloudy here will give a brief overview
about the implementation on ibmc
and
as
we can't trust the hypervisor anymore
the question is which entity can we
currently trust well
we did a poll and the new name of that
entity is ultravisa
problem will be
what's the next entity below that
let's see
and of course
on the Mainframe we have an instruction
for that
so let's have a look at what is this
Ultra visor and what it does
the ultra visor is a trusted entity and
experimented in hardware and firmware
and it basically takes over some of the
tasks traditionally performed by the
hypervisor like the boot process sorry
decryption verifies they put image it
protects the guests from the hypervisor
and from the other guests and proxies
all the interactions between the guests
and the hypervisor so it does a lot of
work
um
this is
little table that shows what we trust
and what we don't trust so
we don't trust the hypervisor as we said
we don't trust the normal guests because
it's basically the same as the
hypervisor from our point of view so we
don't trust them
we trust the protected guests for
themselves as in
we don't care if they shoot themselves
in the foot that's their problem but we
don't assume that they will not try to
do fishy things so we will
um we don't trust them that that will
not try to access each other's memory
for example so we will prevent that the
only thing we really trust is the ultra
visor which is trusted system-wide and
it's the only entity that is really
trusted and here's a scheme that shows
how this how the interactions between
all the moving parts are so normal
guests
are not trusted the hypervisor can read
into normal guests the normal guests in
theory should not be able to read into
the hypervisor but they could
if there is a bug for example or a
malicious hypervisor Nothing Stops that
um on the other hand you can see that
protected guests can only access their
own memory
and they cannot access anything else
the ultraviser can access everything as
we said it's the one trusted entity that
uh can do everything and nobody can
access the memory of the ultra visor
the hypervisor in general cannot access
the memory of the protected guests but
you see there is an asterisk because of
course
you need to share some memory with
hypervisor in order to perform any kind
of IO otherwise yeah you have a very
secure guest but not very useful
um and of course there's no interaction
allowed between the guests themselves
this is the the grand scheme of how
everything works
um basically the rest of the
presentation is to explain how we make
this work
um
so protection as we said
the guest memory is protected by
Hardware unless it's shared it's only
accessible by the owner so the VM itself
and the ultra visor
and when it's shared also the hypervisor
but never other guests never other
secure guests
the guests and CPU state so
some blocks of memory that contain the
state of the CPU or the state of the
guest in general
those are only accessible by the
ultraviser so nobody can access it
read it or write it
the host memory is never accessible by
secure guests so this means that for
example a secure guest
will have a very hard time to do any
kind of breaking out of the box kind of
exploit because a secure guest cannot
access in any way
the memory of the host
the guest to host mappings uh protected
by reverse mapping table so
in case the hypervisor
for any reason malicious bag tries to
map a page
a guest page to a different guest
address that will not work
and IO
as I said the guest shares the buffers
needed for performing IO and it's the
guests that decides which pages to share
of course
I put bounce buffers in parenthesis
because there's no it's not necessary to
have bounce buffers in theory but in
practice this is done with bounce
buffers
finally some emulation data which it's
bounced and checked by the ultra visor
there are some
um some bits some pieces of information
that are needed sometimes to perform uh
emulation of some instructions and those
are copied back and forth by the ultra
visor and they're checked
uh for for correctness
so that there are hypervisor for example
cannot provide values or bits in
registers that are not allowed by the
architecture of course
not not everything is checked perfectly
but this will at least prevent the worst
uh the worst attacks
so
what's left in the hypervisor then the
hypervisor still has to do any kind of i
o so if there is a disk access the
hypervisor will need to read the data
from the disk and provide the device
model probably Spirit i o but
doesn't have to be
of course scheduling
it's just from that point of view it's
just a VM so it's just a on KVM in
particular it's just a process that
needs to be scheduled
this of course means that the host could
decide to never schedule a specific
secure VM that's design loss service but
that's unavoidable because as Viano said
before you can always put the plug
anyway so there's nothing we can do
about that
so the nylon service is never
[Music]
um
is never an issue because
it's always possible there's nothing we
can do about it we don't care
housekeeping for some instructions uh
some instructions uh need some
housekeeping from the hypervisor they're
executed by the firmware or Hardware but
still the hypervisor needs to know about
it
and finally some instructions actually
need to be executed by the hypervisor
for example IO instructions not just
those but in particular our instructions
are handled by by the hypervisor
so let's have a look at
the life cycle of a secure guest
and a secure host and see how how this
looks like now in this complex scenario
so
first the guests boots and they put in
normal standard non-protected mode
and then it loads an encrypted blob in
memory which is the actual protected
boot image
although this the Second Step can be
skipped in some cases like if if you
load the kernel image with the qmo
common uh common line parameter then uh
it's already a memory so you need to
load it but details
and then the guest performs a reboot
into secure mode
um basically the guest asks the
hypervisor to reboot and as a boot
device for for the reboot it specifies
this blob
at this point the hypervisor will call
the ultraviser and say hey I need to
create a secure VM and I need to create
this many secure CPUs and then it will
basically
pass this uh blob
from the guest directly to the
ultraviser the The Blob basically
contains all this possibly contains all
the secrets of the protected guests so
of course the hypervisor will not be
able to do anything with it so this is
just passed directly
uh to the ultra visor which is the only
one entity that is able to actually make
sense of The Blob
The Blob also contains some other
configuration parameters some other keys
that are used for some other purposes
and at this point each page of the image
is then we say unpacked basically the
hypervisor asks the ultraviser to
the crypto page so the page is first
made secure so it's made inaccessible
and then it's decrypted
so the hypervisor will never be able to
see anything inside
once that's done we have a boot image in
memory that is now ready it's been
decrypted and the hypervisor just
continuous execution of the guest it
simply needs to use a different format
for the CPU block because now it's not
normal CPU now it's a secure CPU so some
some things are different necessarily
so
what we normally have in this in in the
in the CPU block is CPU Flags the proper
encounter some registers including all
the special registers
some timer information uh all the
interception data basically the reason
why
there was a VM exit and the instruction
that caused it some hypervisor control
flags that alter the behavior of the
virtual machine
this is of course this is not something
that you want to have in a secure uh in
a protected virtual machine so
we do it differently when the vcpu block
is for a protected CPU
so first of all
when the C when a protected CPU starts
the state is not taken from the blocker
memory that is in the hypervisor storage
but is taken from one of these Ultra
visor reserved areas
because it needs to be protected
except for a couple of bits we have a
couple of bits that are always copied
back into the hypervisor because for
example internet management the
hypervisor needs to know when specific
CPUs are enabled for interrupts because
otherwise
it cannot inject the interrupts
um
of course in some cases some more
information is needed so let's say
an i o instruction there are some small
memory blocks that contain some
information there's some registers that
contain some important information that
needs to be shared with the hypervisor
so the ultra visor will actually on a
case-by-case basis
copy this and this information back to
the hypervisor but only exactly those
bits that are strictly needed to the
hypervisor
and
um
and if needed some information is copied
back from the hypervisor into the guest
and this information and these bits are
also checked
again only for those instructions where
the hypervisor is expected
to go to to provide information to to
the guest
and those beats are also checked uh for
our compliance for architecture
compliance to prevent the
um the hypervisor from playing any
tricks on the guest
that's not all instructional
interception is normalized so the
instruction text is not the actual text
of the instruction it's a normalized
version
so every every time the hypervisor will
see the instruction with the same
registers
of course the values in the register
will be in the right place the ultra
advisor will take care of putting things
back afterwards
there's a new interrupt injection API
because normally
we just read from memory and write back
in memory to perform an interrupt
injection but it is not possible so this
has to be done through the ultra visor
and of course the new interception codes
new reasons to get out of the VM for
example if some instructions seem to be
interpreted in a secure way or if
some cases some instructions have been
already executed but still the
hypervisor needs to be notified about it
because it needs to take care of some
housekeeping
um
so apart from this new intercepts codes
there is interestingly less
to to check because the ultra visor will
take care of most checks
and there's a secure instruction data
area which is also something that
normally is not there
some instructions have some small
buffers that are usually just accessed
by the hypervisor in this case they
cannot be accessed but since they are
small it's also hard to use bounce
buffers for them so what happens is the
ultra visor will just copy those in a
specific page this secure instruction
data area
um
and in case it needed it would be copied
back into the guest
um
is you this is for example used for
console data
serial console and stuff like that or
boot messages
so as I said interrupt injection need to
be done differently this is done through
the state description so before running
the VM the hypervisor set sets some bits
to tell the high the ultraviser please
when you start inject these interrupts
and with these parameters
of course the interrupts are not always
allowed only a few program interrupts
are allowed to be injected those are
like exceptions and these are only
allowed when they are expected
so you cannot just randomly inject a
invalid instruction or a page fault
that's only allowed when the instruction
allows for that to be injected
and of course you can never inject
interrupts into the VM if the interrupts
are not enabled for those classes of
interrupts you want to enable and this
is why the ultra visor needs to
give those few bits to the hypervisor
all the time about which interrupts are
enabled
swapping is is interesting
because
this breaks everything
now the hypervisor needs to read the
memory and save it to the disk and then
put it back when needed so
how we do this we export the page which
means the page gets encrypted by the
ultraviser
and then I'm project unprotected so made
available
this is initiated by the hypervisor the
hypervisor asks the ultraviser to export
the page at this point it's encrypted
unreadable
at this point it can be written to the
disk but
also hash is saved somewhere in a
protector memory area so that we know
once we swap the page back we can check
if the content has changed because we
have to guarantee integrity
at this point the pages can be swapped
okay so the hypervisor can swap the page
to disk and use that memory for
something else
once the page is needed again
the hyper version is to swap it back so
okay reading from disk you have the
Krypton encrypted page in memory
at this point
the hypervisor asks the ultraviser to
import the page so the page is more
secure it's decrypted and the Integrity
of the page is checked
if the check wasn't successful the page
is not imported
otherwise it is and the guest can
continue if the check was not successful
the page is not important means that
basically the guest cannot continue
execution so it's as good as that but
that counts as a denylo service attack
and we said before there's nothing we
can do about it anyway
now let's see what the guest has to do
um
basically it needs to start but
check if protected virtualization is
available if yes loading the blob and do
this reboot this is done
basically put loader stage
um what the kernel has to do actually is
quite simple check if it's running
inside a protected guest if so uh set up
the bounce buffers to be able to perform
IO and then well use the bounce buffers
so the changes in the guests are really
really minimal
um
by the way the changes to the guests
have been Upstream already
well the rest is still a work in
progress
um
go
so let's
let's have a comparison of
what
how this behaves and what the
characteristics are in comparison to our
example with ACV from AMD
um
SUV is already there so you can already
use it while protective virtualization
is not there yet
um but apart from that you can call you
can read the state of the CPU so you can
read the registers unless you have the
es extension which I think the newest
CPUs have it
uh whereas
um unprotected visualization is never
possible
you can always read the memory with
um secure with SUV but it's encrypted so
it's not an issue
whereas unprotected virtualization it's
only readable when it's shared and same
for writing it's only writable when it's
shared in protective visualization
whereas on SUV it's always writable
unless
the SNP extension is enabled which I
think it has been presented last month
for the first time I don't think there
are any CPUs with that available yet
on the other hand
swapping is not supported yet with Sev
it's supported with protective
virtualization migration is supported
with SUV and it's not supported yet
with protective virtualization
let's go
what can we get out of this
um
protector machine virtual machines
I mean means the hypervisor cannot
access the state cannot read or write
the state
ideally Integrity protected as I said
and to put image as well is protected
this is the
the basic idea behind this what needs to
be done yet
is
um
and what challenges are in general yes
um
making memory accessible
requires bounce buffers if you don't
just bounce back first then you need to
make memory accessible and not
accessible on demand which is terrible
for performance
swapping Pages can be hard for example
AMD doesn't even bother to to do any
swapping because it's just it's hard
migration also can be can be very hard
um
go
if you have any questions
you can find us around
don't hesitate
these are our deck numbers
yeah the current
hypervisor patches are on the mailing
list on the KVM list and the Linux s390
list they are in active discussions
and
this year at the KVM Forum we've seen
that there has been a lot of discussion
between the architectures
AMD came out with SUV as the first ones
um
some others will follow I guess arm came
up no no power came out but they
actually
implemented their solution independently
from the s390 version
so get in touch with us
we always
um
want to know the your thoughts and the
newest ideas we want to create a kind of
community around there because most
platforms are working on protected VMS
and they most often have the same
problems
so thank you
okay
[Applause]
now as usual we can field question there
are two lit mics please
okay you have a question go ahead one
sentence just a question please so on
slide 31 discussing the provisioning
with the uh the protected block
what is the encryption key used for that
blob and how does it prove to the uh to
the customer who is launching this guest
that it is actually being launched in
the secure mode
good question
so the blob is encrypted with a public
key
now I mean the blob is encrypted with an
image key but that image key is then
encrypted with
um
the
private key of the machine and for which
you get the public key when you ask your
cloud provider that you want to deploy
on that platform
so on a specific machine
so basically the ultra it's it's classic
public key uh system so the private key
is somewhere in the hardware
and
the the public is public you just
encrypt your image and if yours if your
VM is running you know that it has been
decrypted by A system that has the
private key so
one of this one of these things
yeah okay
okay next question is is there a
question from the internet give us a
sign
no
okay go ahead
uh hi you make the distinction between
migration and swapping from a technical
point of view where's the difference
well I mean for swapping you only bring
out and bring in pages from disk or
network but for migration you need to
transfer the memory and then also the
CPU State and the CPU state is actually
a bigger problem because you need to
have some kind of protocol that
transfers it in a specific way specific
structures and all that it needs to be
Integrity protected and at the end
swapping only does integrity protection
on one page but if you transfer the
whole state you need to have an
Integrity of the whole state for all of
the CPU States and of the whole memory
also you need to guarantee that the
destination is also a proper
[Music]
um
system yeah an authorized host that is
able to run the VM
okay
I see you ask a question please are
there any plans for enlisted
virtualization no no
it's just not possible I mean
VA already have two levels of
virtualization on the Mainframe going
deeper is well we implemented it but we
would need Hardware assistance for that
and more than two levels is
modern our plans as far as I know all
right any other architectures
considering it I don't think so I
haven't heard thank you
I mean of course you have unlimited
levels of virtualization but without
protection
if you want which we don't want
um okay any further questions
okay well let's call this a day let's
have a final big hand for Janos and
Claude York
[Applause]
[Music]
thank you