*36c3 preroll*
Herald: So like many operators, in my group, we actually use a lot of ESXi
servers. You would think that after using these things for 10 years, I would know
how to speak, but I do not. We use these for virtualizing machines. Some of... some of
these actually runs on sandboxes or, you know, run kind of dubious software on it.
So we really do want to prevent these processes from jumping from the virtual
environment to the hypervisor environment. We have today - we have - f1yyy, he wants
to be known by f1yyy, so I'm respecting that; and he's from Triton Security Labs,
and he's going to show us how the exploits that he discovered in the, I think it was
the last Chinese GeekPwn capture the flag. He's gonna show us how these things work,
and was that I would like to help. I would like to ask you, to help me, welcome f1yyy
onto the stage.
*applause*
f1yyy: Hello. Thanks for the introduction. Good evening, everybody. I'm f1yyy a
Senior Security Researcher at Chaitin Technology. I'm going to present The Great
Escape of ESXi; Breaking Out of a Sandboxed Virtual Machine. We have
demonstrated this exploit chain before at GeekPwn 2018. I will introduce our
experience of escaping the sandbox on the ESXi. I will also introduce the work we
have done about the sandbox on the ESXi. Now let's start it. We come from the
Chaitin Security Research Lab. We have researched many practical targets in
recent years, including PS4 jailbreak, Android rooting, IoT offensive research,
and so on. Some of us also play CTF with Team b1o0p and Tea Deliverers. We recently
own the championship at HITCON final. We are also the organizer of the Real World
CTF. We've created some very hard challenges this year. So if you are
interested in it, we welcome you to participate in our CTF game. Now, before
we start our journey to escaping the virtual machine, we need to figure out
what is virtual machine escape? I like to ask some of you that, did anyone use the
virtualization software? If you have used the virtualization software, like VMware
Workstation, Hyper-V, VirtualBox and so on, please raise your hand. Okay, okay,
okay. Thanks, thanks, thanks. Many. So if you are a Software Engineer or a Security
Researcher, you'll probably have used virtualisation software, but if anyone has
heard of the word virtual machine escape, if you have heard of that, please raise
your hand again. Oh, oh, surprised. Thanks, thanks, thanks. It surprises me
that all you know about that, but I have to introduce that again. What's virtual
machine escape? You know, in most circumstances the host OS runs on the
hypervisor and the hypervisor will handle some sensitive instructions executed by
the guest OS. Host OS emulates virtual hardware and handles RPC requests from the
guest OS. That's the architecture of normal virtualization software. And the
guest OS is isolated from each other and cannot affect the host OS. However, if
there are some bugs, or if there are some vulnerabilities existing in the host OS,
it's possible for the guest OS to escape from the virtualization environment. They
can exploit these vulnerabilities. And finally, they can execute arbitrary code
on the host. So this is the Virtual Machine Escape. Then why we chose ESXi as
our target? The first reason is we know that more and more companies are using or
plan to use private cloud to store its private data, including these companies
and the vSphere is an enterprise solution offered by VMware. It's popular between
companies. If you are a Net-Manager of a company, you may know about VMware
vSphere. And the ESXi is the hypervisor for VMware vSphere, so it's widely used in
private cloud. That's the first reason. The second one is that it's a challenging
target for us. There are several exploitations of VMware Workstation in
recent years. Hackers escape from the VMware Workstation by exploiting some
vulnerabilities. These vulnerabilties exist in graphic cards, network cards and
USB devices and so on. But, there has been no public escape of ESXi before, so it's a
challenging target for us and we love challenge. Then why is the ESXi so
challenging? The first reason I think is that there are little documents about its
architecture. The only thing we have found is a white paper offered by VMware. The
white paper only includes some definitions and pictures without details. So let's
take a brief look at the architecture of ESXi first. ESXi is an Enterprise bare-
metal hypervisor and it includes two parts. The kernel, it uses VMKernel
developed by VMware and the User Worlds and the other part, the User Worlds. The
VMKernel is a POSIX-like operating system. And it is uses an in-memory filesystem. It
means that all files stored in this system are not persistent. And the VMKernel also
manages hardware and schedules resource for ESXi. VMKernel also includes VMWare
drivers, I/O Stacks and some User World APIs offered to the User Worlds. And the
word "User World" is used by VMWare to refer the processes running in VMKernel
operating system and the word "User World" means that a group of these processes.
This process can only use a limited /proc directory and limited signals and it can
just use some of the POSIX API. For example, there are some User Worlds
processes like hosted, ssh, vmx and so on. Then this is the architecture of ESXi. I
would like to give you an example to show how a virtual machine works on ESXi. The
VMX process in the User World can communicate with the VMM by using some
undocumented customized system call. And the VMM will initialize the environment
for the guest OS. When guest OS executes some sensitive instructions, it will cause
a VMExit and return to VMM. The VMX process also emulates virtual hardware and
handles RPC requests from the guest. That's how a virtual machine works on
ESXi. Then, how can we escape from the virtual machine on ESXi? If there is a
vulnerability in the virtual hardware of the VMX, we can write a driver, or write
an exploit, to escape from it. The driver will communicate with the virtual hardware
and it can exploit the vulnerability. And finally we can execute shellcode in the
VMX process. So it means that we have successfully escaped from the virtual
machine on the ESXi. So the second reason about why ESXi is so challenging, is that
User World API. The VMX uses many undocumented and customized system calls
and if you want to reverse some code of VMX it is hard for you to understand which
API the VMX is using. But luckily we found two system call tables after compromising
the k.b00 field. There are 2 system call tables we found with symbols so this field
will be useful if we want to reverse some code of the VMX. This is the second
reason. Thirdly, there are some security mitigations here, including ASLR and NX.
It means that we need to link some address information before we start our exploit
to break the randomness of the address space. Furthermore after testing we found
that there is another mitigation on the ESXi. There is a sandbox that isolates the
VMX process. So even if you can execute some shellcode in the VMX process you can
not execute any commands, you can not read any sensitive fields, unless you escape
from the sandbox either. And finally, we think that the VMX of ESXi has a smaller
attack surface. After comparison of the VMX binary between the Workstation and the
ESXi we found that there are some function that have been moved from the VMX in User
World to the VMKernel. For example the packet transmission function in the e1000
net card has been moved from the VMX to the VMKernel. And if you have read some
security advisories published by VMware recently, you can notice that there are
many vulnerabilites existing in the packet transmission part of the e1000 net card.
And all these vulnerabilites only affect Workstation. So we think that the VMX of
ESXi has a smaller attack surface. Now let's start the journey of escaping from
the ESXi. Let's overview the entire exploit chain first. We use 2 memory
corruption vulnerabilites in our exploit. The first one is an uninitialized stack
usage vulnerability which CVE Number is CVE-2018-6981. And the second is an
unitialized stack read vulnerability and the CVE number is CVE-2018-6982. And we
can do arbitrary address free by using the first vulnerability, and we can get
information leakage from the second one. After combining of these two
vulnerabilites we can do arbitrary shellcode execution in VMX process. And
finally we use a logic vulnerability to escape the sandbox of VMX and reverse a
root shell from the ESXi. So that's the entire exploit chain we use. Now let's
start the first one. The first vulnerability is an uninitialized stack
usage vulnerability. It exists in VMXNET3 netcard. When the VMX VMXNET3 netcard
tries to execute command UPDATE_MAC_FILTERS it will us a structure
on the stack, the PhysMemPage structure. This structure is used to represent the
memory mapping between the guest and the host. And it's also been used to transport
data between the guest and the host. Then the VMXNET will call function
DMA_MEM_CREATE to initialize the structure on the stack first, then it will use this
structure to execute this command. And finally it uses PhysMemRelease to destroy
the structure, the physical memory page structure. So it seems that there are no
problems here. But if we look at the function DMA memory create, we can notice
that there is a check before the initialization of the PhysMemoryPage
structure. It will check the argument address and the argument size and if the
check passes then it will initialize the structure. But if the check fails, it will
never initialize the structure on the stack. And finally we found that we can
control the address argument by writing a value to one of the registers of VMXNET3.
What is worse is that in function PhysMemoryRelease there are no checks
about if the PhysMemoryPage structure had been initialized and it just frees a
pointer of this structure. So that's it about it. If we can pad the data on the
stack it's possible for us to do arbitrary address free. We can pad a fake
PhysMemoryPage structure on the stack and then make the check fail in the function
DMA memory create and finally when it comes to the PhysMemoryRelease it will
free a pointer of our PhysMemoryPage structure. So we just try to find a
function to pad the data on the stack. There is a design pattern in software
development, where we store the data into the stack, if the size is small, when we
allocate some memory. And otherwise we will output it to the heap. And we found a
function that fits this pattern. This function will be used when our guest OS
executes the instruction outsb. It will check the size, if the size is smaller
than 0x8000 it will use the stack to store the data. And finally it will copy the
data we send from the guest into the stack. So we can use this function to pad
the data on the stack. Then how do we combine this to do arbitrary address free?
We can use outsb instruction in guest OS first to pad the data on the stack. This
data should contain fake PhysMemoryPage structure and the page count of this fake
structure should be zero. The page array of this fake PhysMemoryPage structure
should be the address we want to free. Then we set some registers of the vmxnet3
to make the check fail in the function DMA memory create. And finally, we order the
vmxnet3 netcard, to execute the command to update MAC filters and then in the VMX it
will use the PhysMemRelease to destroy the structure we pad before. This structure is
a fixed structure with pad in the first step and it will check the page count if it's 0. If
it's 0, it will free the page array of this fake PhysMemPage structure.
So we can do arbitrary address free now by using the first uninitialized stack usage
vulnerability. Here come the next one, the second vulnerability also exists in the
vmxnet3 net card. The vmxnet3 net card tries to execute command get_coalesce. It
will first get a length from the guest, and the length must be 16. Then it
initializes the first eight byte of a structure on the stack. But it's just for
guest to initialize the next 8 byte of this structure and just write this
structure back to our guest OS. So we can link 8 byte uninitialized data on the
stack from the host to our guest. And after debugging the guest VMX process, we
realized that there are fixed offsets between the images, so it's possible for
us to get all the information about the address space by using this vulnerability.
Now, what do we have now? We can do arbitrary address free by using the first
one. And we can get all information about the address space by using the second one.
What do we want to do? We want to do arbitrary shell code execution in the
VMX. So how do we combine these two vulnerabilities to achieve our target?
It's hard for us to do arbitrary shell code execution by using arbitrary address
free. But it's easy for us to do arbitrary shell code execution by using an arbitrary
address write. So our target changes into how to do arbitrary address write by using
arbitrary address free. Then we realized that we need a structure and this
structure should include pointers we can write and the size. So last we can
overwrite this structure. We can do arbitrary address writes usually. When we
first tried to exploit this vulnerability, we used some structures in the heap, but
we've found that we can not manipulate the heap's layout stably because VMX
frequently allocates and releases memory. So we cannot use the structures in
the heap. And after reversing some code of VMX, we have found a structure. The
structure's name is channel and it's used in VMWare RPCI. What's VMWare RPCI?
VMWare has a series of RPC mechanisms to support communication between the guest
and the host. And here it has an interesting name: backdoor. RPCI is one of
them. And the other one we may be familiar with is VMWare tools. I'd like to ask
again if anyone has installed VMWare tools in your guest OS, please raise your hands
again. Oh, not as much as before. So if you use VMWare workstation, you'll
probably have installed VMWare tools in your guest because once you installed it,
you can use some convenient functions such as copy and the paste text fields between
the guest and the host, drag and drop files, create shared folder and so on.
VMWare tools are implemented by using some RPCI commands. And here are some examples
about about some RPCI commands. For example, we can use info-set guestinfo to
set some information about our guest and we can use info-get to retrieve this
information back. Then what happens when we execute this RPCI command in our guest?
For example, if we execute this RPCI command 'info-set guestinfo.a' 123 in our
guest OS. What happens in VMX? It will call VM Exit first and finally it will return
to the RPCI handler of VMX. Then the RPCI handler will choose a subcommand to use by
checking the value of the registers of our guest OS. The RPC tool in our guest OS
will use the subcommand, 'Open' first to open a channel and initialize it. Then it
will use a subcommand, 'SendLen' to set the size of our channel and allocate heap
memory to install the data of our RPC command and suddenly it will use the
'SendData' subcommand to pad the data of the memory we allocated before. And once
the length of the data we sent from the guest re-calls to the sizeof from the
'SendLen' subcommand the VMX will use a corresponding RPCI command handler
function after a string combination. And finally, it will use a 'Close' subcommand
to destroy the channel structure including setting the size to zero and freeing the
data pointer. That's what happens when we execute this RPCI commend in our guest.
Furthermore, there is a channel structure area in the data segment we can use. So
this is a perfect structure for our exploit. Now, you got all the things we
want. We've got two vulnerabilities and we've got the structure we want. How do we
combine this? We notice that the VMX uses ptmalloc of Glibc to manage its heap. So
we just choose to use a fast-bin attack. What's the fast-bin attack? Fast-bin
attack is a method to exploit heap vulnerabilities of ptmalloc by using the
singly-linked list. And it's the easiest exploit method to exploit ptmalloc, I
think. It's also the first method to exploit ptmalloc that I learned when I
just started to learn how to how to exploit. Then after considering the check
existing in the Glibc, we decided to free the address at the Reply Index of channel.
Because by doing that, the Glibc will treat this address as a fake chunk and the Glibc
will check the current chunk's size. And after doing that, the size of the fake
chunk is also the size of the 'channel[N]', so we can set a valid value
to the size of the 'channel[N]' to bypass the check. So we can bypass the check.
Once we've freed this address this fake chunk will be put into the fast-bin linked
list first. Then we can reallocate this fake chunk by using another channel, N+2.
Now we have a data pointer pointed at the reply index of channel[N] and we can
easily overwrite the channel[N+1] by using channel[N+2]. We can send a data to
channel[N+2] and finally it will overwrite some parts of the channel[N+1]. So it's
easy now for us to do arbitrary address write by faking some parts of the channel
structure. Do remember our target? Our target is to do arbitrary shell code
execution in VMX and we can do arbitrary address write now. There are many ways to
do arbitrary shell code execution by using arbitrary address write. We choose to use
a ROP. We can override the '.got.plt' segment. We can fake the channel[N+1],
structure first, overwrite the data pointer at channel[N+1] to the address of
.got.plt segment. Then we can overwrite the function pointer on the .got.plt
segment. So once the VMX uses this function we overwrite, it will jump to our
ROP gadget. So it's also easy for us to do arbitrary shell code execution by using
ROP. So now we can do arbitrary shell code execution in the VMX process. We're seeing
that we have escaped from the virtual machine of the ESXi fully, we tried to
execute some command by using a system call execve, but it fails. We tried to
open and read some sensitive files just like password, it fails again. Then we
realize that there is a sandbox. We cannot execute any commands unless we escape the
sandbox either. The next part come to comes to the how we analyze and the
escape the sandbox. After realizing that there is a sandbox in the ESXi, we reverse
some code of the VMkernel and we find the kernel module named as VM Kernel SAS
control system. And this system, this module, implements the fine grained checks
for the system call. And it seems that this sandbox is a rule-based sandbox. So
we just tried to find the configuration file of this sandbox. We finally found it
at this directory, /etc/vmware/secpolicy/domains, and it
seems that there are many different sandboxes offered by VMWare to the
different processes in the userworld. Like app, plugin and the globalVMDom is a file
for our VMX process and for our VM. After reading that, it's obvious for us that the
/var/run directory is the only directory where we have read and write permissions.
Then we look at the files existing in this directory. We got a lot of pid filess just
like crond, dcui, inetd and so on. And it's also obvious that inetd.conf
configure file is only configure file we can write. What's inetd? inetd is open
source software and it's a super-server domain that provides internet services.
Then we just analyze the contents of the inetd.conf. The content of the inetd.conf
is here on the ESXi. We can find that it a defines two services, ssh and the authd.
And some of it defines which binary will be used by different services. For
example, the authd will be used by the authd services. Also after some testing,
we realize that the authd service is always enabled, while the sshd service is
not. So this is the only configure file we can write. So we got an idea. How about
overwriting this configure file? Or we can overwrite the binary part for authd like
that, we can override the /sbin/authd to /bin/sh. So once can restart the inetd
process we can bind the shell to the port authd is using. Then we just find a way to
restart the inetd process. We analyzed the configure file of the sandbox again, and
we found out the queue system call we can use in the VMX process. Then we just use
the queue HUP to restart the inetd process. Once the inetd process restarts,
we can execute any commands by sending them to the port the authd is using. So
that's the method we use to escape from the sandbox. And here's a demo.
Oh, sorry.
Oh, it seems not, I cannot play this video, but it's OK. You can find it on
YouTube and we created this demo after the GeekPwn 2018, we get a reverse shell
after excuting the exploit in our guest OS. That's all. And if you want to get
more details about our exploit chain, please check our paper here and that's
all. Thanks.
*applause*
Herald: So I don't think I'm actually worthy to share the stage with
f1yyy, that was awesome. If you have questions, we have microphones, you need
to come up to the microphone, line up behind them and we'll take your question.
Meanwhile, does the signal angel have anything? No questions yet. Do we not have
questions from the audience? There is one. Can I have number six, please?
Mic 6: Do you talk to VMWare for this little hack?
f1yyy: We have reported all these vulnerabilities to VMWare after the
GeekPwn 2018, and it has been one year since after they repair it.
Mic 6: OK, Thanks. Herald: That's definitely a relief. Number
one, please. Mic 1: First of all, thanks for the great
talk. I just want to know if there is any meaningful thing a system administrator
can do to lock down the sandbox further so that we can have some preventative,
basically tasks, for our ESXi setups. Or if there is nothing we can do except
patching, of course. f1yyy: Could you repeat your question?
It's so fast for me. Sorry about that. Mic 1: Basically, is there anything you
can do as an administrator to lock down the sandbox even more so that this is
impossible or that it is harder than what you showed?
f1yyy: OK. This is the first question. Your can set the sandbox down by executing
a command on the ESXi shell. I didn't put the command here. I found the command to
set the sandbox down. You can find it by searching the documents about the ESXi.
Wait, wait, wait, wait. I found it, just by myself by using the command offered on
the ESXi shell. It's not documented by the VMWare. OK, I will share this command on
my Twitter later. Sorry about that. I didn't put this command into my slides.
Mic 1: But would this have prevented the attack?
f1yyy: Prevented it? Herald: By doing that change, by doing
that command, would be possible to prevent the attack that you just showed?
f1yyy: The sandbox is used to protect the VMX process. So if you update your ESXi, I
think that it will be safe. Herald: Okay, great. We have a we have a
question from the Internet. Signal Angel: Yes. Does this exploit also
work on non-AMD VTx enabled VMs using binary translation?
Herald: Is it is it more universal than just the AMD-VX?
f1yyy: Yeah, can you repeat that again? I just hear the, okay.
Signal Angel: Does it also work on non-AMD V or VTX-enabled VMs using binary
translation? f1yyyy: Yes, because all these
vulnerabilities exist in the virtual hardware. You will need to use virtual
hardware in your virtual machine. Herald: So any further questions? I'm not
seeing anybody on the microphones. Any further questions from the internet?
That's it then. Good. Please, everybody help me in thanking f1yyyy for this fantastic talk.
*applause*
*36c3 postroll music*
Subtitles created by c3subtitles.de in the year 2020. Join, and help us!