Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/943 Thanks!
A wonderful good morning to all of you
on day four. I hope you made it
through last night.
Halfway intact.
Our next speaker, Mandy Preston,
who's core def
on the team for me, Arashiro s and
release Miniature for its latest release.
We'll be talking to you about the library
operating systems.
The solution for well, if you're dabbling
in micro services and Docker or trust is
too fat over think.
Give a warm round of applause to many.
Hello. Thank you.
If if you can't read this, now is the
time to shout.
But it seems like it's fine.
So today I'm going to talk to you about
ways that we can use library operating
systems and unique Col's to reject
the default reality or
abstractions and substitute our own.
Quick little roadmap of what I'm going to
talk today, talk about today.
This talk is in the Resiliency Track,
which is new at C.C.C.
this year.
And I wanted to talk about why I
think that the concept of resiliency
is very relevant to the work that we're
doing in library operating systems and
unique journals.
And talk about some of the fact that some
of the things, the properties that a
project might have that lead it to be
resilient, especially as
how that relates to whether a project is
understandable.
I'm going to talk about what I think of
as the ultimate bad dependency in your
project, which is a traditional
monolithic operating system.
I'll talk about how you can replace that
bad dependency with maybe a dependency
that fits better into the way that you
usually handle your dependencies.
And they'll talk about how we can do
how we can actually implement increased
resiliency when we take this approach
by using good software,
using good software tools that we use in
applications for the libraries that we
make the Southwest out of.
So to me, a
really important concept in whether a
given project is resilient
is how many humans can disappear
before the project grinds to a halt.
We have a name for this which you've
probably heard, which is bus number
euphemistically defined by
Wikipedia as a sudden disappearance.
If we want to increase the bus number of
our project, the simple answer
is, well, OK, if I want more humans,
if I want to have a higher number of
humans who could go away in my project
would still go. I should get more humans
involved in the project.
And then the bus number will go up and
that'll be fine.
But in fact, it's not a simple thing to
just say a human who's exposed
to my project is
a knowledgeable or competent person
in the context of this project and a way
that we can help people be knowledgeable
and competent people in the context of
our project is to make our project easier
to understand.
But our project isn't just our codes.
So we can go and we can do a ton of
refactoring and we can say, OK, we're
going to use the best practices that we
know of and document the hell out of
everything and use nice language features
and all that stuff.
But we also have to think about the
language that our code is written and how
we how we actually interact with
our code from day to day.
Like how do we build it?
Where do we run it?
And what things do we need in order to
actually use our code in some kind of
meaningful way?
Our project isn't just like the code that
sits in our repository.
It's also the list of all the things that
it depends on.
And you might be saying, OK, well, my
project doesn't have any dependencies or
has a really small list of dependencies.
I understand all of them really well.
They're all really simple.
And I would say to you, that's probably
not true.
It's very likely that you have, in fact,
a really heavy dependency
for your project, which is
a conventional operating system.
So I've made a nice little ASCII
art diagram of how I sort of how I think
about applications there, dependencies
and a monolithic OS in this sort of
holistic view where you think of it,
where you put all these things in the
same category of stuff that I actually
need.
And in this diagram, I've cheated a
little bit by just saying dependencies,
don't worry about them, but this one's
really big. But in fact, it is really
big.
The operating system is really huge.
It's usually almost always got more stuff
than you need.
And the way that you interact with it is
different from the way that you interact
with the rest of your dependencies.
The documentation is probably not where
you normally look for documentation.
You probably if you have a problem with
it, you probably can't debug it in the
way that you normally debug your code.
And when you're writing code that deals
with it, you are likely accessing
it with an API that doesn't look very
much like the API that you like to use
when you're doing application level code.
So to give you a concrete example of
what I mean by that, this is a selection
from Mantu in Linux.
The.
Signature for the function is here.
So this is socket, which we would call
it, which would be the first thing that
we call if we want to do some network
communication and socket.
Once a magic number and another magic
number, another magic number.
And it'll give us back a magic number or
another magic number to tell us that
we've got an error and we should look
somewhere else for another magic magic
number to find out what the error was.
If you program in languages other than
see, this is probably not what you're
used to seeing for a nice interface
for doing some external kind of kind
of stuff and
worse.
So that magic number that we got back
from socket.
We have another function that we need
to call with it, which is also doing
a whole bunch of stuff with magic
numbers, giving us this weird,
out-of-band error.
Error, communication.
And you might say, well, OK, but like,
if you're doing application development
and you're complaining that the C
interface is bad, that's because it's the
C interface. Like, that's probably not
what you're actually dealing with in
your nice in your higher level language.
But in fact, often you are.
So this is the language that I usually
work in is though, Campbell, it's a
functional programing language with a lot
of nice features.
It's considered to
be a higher level language than C used by
most people.
And I'm gonna give you a quick tour
of how to read or
Kemel type signatures.
So this is the equivalent of socket.
It's also named socket.
So that's helpful.
This is a function.
It's going to take some
some arguments.
And these are there types.
The type socket domain is also a magic
number, but it's the least restricted to
a small to a small subset
of possible things.
Same thing for socket type.
We have a magic number for the
protocol type which in
which in the documentation we at least
are told as zero for default.
And then a number for other stuff.
And we get this, we get this type file
descriptor back, which again is
under the covers and magic number.
And on failure, we get an exception.
And if you've dealt with the exception,
if you've ever programed in a language
that has exceptions, you are usually not
very pleased to see that there is
something that's going to be throwing
them. As a matter of course, when you're
trying to do normal stuff, connect
is even worse.
Connect is going to take this file
descriptor that we got back from socket.
It's going to try to map a socket address
to it.
It doesn't even give us any.
The return type is unit, which means
which is basically which is kind of
equivalent to void.
Like, I just went off and did something.
It was side effecting.
Don't worry about it. Look somewhere else
for errors.
In this case, we might get an exception
which will wrap, which will just
wrap the information from a..
From Unix.
So this is really not
very fun to program against in a camel
either.
And we can see why this why this API
is the way it is. It's just a wrapper for
the C API.
It's got a few nice language features
from a camel, but not very many.
And really, all you can do
when you're in the situation, trying to
trying to interface with the operating
system like this is you can write a
higher level API on top of this.
You can build more and more and more
abstractions to get to something that
looks nice to work with as an application
developer in the context of your
language. But somewhere down on
the bottom, you always have to talk like
this.
And what if you don't want to?
So if we're in open source universe,
you can say, OK, Mindy, if you really
don't like that API so much, go
change it, your stupid self.
And I would say that that is non-trivial
for for most people who
are trying to write code that they just
want to run somewhere.
So if I want to make a change to
how I interface with the kernel when I
want to do something, I first have to
learn to program in the language of the
kernel, which is probably not the
language I use day to day.
I have to learn how to
do that. Do that kind of programing in
the kernel, which is another skill set
that I probably don't have.
I have to make it test.
Make it D. Sorry.
Make a patch. Make a decent patch.
And then somehow get that patch accepted.
The community that I'll have to talk to
to do that is a different community than
I normally talk to. It's not the
community of my project.
It's not the community of my language.
It has different norms.
It has different processes.
And it's probably not very excited
about my patch.
That's for me.
That's for either my application or my
language. Like the colonel doesn't care
about whatever stupid userspace thing I
want to make easier for myself.
Generally speaking.
And if you think that I'm just whining
about doing kernel development is hard.
I would say that it's not just me that
thinks that doing kernel development is
hard. You may have heard of the Deptula
challenge, which was a series of
programing exercises for the Linux
kernel.
That was done entirely by mail.
It started from a base, from writing a
basic kernel module.
And the challenges would be gated by
if you if you would basically send in
a patch bond that said, OK, I've solved
this challenge, please give me the next
one. And then some scripts would sit and
think about it for a little while and
say, OK, I think you did a good job.
Here's the next challenge.
So there are 20 steps to the Deptula
challenge.
There were around 19000
people who started. I was one of them.
Of them, 160 finished.
I was not one of them.
And I would argue that a one percent
success rate of people who start out
specifically wanting to do this.
One thing is probably indicative that
it is, in fact, difficult.
And since since we're talking about
resilience, I wanted to bring up another
topic around this.
So members of your project
might want to limit their exposure to
communities that are known for their
hostility and toxicity.
Right. Like they've decided to be in your
community.
They haven't decided to be in every
community that's touched by
the dependencies of your project
and.
A way that we can protect each other is
to limit our exposure to these
communities that we know might not
be safe for people to be in.
And those projects that are famous
for their toxicity do include some very
famous monolithic kernel projects.
So, OK, that all
seems really bad and kind of a bummer.
So let's think a little bit more about
what operating systems are actually doing
for us.
And I've kind of zoomed out from the
view that I gave you before of
our application, its dependencies and an
operating system. I thought a little bit
more detail in here about how we're
doing this communication,
how we're doing this communication with
the things that are sort of below us in
the way that we normally think about the
stack.
And crucially, I've put in this last
little block here.
Usually when we're deploying applications
in this context where we have where we
have some operating system that we're
running on, the operating system
is actually not usually running on what
we consider a bare metal or hardware.
It's normally virtualize.
So there's a thing underneath it.
The hypervisor that is doing
sort of depending on how you look at
it in this view of the stack, it would be
a lower level management of system
resources and then farming these things
out to individual virtual machines.
So. The view that the application has
is that the OS is taking care of memory
management and scheduling and doing all
the network stuff and file systems, clock
video, all of that.
It's actually the hypervisor
that's responsible for most of this in
this in.
When you're deployed in this in this sort
of environment.
So what the operating system is actually
doing a lot of times is just passing,
passing calls to the hypervisor for a lot
of these things or getting more
core screened resources and then doling
them out on a more fine grained manner to
its individual applications.
So what if we can talk to the hypervisor
directly for
the for and get the same sort of
easy virtualizing interface that the
operating system is getting when it's
some when it's running in this virtual
machine context and then use
libraries to take care of the things that
that kernel code would be taking care of
for us that are at a higher level than
just interacting with the device.
So things like networking isn't
just I need to push this data into a
network card. It's I need to assemble
theses people. TCAP flows.
I need to know how to do DNS lookups,
storage and just take all of this data
and write it to the sector.
Over here. There's a
there's an entire file system layer that
we're usually interested in there.
The logic of how we do timekeeping isn't
just, OK, go get the time,
but we can implement these things in
libraries that are written in the
language that we are used to dealing
with.
Now we know how to deal with
and then ship out the
bits, the bits that we that we think are
complicated in a traditional operating
system because we've been told operating
systems have lots of drivers.
But in fact, in this hypervisor context
or not, we can have the language runtime,
we can write small schnozz for it and
then have the language runtime take care
of for us.
This, by the way, is a unit colonel.
So I'm going to show you some examples
of what these libraries actually look
like and what it sort of feels like to
interact with a system like this
in Meraj of us, which is the
library operating system that I work on.
We have a common set of interface
definitions for for these libraries.
So we have a set of module types
that that tell us.
OK, if you are a if you're a timekeeping
device, you should have these functions
that people can call.
If you're an if you're a low level
network device, I should be able to send
a packet.
If you're a TI, if you're a TCAP
module, I should be able to interact with
you in terms of flows.
And then we have a whole bunch of
implementations of those interfaces.
So specific examples of
file system, specific examples of
networks. This is also where
we have specific ways
of handling networks in different
hypervisor is
so that we can be general when we
are writing our code.
And then just deploy to the target that
we actually end up on.
So let's take a look at what this
function that I've been complaining about
in in this in the men
in our C API and
in the camel interface to
the CPI, looks like in
Rajo US, we have a module type TCAP
that says, OK, I am going to give to you,
I'm going to expose to you this function
create connection.
It takes the.
So this a type signature
is quite a bit more idiomatic,
Kemel, which unfortunately makes it a
little bit more difficult to read if you
are not know Kemel programmer.
But I will try to make it.
I'll try to make it a little bit clearer.
So Kemel, we usually carry
around some context or some state for a
thing in thing
that we often call T.
So this is the current state of the
way that we're going to send stuff over
TCAP.
We're going to take an IP address and a
port, and we're going to
try to make a connection.
The return type is
either a flow that you can then write to
with other functions that are provided by
some implementation of this module type
TCAP or it's some error
which is returned and banned.
And so the the type of this
is the this whole thing, this flow error
result is saying, okay, you'll either get
a flow or I will tell you what went
wrong.
And we have this I decide this is
parameter ised over IO, which is a
concurrency type.
Since this might block,
we exposed some stuff for doing
lightweight, lightweight
coÃ¶perative multitasking.
This particular type signature also uses
something that was fairly
just recently merged into
the Camel
Standard Library.
This particular result type was something
that you had to implement yourself up
until fairly recently in
Camel Fallout three.
It was pulled in to the.
It was pulled into the main compiler,
which means, sorry,
let me back up.
We were able to change all of these
module type signatures, which we had
like like custom error types for all of
them all over Marjoe US.
We were able to unify them with this
result type within within, I
think about an eight months or so
of that being released in the main
standard library, which is a turnaround
time that I think you probably won't see
with most larger operating systems
projects.
The implementations also look like video
medical kemel.
So if we look under the hood of what
create connection is actually doing,
it's it's not making any weird buffer's.
It's not it's not digging around in
in global state.
It is just taking.
OK. Give me your tea connection.
Give me your destination address in port.
I'll make a PCV for it.
If it failed, I'll log that and I'll send
you the error.
Otherwise you'll get the flow that I got.
So the way that we actually assemble
these into things that you can really
use.
We have.
So what I've shown you so far, so far
is conceptually
how we deal with these these
these lists of implementations and what
they are. And the list of dependencies
and what they are.
But what we can do what we do in Meraj us
is we take the application, the
application that the user wants to run,
the list of dependencies, a list of
implementations which they can pick when
they're actually when they're actually
building their thing and then compose
those into either a process
that runs on a traditional operating
system. So we have a whole bunch of shim
layers in these libraries that mean that
when you are trying to develop your
application and you just want to run it
locally, you can do that.
It's fine, or
you can have a minimal virtual machine
that will run in the hypervisor context
that I showed you in the diagram just a
few slides ago, which can run
on several different hypervisor
done.
So I'm going to show you a quick demo of
what it's like to interact with that.
I have a Web page.
Well, I have a I have a Web application,
actually.
It is a miraculous, unique kernel.
It has a list of things that it needs.
This is this is the configuration
document for the unique kernel, which
includes a whole bunch of information
about how how
I want to get my networking.
So this generic stack before is just give
me some IPV for Stack.
I do. The user can decide when they're
building the application, how they want
to get it, where my content
is going to be that I want to build in
HGP server.
I want to use tearless.
I can do some customization stuff reports
where I'll get my certificates from for
the tailless connection.
Some more customization for the HTP
ports.
These are my application level package
dependencies. So these are the things
that you might have in a manifest in
another language
and some composition that says, OK, get
all this stuff together.
I also need from the library operating
system.
I need a post X clock.
I need two places for data first.
First off, the actual data that want
me to serve and then something with the
certificates and then give me
something that knows how to deal with a
GDP and we're good to
go.
And the actual
logic for my application is in this
dispatch file here, which
I won't show you because it's a whole
bunch of o'carroll that is
sort of beside the point.
Here's what I want to show you.
So when I invoke Mirage
Configure, it says, OK, I'm gonna go look
at that at that configuration document
and the list of things that you said
you wanted, and I'll figure out how to go
make that.
So part of what it does is it makes me a
makefile. So now I can just type make.
It's gonna go off, make that application.
He'll make a binary called Made Native.
I can run it.
And I've got a Web page.
Well, thanks.
Let's make a unicorn out of it, no.
So this is the
thing that I just showed you.
When I deploy it, I send it
off to Google Cloud, Google Cloud, once
I get the hypervisor and Google
Cloud wants something that knows how to
talk to video devices.
So I'm going to save Maraj.
Please make me a unique girl that will
thrive in this environment.
Make me one of those.
And know what I have.
Is this HTP desktop video, which I
should show you more fully?
So I have 16 megabytes, which
represents the entire virtual machine
that I need to run that same Web site.
And you might say 16 megabytes is kind of
a lot for, like, all you're really doing
is serving Web stuff.
I have all of the content for that page
rolled up in an in-memory file system
there, which is part of the reason that
it's large.
And I should mention
for fairness that the
binary that we built for Unix,
which is actually a simile.
Sorry. Let me show you this.
Let me show you the size of
the operating system.
The binary that runs on top of the US,
it's actually bigger.
So I hope that I
have maybe convinced you at least to look
into whether a library operating
systems make it easier to do systems
development.
And I hope that you have a language that
you like enough, that you think that it
might be nice to be able to talk to you,
to be able to build some of these
operating systems things in it.
And now I'd like to convince you that
being able to use your language tools
makes it easier to do a good job
of building these modules that you might
want to use for these systems, these
sorts of tasks.
So library
operating systems.
There's a really interesting
implementation of the ideas behind
library operating systems in the Rum
Kernel project, which was a project
in Net BSD to take
the take the kernel drivers
basically to make a net BSD be more
like a library operating system.
And the person who put the time in for
doing this tweeted
fairly recently that his original
motivation for doing this was to make it
possible to actually do debugging and
testing for these kernel drivers, because
once you have these devices in isolation
and you can maybe run them in userspace
or even run them just by themselves
in some other kind of space, it's much
easier to make conclusions
about how they behave in the presence of
certain kinds of certain kinds of input.
So the tests that we write for Meraj us,
they look just like really boring unit
tests for any other bit of camel code
that we might write.
This is this is a test of something
that gets so
that is wrong in a lot of implementations
of TCAP, which is options padding.
It is very, very boring.
A bit slingin code that you want to test
a lot and not screw up, especially
if you're in a memory, unsafe language,
which luckily, you know, Campbell, we're
not a feature
of this code is that it is boring.
And the environment in which we run, it
is also very boring.
We just run it as normal in
our normal test framework.
The way that we normally run tests, we
don't know to set up a special test
harness.
We just execute. We just make our tests
and then execute them.
No drama.
Another thing that we can do when all of
these implementations are just individual
libraries is we can decide, OK.
There's the normal library for doing some
specific thing that is gonna do its best
to implement a spec or whatever.
But for my testing purposes,
I want one that's kind of broken.
I want. I want one that does something
unexpected so I can see what my
application does.
When that unexpected thing happens.
So in Marjoe, US, we've come
up with several sort of fun,
I guess, if you'd like to break stuff,
custom implementations of various things
like network interfaces that always have
new packet's waiting.
The really nice thing to be able to do is
always be able to sort of see about the
random number generator that something
might be looking at. So suddenly all your
tests can be deterministic even
if they're not cooperating with you and
they want to go read from read from some
source that you haven't set up.
In the same vein, you can imagine that
having Interpeace sources that always
block is interesting.
Code application code often
fails in lots of situations, like when
the file system is full or a block device
is busy or
you get short reads from the network, or
you maybe you have maybe your DNS has
been hijacked and you want to make sure
that you don't do something stupid in
that situation when you're when
the implementations that you have for
these things are just modules.
It's really easy to drop in one that's
broken in a way that you think is cool.
And some more stuff that we can do that
is maybe a little bit less obvious.
So in a conventional operating system,
you have a whole bunch of global state.
And the way that we're used to
interacting with this whole bunch of
global state about how the system is
running is we open up a shell and we say,
OK, let me poke around in here and see
what's going on in the context
of a unique kernel.
We can we we have access
to the state that these modules
are keeping track of.
We have access to how that state is
surfaced.
So instead of us having to go in and like
try to think of like, OK, what can we
think of the application is doing
something weird in this way?
What are all of the things that we can
think of in the massive set of global
state in our operating system that might
be affecting how the application is
behaving?
We can have.
I mean, how the operating system tell us
what state it's in. And even more
interestingly, why it's in that state.
We did some experiments
putting using
the Library Urman, which is a distributed
data store that presents
interfaces that look a lot like get
to sort of see, OK.
What if every change we made to the
library, to the operating system state
had a commit message?
So everything that wanted to change
anything about anything about the state
would have to tell us why.
And we'd get a log of why.
And if we wanted to change it ourselves,
we could commit to it and say why we did
it and then see what changes in the
running of us and then our running
application.
Another interesting thing is, so I
mentioned that we have that we have
schedulers that we can swap out well,
what if our scheduler left traces
of what it was doing and we had someone
in our project? Thomas Leonard wrote a
really nice visualizer of
all of the decisions that the scheduler
that the cooperative multitasking
scheduler makes in a Mirage
application saves it to a shared memory
buffer. And then you can examine it
interactively in two dimensions to see.
OK, this thread failed.
Why?
And a couple more things.
So you might say, OK, Mindy,
you've convinced me that I don't need the
operating system anymore, but I still
need a hypervisor if I'm going to do this
thing. And I hypervisor is also not a
tiny bit of code.
Like, that's a pretty big dependency as
well. So what
do you think about that smart guy?
And I say, well, we can do stuff
with hypervisor, too.
There's a really interesting project
called Soldo five, which
has a has a component UK VM,
which takes the device requirement
information that we surface when we're
writing these configuration documents
from Rajo US unique Col's and says, OK,
you said you don't need a network device.
I'm not servicing anything about network
devices to you.
As far as you know, this hypervisor
doesn't know what a network is.
I'll put it there's a link to that
project on the inside.
I think it's a it's a very interesting
project and very cool.
And we have a little bit more ongoing
work.
We have a lot more ongoing work for
opening up more hypervisor so that you
can run more unique kernels on more
desktops.
Some work on taking these libraries and
using them outside of the context of
library operating systems.
There's nothing that says that you have
to use them only in this context.
You can use them wherever you want to, do
whatever you want.
We've been doing a lot of interesting
work with cubes OS
cubes. US is a desktop operating
system. It's running on my laptop right
now. It is powering these slides before
you where the conceit
is that you want. You have a whole bunch
of different virtual machines to enforce
separation of concerns between the
different things that you're doing.
And a problem in Cubes OS is that if you
do all of this with conventional
operating systems, you need
a lot of a lot of hard drive space,
which they solve in part by doing some
sharing. And you need a lot of memory to
run all those victims.
So it's really nice to be able to have
something smaller, both in terms of
size and in terms of memory consumption.
So we've had a whole bunch of small
projects to replace small
bits of things in cubes OS that
run on big fat Linux films
with real tiny, unique kernels.
The most, most widely used and successful
one has been Cubes Mirage Firewall, which
I'm using right now.
I replaced
I replaced the Fedora VM that needed
like I think I got two gigs of memory
or something. No, not that much like five
gigs. Five hundred meg of memory with
that you Colonel, that needed 32.
We have a whole bunch of other projects
that are ongoing right now.
If you're if you're interested and you
want to see the slide later, I can pull
it back up for you.
But your idea can go here.
There's plenty of interesting stuff to do
with library opening systems and unique
Col's. There's plenty of space here, and
I'm interested in hearing your ideas.
I think we have like maybe enough time
for a question, but thanks a lot for your
time.
Thank you very much, Mindy.
We actually have time for one question
and we're going to pick the one
from the signal angel via the Internet
because those guys can't catch up with
you later on.
Oh, no question from the Internet that
we can have one question at microphone
one if you're quick.
Have you explored using
capability based languages, for example?
And there's a project called Emily,
which is a capability based
version of a camel that provides
more isolation between
processes. So you can reason more
easily about of
what parts of your application can you
can use what state.
I personally haven't I had heard
of Emily, but I didn't realize that it
had that interesting feature.
So I should for sure look at it.
Thank you.
Kate. That's all the time we have right
now, if you have more questions, please
catch up with Mehndi outside the lecture
hall. And thank you again,
Mindy, and give her a warm round of
applause.