Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/335 Thanks!
Can you hear OK?
Yep, I can hear you OK?
This is good.
So we are here's
Mike from the project and I'm s from if
Mike is going to give most of the
presentation, talking about
the a lot of the motivation for
reproducible builds and a lot of the work
that a number of projects have done in
this direction.
And I'm going to talk
about a particular demo
that I think really focuses on the
motivation and why we care about this
problem for security.
So we have a number of people.
Mike, would you like to describe who all
the people are?
Yeah.
So obviously a lot of work has gone
into this problem over the years,
and especially even recently.
We were supposed to have a whole party up
here on stage today.
Actually, we're supposed to have
Kosoff Steiner from the Guardian Project
NA from the Tor Project and
from Debian.
And we were trying to get somebody from
the who worked on the Gideon system and
Bitcoin up here, too, unfortunately,
for various reasons.
Everybody several those people couldn't
make it.
And Moonah especially is
is having some unfortunate circumstances
that we wish him a speedy recovery.
But he has done a lot of this work for
Debian.
So for background, basically
the idea behind Reproducible World is
to sort of close the circle on
this at those are the promise of
free software, the idea that the
users should have all the source
code to correspond to all the
programs that they run that run on their
computer. And the original argument
by the Free Software Foundation was this
was for individual freedom,
the freedom to modify, the freedom to
understand to to work with the things
on your computer.
And the argument was extended to software
security that you can audit.
And as with many eyeballs auditing
all of the source code that people
use, we can better understand as a
community that the vulnerabilities and
the privacy properties and other aspects
of the software that we all use, both
from the point of view of knowing that
the software doesn't have
vulnerabilities and from the point of
view of knowing that the software doesn't
have malicious functionality, that
unfortunately the for
the vast majority of free software
projects today, the only proof that
the bindery project packages that you
download actually correspond to the
published source code that you can also
download is that somebody said so
either this is a trusted institution,
a trusted individual
or an organization
or several a collection of people.
Ultimately, as somebody compiles the
source code and produces a binary that
you download and you take their word for
it.
So you have a sort of social or
institutional connection where you go to
a website and the website says,
here's the source package, here's the
binary package.
But you as the end user don't actually
have any technical evidence connecting
these to just the fact that they appear
on the same website
and without bill information about
the build system, even binary analysis,
sophisticated reverse engineering
techniques using
either pro or even
manual analysis the machine code,
this verification is almost impossible.
There's a very large amount of code
differences even.
And even if the you try to reproduce
the build system as close as you can,
you can still end up with a large amount
of differences which will explain in
later material.
So as a result of sort of inadequate in
really fulfilling this promise and
fostering trust in and the software
as a true functionality and security, and
we'll try to make this really concrete in
a few minutes.
Now, the most common objection from
hackers and such is probably
many in the audience here is, well, I'm
the developer. I'm a developer.
I know what's in the binary because I
don't know the source code. I say compile
it myself.
And moreover, I'm careful with my
machine, with my operational security.
I know what I'm doing.
Nobody's going to own me.
Or maybe nobody cares.
Only why should I have to worry about
these hypothetical risks of somebody
owning me some way?
And why do they even care?
So, you know, to
try and bring this into perspective and
to add a little bit more rational thought
into this, it actually turns out that as
a developer, even though you may think
your actions are benign, they still are
interesting to adversaries.
In fact, you're a very attractive
target for someone who wants to
compromise large numbers of people.
However, Flake actually this year
presented a very interesting work
offline or offensive work and addiction,
where he drew drew parallels
between addictive behavior and addictive
activity and this desire
to compromise machines and progressively
use what you compromise to gain more
access and more access and
use them as stepping stones.
And this is also been revealed to us
through the Snowden
leaks and that intelligence agencies and
institutions engage in this behavior,
too. They see a target, oh, we
want to go after us so and so and so
we're going to compromise any anything
that we need to get there.
Google, Apple, any
large infrastructure, any like software
distribution and the temptation because
software is created with other software
to get the capability to compromise
infrastructure.
The natural thing to want to do is to get
the capability to compromise other
infrastructure and compromise other
infrastructure. And there's a lot of
sensitive trust because all of you, when
developing when hosting software, when
creating infrastructure, are relying on
other infrastructure for that, relying on
other programs.
And there have been even outside of
of the the Snowden leaks have been
known successful attacks by just
hackers against infrastructure used
by several free software projects, the
Linux kernel, somebody to try to
institute a route privilege, escalation,
vulnerability that was thankfully caught
by Linux is careful custodian
or guardianship of the canonical
source code repository.
Also, RedHat Pache had
their distribution's
distribution servers compromised to
distribute malicious binaries and
packages.
And just yesterday, IAC had
their website compromised
to distribute malware to to visitors.
So this is something that isn't just a
threat from state actors.
This is.
It's very tempting for for any attacker,
so the rest of the talk, we're going to
spend some time trying to thoroughly
convince you that this is these sorts of
attacks against software distribution are
very hard to detect, very possible.
And it can be extremely harmful if
the if the adversary has malicious
intent.
So to try and really drill
this idea of
this vulnerability, source of
vulnerability home, imagine, if
you will, your most to the most
secure computer that you could design
or that you could use for.
Obviously, there's probably a lot of
people who have a lot of opinions on
this. And people like OpenBSD will
want to run, Linux will run around app
armor, want to run.
Are the colonel's probably disable Wi-Fi,
Bluetooth, maybe USB, maybe you can keep
the thing off the network entirely.
Just keep it in a bomb shelter, you know,
like in a Faraday
cage.
So in order to bring this back to
reality, imagine you're
in the realm of software development.
Can your most secure computer
that you in your ideal
scenario, can it still be useful
for software development?
So we're thinking about a developer's
laptop where we're thinking about a built
server used by a real world mainstream
software project.
So in the case of many open source
projects, this computer is often
networked.
The developer, it could be built instead
of build servers that several people have
access to, to upload their packages.
It could be the developers laptop that is
mobile moves from hacker conference
to hacker conference is left in
hotel rooms, is
potentially vulnerable to any number of
physical vectors.
And even in the extreme scenario where
you have an institution that's capable
of maintaining physical security over
this computer, you still have to
ferry data to and from it, often
via USB devices or
cold storage.
And there are several vectors.
And just this year, Carson showed that
USB through the USB torque.
There are several vectors for USB devices
to maintain persistence on
Aragao so called Aagot machines.
Even if you're doing things like
reinstalling your OS periodically to
try and wipe it clean and then in
more extreme scenarios, you'll have to
run windows on this on that
machine to be able to build windows
packages.
In the case of the browser vendors, they
do what's called profile guided
optimization, where they actually crawl
the Aleksa top one million and
get all that HTML, all that JavaScript
unauthenticated and feed it through a
profiler on a machine that outputs
a profiler that a profile
organization file that tells another
machine that may be disconnected, how
to arbitrarily rewrite the resulting
binary to make it faster.
So you can imagine a
piece of malware that targets that
optimizer.
Causing the profiling process that a late
stage in the bill process causes a
profile to be generated that causes
arbitrary rewrites of the
binary to introduce gadgets for
exploitation and malware.
So if you're thinking not just of the
source code as a target of attack, but
really the infrastructure that's used to
produce the software that's used to
produce the binaries as a target of
attack, it's a big deal.
So this gets even worse.
So take your most secure computer
that you've been thinking about and
imagine that
not only are all these attack vectors
possible, not only does it have to be
used in these in these risky
use scenarios, what if
it's extremely valuable to compromise?
What if compromising that computer gets
you access to hundreds of millions of
other computers?
What if it gets you in the case of the
browser wonders what if it gets you
access to every bank account in the
world? In the case of software that runs
the financial system, what if it gets you
access to every Windows computer in the
world and maybe you're not even
compromised? The adversary is not even
compromising Microsoft here.
Maybe they're just compromising a popular
dependency for this for Windows
development or something like Flash
that's on ninety five percent of Windows
computers or in the case of
Debian Red Hat and Bunta, every Linux
server in the world.
And then this goes on, then you can carry
this thought experiment further.
If you think about how much that computer
is worth in monetary terms, if it's a
very if you're targeting servers, remote
Odey against
hardened servers can go for hundreds of
hundreds of one hundred thousand dollars
or five hundred thousand dollars or more
on the black market.
If you're talking about in the case of
the Tor project, censorship
infrastructure, Iran and China
spend hundreds of millions of dollars a
year on their firewall.
So disabling something like Tor can
be potentially worth even more.
And if you're talking about financial
software, just the small
a small example is the Bitcoin community,
the current market cap of all the
bitcoins in circulation is four billion
dollars.
Many of those are offline, obviously, but
obviously still very juicy target.
So I don't know if you want to speak
about your conversations with.
Yeah, so I'm not directly involved in
the Bitcoin development community, but
from what I've heard and I'm sorry that
we don't have Bitcoin developers on stage
to describe this firsthand, from what
I've understood from talking to Bitcoin
developers, this became a really concrete
concern, certainly as
the amount of money
involved in the Bitcoin ecosystem has
grown. The idea is basically that
the Bitcoin transfers are irrevocable.
And so if someone can cause
a Bitcoin client to cause
a transaction, they can steal someone's
bitcoins anonymously and irreversibly.
If you imagine malware in
the Bitcoin client itself, the malware
could sort of wait for a year and then do
this at a certain point and
someone might say, oh,
the developer put that malware in there
to steal everyone's money because the
developer obviously had the necessary
access to do that.
And the developer might say, no, no,
someone hacked into my machine.
It wasn't me.
It was a third party.
Well, people might not believe the
developer or the developer has
an incentive to lie about that.
And so really,
the developers inability to
sneak something into the code or to have
someone else sneak something into the
code late in the development process is a
source of protection for the developer.
And I think this really stimulated the
Tor project thinking about this in the
analagous consideration.
I was very concerned actually about
our build engineers traveling through
various jurisdictions with potentially
with their build machines and
cryptographic material.
And and the laptops have produced
a lot of our packages as we're very
distributed organization.
We don't have a lot of centralized,
secure, physically secure network.
When are you offline computers to be able
to build?
So we had the situation where a building
years were traveling and might be subject
to a course of risk demands to put back
doors in software.
And so I really wanted to eliminate that
completely as a possibility.
So I wanted to get people thinking very
concretely about the
relationship between source and binary as
a vector for introducing bugs and
vulnerabilities.
So I have a couple of concrete examples
just to sort of stimulate thinking in
this area.
This is a real bug from back in 2002.
So this bug is now 12 years old.
I mean, it was fixed in 2002.
This is a very typical bug,
a fence post here in the
open SSA server.
And so the idea is that the programmer
wrote greater than instead
of writing greater than or equal to
because this is something that was
counted starting from zero.
And the programmer was thinking in terms
of counting from one
totally common fence post error.
And so this is the fix that was applied
where the condition should have been
greater than or equal to instead of
greater than. Right.
So it got fixed.
So what kind
of change did this produce when the
fix was applied?
Well, I went through the assembly and
compilation process to look at this
because I had a hypothesis about how big
it would be and in fact, the hypothesis
was right. The difference in the binary
is going to be a single bit fixing
this apparently remotely exploitable bug.
So just looking through the compiler
in one case in the vulnerable case
generated an intel jump if lesser
equal in the fixed case,
the compiler generated jump is less.
And if you look at how those upgrades are
represented, it's
a difference of a single bit.
There's also a corresponding case for
greater than or equal versus greater
then, which is also a difference of a
single bit.
Right.
So this is a small excerpt from the
assembly code of the server from the
division back in 2002.
On the left is the vulnerable
version. On the right is the fixed
version. You might not notice the change.
It's pretty small.
That's the change there, right?
And when we compile it, this is a short
excerpt from a 500 kilobyte binary,
and that's the only change in the binary
as a result of fixing this remotely
explainable bug.
So what that means is you have a
particular concrete case where if you
can flip a single bit in a binary,
that makes the difference between
remotely exploitable or not remotely
exploitable by being present or absent.
Now, I also have a demo that I thought
would also make the sort of
trust in the laptop issue concrete for
people.
So I wrote a kernel load rootkit
just to think about how
much do you know about what your system
is doing when you compile something?
So I have here a program that does
something. Is that big enough for people
to read?
I have here a little program that does
something really important, which is that
it adds to numbers.
We could take check some of the program,
maybe a more cryptographically and
checksum.
Right.
So this is the source code of this
program could look at it.
And Ed.
Right. So it takes two command line
arguments and it adds them and outputs
the result.
So I'm just going to compile that
and what numbers? I'll add seventeen and
twenty three.
OK. Seventeen plus twenty three is forty.
So I'm just going to
install a rootkit on my laptop.
OK.
Using the handy install rootkit command.
So I hope you've memorized the checksums
there, because I'm just going to go
through this process again and take a
look at the code, right?
So there's the code.
Checksums are still the same.
OK, there's the source code.
And let's add 17 and 23 having
compiled this.
And so just to point out quickly what
this particular kid is doing,
if I make a copy of the program
and look at this code, it's
totally normal.
But if that copy of Cat were called
Cissie one, which is one of the internal
subprocesses used by the C compiler,
then the colonel would notice that it was
the compiler trying to read the source
code as opposed to some other program,
and it would actually cause it to open a
different version of the source code.
And so that means that what this cat is
doing is introducing a difference
in the content of files
just at the last moment as you're
actually compiling them.
So if you were to look at
if you were to look at the source code
using any kind of tool on your system
other than the compiler, you would say,
oh, my source code is totally correct.
If you look at it, this shot, once
you say my source code is totally correct
in unmodified and there are no changes
on the disk. Right.
The source code on the disk is correct.
The compiler on the disk is correct.
The kernel on the disk is correct.
All of the modifications are happening in
RAM and then you're getting a modified
binary at the last minute
as you actually compile it.
And the compiler is faithfully compiling
what it's given.
It's just that in this case, the kernel
isn't allowing it to access
the proper source code.
It's accessing a slightly incorrect
version of the source code.
Now, of course, that was just a very
basic proof of concept, Brocket.
I mean, you can't expect to defend
against an arbitrary rootkit by renaming
your compiler to something else and then
hoping that you'll be able to or renaming
your editor to something else and
hoping you'll be able to actually see the
true source code.
A more sophisticated rootkit could, of
course, inspect the virtual address base
of what it claims to be.
The compiler make sure it actually is a
compiler and only act in those cases.
Yeah, and so the basic idea of this kind
of rootkit is to say we want to produce
a specified change only in the binary
of a very specific program, only
at the moment that that's actually
compiled so that the developer will say,
OK, I compiled it myself, I checked the
integrity of everything myself.
This binary is good, right?
OK, so
I think it's going to be very hard to
defend against this if you allow the
possibility that the server or the laptop
that you use to actually make your
binaries may have been compromised at
some point or may be compromised at some
point in the future.
So, yeah. So the idea behind building
usability is eliminating that developer's
laptop or any build machine as
a single point of failure, basically
bringing some science into software
development. For once
you have if you give anybody around
the world the ability to take the source
code and produce
an identical binary
to the one that's being distributed, you
then require the adversary to basically
have to compromise everybody.
And this addresses a pretty wide range
of threat models that relate to the
compilation process itself
in the sense that it's not just about
whether the developer is malicious or
whether a third party is malicious or
whether the developer is under coercion.
Right. It's a wide range of things,
any kind of particular threat that might
result in a discrepancy between the
source code in the binary.
And it also turns out that sort of
development practice is actually very
useful as sort of a canary in the coal
mine to help.
You know, if there's something wrong with
your build machines that build, stop
being reproduced faithfully
there. Something's wrong with the
machine. Either is having a transient
hardware failure or some issue with build
process or perhaps has been
compromised.
So you get that external validation
for free basically from your developer
and enthusiastic.
So how hard is this?
Well, it depends on what you're
trying to compile
for sophisticated, very
large software projects, especially ones
that have custom scripted portions of
their build process that can get quite
involved.
The most obvious differences are if the
build machine is configured differently
as different software.
You have either different compilers,
different optimization flags, different
header files, different library versions,
but it can also extend to
build processes that pull in metadata
from the system.
It's very common to include the build
hostname the kernel version.
The file modification times file.
A lot of times
it turns out the debugging formats
for for LTH actually often include
full path to the reference to where the
source code is expected to be.
The container format,
such as thaat Zip and
Zha and RPK
end up with metadata and file
system source data.
You end up with signatures.
You have entropy that can be introduced
by signatures. If you have any signing
process as part of your distribution and
built into the packages, you probably
don't want to allow third parties to
reproduce your signatures, right?
Obviously, you don't want you don't want
to keep some things some something
secret, such as a material
test driven optimizations is another
problem the browser manufacturer or
producers are grappling with, as I
mentioned earlier, where they have to
try and tune their machine code
for the most popular websites.
So who is doing this?
I think the
most the widest,
most public institution that has
that first made this is a process
for all of their bills was the Bitcoin
community. However, back in the nineties,
the the Free Software Foundation was
concerned with this property.
And I mean, if you trawl through the
ancient documentation man pages
it pre info era, even
you see references to how
to do deterministic linking with R
and D.
So these the chain has been thinking
about this for a while, but it just
hasn't been deployed in practice.
A lot of the toolchain reproducibility
work from the 90s was done by
signees.
When signees was maintaining the new
toolchain, they actually had some tests
that involved getting
reproducibility within the toolchain
itself, like compiling the compiler
twice and making sure that the output was
the same.
John Gilmore said that.
That.
Turned up dozens of boats which Cygnus
managed to fix in various parts
of the toolchain, so it's good for
other testing aspects.
But yeah, there was this concern in the
toolchain in the 90s and unfortunately,
it didn't really spread throughout the
operating system.
And applications were
thankfully, that's starting to change.
The Tor project, I said, was inspired by
by Bitcoin to use
the Gideon system, which I'll get into in
a bit.
The Guardian project is also working on
making sure their packages are
reproducible. Debian has somewhere
around two thirds of their packages
currently already reproducible,
at least in one of the branches, either
unstable or testing.
RedHat is working on this and Fedora,
after all, it is has a very neat build
verify verification system in the works,
which will describe in a bit.
Mozilla is interested and we hope that
developers in the audience and who review
this talk online will be inspired
to try and do this for their own software
projects.
So how does Tor browser do this?
Well, I'm the technical
lead on the browser,
and for those who aren't familiar,
basically we have a branch of Firefox.
We have about 40 to 50 patches on top
of Firefox for third party tracking
and fingerprinting defenses and tor
integration.
We have a Tor client and the add on that
helps make configuration of of
TOR. It's very simple for for novice
users.
And then we have these things called
pluggable transports, which are actually
separate binaries that obfuscate
the traffic for the first half to get
around censorship firewalls in China and
Iran and elsewhere.
And we have a couple of add ons as well.
No skipping everywhere.
So the build system we now use,
Gideon, which was developed, as I said,
was developed by the Bitcoin community,
basically. Well, I'll get into what
Gideon is in a bit.
The output from Gideon produces a full
package set and incremental
update files as used by the Firefox
updater, for which
a single shot to fix some file
lists all of the hashes of all the
individual packages and those Choudhry
six. Some files are signed
by all participating official developers.
We also were very interested in
supporting anonymous verifiers.
So people the inputs are actually
downloaded by default over Tor.
So it's very hard to tell who is who is
a verifier. We encourage anybody who
finds discrepancies in our bills to
report them pseudonymously in our bug
tracker and not reveal necessarily
who they are so that we can
make it even harder to compromise all of
the extended community of
verifiers so no one can make a list
of what all the machines
that are testing the reproducibility and
correctness of the bill.
And also very important for us not to
have it require dedicated hardware,
especially not everybody.
We don't want people to have to purchase
a Mac and a Windows machine
just to be able to reproduce those
builds.
And we wanted the all the build
components and dependencies to at least
be free, as in beer.
So at least interested individuals
could download them. Even for the
proprietary platforms.
For example, the Mac OS SDK is
free, isn't beer.
You get binaries, but you don't get
source code. But we're still able to use
that. But the only tools are, of course,
all free as in freedom.
And as far as what those components look
like on Windows, we cross, compile
and on Mac, we cross compile from
a Linux host.
We use mingy W-W
sixty four for the compiler and Windows
we use hydroxy for the python
bits and Nelsa for the installer.
We have a couple of cross compilers for
Mac that were graciously
provided by Ray.
He's very excellent.
We worked in video gaming video,
the video game industry, but is very
interested in making sure he can compile
for iOS and Mac from his Linux machine.
And we use to produce DMG.
We use make a lot of us and DMG
and for Linux we use some newer versions
of the toolchain which we compile
ourselves for reasons I'll get into in a
bit.
So what is this Gideon Beast?
Basically, as I said, it was developed by
the Bitcoin community.
It's a thin wrapper around the Bunta
virtualization tools and that's
Kumu Cavium and see the
Linux containers.
The compilation stages are
well represented in what are called
Yamal descriptors, which are individual
files that specify Kubuntu version
and architecture, a package list to
install on that VM, a list
of git repos to clone
and additional any additional input files
that go in, and then
an inline baskett that just gets run
on that virtual machine as your build
script. And then these descriptors
can be changed. So the file input files
produce go into a compilation process
to produce output files and then those
output files can go in as input files to
the next stage.
So what does this provide?
Obviously, any sort of scripted
virtualization container or container
system can help normalize hostname,
username your build, pass
your tool versions,
your if it's a full, fully virtualize
system, your kernel and your name
output, and then through fake time
can fix the time.
Some timestamp issues.
It doesn't require, as I said, very
important to us. It doesn't require
delicate hardware.
It can be run on any any Linux machine
that is capable of running Ubuntu in some
form and we're extending that to Debian
very soon.
It authenticates your get based input for
you and it integrates with the Fatime
Linux command line utility for spoofing
timestamps.
So some problems that we ran into with
Gideon to start a Kubuntu only,
we've worked out we've put some work into
making it possible to be
hosted from Debian so you can launch your
Dabby, your VMS from a Debian system.
But they're still right now only can
only be a bunch of guests.
So we're still working on that.
If you're using non input, obviously his
named Gideon after git,
you have to provide additional
authentication for for for those packages
and explain how you know what
the input is, how you
identify the right
and parcel compilation.
So it is a little tricky.
Basically the it creates
a base image and then you have for every
compilation run that you do, creates a
queue to copy on write
secondary image that gets written to for
any modifications.
And normally that that
secondary image is destroyed
for each stage to keep the base clean.
But you can hack around and
sort of play with descriptor stages.
And I believe there's a branch that
allows you to use that secondary
portion to do incremental
builds where say you modify a source code
file and tor you can then resume
the compilation process of just Tor,
but it is still kind of clunky and
time-Consuming to sit there and wait for
these builds and end to see if they
match.
It took me about two months of work
to get to our browser building for all
three platforms reproducibly
and most of that was spent sitting around
waiting willcock for
the build to complete.
Maybe you should have done the sword
fighting.
Oh yeah, I
know I did a lot of that.
Various forms, don't worry.
So yeah. So this means it's sort of kind
of janky, you
know, sometimes of process managements
with due process management issues
with QMI cavium
you can only have one a slave
or Aleksi slave at a time and
it doesn't solve everything.
So in the case of Firefox, we
have Python scripts pulling in things
from all of the file system, assembling
them in jars and zip's
you have all sorts of file system
ordering issues. So interestingly, the
reader system call in POSIX
the order of your directory listing
is not specified. It basically comes out
and whatever ordering the innards were
written to on the file system.
So that means machines of different
speeds.
If you have a compilation process,
especially running in parallel and things
are happening at the same time, the order
of those files getting written, that
directory can be different.
And then a script or zip or
thaat that pulls them all in is going to
end up with issue with ordering issues
in. The binaries are going to be
different.
So the answer to that is, of course, you
sort it, but then you run into the
problem. The different locales sort
things differently.
So you have to make sure you set the
locale explicitly to specify
your sorting order.
We run into ran into an initialized
memory. So as I
said, there was Tussin in Cygnus
that made sure that these the toolchain
had no disability issues of its own.
But on some fork's like the cross
compilers, these tests weren't being run.
And Mianzhu means 64.
Introduce a regression where
structure field or the padding in a
structure wasn't being properly Menza
to zero. And so we were getting random
bytes and parts of our windows binaries
with a similar problem in the creation of
the DMG on a Mac OS.
If got some of those random bytes turned
out to be meaningful.
Oh yeah. And we sent it in terrible.
We spent a long time trying to figure
that out. I actually wrote a script,
a little script lit to look for the
surrounding bytes and then just bang out
those bits to like twenty three.
Twenty three. Twenty three.
And people were shocked and horrified
at that. Enough for anonymous contributor
to show me like, no, there's a problem.
And then you say, here's a patch, stop
doing this.
So that was kind of nice. Thank you.
But not now. More of you are
or scruffy round of applause
for scruffy everyone.
Well they pretty much.
That's our anonymous to our blog reporter
who is possibly a collection
of people that reports all manner of
interesting bugs and patches
as well.
So time zone and you may ask, obviously
can leak deliberately generated
signatures, entropy.
And Aleksi and probably Dr. Container's
will have this problem as well.
For people who are experimenting with
other ways to do this, they won't raptor
kernel where there's a
build issue with live GMP, where it
actually inspects your current CPU
and says, oh, I'll build with these
optimizations for you.
You know, we know you have this type of
CPU. We'll just build with that.
Unless you say no, build all the
optimizations and then
so you get differences in different
views.
And he still has memory initialization
issues that we have not fully tracked
down yet on some platforms.
So since we're talking about we
wanted to also protect our developers
from targeted input delivery.
So we want to make sure that nobody could
compromise our are
the way we download inputs and say, oh,
that looks like the download download for
Gideon, let's feed this malicious source
code. And now when they run the configure
script, we can compromise the build
process that way.
Or that looks like the download for
Gideon. Let's try to compromise that
person's laptop.
Yeah. So that we can cause it to say yes,
everything compiled properly
and some projects don't even provide
signatures for us to do this
verification.
So we have to actually hard code
hashes in the build process.
So in case their servers, their files,
FTP servers or HP servers,
our man in the middle or compromised,
that can't be used to affect us.
Or at least we'll know and see that the
shot is different before running
the config strip.
And an embarrassing list of projects
actually have very weak or no signatures.
Open SSL for a long time was signing
with MI five.
Now they signed with 12 keys.
One of them is owned by Frodo Baggins of
the Shire. Or at least that's what the
email address is.
There's another name on it to the
software.
Software authentication problem in
general is quite bad.
My former colleague from EFF, Chris
Palmer, wrote a piece about it
trying to authenticate a PUDI binary.
Pudi is kind of important because you
might use it to log into all your servers
and they haven't done it download
and they have a
HTP delivered signature
that's on another domain.
And yeah, there's
software authentication in general.
It's pretty rough out there.
Yeah, especially in the Windows world I
think.
Well, but luckily
I think Firefox is probably one of the
most complicated things in the world to
try and build this way.
As a result, our build process is massive
and scary.
Most things aren't that complicated,
especially, it turns out, Android.
The most of this material is prepared by
Hans Steiner of the Guardian project,
but he has been working on on
reproducing Android packages.
And because they're mostly pure Java and
the JDK versions are standardized,
it it's actually fairly,
fairly straightforward to produce a
reproducible Android package, at least
to the point where the signatures match.
So for AP KS, I guess Google
realized that the jar format
based on ZIP can pull in all sorts of
metadata and have ordering issues.
And as a result, the app actually only
signs the contents and the manifest
that describes the package
and doesn't mess with the container.
Now the Lansac people in the audience
probably are all cringing like, oh, now
there's this discrepancy between
verification of can exploit be
introduced by the what was happening at
the zip layer versus what's happening at
a signature layer.
I think the answer is probably yes.
In fact, I believe there's was a very
good talk last year on this possibility
where they've demonstrated that those
sorts of discrepancies can introduce
Turing complete machines
that you can use to actually execute
what you want
based on the just the discrepancies
between the grammars of of different
parsers.
That was depends on how the person is
using. Right.
Depends on the nature of how you're using
that.
But so
nonetheless, afterward, it is
moving towards using this property to
allow developers to build their packages
locally, submit the binaries
with their local key signature
to after ID and as well as the source
code URL.
And then afterward, verifier can
recompile the source code, run verify,
which takes the signature from the
developer's binary, attaches it to
the after I built by an area and sees if
it verifies still and if so,
then it can. The verify binaries can
be published with the developer
signature.
So now you solve the problem with Droid
that currently exists where you're
implicitly trust it. We're all whoever
uses asteroid is implicitly trusting
asteroid and their ability to maintain
control over their key.
There's a single asteroid signing
currently. But this reproducibility
system allows you to have to compromise
both the asteroid belt system and
the developer's laptop.
So let's just think concretely again for
a minute about what the world would be
like without this property or
what the world has been like without this
property.
So without this property, you have all
these developers who have their laptops
in there making these APCs and
then they're uploading them into the
system. And then people are saying,
great, it's open source.
So everything's fine and we'll just
give the case to everyone in the world.
And that also, just to think
further concretely about something that
Mike is about to come to, that's
the way that Debian worked until just
a couple of years ago, that each
individual Debian developer had the
ability to make a binary package upload
that was created on that individual
developer's laptop.
And what you had was that developers word
that that binary package matched the
source package and you didn't have a
technical mechanism that was under the
control of the project.
You just had uploads of binary packages
by thousands of individual Debian
developers that they made on their own
laptops.
So having this for any kind
of package repository is really
a major shift in the kind of assurance
that end users can get about where their
binaries are coming from and how their
binaries were produced. It's really quite
meaningful.
So the effort was, I believe,
spearheaded by Boonah and
Hogar, and
they include so there's so many
different kinds of software, obviously
packages in the universe
that they had to patch several aspects
of their build system and supporting
utilities in order to make sure
everything was reproducible.
I mean, this list here, Deb Helper,
CVS D package,
the Python tools, Java tools,
you can Octave apparently had to be
patched and involved in some packages
built system.
They have a wiki page that describes all
these these problems
in detail and is very well informative
and invites anyone to
help out with that effort as well.
And as a result of this, I believe they
have somewhere around 60 percent or two
thirds of their packages currently
being built reproducibly.
And they have a little cool little
jenkins' graph that shows that slowly
has been going up into the right for
the past six or nine months since
but since people started taking this
effort seriously and all the individual
package maintainers have been working
to make their packages reproducible.
And that really protects Debian
developers in much the way that you
described that the reproducibility
protects for developers and produces
a much diminished incentive to
compromise the developers laptop
or to try to coerce a developer to
do something improper in a software
release, at least at the bindery level.
Yeah, unfortunately, we have a situation
where you could find any one of a
thousand people who just left their
laptop at a hacker conference laying
on a table one day wasn't paying
attention and allowed you to stick a USB
key or something in.
And then you could compromise the entire
Debian population just from one of those
thousand people or so that were building
packages.
And now that's no longer possible.
And it's getting to the case where you
won't even be able to compromise their
their central billed system anymore,
or at least that it would be detected,
right?
Yes. In a way that you can send packages.
Now, they took a slightly different
approach to expedite this process.
They strip out differences that were
that they found common to many packages.
There's actually a strip
determinism helper that removes
time stamps and handles common
issues of file formats and normalizes
these things and basically
alters those.
And it's use, I
think, is a large part of what got Debian
so far in terms of all of their packages
being reproduced.
But because it modifies the binary,
it itself can be as a target for
compromise, another single point of
failure.
And you introduce
this trusting trust
attack where if you can
compromise the things that build
determinism that can be used to propagate
a back door for those who are familiar.
The trust and trust attack was first
described by Ken Thompson in the 80s
in his famous paper
Reflections on Trusting Trust.
What he did was took the C compiler
and wrote a backdoor and source code
that went
into a binary version that went out in
one release and then removed that
source code from the compiler source
code. But the back door was such a nature
that likes what Sutt demonstrated,
it could insert itself
into the compiler source code.
So it's a self-propagating vector that's
only visible in the binaries and never
visible in the source, but it can
propagate itself through multiple
generations of the compiler.
And that's still
a considerable concern about the
integrity of our infrastructure.
And there's research,
which you're about to mention
by David Wheeler, describing one
means of addressing this.
And we actually found that the same
attack was actually described
by Air Force researchers in nineteen
seventy three and
that they said, oh, someone could make a
self propagating compromise
in the software development toolchain
and then we wouldn't notice.
And then all of the things that we built
could be compromised.
So this is a concern that people have
identified. Back in nineteen seventy
three and the Air Force researchers
suggested that this would be a powerful
tool for espionage that someone might be
tempted to do.
And I think that's still the case.
Right. So David Wheeler actually
found that what you can do is you take to
independently develop compilers and you
only need the binaries for this.
They can be one of them can be
proprietary compiler.
And then you use those two compilers
to compile the source code of an open
source compiler and you produce two more
binaries.
Now, you have two binaries, one of
which may or may not have have a back
door. And now you use those two compilers
that were compiled from the same source
code, but with different compilers to
compile that same source code again
to produce two more binaries.
Now, those two binaries should be
identical if there is no back door
in or unless
there is the same back door and both of
the binary compilers or there is no
backdoor.
So, David. This is really great, if you
like, saying the word compiler.
Yeah, so Bruce Schneier has a very
good description of this that describes
all the security properties.
But the really important property
is that now the Debian is working on
reproducible builds.
It turns out that, as demonstrated,
this sort of back door can't
or can propagate. It can propagate
between the kernel and the compiler or
the compiler and usually
the tools or the link or anything on the
system. And David,
Dr. Wheeler only proved that
his two compilers didn't have back doors.
He did not rebuild an entire build
environment.
So we also have to trust the operating
system.
Right. We still and he trusted the system
that experiment and concluded there was
no back door and GC, but
we still don't know if there might be a
back door that is capable of propagating
between the kernel and the compiler.
But once with Debian reproducibility,
what we can do now is if
we have a base system that can be cross
compiled to all these other
architectures and it's still
reproducible, you can take the arm, build
PowerPC, build the MIPS, build the intel
build and you can take the hard
kernel, the K FreeBSD kernel,
and you can have them all across
compiling and verifying each other.
And now you force the adversary to
be very forward looking in a way that
they would have to have anticipated.
Oh, we need a compromise, have a
self-propagating back door that has so
many copies of itself, the connected
able to infect every architecture, using
every tool and survives by
infecting every architecture in the
compatible way.
And turns out you can also throw in
proprietary compilers using Dr. Wheelus
technique and then force the adversary
to have to compromise those as well.
So we're getting very close to being able
to actually do this kind of assurance for
essentially the first time ever.
Yes. So we're almost about to be able
to prove that the integrity of software
to the level of the hardware, which
then.
Which, as you know, is then a whole new
set of turtles all the way down, but
we'll get that, too.
I think Bunny Wang is still working on
that one.
Do you want to talk about.
Oh, yes. So then if you want to talk
about the software distribution problem,
I think you.
Oh, well, I think this could be a whole
other talk. But I mean,
there's also this problem about when you
distribute software updates to people,
how do you know that everyone is getting
the same update?
And I think that's a pretty significant
problem in its own right, because this is
talking about how can people
who check verify that
binaries were produced from a particular
source code version.
But we also have this other family of
problems about, well, is everyone
actually getting the same update or
could, for example, the Tor project put
out one version of the browser
and another version of the browser and
have a malicious back door, one of them,
and give that to a small subset of users,
perhaps in a reproducible way,
but still in a way that would compromise
those users. So I think there's actually
for the concept of software transparency
and software development, transparency
generally, there are actually a few other
problems on top of this, not just making
sure that source code and binaries match,
but actually making sure that everyone
gets the same thing.
Now, the core problem here, in a
theoretical sense, is one of distributed
consensus.
Everybody has a potentially different
view of the Internet, and we're
seeing this more and more as there's more
censorship regimes and interception
hardware.
Just because somebody else can download
the canonical Tor browser from one
location doesn't mean that
somebody in China, when they try to
download it, they're going to necessarily
get the same packet packages.
So in order to try and
ensure this, one of the things that we're
planning on doing is having our upset
updates authenticated in addition
to signatures by the Tor consensus.
So there'll be a new URL with a hash
that itself contains the hashes
of all of the update files for the
browser.
And we can further strengthen that
by either storing records of the Tor
consensus in the Bitcoin block chain or
using the Bitcoin block directly, or
using something like certificate
transparency to record the Tor consensus
or the package archives.
And further ensure that everybody
sees essentially the same consensus view
of not just the terror network, but also
our packages.
So we have some links here that are also
in the slides we've just uploaded about
an hour before the talk
to the events
page.
So if you're interested in working on
this problem for your own software
project, there are some useful things.
The Web browser to design doc has
a section on builded security that
describes getting in again, the
Droid verification server.
Again, that's not complete yet.
So if you're interested in Android
development, I'm sure they'd love some
help making sure that all the all
the packages are ready to be user the
system.
And also, I'm sure, would love your help
with any any packages making them
reproducible.
And then if you would like more
information on the diverse
double compilation and how we can get to
true software
or binary integrity all all the way down
to the hardware, these two articles are
excellent. Boucherie give a very succinct
description of Dr. Wheeler's
double compilation and that LWR an
article actually points out, Oh, well,
you need to do not just the compiler, but
the whole operating, the whole operating
system.
I would also recommend Mike wrote
some very readable and
very useful blog posts on the tour
blog about the motivation
for all of this and about
how the project succeeded in doing it.
It's much the same material that you've
just heard, but if you want to see it in
written format, there
are a couple of great posts on the blog.
And again, Thomas DeLeon's piece
Offensive Work in Addiction, talking
about the incentive that an attacker
would have to
compromise infrastructure as a means of
compromising infrastructure, as a means
of compromising infrastructure and the
sort of addictiveness of, hey, I can get
more and more power over information
technology as a whole because of
the trust relationships between software
projects.
Those are contact info if you want to
email or contact.
Anyway, I guess we can open up for
questions if anyone
if you want to line up at the microphones
here, if anybody has any questions for
us, we have about five minute.
First order questions from the Internet
is, can you approach one microphone?
What if you leave?
Please take all your trust with
you and also take other people's trust
with you if you see by your own eyes and
approach one of the microphones if you
have questions.
So, um,
I have one the first question,
if I develop a computer is compromised
and
sticks and
an anniversary
could change a small part of
the code to make the code available
to attack.
In this case, the court could be
compromised from the very beginning.
Why would reproducible build
help to see this manipulation as a
computer would have to compromise code.
I guess that's referring to source code.
I think so, yeah,
you want to try to address that?
I mean, I think one of the problems that
we really do have in
terms of people introducing
vulnerabilities into systems, if
you look at the Obfuscated
V contest or the underhanded
C-code contest, there's an annual
contest for writing malicious
source code that can pass an
audit.
And it's very scary because you look
at some of the results and it's like, OK,
this code here has a malicious
functionality.
That's as follows.
And here's the source code.
Can you see it?
And people have managed to make some that
are very hard to see in the source code.
I mean, I think the best, simplest answer
is the one that the free software
community always gives is that
many eyes.
We hope that by having the software open
and having especially incentive programs,
vulnerability reward programs sponsored
by people who rely on that infrastructure
for their for critical things
to sponsor disclosure,
we hope that those sorts of mistakes can
be found. And also things like get
better source code integrity policies.
There's a lot of projects that just throw
tarballs up on FTP sites and
don't have any authenticated revision
history.
That's very dangerous, I think, for this
sort of attack. So things like GET and
other distributed version control can
help prevent that.
You have to compromise all the get repos
and get it in there or there's at least a
commit record in all of them
that shows that things can produce.
One question from microphone number two.
Right.
So I had a question where
you say that you hard code sha 256
hashes in the build process to sort of
verify that upstream is OK.
Have you considered the possibility of
actually adding in, like
signed get commits and then
having something in the build process
that will verify those based on hard
coded keys?
We do. We do both.
We have the key key rings for
all of the git repositories that actually
sign their comments
and packages that sign
their first releases.
So we use the those signatures
to verify the inputs where we believe
they're strong. And if we think they're
sketchy because there's 12 developers who
can all say the same thing or they sign
with MI five or whatever, that's when we
introduce the SHA hash into our repo
and we also sign the guest tags
that we build from for in the case of a
browser. So when you check out something
to build for an official release, it has
my signature or Gheorghe signature
on that actual commit.
OK, thank you. OK, another question from
the Internet from microphone three.
Um, question number two, what
procedures are there where
they are when there is a failure?
Notice to compromise
keys, block change or how to roll
back multiple steps.
I'm having trouble interpreting
the question, if there is a key
compromise, what procedures
are there is a failure
notice to the compromise
key?
It depends on which key.
I mean, there's a lot of key material.
There's the keys for the inputs
question to get in the direction
of what happens if the key is
compromised.
So that is why we want to extend
the software update authentication to the
Tor consensus in the Bitcoin block chain.
The key is that sign the software that
you download, you have
to compromise all of the builder's keys
because it's now multi signed.
But if you manage to do that,
we hope that the Tor consensus, which is
itself signed by a different set of keys
and can hopefully one day be
authenticated by another consensus
process like Bitcoin Blockin.
All of these systems provide security
like layers of security that
hopefully someone somewhere can say, hey,
this is a signature for this
package claims to be a valid signature,
but yet is signing something that is
not a hash that was published elsewhere.
So far that hasn't happened.
It's possible it has happened and nobody
has told us. But we haven't seen any
evidence of of that with the Tor software
at least.
OK, one more question from microphone
number two.
Hi, great to talk to both of you.
My question is for Seth,
is the install rootkit
program open source?
Anyway, I knew we were going to get this
one.
So.
I wasn't really planning to publish that.
We did give a version of this talk at
Mozilla and one of the Mozilla developers
apparently actually implemented it,
so apparently there is a version from
one of the Mozilla developers.
I'm not sure what it's called, but it
was actually out there and has this
functionality.
But, yeah, this is basically I think
there's a frak article from the 90s that
describes the techniques.
I mean, you're hooking the open system
call in the kernel
to see if the process that's opening
the file is named one.
So it's a really naive implementation of
this sort of thing. Again, real
sophisticated rootkit would do things
like also inspect the current process,
address space to make sure it really is a
compiler and other such things.
So, I mean, I know that people do want to
like referral their coworkers and so on.
So if someone can convince me that
there's a real gap that
existing rockets are just not
suitable, then
we can consider cleaning this up and
publishing it some.
So another question from over here.
Hello. What about
the role of the Hlinka and which one is
usually used of the two?
Linkous from Bin Utils,
I believe we started, I didn't get to
actually describe it in detail.
I believe we switched to the Gold Linko,
which is, I think the new Lakers.
That's the new one. The new one.
So it turned out the old one had an issue
with the implementation of SHA one
on thirty two bit machines where
if it was taking the build ID
for debug link to link the debugging
symbols to a stripped binary, you
can have these things in separate files.
And then there's this building that
contains the shot, one of the
the build to associate
the two that
sha one implementation in the old Linko
had some sort of issue with very large
files. We ran into it on Lib's ruled out.
So possibly because with the bug symbols,
that beast ends up larger than four
gigabytes and we ended up with
random values for that shamash
not different values than the official
sha hash, but random values for
that shot. So we had to switch off of
that the old linker and
start using the new linker for that
reason. I think the new Linko also has
better memory
overhead properties for linking very
large things like Lib's, the Firefox
Library.
So one more question from the Internet
default question.
How do you propagate a vector among
compilers?
You can, uh,
you can use fingerprints, but they are
not for you to future proof
and detecting is a piece of
code is compiler is quite
enforcible from the information
for theoretical people.
We yeah.
So the the basic point is
that it might be implausible
to have a real world vector that
propagates itself through very
heterogeneous code bases, especially
compilers.
And I think that's absolutely correct.
I think that the
more realistic scenario, if you look at
something like Thomas DeLeon's
presentation, is actually sort of
an actively maintained backdoor
where someone is compromising not only in
code bases, but servers
and is periodically logging into the
servers and manually intervening in the
development process.
So I think the real threat scenario would
be that someone has compromised
developer workstations or built servers
inside various organizations and projects
and periodically adjusts the back
door in order to maintain access.
That's the sort of whereas you don't get
and you can even get that ability against
Aagot computers. You just keep seeding
the parking lot with new USB keys that
have the malware on it and they propagate
through all the computers until somebody
plugs into a USB key that goes into the
build server. But you don't get that
property with a compiler because it
has to be baked in that compiler once.
And then all of the binary versions
of that rootkit have to be there.
And you get in the diverse double
compilation, you actually get to use old
compilers so you can go back into your
archives and pull out a compiler binary
from nineteen eighty three and
use that as long as it implements the
subset of C that GCAS
can compile g, c, C, you can use that
one. So I think this area would have had
to back gone back in time to nineteen
eighty three to get their multi
architecture support
introduced for an architecture that was
developed later.
Like I don't know what would be
armed was Armony one.
So I think the intuition of the person
asking the question is that it's very
difficult for a vector to propagate
without human intervention through
heterogeneous compilers.
And I think we completely agree with
that. And in fact, that difficulty
becomes a part of the safety strategy
and that the real attack strategy is
probably more manual on the part of it as
we are running out of time.
One final question from you and please,
the short answer.
Yeah, I would like to know how well do,
uh, multiple threat works was
reproducible, so he used more
than one core to build.
Your binary is just built
in general, still usable.
That is where you run into things like
the file system ordering problem, where
the different make threads
or processes will be writing files into
the directory in different orders between
different machines, depending on how many
processes running.
So we do use multiple
cores on in the browser build process
and we solve that through sorting.
We sort everything before we make any
archive of tar or zip or
jar and during bill process.
And that takes care of that.
Thanks.
OK, thank you very much.
Please get another round off to the.