Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2690 Thanks!
[Music]
hello welcome to the bits and boers
Stage for the next talk please try to
find a place in the
shadows have your water ready and so on
uh yeah I should remind you if you
leaving camp and you still have stuff
you don't want to take back and it's too
nice to trash you can go to infodesk and
try to donate it there and they will try
to do something useful with
it um as uh I guess the rumors have
already spread that there's a little bit
of Co going around it's not more than
what would have been to expect it as uh
such a huge crowd if you have questions
regarding that you can go for test at
thir and there are masks and whatever
stuff available at third heaven and
infodesk
so welcome to the bits and boom stage
for the next talk and for that
I uh welcome hulo on stage and he will
talk about reproducible builds the first
10 years give him a warm welcome and
enjoy the
talk hello thank you
um yeah I'm holar I live in Germany I
work a lot on Debian and since a long
time also work on reproducer builds
since 10 years even though I work on De
I really love Debian I try to make all
software
reproducible and this is a pretty
concept complex toic topic so just talk
to ask me anything anytime I'm happy to
talk about it and I will miss lots of
things in this talk so reproducable
builds um this is not my talk this is a
list of people working on this has been
working on it it's from the web page
it's about 100 people but I think the
list is at least double as
long so who of you has he about
reproducible builds and why it's
useful okay
nice uh who has contributed or is
contributing ah some people
nice who knows reproducible bits have
been around for more than 10
years few for more than 30 years cuz
it's really an old idea actually who has
he about esom files hey ESP bomb is some
fancy new thing the the US government
wants um software Bild of materials and
we have them since 9 years as build info
files they're basically the same um just
the format is a bit
different and yeah we need you we need
people in all Upstream projects and
elsewhere to um make sure that their
software builds reproducible so we need
your help um my it's hot um and the goal
of the talk is to get you involved um
because it's we have done a lot but it's
still a lot to do
um so
introdution um the problem is is that
source code of free software is freely
available but nobody use a source code
we all use binaries and there's no one
really knows how they really correspond
P correspond even those building the
binaries because you need to trust the
computer and the bill chain and
whatnot so there's various Supply um
chain
attacks um and in 2007 somebody
suggested on Debian devil um to make
create reproducible binaries and that
there was a huge threat with 50 males
and people said this is not
possible and the same even happened
earlier in the year 2000 same idea it
was shuted down quite quickly and then
in 2017 17 John Gilmore from Psy pyus um
wrote that GCC was reproducible in the
early '90s on several architectures and
not only GCC but also the whole uh tool
chain there and that bit rotted
away um and now 5 years ago the GCC
people were weren't even aware anymore
that it was reproducible in the '90s
now fast forward to this year in April
there was a mail on the wire guard
mailing list that their VPN client is
now reproducible on their web page the
build is identical in Google Play Store
and F Droid and that's pretty wow and we
were not even informed that just did it
and it's pretty pretty cool I'm really
happy
um
[Music]
so our mission is to enable anyone to
independently verify that a given Source
produces bit by bit identical
results um so we an important building
block in the supply chain to make Supply
chains more secure nothing more and
nothing less an insecure insecure
software is still insecure
software um just distributed
securely and then so that's it has
nothing to do with making the software
more secure in
itself and by now this has been widely
understood we have documented this on
the web page the resources are like 50
talks we've been given on various
conference in the last 10 years doc we
have common pitfalls and problems and
solutions there's about 10 or 15 um
scientific Publications by now and there
was this White House Government
statement in 2001 which requires these
es bomb files for governmental software
and on page 42 they also required
recommended reproducible builds andu
builds really are verified es bombs
because ESP bomb is just a statement
from the manufacturer this should be in
there and everybody can lie about this
and reproducible bills gives you
verified ES
boms
so we've made a definition to really
Define when is a build reproducible and
a build is reproducible if given the
same source code build environment and
build instructions any party can
recreate bit by bit identical copies of
all specified
artifacts
and for attended this to Define what
build environment means what's shortcut
means and this is there on this page so
that we on all on the same page when
discussing
this and how did we get there was money
was a reason and at what
Snowden why money Bitcoin Bitcoin I
don't know when it since Bitcoin is
probably 2009 or something and 2011 it's
older but in 201 when Bitcoin has a
market capitulation of4 billion US
dollar they wanted to be secure that if
there's a melish back door Bitcoin
client which stole steals all your money
that they could not be BL blamed because
the sources are available and it's not
back to so they made Bitcoin
reproducible in
2011 which then was an inspiration for
the tour browser people um and Mike
Perry made tour browser available in
2013 directly after this no
Revelations and that was is Firefox tour
browser and that's one of the biggest
software projects in the world it's like
70 megabyte the binary and it was
possible to make this bit by bit
identical
um and of course it was not only
this but many many people did a lot of
work on
this um so
Luna um fellow Debian developer made a
workshop at decom 13 in 2013 and
gathered 30 Debian people and discussed
whether it's
possible and then there was another buff
at Decon
14 and there are patches for D package
to sort the files
inside deterministic order and create as
a build info file so that the
instructions what is needed what's the
build environment what are the build
instructions to to build a
package and in September 2014 I started
systematic builds of Debian packages we
do them twice at first I just did 100
packages in a wire Loop and then I just
bit wrote a bit of code and made um
tested all of
them and then Mike Perry and S sh
gave a presentation at CCC Congress in
2014 showing my graphs and I was sitting
in the audience did not expect that it
was really
nice and I'll show you some slides of
this talk because it's really nice
summary why it's useful so most people
think um or there a false is that users
should have the source code and build
the software themselves and but they
only prove that the binary comes from
the source as somebody said
so and that is probably not
enough and most people think also I know
that what's in the binary because I
compiled it myself and I'm upstanding
careful responsible I always watch my
laptop why should I have to worry about
hypothetical RIS risk in the contents of
my
binaries um I'm not that interesting but
developers right code which is used by
many people sometimes millions of people
and billions and they're interesting
people so there's been um theoretical
and practical attacks against
infrastructure used by Linux um BSD
anything also has been in the Snowden
papers the designs from the CIA how to
attack this and this has been found in
the
wild and they also showed nice
compromise that is hard to attack um
extremely possible extremely har harmful
imagine the most securest computer in
the world
is this computer still secure if it's
networked accessible by others you put
put USB devices in there um it runs
strange Java code um several nation
states wants to access
it um what if access to this computer
gives you access to hundred of millions
of other computers bank accounts every
Windows computer in the
world and or what money you can gain
from
it um
so they showed a small back door in open
SSH a remote exploit where the diff is
um greater equal or it was greater and
it had to be greater equal and the diff
in the binary is one
bit so if you look at this 500 kilobyte
one bit decides whether this is secure
or not
secure um and that's really hard to find
and that's also why we say everything
has to be bit by bit identical not there
should not be stuff which we can ignore
because it's easier to say everything is
the same they also build a nice kernel
module which modifies the source code um
when the compiler builds it but not when
you look at it with an
editor um so the source code is fine
unless you compile
it um and that was the graph they showed
the green packages are reproducible
packages the are the yellow one uh
orange ones are unreproducible and that
is from 2 end of 2014 till the end of
2015 and you can see there we reached
probably 70%
reproducibility in Dean back then
[Applause]
already so 2015 Luna and myself gave a
talk at Fem invi in the free software
World At Large to collaborate and Tackle
this
problem and here at CCC cam Luna gave a
presentation showing many problems in
their Solutions like what to do with Gip
what to do with tar whatever there's
mostly options
needed and we had a summit in ethens
where 15 projects I think were there to
collaborate and in 2014 we wrote the
sour State EPO specification which I
explain in a second
and we started writing diff
scope so but first the reasons for
unreproducible
we found types of issues and we have
documented them Luna gave this talk to
explain them and we made an
unreproducible package which shows the
problems you can have because it's way
easier to show how not to do it than to
show how to do
it
um and so in the last 10 years we've um
submitted 2 and a half th000 um
reproducibility related bugs um
500 there's a worest 500
Buck 500 bucks are still
open but we also did more uh we also did
submit 20,000 bucks about mostly
packages building failing some SCE so we
found other problems while continuously
testing um that be in this
case and there's more unexpected
benefits of reproducible builds because
if the software builds reproducible then
the compiler can catch a lot of more
things so it's way faster if it's
especially if you build big projects and
so it saves time for the developers and
does money for the
companies and for software development
it's also good to see if the change only
really has the desired effect on this
object on all other
parts and license comp comp compliance
you can only be sure you're running free
software is the binary comes from the
source and the only way to find it out
is to rebuild and have it identical
result and yeah you get ver reproducible
verified esom
also so defos scope who knows about
defos scope
here few few way too too little who uses
defos
scope same people more or
less so defos scope tries to get to the
bottom of what makes files or
directories different it will Rec
recursively unpack archives of many
kinds and transform various binary
formats into more human readable form to
compare them
so it has text and HTML output and
can supports many file formats so it's
whatever Berkeley database Debian
packages elf binaries husu files ISO
images JP images pgp signatures
whatever and if it doesn't um know the
file type it goes back to hexom
comparison there's fuzzy matching to
handle renamings and much more and I
show you a bit different scod
so this is uh https everywhere uh
Firefox extension with two different
versions and you see on the left side
the 56 version on the right the 57 or
maybe you don't see it cuz the light is
shitty
um and it will show there directly the
diff in the file to what's in there so
it takes a tar archive and goes in there
blah blah blah have to be faster I know
um
and
so that also shows you nicely the
divs there's also trioscope where you
can upload two files and Theos scope
will then run for you as is the web page
and you can install it everywhere s
State Epoch
um I skip this because a bit F so time
stamps are meaningless because they vary
at each build so Source state Epoch is
the time of the last modification of the
source code because that is
useful and it's supported by a lot of
software today like 100 softwares all
the compilers many documentation
Frameworks and we have a specification
which is like one or two kilobytes in
size and is very
stable so we had many Summits we'll have
another Summit this year in
Hamburg and there were like 40 Projects
by now um which is really cool to
collaborate which brings me to funding
um we are a software Freedom conserv
project now since 2018 four people are
funded by this we need funding for the
summit this year in Hamburg and we need
funding for our general work thank you
for
funding um I skipped the most of the
things now this is the result for Debian
unstable um for the last 10 years so
it's pretty stable Trix is pretty boring
cuz Trixie is just 2 months
old and this is the numbers for Debian
so
we oh it's harder to read the
interesting part of the unreproducible
packages which are still way too many
1,100 but it's only 3.3% of the software
archive the rest is reproducible and so
each release Dean release we get 3,000
more packages but they are mostly the
same number of unreproducible
packages um
we managed to change Debian policy in
2017 saying packages should be
reproducible I now want packages must
not regret and then new packages also
must be
producible
and then in 27 or whatever all packages
must be
reproducible and I would also like to
see this faster to have it two years
earlier that this year we already don't
allow new packages which are
unreproducible
and this 100% is a political decision
it's not
technical um so we need to change deian
policy and say whatever the last 2% we
just kick
out um deban needs to change the policy
not we
we and 100% reproducibility in theory is
not enough because we need to rebuild us
and for Dean we need a working snapshot
service transparency locks also that we
know which binaries we have installed
also for the non-reproducible packages
this is useful and we also need binary
transparency locks and this is true for
all projects not Debian I skip the other
projects a bit now they're all
um all in theory it's nice even Fedora
has now enabled
finally um in macro to enable um source
that EPO when building RPM
packages so today many projects support
reproducible builds but it's unclear
what it means how it's enforced and how
users can know and be confident because
it's just in theory we can do it but we
don't really do it I call that
reproducible in theory or CI and that is
a massive success this was thought
impossible 10 years
ago um and this is again from this
2014 which is all the others part
infrastructure should be created
independently this has not been done yet
nowhere
really so in theory we are done in
practice we are just done in
theory
um so we need Rebuilders we need to
store the results somewhere and we need
to Define criteria how tools should
treat that data and then we need those
tools
also and those missing 5% are also still
crucial
um at least 300 softwares need to be
fixed to have the software which we all
run so in summary many projects support
reproducibility in theory but it's
unclear what it means in practice and
how users can know and be confident and
this is huge
success
[Applause]
and now we need to finish this last bit
we need create infrastructure for
Rebuilders we need create infrastructure
to processes and tools um to securely
use those
results and we need project level
consensus and commitment to reprodu
builds and
practice thank
[Music]
you thank you hulga I think we have
about time for one very urgent question
maybe
two thank you great talk so you
mentioned at the beginning some caveats
and and like known issues what is so is
one of those the fact that when
everybody runs the same binary one
exploit is to own
everybody well that is a side effect of
course but the result is not everybody
runs scramble
software layers of protection or
whatever you want to add there but it's
not I guess we can take that so uh you
mentioned Rebuilders which are like a
really interesting concept are existing
projects which make it easy to for
example set up a Rebuilder which
automatically rebuilds like maybe old
dban package and some others and then
exposes them in a standardized format so
for example clients can um like have a
list of uh Rebuilders they trust and
then check with all of them if the
package is correct before installing it
we have we have at least two Rebuilder
implementation which are both used by
different people um for Arch Linux is
really nice working for Debian we need
the snapshot service which is not
working so Rebuilders for Debian don't
work at the
moment
for
so with that I guess we have reached the
end of our time and so
I want to thank hulga again for giving
this a nice talk and then enjoy the rest
of
[Music]
camp