Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2628 Thanks!
[Music]
all right ladies and gentlemen we're
going to have an interesting bugs in the
blockchain brought to you by Louie and
Gabby hello everybody um yeah thanks a
lot for coming it's uh it's still still
early I'm glad everyone could make it um
so our talk is called Don't Panic bite
blocks bugs and I'm Louie I'm a security
researcher I've been working on this
topic of finding bugs in blockchain for
uh almost two years now and I also
focused a lot in the past uh past two
years on the tool development around uh
around our um audit efforts yeah uh good
morning everyone I'm Gabby um thank you
for making it this morning we know it
can be a bit difficult after like 3 4
days of of Camp um I'm a security
researcher and currently I'm focusing on
finding bugs in
blockchains all right so blockchains
they've been in the news quite a lot in
the past in the past couple of years uh
not always for the good reasons a lot of
time it's because a lot of money gets
stolen from somewhere or a lot of money
gets lost and it seems like a lot of
attackers have a big u a big Advantage
uh in the field so as security
researchers we're kind of trying to be
proactive and we want to share with you
today the knowledge of how we find bugs
and which kind of bugs we find in those
uh blockchain systems so that you know
everybody can learn from uh from our uh
um our work so today we'll be going
through five common types of uh bugs
that we find in a specific framework
that we'll talk about a bit later and
how to find those bugs so this is uh
kind of a small shocker bug that I want
to quickly present to show you how
important this um this auditing process
can be for blockchain systems so this is
was an ethereum project that because of
this one line basically crumbled on
itself this wasn't a hack per se it
wasn't like an attacker stealing money
or something it was just one line that
made the whole project collapse in
itself and the everybody that had put
money into the project just um basically
lost everything and the code was brand
new UN audited the the developers said
guys please don't put your money in yet
uh we want to you know finish up stuff
and people still still uh put way too
much money in there and uh basically
lost uh lost everything so yeah just
because of this one line which is a uh
an overflow in a multiplication that
made it so that the developers or I mean
the community um lost control over the
governance of the contract and yeah made
it impossible to recover um Everybody
lost uh lost
funds so as an attacker what can you
gain from blockchains um the first
obvious way is just like stealing funds
if you are able to find some kind of a
bug that um allows you to steal funds
from one address or a contract or
whatever you can just take those funds
or these assets put them on an exchange
exchange them for uh real money and then
um you know walk away with with your
with your gains um then the other way
that we'll also be um looking at today
is that if you able to mess with the
availability of the chain or you know
Make it mess with the with people's
expectations of what the project should
uh act like then you can diminish the
overall value of the chain and if as an
attacker you knew that in advance and
you shorted the the token in some way
then you can also gain some uh some
money this way and this actually might
be even harder to trace than the uh than
the first uh than the first way um there
are a few other ways an attacker can
gain um uh gain money from attacking
blockchain we were not going to cover
them today but these include for example
defi attacks so decentralized Finance um
also and you know attacking the
underlying it infrastructure which you
know uh um Can Happen of course uh
social engineering which has happened a
lot in the past couple of years and uh
yeah also just you know uh uh Ruck pull
uh from the from the from the from the
developers uh Financial
scams um so yeah we'll first be giving
you a uh intro of these of these
blockchains uh then uh we'll uh explain
the five types of bugs that we wanted to
outline to you today these are not
exhaustive it's just five types that we
find interesting and that we think are
uh that you might uh you might enjoy
learning about we be showing you
examples of real world bugs that we find
found in the past couple of years and
then we'll show you how we can use
fuzzing to actually find those bucks uh
proactively and effectiv
um and yeah go ahead Gabby thank you um
so today we are going to focus on the
latest blockchain technology um which is
called the third generation uh but let's
just quickly go back and check the the
other two ones so the first generation
it's um the Bitcoin um basically you all
know it um it allows you to transfer
assets um from one party to the other um
and compared with the other two
generations it's quite uh it has quite a
simp publicity let's say um also
currently there are not too many bugs
that are being found in in Bitcoin and
probably this is also because um it's
quite simple let's say um then the
second generation came out and it
introduced the concept of smart
contracts so basically now you are able
to execute code um on top of the
blockchain so um you can write um your
contract your agreements and they are
going to be executed and stored in the
blockchain um this also introduces the
possibility of creating business logic
so basically companies actually started
to develop themselves um on on the
blockchain um now the idea with the
second generation is that you have the
smart contracts that are all being run
on the same core the third generation
cames out and says okay instead of
having one core um and a bunch of smart
contracts let's have um let's empower
the developers to actually create their
own blockchains um so they can Define
the C themselves um and in this case
basically um we are going to end up with
a bunch of smaller blockchains um each
one of them targeting a specific use
case um but now the question is how do
you make all of them communicate with
each other and actually um being an
ecosystem um so the third generation Al
also introduces the interoperability um
so all of these smaller um blockchains
can communicate with with each other um
also um there are multiple um Frameworks
um that are built um on on the third
generation um and today we are actually
going to focus on um one of them and um
it is called substrate and we are going
to focus on it because basically we have
uh our experience is the most with
substrate so it's just easier for us to
give you real life examples um of bugs
and attacks um but all the things that
we are going to present you here um can
be applied to other blockchains as well
um the team that is building substrate
they are also running uh two blockchains
Paul cadot and kusama and probably a lot
of you um know more about pokadot than
the name of substrate let's say um but
besides um these two projects in the
ecosystem there are another 150 projects
that are using the the framework so
basically now we are faced with the
research question on how do we actually
make sure that all these
150 um projects are actually secure um
and how can we um automate this in a way
because um at this uh at this scale
basically you need to to automate as
much as
possible um let's have a quick look um
at the architecture um of substrate it
has um all the usual components like
peer-to-peer networking RPC Telemetry
and so on but today we are going to uh
focus on the wasm runtime um which
basically um it's the the the core of
the blockchain um and uh in substrate
but basically this runtime um it's being
built using modules um these modules in
substrate are being called pallets um
throughout the presentation we're going
to try to use the term modules so we
don't introduce too too many new terms
for you um and the idea is that
basically when a developer wants to to
create a blockchain um they can either
um use
um like cherry pick actually a bunch of
these pallets that are already um
provided by substrate so you can choose
your consensus you can choose if your
blockchain uh will be able to run smart
contracts or not uh Randomness and so on
um and on top of this you are going to
come up with your own uh modules that
are going to bring um basically your own
business logic um okay so now let's just
um quickly look actually I I I suppose
all of you came here for seeing the
attacks and the bugs so let's just dive
into
that yeah let's go so we'll be looking
at five major classes of bugs that we've
defined uh today so uh attack type a uh
bug type A would be like wrongly uh
priced transactions so if a transaction
doesn't have the right gas associated
with it then maybe we can exhaust the
resources of the chain caus in some some
kind of a denial service block type B is
we're going to look at logic bugs and
unsafe arithmetic to manipulate the
actual program flow this will hinder of
course the Integrity of the chain
because you might be able to tweak a
number that you weren't supposed to be
able to tweak but as we'll see it can
also impact availability because those
numbers might be associated with
availability that the numbers that you
tweak um bu type c will be looking at
the reachable panics uh this is a very
rust specific Buck type that we'll be
taking a taking a look at um and and
buck type D encourage usage of standard
pattern so basically misconfigurations
in the um in the blockchain code uh
because you know we we said that there
were quite a lot of things you could
import and if you import stuff with the
wrong configuration then you might be
vulnerable to uh any kind of uh any kind
of misbehavior and bug type e will be
looking at storage bloating so that's
like inserting a whole lot of uh whole
lot of data into the blockchain storage
and um and cause some uh some
availability
mishaps so first wrongly priced
transactions uh the idea here is that
operations on those substrate based
blockchains are benchmarked and then the
Benchmark uh time basically is
translated into a uh
[Music]
a a gas fee that you have to pay in
order to um to run the operation on the
network so whenever I want to run an
operation a it has an associate G gas
fee of of N and I have to pay n in order
to uh to run a and if a was not well
benchmarked then I might be able to spam
it and then the um the P the node that
tries to run the whole block might just
like not finish running it in time
because it uh it it it estimated how
much it would need to uh how much time
it would need beforehand and if it's
wrongly estimated because the Benchmark
was wrong then you might be able to make
it uh basically um
it's it's not a timeout but at least
you're going to slow the whole thing
down and maybe cause some people to uh
to be um blamed for slowing it down uh
that were not uh not
you so I'm going to talk about a very
cool bug that is related to what I just
mentioned but not exactly what I just
mentioned because this bug that we found
I mean our father found it actually it
applies not during during the running of
the actual operation but during the
Computing of the the weight of the uh of
the gas cost of the function so you can
see on the left this is the um the gas
computation of the of an operation
that's called pseudo as and this Pudo as
takes as an argument a another operation
to run as pseudo and to run as root and
this of course this sud sudo as can only
be run by you know for example a
governance call um it cannot be run just
by anybody it's also used for
development uh for like you know
debugging but in real life chains it
also exist and it's can only be used by
by governance basically but this um the
the computation of the gas cost of sudas
has to take into account the computation
of the gas cost of the child operation
and if you look at the way that the
computation is done it calls the child
computation uh computation function
twice and that means that you can
actually get an exponential number of um
of uh function calls just by nesting
those pseudo as calls or any other uh
any other call that that was vulnerable
to this and there were there were a few
you can Nest them and then for example
if you Nest them 41 times you get more
than a trillion leaves and basically the
node that I is going to try to run that
it's not even going to go to the logic
of the code at any point so it's not
even going to have
time to check at any point that you're
actually who you're claiming to be so
you can call this as just anybody and
it's just going to time out and
completely hang during the weight uh the
weight computation uh so yeah this was a
very cool bug that we found a couple of
a couple of years
ago um okay so if we are going to have a
criminal that is um looking more into
how to manipulate the um the flow of the
program there are two ways of doing this
the first first one is taking advantage
of the usage of unsafe
arithmetic and as an example um here
it's part of a function um that can be
called by any user um now this function
takes um a few arguments one of them
being a vector of messages these me
messages have to be um processed by the
blockchain so before processing them um
the blockchain is going to calculate the
weight of each message then sum of the
weights uh sum all of the weights up um
and then it will um just check hey is
the sum of the weight um smaller than
the maximum weight that I'm supposed to
to to be able to calculate and in case
if it's not it will return an error now
the problem uh in this code is that the
sum function it's being used to um to
calculate this weight and the sum
function is not um using safe
arithmetics so it can actually overflow
so in this casee an attacker um can come
in and actually execute this function
with a large um Vector so basically uh
when the sum is going to be called um
the weight calculation is going to
overflow and drop around and in the end
we are going to end up um with a very
small weight for actually a super large
amount of work to be done so the note
will consider hey I can actually execute
this in the um amount of time that I'm
supposed to but actually it won't be
able and it will time
out now in order to fix the previous
vulnerability you just consider hey I
will just use um safe arithmetic as the
checked function uh but that's not
always fixing the the issue because uh
when you use um safe arithmetic um it
will actually return you an option so
you have to prop properly handle um the
error that is going to be returned in
case of an overflow um in here the
developers are using um the checked
functions which is fine but they are not
properly handling um the case of an
overflow so um the function uh that can
be called here by uh by a criminal um
allows you to um control the begin end
of the message count parameters so
basically if you are going to call this
function with an end value of u64 Max
and the other two ones as zero um in the
end the blockchain will try to allocate
a vector with the capacity of u64 Max
which of course is going to crash the
node um and it it it will end up with
availability issues in the
blockchain now looking into the Logic
box um in here we have um an example of
a blockchain um that basically uh uh a
user comes in and says uh the user
actually has to um validate their their
identity so a user will come in and say
um hey this is my identity then um a
regist uh will pick up the the request
from the user will check it and it will
say yes your identity is um legit or not
um now what the the regist will
specifically say yes this user provided
a valid identity but it won't say
um that it provided this specific
identity it will just say that the user
provided an identity right so in the end
what a hacker can do is um submit a
request um for a legit identity then
monitor U the network and see when the
registra actually comes in and says yes
this identity is legit and um execute
another transaction very quickly um with
a spoofed identity and a higher tip so
basically um in in the same block the
hacker um will uh will change their
their identity with a spoofed one and
the registra will still consider that
identity as being
valid all right now moving on to
reachable panic for availability issues
now this is a uh these reachable panics
it's a bit more complicated now like
there are some changes in the code that
make it uh not so clearcut the nodes are
not just going to panic in in most
situations but sometimes they do and we
we want to we want to include it for uh
um because we think it's still still
very interesting so in Rust because all
of these blockchains and the codes that
you've seen so far are coded in Rust if
you can't handle some kind of a result
uh that that uh that you that you did
you have the option to unwrap the result
and then uh if the result is good is
valid then the program FL continues if
the program uh if the result is invalid
that you decided as a programmer that
the state of the progr program was not
recoverable and you just panic and this
is very useful in a development scenario
because you're you just just don't care
about uh if the result is going to be
there or not and you can just put unwrap
everywhere as a developer and it's going
to speed up your development and you can
come back to those uh to those unwraps
later and maybe add some uh some um some
management of these edge cases but
sometimes those unwraps get forgotten in
the Cod and this was a situation where
in some kind of a decentralized exchange
there was a um a conversion function
that was doing a checked multiplication
and the programmers had forgotten to
handle the the case where the checked
multiplication was returning a nonv
valid result like if if it was
overflowing and it was just unwrapping
and then the whole basically the whole
node might just go boom and uh have to
restart its um restart its
process um yeah so that's uh uh these
are perfect targets for the fers by the
way um that will be showing up in a
bit um okay so in the beginning of the
presentation I was telling you that um
you can create your own runtime just by
cherry picking some uh some modules that
are already provided uh by the framework
um but now when you actually create uh
and import these modules you are also
supposed to um set some parameters for
these modules and there might be a bug
in there actually if you don't set these
parameters correctly um in substrate
each account uh has actually it's
represented by a structure and inside
we're going to find the balance of the
user um some internal counters and you
can also enhance this these structures
uh with more parameters that are
specific to your project so basically
each account um is going to to take some
space in in the storage of the
blockchain um now the idea um is that
you should not be able to to store such
data and have such an account if there
is no token in your account and for this
there is a parameter that is called
existential deposit and basically this
parameter will say hey um this account
is allowed to exist as long as there is
um at least an x amount of tokens in in
that account um now it often happens
that um this existential deposit is
being set to zero or to a small value
such as during the development process
um is just easier for the developers to
to test everything um but if it if this
goes into production then a criminal uh
will be able to come in and create a
super large amount of accounts um for a
very small fee or also for for free um
so basically in the end bloating the
storage of the blockchain and making it
uh run slower and slower in in the long
run all right now we're going to move to
storage bloating as a class of issues
itself I mean Gabby already explained
most of the issue uh already um it's
going to the issue with storage bloating
is that if uh you as a random person try
to join the network in the future with a
full nodee or something if you have to
download terabytes of data to actually
uh make the node run it's not going to
be easy it's not going to um yeah it's
not going to make it uh univers
universally available to uh to to
everybody if you know you have to
download terabytes of data
so yeah so we take these uh this this
class of book seriously when we review a
project um if the fee for creating some
kind of of storage item is way too low
then we uh we report that to uh to the
teams and here uh in this specific
situation there was a um a module under
construction that was uh handling
Bridges to other chains and it was not
uh ready yet but they wanted still to
publish the code without the actual
bridges Part uh implemented so they just
had the registration of bridges open to
the public so that you could
pre-register your Bridges with the
identifications you wanted and then in
the future you would be able to uh to
send out those uh to to to create those
bridges for real the thing is here they
had set the required state for uh the
the require the amount that you needed
for creating a bridge way too low like
this is 1 million and it basically means
1 million of like thousands of a
fractions of a of a fraction of of a
millionth of a scent um so in the end
it's like um almost nothing in um in in
Euro terms so you could just register
spam this bridge registering and it
would allocate quite a big amount of
data in the on the Chain but most
importantly in the future when those
bridges would actually be uh imple
implemented for real this amount of data
would actually probably also go up um so
yeah this uh this is a classic um
storage bloating
issue okay so now that you are aware of
um what type of bugs um are we looking
for in the blockchain you might be
wondering how can I actually start um
finding some of this and there are at
least three techniques that you can use
um two of them are automated and the
third one um it's the manual code review
uh we're going to start with the third
one so basically if you are already
doing uh some manual code review um it
should be quite easy for you um to move
to um to the blockchain world um but you
have to keep in mind um that a different
threat model will be applied here um
because for example um in a in a Android
application let's say um if you have a
crash okay that's fine the the da is
going to crash you just open it again uh
but in a blockchain if you have a crash
that might actually be a critical
vulnerability in the end so um for the
manual code review you just need to to
change a bit uh the threat modeling and
um look for different stuff um then
keeping in mind the five um um classes
of bugs that that we've showed um you
are kind of on a good track to actually
be able to um to start finding this bugs
as as well um mostly um using fuzzing um
and for this we are also going to um to
give you a demo about fuzzing um mostly
we are finding um a lot of bugs with
static analysis and fuzzing um mostly
all of them I would say uh but of course
the manual review is all the time
complimentary all right so very quickly
what's fuzzing and how do we apply to
the to these blockchain project
um we first thing is when you take a
program and you feed it a whole bunch of
of inputs and you wait for any kind of
uh uh misbehavior to happen or something
that you've uh you've uh aligned to to
trigger and in this situation the
program is the blockchain runtime itself
so the part of the blockchain that
executes transactions one after the
other and that interacts with the
storage and everything else and our
fuzzers are coverage guided fuzzers and
they're going to look at the code
coverage inside of the blockchain uh
inside of the blockchain application to
see what's interesting and what's not
and yeah it's it's going to find really
complex stuff quite fast and that it's
uh it really works very well in this uh
in this situation so to make it really
quick the fuzzer is going to have a a
bunch of of potential inputs it's maybe
going to apply some mutations to it um
then the the fuzzing harness will decode
these inputs so like take one input and
split it in a into a few transactions
decode those transactions and then run
them one after the other maybe skipping
a few blocks here and there uh for
timing ISS for timing purposes and then
um and yeah the the target is going to
just run the program and if we find some
kind of an overflow some kind of a crash
some kind of a timeout or any other um
any other pre-programmed um uh values
that we've that we've set to to trigger
um yeah it it's going to it's going to
report a crash and we can um we and
we'll take a manual look um
o all right now time for the demo so we
launched a fuzzer uh bit before the talk
we hope it found the crash that we
wanted it to to find um just to show you
this is the the interesting uh one
interesting part of the runtime code
that we're going to the fuz this is the
the template runtime so like the one
that you're supposed to start with if
you want a normal substrate stack and
these uh Define the the different
modules that you're going to have in
your chain so the interesting one here
is the template module which is as it
says a template module which is defined
right there and in this template module
you have two calls the first one is
called do something it's going to take
an uh a u32 as as an input and it's
going to put it into a storage item the
second one it's called cause error it's
to show you how to how uh how an
operation can cause an error and return
an error in the safe way um and it's
going to take the value that you put
into storage if it was nothing then it's
going to return an error if it was
something then it's going to increment
it now the original code for this is
checked uh a checked addition um and the
the code that we actually uh we replaced
it with to find a crash is just new is
equal to Old plus one so unchecked math
the classic you know classic mistake
that we see over and over again and then
it puts that new value into storage now
let's look at our uh fuzzer for a bit
it's like 300 lines of code it's open
source I'll show you the um I'll show
you
the um the repo in a bit and yeah you
basically decode stuff and then run the
let me show you the running of it you
just run the um run the extrinsic here
and the rest is basically um stuff that
we've added over the years to make sure
that the the the
um to make sure the situation is perfect
for our fzs to find uh crash so let's
look at how the fers has been doing uh
it has found a crash perfect we are
going to be able to show it to you then
and this crash hopefully is uh running
the do something extrinsic first uh to
do something operation with a with u32
Max and then cause error and this Should
Crash the chain because there was an
overflow oh yeah one thing I forgot to
mention is that the fuzzer is uh has
some checks enabled for example this
overflow check because in a real world
scenario like the blockchain would just
make the value overflow and you wouldn't
see anything except for the value that
has overflowed but in our situation we
want to catch those so the fuzzer will
actually um catch those overflows as
crashes all right so uh this is Ziggy
it's our uh in-house manager of fuzz uh
it launches an AF instance or a few I
think there are like 40 of them on this
server and also hongas in parallel and
then they share Corpus uh so that they
can uh they can learn from each other
and yeah let's try to run the the crash
that we
found all right what does it
say it says that it tried to do
something with a value that looks very
much like u32 Max and then it ran cause
error and yeah there it crashed to
attempt to add with overflow so we found
the bug that we had planted there in our
uh in our template and this is a very
easy or uh quite easy way to find a lot
of bugs in uh in many in many blockchain
systems now to show you the impact that
this have had uh oh yeah first this the
the harness itself is open source uh
please go check it out uh on uh on in
this repo the the Ziggy the tool that
you saw that we used to to manage the
fuzzers is also open source and you can
find it there and now back to Gabby um
okay so now you are aware of um what
kind of bugs we can find them um Louie
just showed you that the the fuzer that
we are using it's also open source so um
with these two you are kind of on a good
track to actually start and finding bugs
by by yourself um and in the past one
year and a half uh we analyzed um 45
projects um in the ecosystem and most of
the bugs that we found were actually um
f in a semi-automated way um most of
them using using the fazer um harness
that we we've showed before so um the
idea is that you can actually start and
playing around with this um you can
contribute to the ecosystem um and um
you can do this um out of curiosity of
course or you can even um look into some
of the projects in the ecosystem that
are running um buang these
programs so in the end um while we like
in in the past one year and a half two
years um we developed our tooling um all
the time in order to be able to automate
as much as possible and cover as many
projects um as as we could um but of
course as I said in the beginning there
are over 150 projects in the ecosystem
we covered only 45 so far so um we are
kind of aware that in order to have an
impact we need the the help of the
community um for for this and this is
why we also open sourced the um the fer
and we encourage you to start playing
around with that um you can consider it
a new game a new puzzle whatever get
gets you excited about it um also um as
I was saying about the manual code
review um the ecosystem itself um kind
of needs uh a different threat modeling
than than the usual um Security review
reviews that we are doing um so the
entire ecosystem would benefit if um
both the developers and the um the
security researchers will start um
collaborate more and and share this new
this new threat modeling with with
everyone so with that being said um
thank you very much for making it to to
our talk this morning um and um we are
waiting for your PRS comments or
anything on the open source fer thank
you
we still have some time for a few
questions if you have a question please
line up at your at the microphone over
there signal Angel is the internet
already awake no the internet is still
asleep they do not have any questions
anyone else from the audience with a
[Music]
question I didn't understand the
incentives for paying 15
$25,000
to uh use 1 gab of storage on the
blockchain like what's the attacka model
who would want to do that that's that's
a good point uh yeah I heard other
blockchains right there I guess yeah you
have an incentive to make your
competitive competitor project as uh
hard to get into as possible and it is a
lot of money this 25k for 1 Gigabyte but
if you compare it to other uh to other
blockchains for example ethereum uh you
would need to pay like tens of millions
of dollars to actually put a gigabyte on
that blockchain uh which you know then
the idea is that you make it harder for
other people to uh to to get into the
ecosystem and I mean this you know this
uh this issue was probably low or maybe
medium severity probably low uh but
potentially you could find something
that scales even better uh and where you
can actually go uh go hard on the on the
storage so um isn't don't do don't uh
like asynchronous programming paradigms
and things like this usually say that
you should embrace panics and and
shouldn't like isn't it a major failure
to to try and exclude all the panics
yeah that's an interesting uh an
interesting idea I guess um in that
scenario some uh nodes are incentivized
to uh are incentivized to publish blocks
because they will get some reward out of
it and if you make them if an attacker
can make them unable to produce them
they might even get punished for it so
um yeah I guess in on a theoretical uh
an a theoretical uh way I agree but the
incentives are aligned so that an
attacker wanting to make a note Panic
will actually um you know cause some a
bit of Mayhem in the in the
process thank you for your talk let's
give another round of applause to Louie
and
[Applause]
[Music]
Gabby