Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/684 Thanks!
OK, our next talk is going
to be by her book
in search of evidence based
I.T. Security, and he wants
to do most of the introduction himself.
So this is a very short and brief moment
for me up on stage.
Enjoy and give it up for Hanno.
Yeah, hello, so I
cannot work I am working as a journalist,
and I like to say
I like to avoid the term security
researcher and I hope during my
target will become obvious.
Why that's the case.
Um, I write articles mostly
for Gulam, uh, usually about
I.T. security topics.
And I ran the fuzzing project where
I try to improve the security of free
and open source software.
And this is funded by the Linux
Foundation's Core Infrastructure
Initiative.
And I also write a monthly newsletter
about this.
So, um, as I work
in I.T. Security, I occasionally go
to security conferences and
not just conferences like this one,
but also conferences where you have
a vendor area where people are trying to
sell it, security products.
Um, so I have a few pictures here.
Here someone is selling Next Generation
Apte defense
because, um, someone is selling
something with artificial intelligence.
Um, someone is asking if everything
is moving to the cloud.
Why isn't your security.
And this vendor is saying the
only vendor with guaranteed protection
from ransomware.
And when I see these things,
I am a bit skeptical.
Um, I'm not sure I
feel many of the towns don't have a real
meaning.
They feel like marketing terms.
I don't really know what they are doing
or if if I know what they're
doing, it doesn't feel right.
Um, and I'm not
the only person skeptical about its
security products, so I don't know if,
you know, this guy was Travis Omondi.
He's working for Google and lately
he's been looking at security
products.
And what he found was that many security
products are not very secure.
So, for example, he found
out that Hamas was using some open
source code and they replaced the
stolen copy with copy
and introduced a buffer overflow.
But it's faster.
So great.
Um, Trent Microaggression
did accidentally left a remote debugging
server running, um,
or Palo Alto Network had a memory of
corruption because they shipped a Web
server that was no longer supported.
Um, and here he was trying to
contact Avichai.
And also your code makes zero
sense.
Um, yes.
And we have headlines like this
for, uh, PC versus antivirus
software could make your company more
vulnerable.
And on the upper right,
antivirus tools are useless.
Box ticking exercise.
First, Google security check.
And here are two
tweets where there was recently a quite
heated debate about the value of
antivirus software.
A trust issue is a Chrome developer.
He compared antivirus to homoeopathy.
And, um, April
King, who is from Firefox, said antivirus
because of security issues for Firefox.
Um, and this is from, uh,
where Google asked, um,
users and I.T.
security experts what they think is
are the most important things to do about
it. Security.
And you can see the users had antivirus
software as the very first thing.
Uh, and the security experts don't seem
to think that's so important.
It doesn't even show up in the top five.
So
we can conclude that there's considerable
disagreement, whether it is security
products and especially antivirus
software is actually a good idea.
Um, so
how do we know actually if these things
work and
to investigate, then I'd like to
talk about something completely
different.
So this industry likes
to use medical analogies.
We're talking about viruses.
Viruses are usually something
from medicine which affects people.
And here's another form of antivirus.
It's a vitamin C pill.
And here's a person having a common cold
sneezing.
And yeah, many people think
it's a good idea if you have a common
cold that you should take vitamin
C pills.
Um, unfortunately,
it's probably not very useful.
Um, and why
do we actually know that?
Um, obviously we know this
because we have science.
We're using science to investigate
whether things work.
And for the vitamin C,
here's some quotes from a study
which says, OK, regular ingestion of
vitamin C had no effect on common
cold incidents in the ordinary
population.
So if you're like an average adult person
and take vitamin C,
you're just as likely to get a cold than
everybody else.
However, it may be that it broadens
the duration of your cold a little bit.
But if you take the vitamin C only once
you already got a cold and it has
no use at all.
And the study here is from the
Cochrane Collaboration, which is an
organization that's doing a so-called
meta analysis.
And I will come back later what that is.
But it's generally an organization.
I think there's widespread agreement that
the Cochrane Collaboration is creating
some of the highest quality scientific
evidence in medicine.
Um, so if we want to
know if, uh, medicine
or also like something like a food
supplement, like about pill works,
what's usually done is a so-called
randomized controlled trial.
And that is where you take a group of
people that may have some
some illness.
And then we split them randomly into
groups. And it's it's crucial
that this is done randomly because we
don't want to have some
statistical thing that we chose one
group that is maybe
more sick than the other group to begin
with and then screws
with our results. So we need to split
them randomly into groups.
And a simple way would be, OK,
one group gets medication, the other
group gets a placebo, and then we see
what happens.
In reality, it's usually more complicated
because usually we have a situation
where we already have a known good
medication and we have a new medication
and we just want to know if the new
medication is better.
So we compare an old and a new medication
and we also may have an
alternative to medication like
exercise or dietary changes.
And you may want to know, OK, maybe
we have a medication that works, but
doing exercise works even better and
maybe taking both the medication and
exercise at the same time works even more
better. Um, but this the general
idea. So we randomly split people
into groups and test what happens,
but then we usually don't really care
about a single study because they are far
too many things that can go wrong.
So what we usually care about is all
the scientific evidence we have as a
whole. And that's why we're doing a
meta analysis, which is we're
trying to search for all the studies that
have been done on a particular topic,
ideally randomized control trials,
and we combine the results.
This is obviously sometimes
complicated because we might have studies
for different groups of the population.
They cannot always be easily compared,
but that's the ideal idea.
So we have many studies and then
we combine the results and look at the
whole body of evidence.
Um, yeah. So we call that evidence
based medicine.
So ideally, we want
to make all decisions based on high
quality scientific evidence,
which very often a meta
analysis qualifies.
And now
I want to point out that
one shouldn't have a too idealized view
on science because there are many
problems, um,
here. The top one is actually
the most popular open access paper
of all times, which is why most
published research findings are false.
It has been published in 2005,
and that's actually not very
controversial. So, um,
and the middle one points to an issue
that has been debated in recent years
a lot, where there was
a big experiment to try to replicate
studies in psychology.
And they found out that they were
only able to replicate
the result of 37 percent of the studies.
So the majority of studies, it seems
either they are wrong or the replication
was wrong.
But it seems there's a problem.
But this is not only affecting
psychology.
The you have the same problem in many
fields of sciences.
For example, there's a very similar
debate in cancer research.
Um, and the lowest one points to
something that many clinical trials
findings never get published, which
is also a very important thing to
consider, that the science we're
seeing is not all the science that
has happened.
We very often have a situation where
people do a study and then
based on the result, they decide
whether it's interesting and gets
published or whether it just gets
thrown away.
So, yeah.
So if you want to
evaluate what's good or bad science,
then there are some things we
can look at.
Something very obvious is if we have a
very small number of research subjects.
So sometimes you see studies where people
say, OK, we've tested this with 10
people.
And I say, OK, that's maybe not very
meaningful. It could be just coincidence.
Could be a statistical glitch
then, which is a very common thing, not
only with the quality of the science
itself, but also with the reporting
about science, like when the media
reports about it, is that
correlations are reported as
if they were causal results.
So what's happening here is if
we have a set of data and we may find
out all the people who have property
also have property B, then
we could conclude, OK, A causes B,
but it could also be that B causes A,
or it could also be that there's
a so-called confounder, which means we
have something completely different that
we may not even know about that's causing
both A and B, so
and this is generally a problem
in all studies where you're using an
existing dataset and trying to find
something in it.
And that's why we are doing these
controlled trials where we are split in
groups randomly into two groups
so we can exclude that there are some
some other factor that's happening
here.
Um, and then, yeah, sometimes we
only have a single study or very few
studies.
So we usually want
good science to be based on many
studies. We want science to be replicated
independently.
Um, and
then we have a thing that's called
publication bias.
And that's what I mentioned earlier, is
we don't see all the studies that are
done.
And we may have a situation where
a pharmaceutical company makes a trial
on a medication and it turns
out the medication doesn't really help
and then they don't publish the trial
and then they do another trial.
And there it seems like the medication
helps and then they publish it.
And you can see that, like if you only
see the positive studies and you
don't see the negative studies, then
and then you try to combine these
results, like in a meta analysis,
then you get a skewed result.
And another problem is
called outcome switching.
And it's kind of related to fishing for
results, which is you may have collected
some data and but
it doesn't match your theory.
But then you could try.
OK, maybe if I
just use a selection of
my data, I might may
maybe I can prove something similar to my
theory.
And if you look long enough, you can take
some random data and you will find
something that looks like a scientific
result.
So it's not generally a
problem to do this, but it should be
transparent about it.
If you were first searching for something
and then later you're searching for
something else, you should at least make
clear that you did that.
And ideally, you would want all
of these studies that are somehow based
on statistics, that are empirical
studies. You want them to be
preregistered, which would mean that
you would publish before you even start
collecting the data what you're about
to do. So you could say, yeah, I want
to study this medication.
I'll do a randomized control trial with
these groups.
And then I publish that in a trial
register because then if
you later changed what you were studying,
then other people can see that.
So it's transparent,
but we're very far from that in medicine.
This is happening usually it's
still not ideal.
There are still a lot of problems with
this, but in many other fields, this is
not happening at all.
OK, now let's get back to I.T.
Security.
Um, here's an empty slide and
it's not a mistake. It's intentionally
empty because it's also the complete
list of all randomized control trials
that have ever been done on security
software.
Yeah, um,
there are some people who are doing
something that may look a bit like
scientific tests of antivirus software,
but I fear the methodology
that's used there is extremely flawed.
Um, so what they usually do
is they they have a collection
of malware, which is hopefully
somewhat representative of real malware,
and then they try which software
detects it and which not.
Um, this is a lot of problems
because, for example, if you detect them
measure, that does not mean
if you wouldn't detect them ever, that it
would infect the user.
It could be that the ever tried to
use a browser exploit and
the exploit is only in an altered version
of the browser that the user is no longer
using.
And they usually completely fail
to to consider the idea that you
could do something else then antivirus
software to protect yourself, like
you could use regular updates and
application wirelessly.
Um, so these tests kind of
have the idea that antivirus is
the only thing you can do and the only
thing that matters is comparing different
products against each other.
Um, they usually do not consider that
antivirus software itself could be a
security risk.
But most important of all, they
usually don't test with real users.
They are testing in some kind of lab
condition where they say they are
simulating what a real user is doing,
but they are not testing with your users.
Um, and it's it's quite widespread.
That is the question of forms of
statistics in I.T.
security.
Uh, one very notorious example is also
CV counting when people say so
severe is ah, I don't know if everybody
knows that these are identifiers for
security vulnerabilities.
And what some people tend to do is say,
OK, Windows had that many CBD, Linux
had that many CBD.
So clearly Windows is more secure than
Linux.
This is completely flawed because these
severe identifiers don't
even try to be complete.
And if you don't believe me,
there's a talk from the guy who invented
the CVA I.D.
where he thinks you shouldn't do these
kinds of statistics.
Um.
So
my feeling is that security
is largely not bias based on scientific
evidence, and it's a bit of something
that bothers me because I work in I.T.
security and I'm a very scientifically
minded person. So when someone tells
me, hey, this is healthy, then
I say, do you have some studies to show
me? And if you don't have to study it,
then I don't believe it.
And at the same time, I'm working in a
field where if I ask this question, the
answer is just the evidence is not there
very often.
Um, and now you
might say, OK, but aren't there plenty of
scientific papers and conferences on I.T.
security?
And, um, here's
a list of some of the most cited papers
and a quick remark on that.
Like counting the citations of papers
itself is a very controversial
thing,
but I cannot go into that.
But there's a whole debate about whether
you should use something like an impact
factor or whether that's a bad idea.
But at least I think it tells us, which
are the scientific papers that other
scientists care about.
And this is a list from Google Scholar,
from papers from 2012 and 2013.
So the first one here says Candidate
indistinguishably obfuscation and
functional encryption for autocrat's.
Now, I could ask if we have an
average user who is using the Internet,
writing emails, using a Web browser,
using Facebook, whatever.
Um, how does this matter for
him?
If you have an answer for that, I would
really like to hear you talk to me later.
And I think you could ask a similar
question for all of these papers.
Um, I had to go to
No.11 where I found something
that sounded like it was actually about
real software.
That was a paper about Android malware.
And also at number twenty,
I found another paper that was about real
software, which was the lucky 13 paper.
And that one made me kind
of question myself, because this is the
kind of paper that I usually care about
because I do a lot of crypto stuff.
And this is a crypto attack.
Um, it's a timing attack.
And it's really hard to pull that attack
off.
Um, and
it's so hard that I am almost certain
that this attack has never been used
in the wild to attack a real user.
But these are the kinds of papers
we find interesting because we say,
oh, they were able to pull off this
interesting attack.
That's great. That questions all the way
how we did encryption
and it had a pretty big impact.
Yeah. I'm also proud that I found a
little mistake in that paper.
Actually, it's not very important,
but so.
Yeah, so I
yeah. But in this whole list with
26 papers that were the most cited
papers, there was not a single paper that
was doing anything with real users.
What they were trying to see what's
happening when real users act with
the Internet, do something about
security.
So it seems the users not really
something a security researcher cares
a lot about.
Um, so my feeling is
most academic research in I.T.
security is comparable to basic research.
When we talk about, I don't know, human
morphic encryption are indistinguishable
obfuscation.
And these are crypto theories that may
lead to some interesting products for the
future.
And that's fine.
I mean, basic research is totally fine,
but I feel we completely missing the
applied research.
And if we do like the more if we
look at the more practical research, I
feel it tends to go into interesting
problems, but not the most important
problems, which is also
kind of fine. But I feel there's
a whole big area we are missing here.
So what would we do
if we would say we want to do a
randomized control trial on a security
software because the OK, get a
large group of users and randomly
split them in groups?
We have some groups that use some
different I.T. security products.
We could say one group uses some
alternative treatment, which could be
something like applying regular
updates and doing application wide
listing, which is generally considered
the most viable alternative to antivirus
software.
Um, and we can't say one group
gets a training where we say, OK,
don't click on these email attachments.
Um, I have to say here, I don't
think training users is a very,
very good strategy, but I think we should
test that anyway.
And then we could have a placebo group
where we say just do the same thing you
did before.
Um,
and then we try to measure security
incidents, which maybe
be. To even decide when a security
incident happened,
then we could also try to measure
what side effects does this have to
things crash? Do things get slower
or what does it cost?
We have some downturns and
then after some time, we compare the
result.
Now, I have discussed
this with a number of people before I did
this talk. And the first reaction
that usually comes as some form of this
is really harsh and there's this problem
and that problem and that problem.
And I totally agree.
This is really hard.
Science is hard.
That's just how it is.
But it doesn't mean we shouldn't do it.
So some problems that would show up
for you could ask, what about the ethics
of such a trial? Because you would say
you give some security products to some
people and not to others.
So do you put them at risk?
But if you think about it, that's a very
comparable situation to medicine.
If you test a medical track and
then you give the drug to some people and
you don't give it to other people.
So it could be that this drug helps some
people and doesn't help the others.
But it could also be that this drug has a
risk and that people
suffer from taking that drug.
But we generally have the idea in
medicine that if we don't know whether
drug helps, then testing it
is the ethical thing because it will help
many more people in the future.
Um, yeah.
Then we may wonder how do we
reliably measure what's an incident?
Because we may have situations where
it's not even clear what was
a heck or you have been hacked
and you don't know about it.
And you probably get
very different results whether you
have someone who is just affected
by the normal everyday
Internet of thing or someone
who is targeted, someone
who is targeted by a professional
attacker and many
more. I'm not saying it's easy, but
there are many problems to be solved.
Um, and, uh,
I think the the security applications
and antivirus software are just an
example here, I think there are many
things that we could test for such tests
like there are debates about the safety
of programing languages is
rather better than C++.
I think so. But I would like to see
studies on it or application
security like this browser, a
more secure than browser B could
be tested.
Um.
And finally, I want to bring up an
example, which I think in some sense
both a good and a bad example.
So this was a tweet from the
FTC, the Federal Trade
Commission in the US,
where they say, encourage your loved ones
to change passwords often and some
other things.
So, um.
Oh, um,
yeah, so at some point they found out
they actually had no scientific evidence
for this recommendation to change
passwords often, and they tried to find
out why are we recommending this?
And then they said, OK, we were
recommending this because we are doing it
ourselves. So it must be good.
Um, um.
And then they basically recommended
the opposite. They said, OK,
we we have reconsidered this.
We looked at the evidence.
We have some studies that say that
mandatory password changes are not a
probably not a good idea and maybe
you should not change your passwords on a
regular basis.
So, um,
I looked at the studies that they were
citing for that, and I was not
completely convinced.
So I felt the quality
of these studies was not very high.
So all of them were based on
on observational data that they
didn't make any intervention where they
put people in groups.
But they they did things like they had
password data from a company
where at some point they had a password
changing policy and at other points not.
And, um, then
there was one which was trying to make a
theoretical model of password breaks
and password changes and how much that
matters.
But the basis of all of these studies was
observational data.
And then also some of these studies tried
to measure things like password quality
by the entropy.
And if you think about that, that's not
really what we care about.
What we care about is whether our data
gets hacked.
We don't care about the entropy of our
passwords. Maybe the entropy of our
passwords is an indicator that we have a
good password.
But it's only one factor.
And there are other things like if we
use a password, that's also bad.
So maybe people use a strong password,
but use it for many different services
and it's also better.
And in medicine, there's
a term for that. And that's a surrogate
endpoint, which is when you're measuring
something that's not the thing you really
care about, but maybe an indicator of
what you care about.
And that's generally considered a lower
quality of evidence.
Um, so the good thing
here is the FTC found out that they
didn't have scientific evidence for their
recommendations and they said, OK, we
have to look at the scientific evidence,
but they're not. So good thing is, I
think the quality of the evidence was not
so good.
So the real conclusion would be maybe
we just don't know what we should do.
Some proper studies on that.
Um, so finally,
I have some, uh, things
for I think like I think
this is the right approach, but I think
it has some limits that should be
considered.
Um, there are
things that we want to protect ourselves
against a threat that we cannot
really measure because they may be just
future threats.
For example, currently we have a debate
about post quantum cryptography.
Do we need to protect ourselves
against quantum computers?
There are no quantum computers today,
so we cannot measure any attacks with
quantum computers.
But we still may want to prepare for
that.
And there are things where we have attack
scenarios that are very obscure where we
say, OK, what if a nation state
is compromising a Debian developer
and he gives me a different package than
the one he is telling me,
which is something that the reproducible,
built community is trying to tackle,
which I'm
not sure if such an attack ever happened,
but it may still be something you want to
think about against protecting so that
there are definitely situations where you
cannot do this approach with a controlled
study.
Um, and one more thing.
There are sometimes claims that are
simply against that violate basic
scientific principles.
For example, if a vendor promises for
protection from malware, that's just
a lie. It's simply a lie because
that's impossible because of
the so-called halting problem, which is a
very basic theorem of computer science.
And there's a related debate in medicine
where some people argue we shouldn't even
study something like homeopathy because
it simply cannot be true based on the
laws of physics.
So, yeah, that was my last
slide.
Um, so I think today
think security is often very often
not based on scientific evidence.
We rely on experience, we rely on
experts are even worse.
We may rely on marketing.
We should have evidence based IP
security. But right now we don't have the
science to do that.
Yeah, I think.
I said we likely don't have time
for question, but if someone is very
quick and runs up to the microphone, then
we can take one.
Yes, microphone three.
And if you do, you hear
me? Yeah.
If you go to medical studies, it's
actually the highest quality that you do
double randomized controlled studies,
which means neither the experimental
nor the participant knows whether it
takes placebo or the actual medication.
You think that is something that you can
actually implement because people act
differently if they know they have the
placebo or if they have the drug
greatly interferes.
The randomized control, that's
fine. So so BLANNING studies,
if you can do that, depends on your
situation because there are situations
where you cannot blind if it's something
that the user has to actively do, but if
possible, you are blinding us better.
OK, sorry. Unfortunately we don't have
any more time for questions, but
as I said, Hannah will be prepared
and ready to answer some questions right
here. Thanks.