Herald: So could you give a warm welcome of applause to Stephan Gronke who will be
talking to you in one minute.
*Applause*
Stephan: So hi everybody my name is
Stephan Gronke. I'm a software developer since about 15 years, working in solo
projects and larger teams and smaller teams. So mostly my development stack was
JavaScript and you will find some of the tools that I mentioned coming from this
world, but I'm very sure you can also find something for your project that applies
here. Here's my email address, my PGP key and my favorite social network account.
Yeah, so a little spoiler what will happen today: I will talk about development
process exploitation. So that means if you are developing your software and somebody
joins your team and since you code for review it could happen that it executes
code on your machine without your knowledge. There are a few things that are
really hard to catch or I found hard to catch and I want to share. Maybe you have
the same problems and you find that the same mitigations apply for your project as
well. I will then continue and... let's start with a software development process
that's a small cycle. So first of all it starts with an operating system: you need
to have a computer to write a software and that's something you need to trust first
off. If you... yes, so your operating system contains keys and credentials, it
contains the source code you want to develop and the tools that you have in
place. And the major risk is that tools are vulnerable to some exploitation or
that your host is already compromised and you write a software, commit it to your
coworkers and it isn't what you intended to write. That's a larger problem here.
After you start writing code, the editor is kind of the interface that I have to
write the files and edit the code. I find it kind of complex to use an editor. On
the left you can see that many of the editors come with a package manager
included, which is a good sign for the complexity that these tools have. And I
don't know what tools you need but they support you in development so it's very
good to, for example, have code linters and auto-completion in place to write
better code. At the same time it can be a problem because they can execute code
unattendedly. We will see in a moment. The mitigation I came up with for the editor
part is that you have a virtualized environment where you run your editor. So
when something happens and and it is compromised, not your root system is also
compromised as well. You want to monitor all your config files that you have in the
project and you want to get awareness what exactly happens on my system when I run
and view this code.
The next part you will probably use as a shell integration. So as
soon as you open your repository some of the shells I saw just tell you what
branch you're working in and what files were changed and so on. So that's
something that comes very neat if you're developing but it can be a risk as well.
So my opinion on the shell integrations is mostly that it's made for software
development on your own system. So when you write the code and you can trust it
it's not a problem to use those tools at all, but as soon as you get sources from
foreign developers it can be a problem. So choose your tools wisely and don't execute
code from others if possible. The versioning system that you commit your
code to is also a very good choice. For example Git and yeah Git can execute hooks
on different occasions. For example when you check out new code, if you commit and
so on. That means if you managed to clone a repository and a ".git" folder is
included or a ".hg" folder is included, it could mean that your operating system
decides to execute whatever is in the hooks. It's not possible to store a ".git"
folder within a Git repository but it's possible to store it in a Mercurial
repository or in a SVN or something, and then your shell integration won't know
what the original source was and will execute it anyway. One thing that was
introduced for example from Visual Studio Code, this October, is that they now
support git hooks - which is a great feature, right? *chuckles*
The mitigations
against this are pretty easy: you can either set a different hooks path, which
is not within this project repository so that you don't execute git hooks at all,
or you can use that little wrapper here that you see, to for example check at
least that there is no file that is a git hook within that folder before you execute
git. It's a very good choice if you want to protect yourself from that
vulnerability.
So after you committed the code and shared it to the versioning
server, you probably are going to build it automatically. So some services like
Travis CI will run it, will run it for you. So they will run tests, they will
compile the software and also they do the package versioning and deployment to some
other places. It becomes a problem if you can't reproduce the results from your
built runner, because it's an system you don't control sometimes. And as soon as
you get the binary result from it - if you compile the software that compiles to
binary - you need to check that result somehow, because somebody could have
altered it without your knowledge and then you will ship it to your users. Also a
problem on many of this build workers is: you want to have this process very fast.
So that means you don't want to wait until all the dependencies are installed and the
great service is that you have caching in between these projects. This means that,
for example, if somebody managed to inject the version to the cache of some CI
system, then it will eventually show up in other projects as well and you can pivot
across the projects. Usually if you have a build environment it has access to some
kind of development key. Mostly if you get pull requests from externals, the keys are
stored encrypted and you don't have access to them, but as soon as somebody has write
access to your repository also the keys could be leaked. Let's make an example:
you have somebody offering you a software and you don't give permission to edit the
master branch of the repository but as soon as you open an open a branch anywhere
and make a pull request Travis CI or other build runners will use that and decrypt
the passwords for you and give you access to the credentials which you can then
print or do whatever you intend to.
Yeah, and for me the best option here would be
to have reproducible builds because then you can use different of the build workers
and compare results somehow so that you see if one gets compromised the other two
will tell you: hey, there's a different result, have a look please.
That would be great.
And also the build steps; I mentioned building, testing and packaging
the software are totally different steps so what you can do is you can have one
compartment per step so that you can have a data at a finer level and see what
happens here. After you compiled the software, you built the software, you need
to ship it to the user somehow so either you store it in your own server or most
often you use a CDN. You just put it there and it's the asset that's lying around.
Your users will come around, download it from here and execute it, so what is the
problem here? The problem is, that if you have an URL it's very hard to prove that
it's actually from the real maintainer. If you call your software like if - if you
call your account like a different project, then people won't be able to
notice the difference somehow. What you can do to mitigate this, is to publish the
URLs that you legitly using and also sign your assets so that users can check is
that something that the developer intended to give me or is it something that is
really ... that is really not intended.
So, ...
Yes. And the next part is you need to reach out to your users
so you make people aware, that there is a project they can
check out, they can clone and usually you have the package registries. A few slides
back you saw that the package managers are also included in the editors, so that's
also something where you can ship the software but the package manager I was
mostly looking at was for example NPM. There was an interesting occasion where
somebody had a project called Kik.The company Kik then tried to take it down and
the person just ignored it for the moment but then Kik reached out to NPM directly
and they deleted the repository. In consequence the developer removed all his
projects from the versioning server and a few hours later malware showed up with the
same project names, so that means if you have a software that uses that
dependencies and somebody freed up names it would affect your repository as well
and compromise it. That's something that needs mitigation. I think the best idea
here is to not only identify the project by a unique identifier but also
have a GUID or an or a unique identifier per project that does not change, so that
you can make a difference. That's something that's up to the package
registries to implement. That's not something we can do as a user but it's a
very common case - it's a very common case that these packages fluctuate.
So for example, if somebody deletes it, you don't have a backup of that.
A very good idea is also to store offline backups of every package that you
check out and that you install to your software because it's very bad
if you want to maintain your software and you figure out there's something, there's
something missing and I can't recover because it's deleted.
Yes, software developers have some needs during their work.
I want my tooling to perform if my code editor for example is in the VM and
the VM is slow, that's something that's annoying all over the process. So then on
the other hand the velocity is something that your manager will require from you if
you write commercial software or you try to get something done and you can't spend
all day to work on chores and improve the repository, the versioning and so on.
So that's something you need to deal with. Another big factor for me is the
reliability. So as soon as your software goes down and you are in holiday or
something, anybody else from the company or from your team should be able to
recover what was there before, also known as the bus-factor and, yeah, if you have
convenience like for example Ruby on Rails gives you. It gives you a very good,
very easy start in the projects and that's something you don't want to break by
making it too complicated with a development environment. And also
something I've found to be more annoying than helpful is, if you want to pair-
program and you have a very compartmentalized environment, it's very
hard to share the resources that you need to talk about with other developers,
expecting you're not in the same room but working remotely, what is for me most
often the case.
A large problem that I saw is, if you underhand somebody code,
if you go ahead and and check out code from any online resources, it's
sometimes very hard to tell if the code that you see in your, e.g. Git diff,
is what you really would expect to see. I have some examples here, which can
show how this could work and how this could look like if somebody tries
to inject code to your repository, that you don't see. First of all, let's start
with something easy, that's phishing. What you see here on the slide, on the left
side maybe you see the cursor. That's not a full path, that is just a domain name.
The slashes in here are UTF-8 characters so that thing here resolves to a hostname,
and if you control this host, you can get a certificate for it and then the example
below you see, how it would look like, if you install it. First I have a host that's
just running a web server on port 80, so that you can see the result. Okay, I was
cheating a little bit. I was putting the domain in the /etc/hosts so that I don't
have to buy it, for just showing it. It's strange that dot zip is a domain actually,
but then if you install it, you would see that you can send somebody a very
nice-looking link which looks like a totally different project, but it's
pointing to your server instead; and I found many of the package managers having
the nice feature of executing PostScript hooks, so that means, if you have
installed it, it will run some commands afterwards for you.
Then there is invisible code. If you go online somewhere, find in a forum or in a
blog, you find an article and see, "hey, that code is actually solving my problem",
you go ahead and copy/paste it. So, on the left you see the source code how this
would look like in HTML for the blog. On the right there's the result. So you can
go ahead, you can copy/paste from it and if you paste it to a text area, you will
see, that the result is something that you didn't expect. For example, if you copy a
large chunk of code, you won't go ahead and review it on your local system again
and that could be the compromise for your project.
Another example here is, you can use ASCII characters,
the control characters to influence the output in your terminal.
So if your terminal also supports the legacy of ASCII control
characters, you can use that to just revert the line and override it with
something you wouldn't expect. What you see on top here, that harmless script is
the file. It's a little larger than you would expect for just a echo foo, but not
something you would notice when you just see it. Looking at it from a hex editor
you can see, that there is something more going on than just the foo and if you
actually execute it, it will not print something, it will create the pwned text,
which is a good example for you that your host was compromised in this moment.
Another example I found online, so credit to Ariel for this, so there is a byte
sequence you can use so that this even works in a Git diff. So when you're
working exclusively in your terminal and you're not doing reviews on GitHub or some
graphical tool, it could be the case that you don't notice what was going on. What
you can see here on the left is, I created an empty repository, I added a small
script and in the next step down here, I added some improvement to the script,
which is actually the malicious commit that's here in red. Afterwards, I just ran
a Git diff on the code and I see that there is only no backdoor, oh sorry that
should be okay in the updated slides. So, you don't see the evil.sh that it's
executed as well, if you run it. That's something I consider very dangerous.
So, some mitigations: the best thing you can do, is to make it expensive for your
attackers to compromise or try to. So as soon as you have the chance to notice
what is going on, also retrospectively, you can at least burn the capabilities and
tell others how your project was attempted to compromise and that's something
that is, in my opinion, the best mitigation against this complexity.
What you can also do, is you can test your software from external services directly,
which will tell you, if some compromise happened. For example, Git has it newly
introduced. They will check your packages, the dependencies and will warn you about
some vulnerabilities that are commonly known. The best thing you can do on your
local system is to build small compartments, so that if some compromise
happens, it doesn't affect your full host. Also not all your projects you have
access to. And it's very important, that you have backups on a different system
than the hosts you're working on. So if the compromise happens you still have
access to the original data and can compare it and do some forensics on this.
So, the intrusion detection forensics; there are some great tools available.
For example my favorites are DTrace and Opensnoop. You can monitor changes and
access on the file system or on your system at all. And you can e.g. set some
rules for your projects that are specifically matching. So I am
not going to share some rules that match for all projects, but you will figure out
what is e.g. important. Very good start is e.g. to Opensnoop for
/etc/passwd if there was some access, then you can e.g. say that's not something
what my software would do. And again it's very important to have the backups of this
because in the moment where you execute it, you can't trust your host at all.
The idea how to achieve this is, if you have a VM per project for example,
you let it run for half a year, you don't approve the situation. Instead
of having one system that you need to update the software to, you need to update
afterwards all the projects that you're working on frequently and that's something
that's easy to forget. So it's dangerous. If you assume that every time you run some
command or every time you work in a project, you spin up a new server entirely
from scratch, install the dependencies and so on that's something that's not a risk
for you. Also, if you have for example a virtualized server environment you can
have memory dumps at any time, you can monitor the network and you can also diff
the filesystem. For example, if you stop the server and just compare it to a
previous version and see, "hey, here is something that was changed that I didn't
plan". It's great to know.
Very important is also to separate your accounts. E.g., if you see large GitHub
accounts, people are making contributions every day since years.
So it shows that the people have access to very, to many projects from
the same machine. And the permission model from GitHub for example, allows you to
store an SSH key for write access. But it automatically has access to all the
repositories you control. So the best that you can have here, is to make a
new GitHub account for, ... or to make a new account on that versioning system
that only has exclusively write access to that single repository.
So when you work in your compartmented system and you want to
upload or pull changes, you can't influence other repositories. That means
compromise doesn't spread across all your projects and so on, which would be an
invitation for malware somehow or ransomware. And you get a better
permission model if you create a GitHub organization. In this case you can also
limit your own access in a better way. So my recommendation is not to work on your
personal GitHub account but create an organization for your project.
Something many projects are missing are to find responsible persons for security
and to clearly communicate what is the plan for incident response. Small example:
If you have a new project and you find a vulnerability, you would like to
commit it, but you don't open an issue publicly, because then everybody, every
user would be affected. You try to reach out to some developers and if you don't
have any clue how to securely achieve this, that can get you into trouble. And
there are quite a few projects which don't communicate this and some of them don't
even respond to their security@ email address, which is bad.
In this case I told you what I saw from my experiences of working on the projects.
So that's basically my summary of what can be harmful
and what can be good for your project.
*Applause*
Herald: Thank you and we now have time for Q&A. In the room you can line up behind
the microphones and I can see we have a question from the internet already.
Signal Angel: What about Git signed commits? Any thoughts on that?
Answer: So as soon as you have signed commits and I find that you also email
with the same PGP key, it's very interesting that you have the PGP key on
the same host probably, then you have your Git executable. So if somebody executes
Git hooks, they can steal your PGP keys from this. I didn't find any tutorial
online which explains you, how to make it manually, so that you don't use the Git
for signing the commits. But I think it can be very good to sign the commits,
but it can be also dangerous, because your email communication can be compromised.
Herald: Microphone number four. Q: In the Git diff you showed us, there
were some control characters. I think Git diff pipes to less by default, so
shouldn't they appear there somewhere? A: No, they don't. I just checked with the
latest version today. So, that's something that, well, we can also click on the blog
and see, if there is the video available.
Yeah, it's very hard to show from my HTML slide how this works.
So this video animation, ... maybe we can enlarge it a bit.
That's how it would work.
So most often, yes, if you pipe to less or you use a hex editor to review,
then you would notice, yes.
Q: I somehow remember that, maybe it only shows for longer diffs, but I think
when I type "git diff", I can scroll around.
A: Ah, that's interesting. I need to try. Herald: We have a question from microphone
number one. Q: You mentioned Travis having access to
hidden variables and you being able to leak those variables during pull requests.
What are your suggestions to mitigate that?
A: Don't give people write access to your repository, not even to branches that you
don't trust. So, as soon as they have write access, they would also know the
secrets behind the variables in this case. I like the security model, because if you
for example get contributions from outside, nobody can trigger that and steal
your keys. But as soon as you build it on your own
branch somewhere in the repository that changes.
Q: Yes, but if you submit a pull request, you don't necessarily have to have write
access to that repository. A: Yes, that's what I mean. If you come
from outside and it's not within the same repository, the secrets are not decrypted.
So you can't run the steps. For example, you would not like to deploy directly from
a foreign branch, somewhere. Herald: We have a question from microphone
number four. Q: You mentioned the problem with
different compartments and how to exchange those environments without people. I think
that problem has already been solved with Vagrant and some kind of provisioning
software like Ansible. Do you have any experience with checking
those results of those Vagrant boxes that are automatically provisioned, like
having some server spec software to check those environments afterwards, or having
some kind of hashing, how to find out, if they have been reproduced the same way and
or if they have been any exploit used in that process of setting up the Vagrant
environments. A: Yes, so different levels you can look
at this. There was some, ..., I try to find it, yes. You can for example memory
dump at any time, if you have the hosts trying somewhere or was your question
exactly that you want to check, if your environment that was spun up was not
compromised yet? Q: Yeah, there has to be some kind of
process, how to verify that the produced environments are the ones you expect them
to be, or if they have been compromised and the problem is, I have used those
environments and tried, first I tried the full disk encryption for the Vagrant boxes
but the problem is, it's always the same key for the encryption, so that doesn't
work and even as you mentioned, you can have a memory dump, so you can read out
that key so there's no real possibility to set up a Vagrant box that can't be
tampered with afterwards. So there has to be some kind of hash sum to compare those
produced results. A: Yeah, so as soon as you have a
reproducible build and the result that you, for example script languages are much
easier to achieve, because then you can just diff the filesystem directory and
see, if there was some change. What I would do in this case, is to run multiple
services and compare the results, if that's possible. For example, you have
these reproducible builds, then run it on a few servers which are independent and
compare what you have. Herald: We have two more questions from
microphone number one and only a few minutes left. Microphone number one.
Q: So, what's your recommendation for handling credentials in application
configuration files? We need often some database user and password or something
like this in, say Spring Boot Application YML, or things like that?
And is there any best practice or any framework which can handle such things or
we need to explicitly encrypt these credentials in this application and then
decrypt for itself in the application, but then you need symmetric keys, or?
A: Yes, so Ansible for example comes with a mechanism that's called Ansible Vault,
which encrypts that with a passphrase that you can enter in your command line as soon
as you touch the file. For example, if you want to run Ansible then, it will ask you
for the password when starting up. So, if you want to share that password with your
developers everybody has access to the same keys, I would prefer to give
everybody, so every person in this team or even every device a different key, if
that's possible somehow. That's what I was trying to mention with
the GitHub accounts; that you don't use one GitHub account but you use many of
them. If you, ... yeah.
Herald: We have one more question from microphone number one and then a question
from the internet. Q: Yeah, my question was more about, I
mean some of your recommendations are low- hanging fruits, but some of them it's
like, it's just impossible. I mean it's not sustainable, like it's very hard to
maintain and so I'm wondering, if you use all of them every day or just part of them
or do you just leave like an ??? at the end?
A: It depends on the project. So what I try to do on my development system is to
have these compartment, so that one compromised project would not affect
others. Because I'm not the only person checking and merging the code, so and
that's something that gets quickly too much for one person to review. So, I
can't review all the code that I'm running currently on my computer, that's true. But
I can try to mitigate what the impact of this will be.
Herald: And the question from the internet. Signal: What tool would you recommend
for diffing a file system? A: Diff. *giggles* Well, it worked for me
so far. Or what exactly is the question about? Um, maybe you want to see, if did
the hash change in the files? So when you have e.g. the script file one and
the script file B and they have a different hash sum, that's something I
would consider something I would look up manually. So as soon as I have an
indication that there was something wrong, I would look it up manually and use any
tool that I have. Hex editor or whatever is available.
Herald: Good. We have less than one minute left. Are there any final remarks?
Stephan: Thank you. Herald: Thank you very much.
*Applause*
*postroll music*
subtitles created by c3subtitles.de in the year 2019. Join, and help us!