Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2535 Thanks!
[Music]
um welcome to the bitten boy stage um
we're continuing our program with um the
next talk by petus and Phillip who are
explaining to us what the search wi
project is and what it is doing for
civil um Emergency Operations with
drones welcome um enjoy the
[Music]
talk yeah hi and welcome uh um my name
is Phillip and who are you p zilios and
we are part of the search wi team and we
built um drones uavs unman aerial we
vehicles for search and rescue um it's
really nice to see you all here uh the
great interest in our work and um we are
really happy to present our work here
thanks to all the people who made this
talk possible to the whole bits and Bo
Village um
yeah so short overview about our talk so
we will talk a little bit about
motivation why we do it why it's
important to do it um then we will show
you in detail our drone how it works um
how it looks like I mean you can see it
over here already but uh we will also
present some pictures we talk a little
bit about our organization we have a
Association now um then we will talk
about uh the past and the present and
maybe a little bit about the future and
then we also go into detail about the um
image processing which we do um and
future work stuff we want to do in the
future so what motivates us more and
more people are fleeing um are on the
move for several reasons one reason is
war and Terror
poverty climate
change um there are a lot of reasons why
people flee and um some people flee over
the O the ocean because they have no
other way to get uh to safety and here
you can see a map and some numbers about
people on the Move what's important to
see here is that a lot of people die so
there are very dangerous routs for
people so it's our our respon
responsibility to help these people and
what you can see here too is um that
that there are multiple routes um even
through the Mediterranean Sea there are
multiple routes which people take um for
example there's a route via Libya which
is a big war zone then there's a route
via Tunisia then but then you also see
um on the left uh there's uh route via
the um May
gibralta um but also people come come to
Greece and what you don't see here at
the moment is for example the English
Channel or um the Atlantic route where
people go to the Canary Islands for
example from syal um
yeah it's very dangerous and uh we
should support these people what we see
at the moment that um almost all states
uh withdraw from the from the sea they
leave it to uh a civil Fleet to support
people on the move and um here you can
see a few organizations uh which we are
actually working with um they have very
different um vessels here on the left
it's an old picture actually now the
ship is run by um SOS Humanity but
before Seawatch had the Seawatch fear um
then you can see the Alan CI the ship of
SEI um also not uh present situation
right so this not ship is now owned by
somebody else but um there's also rescue
ship a small Sailing Boat uh which
operates right now in front of um
lampedusa
so what we want to do we want to support
a lot of NOS with in different
situations with our system um and this
then has some implications for our work
of course there's the government or
multiple governments doing work uh in
the Mediterranean too um but what they
mostly do they fund uh corrupt and
um yeah bad people people um which you
can see on the left is the so-called
Libyan Coast Guard um they get for
example infrastructure for example
vessels um and then on the right you see
that um frontex uh EU um border agency
more or less um they uh operate drones
in the Mediterranean Sea um or above the
sea um there you can see some tracks of
these drones but the image data is
actually not
um public so um they only use it to
inform for example the Libyan Coast
Guard about cases and then they start to
do pullbacks so they bring people back
in in in the war zone um and not to a
safe place where they actually
belong um here you can see uh get an
impression of the area where the NGO
boats operate in this the med
Mediterranean Sea in front of lampedusa
and Malta so it's quite a long way with
a shitty boat to to get to a safe place
and then you can also see that's a very
big area um the ships have to
cover um and right now uh it's quite low
Tech what the ngos do um they go on
Rusty ships they go to Sea and then they
use the ice to check for people um on
the water sometimes or most of the times
they use binoculars then the next
technical um device they have is the
radar but the radar not always works or
is not as great as we want it to be
because um they are rubber boats or the
the the people on the move they use
rubber boats so there's no real
reflection of the radar signal so it's
really hard to um make out cases by
radar then uh the next Tool uh we have
are airplanes some ngos operate
airplanes over the uh sea um they of
course can spot people but they use
binoculars eyes or cameras to spot these
people and they can only operate in very
good
conditions then of course another tool
we have is a mobile phone but uh the
mobile phone only works for a few miles
uh out into the ocean and then people
are more or less disconnected some
people have um satellite phones and can
um call for example alarm phone um alarm
phone is the hotline for people on the
move so you can call them and tell them
hey um we are in distress please um
forward this information to the Coast
Guard or the
Fleet um yeah but of course you have to
have a for example a satellite phone
some countries uh forbid satellite
phones so it's in some situations hard
to get satellite
phones um yeah and then it's possible to
use satellites to detect cases on on the
water to um but this is uh really
expensive process and um it's also not
as fast as for example an airplane um
yeah and nowadays we start to use drones
to find people but uh to this I come
later so here you can see for example
how people on a ship um try to find
people um yeah but as you can imagine
this on a ship a moving ship rocking
ship is uh really Tire tiresome process
and also prone to errors we actually
want to show the video h no okay so yeah
what what we do we built uh drones for
search and rescue um here you can see
our drone in
operation um but how did this all start
it actually started in this wonderful
Community some people saw that NGS and
other people um started to do search and
rescue at Sea with with this quite low
Tech approach and um these people out
from the chaos Community they thought
okay hey um we have skills um and
there's quite a lot of free software and
open Hardware which we can use to build
a drone that helps in this um use case
and so it actually started uh 20 um
2017 um people gave talk here um and uh
the project
started um
yeah so what's happening right now so
yesterday evening um uh we got a call
from one of the ships um they have a
drone near to lampedusa on the on the
vessel and they um did a search search
flight from The Vessel so um
the drones are actually in operation and
then there's another ship where we have
two drones um which are right now in in
the so-called um Z Zone um so the Zone
where we actually search for people um
and we are waiting for next um next
flight of one of our
drones yes
um one video come now Okay so
um right now um three drones are in
operation but in the past of course we
did um missions and test missions um
here you can see a small overview um so
we did the first um Mission uh with
rescue ship um they operate from a small
sailing vessel I think uh the the person
who actually did the mission is also
somewhere here um
then uh we tried to do a mission with CI
but
unfortunately our system failed before
we actually uh went to sea and then you
earlier saw the Seawatch 4 so we went on
a short trip uh to the shipyard uh with
Seawatch 4 um where we intensively
tested our system um so one of the
latest prototypes and um then the last
two missions we did with Mission
Lifeline they operate uh near the canari
islands um and they um operated the
Drone in
2021 for one week and then now they
operated um two drones for 3 months um
yeah and um now I want to tell you a
little bit about the Drone um and we
want to show you a short video about the
process so this
yeah but I guess it's a little bit hard
to
see yeah but the idea is that we operate
at Sea therefore there's a lot of
water um
for example from a small sailing
vessel um normally we give two drones to
the vessel if there's enough space um
then we have a small tablet which we use
for a planning of the actual search
pattern so normally we know about the
case um and or know some some
information about the case and want to
fly somewhere and fly a search patter
there um yeah it's not a big
drone um you have to invest uh right now
€1,700 for for one drone um it's quite
wind resistant here you can see the
hands hand launch of the
Drone um so you just throw the Drone in
the air and then it starts to
automatically fly the the search pattern
um we fly in 550 M of course we can also
change this but in this height um we see
uh the most
area and then our cruise speed now is
actually a little bit uh
faster um and then we see an area or um
ground resolution is 20 cm per pixel
right now here you can see how a search
pattern uh could look
like and we can cover um
roughly 200 square kilm with our two
cameras and then we have a base station
where we always have a connection to the
Drone and can also change the flight
pattern in Flight if there's need for
that and then the Drone automatically
comes back to the ship and then the
Drone pilot initiates the landing um
right now we do the Landing by hand so
we have a remote control um and land the
Drone next to the ship and it's even
possible in very rough conditions for
example waves of 2 met are not a big
problem then the Drone actually lands in
the water because we don't want to
endanger any people on the ship um and
then we start the image download and the
automatic processing of the images um um
here you can see an old version um of
our
system um yeah and one crew member goes
through all the images we
have
okay yep next one so just to recap we
actually have a flying camera we can go
100 kilomet it's waterproof because we
land in the water um the flying is
automatically so it's not
autonomous um then we have computer
vision but it's offline um and of course
it's all free and open source software
and Hardware so if you want to build a
drone you can do this here's short
overview about our system so we actually
have this Frame we buy it in Asia then
we have a waterproof box you can see in
the middle then we have a flight
computer um based on an STM 32 um then
we have a Raspberry Pi with two cameras
connected
um the Rasberry Pi then gets data from a
flight computer and then we can fuse
this data with the image data and for
example then tell where we took the
image or at which time then of course we
also have a a Telemetry module or
transceiver to stay in contact with the
Drone but we only use this for for
example POS um position data and uh not
for image transmission
yeah that's and then in the front of our
plane there's a big battery which weighs
more or less 1 kilo and in the back
there we have the motor and uh of course
we have a few servos to control the
plane then here you can see how we
actually do the planning so we use free
software here it's called C Ground
Control it's uh CT app um and you can
draw a pattern which the Drone then
actually flies
through um and then we also developed a
little bit of software to actually take
a look at the pictures uh we take and
here you can see maybe a picture um with
uh one big ship and then maybe you could
or normally we ask people to count the
ships in the in the image I think this
now really hard um but what you can see
that it's really hard to actually see
the the small ship um so the big ship is
the Seawatch 4 and then we have a small
rip um which is next to the Seawatch 4
but it's with our camera system it's
really hard uh by human eye to to make
out um this ship um and therefore we
have this uh system to take a look at
images and then also the image
processing to give hints um in the image
uh to to actually see the case and here
you can now see for example that we
highlighted this case you only see a few
pixels um with of the case um
yeah
so and now I give over to julan I
think thanks yeah I'm uh I want to show
you few bits about the boat detection
process uh we developed at search Bing
um you saw already in the previous
images the like the result of it so we
have a software which shows the image
and then overlay with the rectangle
around where about is and uh the steps I
want to show you are uh yeah describe
this process how to generate uh these
boxes and um there are a few um things
we thought or we we developed some
knowledge in and I want to share this
with you so um we can roughly uh um
create uh or build this process and it's
built out of five steps it's uh the
first step is taking the actual image
then we download the images after the
plane has landed and uh then we do the
actual boat detection after this
detection we do a four first fourth step
it's a offline tracking which combines
all the information we uh created in the
previous step and then in the end we
visualize this so how do we do the
actual
uh image capturing um as Philip already
said uh there is a um yeah raspberry pie
board in a plane which has two uh
cameras connected to it um it got 8
megapixels and this results in our
altitude of 550 m in a ground resolution
of roughly 20 cm per pixel to 35 cm per
pixel and um we can cover uh with both
cameras uh like a sweep WID of 2 km MERS
and
yeah we take one photo every 2 seconds
and we also thought a bit about the
exposure control because normally the
exposure control for cameras is very
optimized for big objects inside the
image but as we want to detect very
small objects they tend to get
Overexposed so we really try to
underexpose the images to get um not uh
overexposure of the boats to to have
some more contrast in the image so the
second step after the actual image
taking is then already the download of
the image which is done after the plane
is landed next to the ship we get it on
board and then we download the images
this takes about uh 35 uh minutes and
because it's a a lot of data actually
it's 22 GB because of the high
resolution and um we also use some kind
of priorization to get first the most
important images uh um inside the data
we or the images we
created in the third step we uh do do
the actual detection of the boats so um
finding the boats inside the image and
we realized this is very much important
step to do because we had some um yeah
users actually and they tried to um
check all of those images manually and
it was a very um yeah tiresome and uh a
tiresome process and they yeah humans
cannot keep that attention span for a
very long long time especially if you're
on the sea and get maybe seasick and you
cannot re analyze manually this huge
amount of image so we think uh Pro
algorithm which uh assists this process
is very important and therefore we
decided to take a or use a yeah some
algorithms from the neural networks uh
family some say it's AI but um it's more
I think like a pattern recognition
algorithm and um there's one problem
with these algorithms that um they uh
are more or less a blackbox which yeah
it's hard to understand and that you
cannot guarantee as a programmer that it
works as expected or intended so even if
there are some boats um detected we
cannot guarantee that other boats which
might look similar are then detected
afterwards uh again and also those
algorithms need a lot of data which we
don't have because we cannot fly that
much at the sea and also we cannot
create images of every possible object
which could be out there so um we don't
generally trust these algorithms and we
always tell the people who use our
software that they still need to check
every image and just use this um
algorithm to um like rank or prioritize
the images to get the most important
ones first then check those uh according
to the
prioritization and um this makes it then
possible to actually um get through the
images in a um meaningful time otherwise
it's nearly impossible and uh yeah just
the detail maybe the object detection
algorithm we use is called faster rcnn
it's a quite old one but it works the
best for us in in our use case because
we have a lot of small objects and other
more modern ones uh don't work that well
on um yeah small
objects then the next step is the
offline tracking this step combines all
the information from the previous ones
so it takes all the detections and
combines them in a meaningful way and
you can see in this uh example here um
we um we have here a plane uh at five
time steps and uh you have a boat on the
on the Sea on the ground there and
um uh in the first time step the boat is
not detected otherwise there would be a
box around it and there are also no
other waves which might have been um
false positively detected so in the next
time step we fly there is a wave on the
left side and the boat on the right side
we detect the wave and uh for us as
humans we can see this is a false
positive um and we create a track out of
this which says okay we we use this
information from the image calculate the
real position and uh we remember this
position for the next time step then and
then check if the boat or wave in this
case appears again and then we can see
if it's a going because if it's uh going
uh the next time step then we know or
two time steps then we know okay it
might not be a real boat because a real
boat differentiates from a wave that it
um stays there but the wave disappears
so let's see what happens so um at times
step three they are now we detect the
actual boat and the wave again as a yeah
true positive in this case then we have
a Time step for the boat again it gets
detected and at time step four five we
uh only detect the boat at as true
positive it gets marked with a green uh
um yeah cross and this says okay this is
a real boat because it was uh detected
multiple times in contrast to the wave
which disappeared at some point and is
therefore a false
positive
okay now the last step is the
visualization so we now combine all the
information and visualize this and you
can see here in the center that's the
image there you have a boat and it got
uh like a rectangular box and a circle
around it this is the detection and the
tracking information on the right hand
side you see two lists uh one list for
each of the cameras inside the Drone and
Below there's a small map which shows
where the Drone is going
to okay just a few uh more bits about
the future work we going to take with
the Drone so we want to um put better
cameras inside of it it because right
now it's only 8 megapixels in a quite
old sensor and with a new modern camera
we might uh be able to detect uh more
boats or smaller ones and so on we also
experimented with thermal sensors uh
which would enable us to also fly by
night because right now with the regular
RGB cameras we only can fly on the
daytime and this limits us in a lot of
use cases so uh we tested those small
thermal sensors they are like 200 to 300
they fit into our small plane which you
can see here um but we haven't not
integrated them in our real usage planes
yet but um it works quite okay the the
image on the right hand side yeah it's
very hard I think to see but on the top
it shows the RGB image and on the lower
side it shows the actual thermal image
and there you maybe in the Stream can
see that the actual boats and even the
small booys which are also there they
are inside the uh you can see them in
the thermal image and uh therefore it
would work to use these uh cheap sensors
also in our
drone
okay online boat detection is also one
important thing we want to build in so
right now we only work uh or to do this
detection of the boats after we landed
but it would be also nice to directly
get information about where the birds
are the boats are while we are flying
and uh there is ongoing work on this and
uh we hope we can implement or or um
deploy this soon and uh we can analyze
one image uh with the half of the
resolution with uh in 500 milliseconds
and it gives us already a rough
information where a boat could be and
they could um alert the
crew
okay long range image download is also
an important thing um as I said already
we only can download the images uh um
after we landed but it would be very
important to also give the ngos the
ability to get the images while they are
flying and um yeah we worked on this uh
a lot in the recent years um there are
multiple possibilities but we um ended
up with using uh just regular Wi-Fi
because this enables us to download
regular images um um through it using
TCP and to like streaming video or
something and uh yeah we used their
offshelf hardware and we tested it with
um quite okayish results for our use
case 430 kilom so maybe um or we hope
that we can um also put this into usage
soon
sorry
okay um another group uh in our um or
another part of our group also in
parallel work on a completely different
concept um building instead of using uh
like the um foam based drones building a
completely different one using fiber
material and it's uh it's it's currently
just a few days ago they flew some test
missions uh on the um um Baltic Sea is
it Baltic Sea I guess so and it was
successful so maybe we can soon also um
get a more robust version than the foam
based version because um it tends to if
there are some wakes it might get some
damage but the fiber material is way
more um stable in this uh
regard
okay yeah
okay okay here um yeah here you can see
how damage could could look like so it's
really um not nice to hit the ground um
but but if you want to experience this
firsthand you can of course participate
in our project um we are always looking
for new uh um members uh to support our
project um but you can also just spread
news about our um project tell people
about our project um we also take
donations so we have an association at
German Association nonprofit Association
so if you have some uh money somewhere
you don't need please send it our ways
um and of course we have a lot of
technical but not only technical um
problems um the whole aspect of bringing
the system into operation is is a big
process and we also need experts for
example in operating systems in maritim
environments so if you have any
experience please join our project um
yeah so we over there we have a booth so
if you have any questions you can come
there um and we can talk in detail about
um challenges um you can tackle with us
and now there is some more time for
questions um yeah do you have any
[Applause]
questions I was waiting for your end
slide but it didn't come thank you for
the talk um I see one question at the
front
already um there is my colleague coming
oh sorry I I was supposed to give the
microphone
here um you already spoke about on
device detection what is your biggest
constraint here is it the power budget
is it the wait is it the runtime why is
that something that you hadn't done
before is or is it just time um yeah
it's time also but uh also as we run on
a Raspberry Pi it's not very or it's for
image processing it's not the best
platform but I think for all the other
things it's the best platform like
regarding maintenance and stuff uh so
it's not like super powerful for this so
we needed to find some way to find the
um usable option for online B
detection uh to small questions how do
you seal the 3D pinted case with epoxy
or yeah we tried this to but didn't work
out because um epoxy doesn't stick to
some of the uh printing filaments um so
right now we use SLA printing um we
actually buy the waterproof box we built
in Asia um so there service to print um
these boxes and um yeah this
works works quite
well and isn't Wi-Fi using too much
battery um no I think it's maybe 5 Watts
or so we actually use with our Wi-Fi
module um yeah but of course it's a high
power Wi-Fi module and we operate at c a
little bit like Pirates um so we have
flying Pirate Radio somehow um and uh
yeah of course it consumes a little bit
of
energy question over
there so this is maybe slightly
different but um have you thought about
doing long endurance system so I
actually thought about some of this at
one point in my life and I thought about
building a long endurance flight that
stays up for 24 hours and basically
scans nonstop it's not impossible I
think to do um when it's bad weather you
might have some some trouble but uh if
you fly higher you build a bigger train
I think it's not
impossible yeah it's not impossible but
of course we have to somehow manage our
time so uh we focused on this special
use case um start from the NGO vessel
and then coming back to the NGO vessel
and flying I mean we have this this um
airplane and this Frame because we also
want to operate this drone on different
vessels
um but of course it's possible to build
bigger systems and in the beginning we
saw quite big systems from frontex and
so on um and of course we can try to
build bigger systems um one big drawback
is maybe that if we build big systems um
then we have to operate them from land
and then of course there's new attack
Vector um the whole regulation and um
for example the airplanes they face
these problems um but maybe we are in
quite good situation with the um EU um
drone regulation so it's but it's
something we are thinking about and um
if you are interested in that we can
talk about this
yeah and yeah are there and the project
specification and the software published
somewhere uh yeah we have a Wikipedia
Wiki Online U Wiki JS and there
almost everything documented I mean yeah
it's some parts are missing which we
don't have time to but uh yeah normally
you should be able to build a drone
using this Wikipedia Wiki not Wikipedia
hi uh you mentioned rescue missions um
do you see your project being used
Monday uh for patrolling uh Seas
especially when it comes to law
enforcement uh and Protec environment
against uh say illegal fishing
yeah we I mean technically this is
totally possible with this but we
totally focus on the um yeah rescuing
humans right now and cooporate with the
NGS and
yeah thank you uh two questions first of
all have you tried an Nvidia Jetson
instead of a Raspberry Pi uh yeah we
thought of this but it's too big
actually for our plane right now um I
have some lying around at my home but uh
it just doesn't fit and but it would be
nice platform um for us actually um it
could be it's also possible to make it
smaller by just creating a custom base
bird so maybe at some point we're going
to use it especially if we have bigger
planes and second question have you
thought about looking for Wi-Fi pings or
like phone pings from people on boats
cuz that can travel quite far in the
ocean um Wi-Fi pings not but we thought
of using actual GSM uh stuff for
detecting like GSM signals there was
also a test of us on the Baltic Sea but
we weren't that
successful um and we haven't gone into
this again but there are companies
building those systems for um aval
launch detect or people under aval
launches to detect them so it should be
possible um but as far as I know there's
nothing for open source like o how's it
called the open source cheese MST
um yeah the the one open source cheese
mstc everyone should know about osom um
so I haven't seen anything yet there
yeah but what we actually did we test
the base station at C so um actually we
are interested in not only drones so if
you have any other ideas you want to
explore with our association of course
you are welcome um so for example we
tested a base station at C but
unfortunately we don't have so much time
to test all the ideas um we come up
with so uh I wonder how well does your
drone do with bad weather I mean the
worse the weather the more urgent rescu
is but the less stuff works typically
yeah so um actually the way Wes are not
a big problem so as long as the waves
are not crashing we can more or less
land safely um I think in regards to
waves this more question do you want to
deploy your your crew into the water um
and then we can fly for at 25 knots
maybe even 30 knots of wind um so the
whole system is uh very able to to
yeah withand bad conditions uh of course
it's waterproof so we can in theory fly
uh when it's rainy um
yeah thank you um how many uh
uninteresting boats do you have in these
areas I mean of course there are so you
mean the boats which are not cases I
assume um yeah of course there are for
example fisherman there's also just um
Merchant vessels going through um yeah I
mean some I can't give you any numbers
to can you give any numbers no I I don't
think so but what our software can tell
you for example also can tell you the
size of an object in the image so you
you can for example uh assume that the
ship which is I don't know 50 or 100 m
long is of course not the case um but
with um smaller ships um of course
there's the possibility that we have a
case or or fisherman is maybe case and
so on
thanks we still have a bit of time um
there is another question over
there thank you how many people have
been
rescued yeah um so it's actually this
year the first time we really use it in
this um mission context really um so uh
beginning of this year in March we
operated the drone at near the canari
islands um but there are not so many
cases so um the Drone actually only took
pictures of uh of the ocean so we can
only say Okay um we didn't find somebody
but we covered a lot of area and then
right now the um the drones are yeah I
mean in the next days they maybe will
find the actual case but until now the
Drone didn't find any
case but of course it's good to also
find no
case
okay any more questions there is one at
the
front um that might be a hypothetical
question but how hard is it for you to
actually find the boat at Sea right so
it takes half an hour for the Drone to
get back half an hour to download the
pictures an hour to go through them all
and then what you know is where the boat
was 2 hours ago basically and then with
currents and them moving and you moving
like is it realistic to find them or how
long does it take you to find
them um yeah we don't know so the
Assumption that's takes roughly 2 hours
the whole process is right um of course
we don't know because we
never found somebody um but you might
have tried or yeah I mean this 2 hours
is the time it takes and and then of
course you can make assumptions for
example in the Mediterranean all people
go to the north at least if if they have
a engine on um and then you know maybe
the speed we can also do assumptions by
the image data about the speed of of the
ship so we can tell okay they maybe go
north by I don't know three kns um and
then um it's maybe not not so
unrealistic to actually then find find
the the case um
yeah I think we still have two more
minutes I'm checking with the signal
with the video people if there are any
more questions I see you one here at the
very front maybe I'm giving just my
mic hello have you thought about um if
if you do successfully find a boat and
it's a couple of hours later maybe
having a a shorter range one that can
maybe drop a a starter or a radar
reflector to the boat in question so
that it makes it more easier for
Rescuers to
find yeah I guess we thought about this
two at some
point um yeah but right now there's no
plan to drop anything um yeah I think
maybe not from my point of view it would
not be as necessary
um as might seem um yeah but of course
it's possible um I mean if we have a
little bit bigger drone we could think
about something like
that yeah I also think it's instead of
yeah trying to build a system which
throws something I think it's more
important to just build a live
transmission system that you get this
information instantly and then you can
directly go there yeah but actually
there are drones big drones which can
drop um how do you say um rescue Islands
or rescue systems um floating devices so
this would be possible and would be
maybe a demand to to our politicians
that they um start using this and don't
operate only
cameras okay I would say that was it um
if you have any more questions go visit
the search Booth thank you both very
much for this presentation it was very
insightful see you
around um come to our next
talk thank you very
[Applause]
much