Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1090 Thanks!
[Music]
and now our next speaker is Yan who is a
computer science student in aan and he
will talk to us about circumventing
video identification using augmented
reality please welcome our
speaker hello we are talking about video
identification so let's first find out
what is video
identification video identification is a
process to identify a person remotely so
that we know um he is who he says he is
and this is done using a video chat like
on
Skype and especially this is done
between a call center
operator and a user and example use
cases are like if you open a bank
account uh purchase a loow a loan uh buy
SIM cards or insurance and in Germany
for some of these use cases this is even
um this is even required by
law and now I come to the story how I
had the idea of the this uh this process
what I'm going to show you I needed to
create a bank account for my for me and
my flatmates and because I'm lazy I did
it online and uh so I went to a bank
entered my
information and um was greeted by our
call censor
operator um who also asked me for my
information again to verify that that
that's me and then
still happy I was asked to uh present my
ID
card and uh then we came to trouble
because they wanted to see some
watermarks on it and I was on my laptop
and the lighting in my room was not
perfect so I had to unplug it move
around do some dancing until they could
see the watermarks and in between the
Wi-Fi dropped of course so I had to do
it twice but um in the end I got it
working but I noticed okay this is not
fun I was annoyed and got thinking okay
this stuff can can really be faked
somehow this is not really secure just
checking my ID card using a webcam this
is not what it's made for so I began
tinkering around and I'm going to show
you what I found
out now uh back to the workflow of video
identification the German Institution
for financial oversight uh gave out a
letter with the exact workflow what they
have to ask and how and they consider
the identification successful if three
watermarks are correctly identified on
webcam screenshots if the text is
correct if the text matches the personal
information I gave on the on the form
and if the image on the card matches the
person in the webcam feed additionally I
have to move a finger in front of the
card and I have to move my hands in
front of my face I'm not sure why and
the agent is supported by tools meaning
uh when he sees the watermark he presses
screenshot that's why you have to hold
the card for a few seconds in exactly
the right uh position and angle
and and uh take screenshots for
reference so what's uh that's for video
identification what are we going to do
now is um I'll show you how we can fake
this webcam footage but only as a proof
of concept meaning no perfect results
you you see some small errors and I only
show a still images while I I promise a
video footage work works as well but the
lighting conditions have to be exactly
right and uh the workflow we go through
is first we create fake ID card images
like textures templates and then in the
second part we um use the textures to
render the card
realistically and as I'm not allowed by
law to create fake ID cards with real
people I have bought my stuffed animal
here this is Hank it's an octopus small
octopus shouldn't have an ID card never
had one but we'll make some for him and
as to um manage your expectations I um
give you a small um hint and what we'll
end up with this is me holding his ID
card with a watermark clearly genuine
real card his name is on it uh even some
signature everything
fine and so now let's go on and talk
about how we generate textures we we
scaned some publicly available documents
we find out how to remove the the sample
text er
and so on we find out which fonts are
they used on the real documents we will
talk about the watermarks we'll talk
about how to generate our own watermarks
and then we'll talk about rendering the
card okay so there's a a law called
personal
for and it of course contains some
sample images small on the page but if
you right click it and press save as you
get some from highr sample images for
use awesome without any watermarks
everything correct uh we can work with
that uh same for the back side um but
now we we don't have the water marks and
if you scan your your ID card you have
garbage basically this is not good but
the the bundus strer the company um
responsible for actually producing the
ID cards
um gave out a nice flyer with uh a list
of all uh waterm marks visible and
invisible ones and now it's up to you
guess what happens if you right click on
the sample image and press
safe
yeah okay so uh we got some nice uh
sample image we have a map of all
watermarks on the ID card
we can use this um as a as a baseline to
to work through the watermarks same for
the back side we'll now do a quick recap
of the water marks especially those with
images on this uh flyer from the bunus
so we have repeatable background
patterns we have um microscopic prints
which you don't see on the
webcam we have a UV print which you
never see on the webcam we have a
variable color print which you may see
but won't see on the webcam we have a
contrast inversion uh hologram and did
you notice that this is pretty high
resolution so we can just redraw it in
in [ __ ] or photoshop and have our own
sample uh Watermark to work
with um next we have some uh some images
like the holographic portrait which is
I'll show which I'll show you later how
to fake which is the only one I'll show
how to fake because the others are
homework for you and then we have the
eagle the hexagons above the photograph
of which you only see some and at eight
you see um microfon so which is
basically word art because no document
is perfect without word art and next we
go to the back side again repeatable
background patterns okay a microscopic
print which you won't see you V fibers
which you never see okay some EMB things
I mean like you can feel it but the
webcam can't feel so you won't see
it uh and um like if you we we later
need to remove this text there and if
it's too hard we just use the address
change sticker and put it on top of it
and then it's no problem for us because
the sticker is very easy REM moving text
there no problem okay and uh we also
have this this tilt uh image the effect
is like from a Happy Meal at McDonald's
when you get some some some tilt image
there it's exactly the same
and and uh finally the machine readable
area but that's not really a security
feature our aners security uh threats
personalize with your own text but it's
so small you don't see it in the webcam
okay so to sum it up all security
features we remove those which uh you
basically don't see on a webcam and now
we we remove those that are basically
already in the in the pictures I showed
you the sample images and now we are
left with seven seven security features
of which we really need to take care of
because you can see them on a webcam and
they're Dynamic when you tilt the card
they have to change somehow and um this
is basically what we what we really need
to fake the the hard part and and uh to
sumon the security features up only some
features can be verified UV light is not
available for the video
identification and when we the bad guys
want to do some video identification and
want to fake it we can of course enforce
some low resolution make a bad image
quality like compression artifacts make
the footage blurry or make bad lighting
conditions or when we have some trouble
we can drop some frames no problem bad
internet so we are on the on the on the
on the easy side of the
webcam okay now we have the sample
textures and we need to remove the text
there how hard could it be okay we have
repeatable patterns all over the ID card
which means we just use the Clone stamp
from Photoshop and clone clone all the
text away completely it's it's just this
simple step no magic involved you just
clone the S the the stuff beside it and
draw over it and in the end you won't
notice because text Will over they they
be over there and you can blur the stuff
and doesn't matter really and I I I
won't show you the the real template
because this is not legal for me but
this is basically the process nor uh
normally there would be shown mustan but
I removed the mask there just to show
that that's the way it works next we
need to identify the fonts actually used
and there are two fonts are really
important for us and the first font is
also used in a machine readable area
it's standardized you can look it up on
Wikipedia it's called OCB and it's used
in all all the areas marked in red so we
already have the font and we can simply
draw over it when we removed the sample
text before good now we have another
font which is used for the personal
information it's not the same if you
look at the number three um do it at
home um you can see that's different
it's not the same font and okay it's not
documented if you Google it no
chance but there's a reverse font search
engine and if you take a good uh sample
crop it out of the image put it into
search engine you find a font called sex
mono regular and this matches exactly
the font they use on the ID card and if
you try to draw over the real ID card
you'll notice they changed some stuff so
they have to customized this sex monor
regular font for example the spacing
between the characters but this is all
stuff you don't see on the webcam so who
cares and uh now we talk about the now
we have the fonts and now we talk about
the watermarks and if you look at this
comparison the blue area is what we fake
and the area on the left is the photo
the base photo and you'll notice the
dark areas of the photo are really not
really considered in the watermark but
the the the the bright areas they are
really green in the watermark so we know
okay they maybe have a black white image
where black are dark areas are not
really considered in the water mark and
the bright ones are and we see those um
those Eagles on the left side of the
watermark in the blue rectangle and
those are not considered in the
watermark as well so they must be Black
on their base texture and this is what
I've come up with um the Eagles you can
simply draw using [ __ ] you have highr
bace uh a highr image here which you can
use so you draw the Eagles and you
generate your
picture uh that's that's basically about
it for generating a watermark based
texture and now uh we'll go on with how
to implement the watermark actually and
it's very very easy I use the Fong
lighting model it's it's the first
lighting model you learn in a computer
Graphics lecture and the specular comp
component which basically tells um if
you have a blue cup or of porcelain and
you shine a light on it the the white
parts are specular and if the specular
part is above a specific threshold I
basically fade in my Watermark and this
is already this nice effect I didn't
have to do more it's like three lines of
code and you're done
now I'll show you the the base textures
which I generated from for my uh for my
stuffed anima here um you don't on the
left side you don't really see that this
is faked or that I I changed a removed
text
beforehand because this fine repeatable
print is so small that's no issue for us
so to sum it up information to build a
sample Baseline texture templates is
public in information about the used
fonts is public the re repeatable
background patterns can be used with the
Clone
stamp and um embedding high resolution
images in PDFs is maybe not a good idea
for ID cards but yeah okay and now we go
on with actually rendering the ID card
in a video stream and there's five steps
you first have to detect it somehow we
have to create a dummy card which will
detect in this stream actually detect
the card remove the skin if if we moove
a finger in front of our fake card and
then we have to put everything together
this is the easy part
basically so there's something called
aruku markers it's like QR codes which
you can arrange on a on a piece of paper
on a plane and you can um you can detect
them individually and if you detect if
you know how the board is laid out which
marker is where and you detect a few
markers you can calculate how the board
is um laid out on this on on the
image and um you can use this for
getting the
orientation and if if you take the image
on the right we have the pose and if you
want to render something in 3D over it
and you would just do it it would look
like the image on the left where the
hand is missing this is bad because we
would fail the video identific
but if you if you detect the skin
somehow in front of the board you could
not render over it and this would look
pretty realistic and this is what we've
done now I'll show you um my fake uh ID
cards which I've made uh can we can we
get the other camera please um this is
my first prototype made out of
cardboard very simple it has markers uh
it's like 1 2 3 4 it's like numbers
those markers are different and if I
change the site those are different
markers as well so I can show the back
side of my ID card later and then I I
ordered uh um a printed plastic card
online 10
bucks and uh this is what I've worked
with then it's it's it's exactly exactly
the same size as the real ID card okay
now we can go
back now we have our mark boards and we
can start really uh faking our video
feed so on the left you can see the
image we work with on the right you can
see what the detection algorithm put out
so we have the pose we know how um
because I've made the card I know how
large the card is and we can work with
this to detect the skin in front of the
card this is the next important step
because we don't want to render over the
Skin So with this information on the top
you can see see the skin the card
boundary mask which basically tells me
okay in this area of my source image
there is the card and only inside this
area I start to try to detect skin and I
do this basically by looking at the
colors inividual pixel colors and
checking if the color is within a
specified range of a specific color
color space which is good for detecting
skin after that I blur my detection
image and and detect the Contour of my
finger because um if I would not do this
sometimes the finger would have holes in
the skin mask and my finger does not
have holes and this would not match
reality okay now um we are basically
also coming to the end and this is the
image composition Pipeline on the top
left we're starting with the source
image then we're going down to the
marker detection so we know how the card
is laid out in the
image and then we do two things with the
with the known information of where the
card is first we we render it how I
shown you in part one and then we'll
create a card boundary mask to do um
then skin detection on the source image
only within the card boundary mask and
then we put everything together we take
the source image and say okay write the
rendered card over it but only where
there's no in and this is basically it
this is the
process and um the final result what
what you can see here is there's still
some work to do like the card has some
some black uh border around it but this
is only a follow shop issue and you can
see at my finger it's a bit fuzzy this
is also an issue and in a video stream
this would like be fuzzy all the time it
would change its fuzziness and stuff
like this um there's this some work to
do and the card looks too good because
you you you would need to blur it or
something and you would need to add some
more um lighting
effects um also the generated marker
board is not perfect according to the
paper which uh I've got it from I need
to order the markers differently they
are one two three but you want to order
them so that they are different the
neighbors are different also the skin
detection does not work reliable in
varying lighting conditions here at the
Congress I have not found good lighting
conditions for for creating a sample
video you need indirect lights so this
light would be very bad so a demo would
not be possible and I would need to um
Implement more water marks the other
six so in conclusion video
identification the process of
identifying a card with a webcam in
between is not really the way it was
meant to be cards are not meant to be
verified with a webcam in between and
with enough preparation video
identification can surely be bypassed
because for this I only took um two
months of work in my spare time and I'm
really a normal student with no special
knowledge in any of it and I've noticed
the general trend is you can't really
trust any video at all and that's for my
talk thank
you
[Music]
thanks for the great talk um we have a
lot of time for questions so please line
up at the
microphones and ask away also if you're
on the stream you know the drill in the
ISC or on Twitter you should be able to
ask
questions mic number two please that's
you yes okay hello my name is pow um I
really like your
presentation uh I have a question uh
don't you plan to release uh I don't
know like some kind of generator for
these VDS so we everybody can use it I
think this would be
illegal uh for uh for uh academic
purposes of course
um well you have the
blueprint I think it would not be hard
to to build it yourself it would take a
week may be so just heck away I'm not
stopping
you but it's also not your
fault mic number four
please did you ever use any uh Community
sourcing for the watermark sources or
did you do a lot of independent
research um I didn't understand the
question when you were identifying your
watermarks you had like 1 2 three all
the way you know number
did you go through did you use the
community to help you identify that or
did you just do completely independent
and I used the flyer from the bonus and
they they they did that they numbered
The
Watermark thank you so I did not number
them mic number one please wouldn't
wouldn't the skin detection be easier if
the example card would be in green and
blue so that there's a clear color
difference between the skin and the card
yes that's exactly the case and um this
is further work like testing different
colors but um when I ordered this this
sample card uh I ordered exactly one and
the the printer the printer the online
printer service they blocked ordering
just one card after I ordered
it maybe they made a loss or something
so I had no choice to order more more
cards cheaply would have cost cost than
€ 100 instead of
€1 so that's a further
work thank
you mik number two please uh thanks for
your interesting talk and um in the last
slide you mentioned that uh video can't
be trusted and I'm thinking about the no
networks and the Deep fakes or the Deep
learning uh do you think this process or
this webcam image video could be faked
with uh some deep fake so that you
replace the
so that you don't have to print a real
card and render that stuff or simply uh
show a video and and replace the card in
the video I think a card is more
structured uh to be too structured to be
faked with a neural network maybe you
can augment this uh this workflow for
example you um you could use neuron
networks to detect the skin in front of
the card maybe this is more reliable
than my Approach of just looking at the
pixel colors
number four please um hi thanks for your
interesting talk um two small questions
have you tested it yet in the video
stream what happens if you turn the card
around because I would uh uh suggest
that there you see a small image with
your sample um numbers on the card in
that moment um and the second question
do you have um ask any
official um government or the Bund strer
yet what they think about the security
for the video
ident okay for the first question is um
the the card has different markers on
the front and the back so yeah I can
turn it around and see the back side but
in the process of turning it around of
course when the card is very steep to
the camera um I could not detect the
markers but we have a bad internet
connection really and we just dropped
the
frames they won't notice and I have not
talked to any officials about this
yet number three please thank you for
your talk
um how do you make the lighting change
when um the C moves like you seem to
have used a static image you just
inserted into the um detected card so
how do you make still make the lighting
change and render it uh the card is
rendered using op so I can use all op
features for this and in my Shader I can
simply well the Shader computes the
lighting independently for each frame so
I can move the card around and the
lighting adjusts accordingly but I
couldn't show this in a still
image number one
please so you've made a pretty
impressive effort with trying to delete
the letters and recognizing the pattern
behind it I think so I think that's
great but um have you tried the existing
tools like even Adobe Acrobat does a
very excellent job sometimes I'm
surprised with the OCR and the the text
replacement so
uh no I I just did the manual approach
because I saw okay works good and it
works reliably and I only need to do it
one time so that was okay for me acrobat
does an excellent job with the letter
replacement in OCR I would recommend you
to take a look the the full version of
acrobat yeah okay thank
you number four please hey so um when
you actually do video verification if
you combine that with also checking with
a third party are you able to also
generate valid identifiers and have them
um be be are you able to swap the photo
out and then still fool people um well
that depends on the third party they use
for verification of course we don't know
what they use so we could not fake this
then I also only used like a random
identifier for the ID card because I I
can make them up uh reliably and
correctly mic number one please uh have
you made uh any kind of research uh how
you can replace your life phase two in
in the sample so that you can for
instance open a bank account as your
mother or something like
that I think you could basically do this
the Deep fakes approach of just
replacing the face and then use the face
for the generated textures as well or I
thought about maybe using some uh
something like the unreal engine to
replace the background and the the the
the person as well it's reliable enough
and you can blur the image and stuff
like this so in with future in future
work you could try this you can make the
image noisy blurry low resolution and
then you can also use a computer
generated
person do we have any more
questions and also does the internet
have any questions the internet has been
uncharacteristically
quiet no okay number one please um yeah
maybe from your experience what would
now be the recommendation to to this
video identification approach so okay
don't use it at all maybe of course but
you seems to know how to fake it but
what would be really hard for
you well you this is work of two months
so if you take a team of people who know
what they're doing and let them work a
year they can make it perfect and what
there's basically nothing in a webcam
that would be really hard to fake and my
uh recommendation would be to use
anything but video
identification okay please thank our
[Music]
speaker
[Music]
oh