Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2731 Thanks!
[Music]
So yakob kilan currently works at the
colog International School of Design
investigating the interplay between Ai
and design he's going to tell us how to
exploit brain plasticity to optically
explore the depth of surrounding
environments the talk will give a brief
overview of the study and design of the
unfolding space glove enabling blind
people to explore the depth of their
surrounding give it up for
[Music]
yob thank you for the nice introduction
um first of all I want to introduce you
to
Ben Ben is wearing a device um that
displays the environment as vibration
patterns on the back of his hand he
actually has um an a different name in
reality he's not Ben um and he
participated in my empirical study in
which I tried to find out how well this
device can be used to master a specific
obstacle course like the one you can see
here in the video Ben actually is blind
with some residual Vision but like all
participants he is got blindfolded to
ensure absolutely no information reaches
him uh to fulfill this task anyhow Ben
seems to have no big trouble getting
through the course and that is because
he learned how to interpret the
vibration patterns in a visual way
Studies have shown that people who wear
similar devices use their visual cortex
to interpret this data so um this
process is actually just possible due to
the so-called neuroplasticity of the
brain um which I will come to in a few
moments hi everyone my name is Jacob I'm
from cologne first of all and I don't
want to talk too much about my
background that's not why we here today
um but it might be interesting for you
to know that I am um I didn't have any
IT training actually um I'm not a good
programmer I actually studied design and
I also work in that field so very often
I'd be the nerd among the people that
are around me because I'm into techy
stuff and many are not um funny thing
however is that depending on the
perspective I might also be the nerd in
this room here because of my lack of
knowledge um my lack of knowledge and
Tech and my Affinity with design but I
think that's fine and we'll find a good
way of exchanging ideas um
today I am constantly searching and I'm
in love with the idea of inter
subjective methodological research which
makes me a Searcher and a researcher at
the same time and finally I love to
tackle
problems I love to repair and I love to
make things things and um in the next 30
minutes I will show you one thing that I
made um that all started with this
picture of me in the right um about 5
years
ago and the end of the project actually
was this and this will be the span of
the talk now the unfolding space glove
which I will talk about is a varable for
visually impaired people translating 3D
Vision into heptic
stimuli
the talk will cover four topics first of
all I will spend much time on explaining
the theoretical background of the
project and then the glove itself and in
the end I will give you some insights
into the study I conducted and how you
can read uh the results and some
findings and an
Outlook so let's start with the theory
right
away and to understand how we perceive
the world around us we first have to
differentiate between the concepts of
sensation on one hand and perception on
the other
hand
sensation can be described using these
well-known symbols representing our
sensory modalities our receptors so to
say um for example the ear for hearing
the eyes for seeing and so on and for
sure there's more than these five
classical ones but we'll stick to the
we'll stick to these for now for sake of
some
it these receptors now pick up so-called
sensory stimuli of the physical world
around us Sensations um that is specific
properties about the world such as the
brightness and the color of light rays
reflected or emitted by objects in space
around
us they translate these into electrical
signals nothing more than that and they
sent them to the brain and now these
days we are very good in observing these
these signals studying them helps us to
diagnose and to treat possible disorders
of our nervous system and gives us a
great insight into how our brain
works at least to some stage we can even
build artificial perceptors by imitating
them for example the widespread cckar
implant for people with hearing
loss what it does not explain however is
the somewhat philosophical question of
how we end up with having meaningful
ideas based on these signals because
until now it's just electrical signals
right so in other words why do we
perceive blue as being blue and why do
we perceive a cat as being a cat or a
vinka
cater this is all done by our perception
and this is happening in the brain which
organizes an interprets interprets the
information and ultimately gives us our
conscious experience of the world
now for a very long time it was assumed
that there is a specific brain area and
this is very simple of course we don't
have more time for that now um a
specific brain area for each modality
that we have so for example the sensory
perceptor the eye was connected in an
unchangeable way to the visual
cortex this would explain why
information arriving in the visual
cortex would be perceived visually and
not for example atively just because it
arrives in that region of the brain so
the region of the brain somehow defines
the quality of the data that comes
in now if one of the receptors were to
fail for example you say you say you
lost your site due to an accident you
would have no visual perception at all
because you would have no way to
transfer visual information to the
actual visual cortex because there's
just one connection between the eye and
the visual cortex and this is not
available anymore right and these
traditional theories as we will call
them for now have a number of
explanatory problems exponentory gaps
they're also called and we will have a
closer look into one observation one
more recent observation of the brain um
that it fails to
explain to illustrate these observations
pick this blue area in the brain and
again this is very simple of course um
and think of it as being the language
center now if there's any damage to that
part of the brain for example after a
stroke we now know that the brain can
reorganize itself and even in adulthood
relearn the Lost skills in another part
of the
brain the skills won't always be fully
restored of course but in generally in
general it's
possible so this movement of the blue
dot here moving from the damage to
another region of the brain is it's
called
neuroplasticity it is the ability of the
nervous system or brain to adapt to
changes in the environment by
reorganizing neural
connections now modern approaches
generally take this into account and
again we can't go into detail here but
in essence they say that sensory
information arriving in the brain shown
here with a white line
does not yet have a specific quality
instead the brain examines and now stay
with me because this is getting a bit
tricky the brain examines the sensory
stimuli for the characteristics of
change due to bodily
activity that means that if you move
your eyes and some piece of that
incoming information of that Highway
this data stream of uh informations from
all your sensory perceptors The Ear The
Eye and everything if some of this
information changes in relation to that
very movement you just did you can be
sure that this part of the information
must be of visual nature and that is why
you can actually put visual quality to
it visual qualia how it's called also um
that's actually the reason why you can
pick out the visual quality and see in
the
end this may be difficult to understand
in detail but it can be summarized by
saying that it requires a body and M
action to
perceive this is why these theories are
often called an activism or embodied
cognition in cognitive
science now let's try the thought
experiment again let's say the eye has a
dysfunction and no longer receives
stimuli you can't see because there is
no
information and here's the clue we are
now simply feeding the information with
visual quality so images
pictures into the tactile
sense the brain now lears to learns to
interpret this information as being
visual because of the just discussed um
Associated movements and characteristics
and we can now only not only use it for
everyday tasks but also develop a visual
like experience with it Studies have
shown
this this concept has a name a long
history of research and yet it's not
widely used or widespread sensory
substitution means that if one sensory
modality is missing the brain is able to
receive and process that missing
information by means of another modality
another uh
perceptor receptor
sorry now looking back on our graph um
one last time there's one thing we need
to do to make this happen because we
cannot feel visual images we cannot feel
visual um data so we need a translation
device a device that puts the visual
information into our tactile sense kind
of and that little error with the circle
here um is such a translational device
it's a sensory substitution device in
short also SSD um and that's what
actually actually my project is all
about I built a sensory substitution
device the unfolding space glove is such
a
device
so in the next session I'm going to talk
about the device itself the glove where
does it come from how did I build it um
let's just dive into
that in 2018 I started to do research on
navigation Aid systems that use spatial
information 3D information from a 3D
camera because back in the time sensory
substitution projects in science mainly
used standard cameras only so gray scale
or color cameras um that seemed outdated
to me and I saw potential so I started
in researching that during my bachelor
thesis so I experimented with different
forms of application so in the beginning
it wasn't sure that it's vibrations only
I Ed different means as well um but I
pretty pretty quickly settled on the
back of the hand as um and vibration as
the medium as the
concept basically this rough prototype
that you can see here um emerged within
just a few days um but back in time as
you can maybe think was a bit cumbersome
and clumsy because it was difficult to
put it on and off to all the taping um
and also it was very slow so you
couldn't really feel um the outcome but
it was a starting
point then still 2018 um a few weeks
later this functional prototype was the
end of my bachelor thesis um all the
components including a laptop were now
in the backpack um it was still bulky
and impractical was Heavy you needed to
attach a lot of cable to make it work
and uh a problem with this prototype and
this is just another close-up shot of
the very same prototype was um it's high
latency um I probably do not have to
tell you what latency is but in this
case I mean the delay between your
physical movement the movement of your
hand for example um and the feedback
that you can receive as vibrations and
as we learned just in the first chapter
this relation between movement and
feedback is key in learning so so it
can't be delayed it wouldn't work
well so I continued to develop the
device further so here you can see an
intermediate version of
2019 that does not need any backpack
anymore we have a small um arm armband
that you use for running um where I have
a raspy Pi instead of a computer and a
battery pack um so it was easier to put
on and off it wasn't that heavy um and I
also did in the same time many
improvements on the softare Ware side so
now we're in 2020 2020 um so software
you can't show that easily but I also
tried to streamline um the hardware more
so now we just have this one Computing
unit consisting of a 3D printed housing
a slim down Rasberry p and the battery
pack again and now it was only a single
cable connecting the two things still
the boot app time was too high and I got
new problems introduced by using the one
cable so a lot of work to do which I
then finished in 2021 with the final
prototype that I also will present in a
few minutes
um I was able to draw on a small grant
for professional advice and also for
purchase of components so maybe at that
point many thanks to all the people that
helped with the project that supported
me partly in their jobs partly in their
free time some of them are here in the
room um thanks a lot uh for your help
help so now all the components um are on
the glove itself we have a carrier board
um the computer the motors everything on
the glove and we now just have a USBC
cable for power supply um so you plug
that in um and the power bank still has
to reside on the upper arm uh then you
need about 10 seconds to boot the system
um and you're ready to go the biggest
difference again to previous models is
uh under the hood it's the performance
um the latency between input and output
is now less than 50 milliseconds um
which is no longer noticeable in
practice because vibrations also have
some latency um and and then of the 50
milliseconds 90% is uh due to the camera
technology involved so you can't go any
further here you would need different
technology the exchangeable units when
you can see here in the this in the
picture um from top to bottom have an
IDC socket which is the connection
between the unit and the glove so here
the um voltage to the motors uh flows
and um second point is the time of
flight camera and short tough camera I
spoke about that again so um this is a
3D camera that actually determines the
distance of each pixel um by measuring
the time that a light impulse needs to
travel from the camera to the object and
back um I needed a computer which in
this case is a rasir p Computing module
compute module 4 um this gets very hot
actually if you do the research in an
August with 30° so I needed a heat sink
an active cooling on top and finally the
most challenging part for me at least
was um the custom circuit board um that
carries all the components so um the the
compute module on one hand and also um
the drivers for the vibration Motors
because you need need specific drivers
and you can see those drivers here on
the board um and also the board was
built under highly professional
conditions as you can see in the upper
right picture again with the help of
friends who know how to do this
process of course the whole project is
open source you might have guessed it
already so um like the rest of it this
is the open hardware part of course uh
you can find all the exports renderings
and schematics um online in the hardware
repositor there's two repositor and of
course you find also the key card files
if you want to work with that um then
the building of the glove um took like
many steps first of all you need to
select the textiles um uh you need to
develop a robust construction of how to
put the cables how to do the shouldering
points so that it don't break if you put
on and off the glove people don't really
um they aren't really careful with
putting it on and off so you have to
think that first um and all this
tinkering um is also documented so
there's a step bystep building
instruction of the glove that you can
find on
GitHub um so also if you don't want to
build this glove but have something
similar in mind this would be maybe a
good resource um to
start then last but not least
challenging for me at least the code it
is written in C++ because of the camera
Library um and if that wasn't complex
enough for me I also had to integrate
multi-threading over time again with
help of friends um and I build up a
cross compile environment to be able to
iterate and develop quickly if you're
not a good coder iteration is
key you can also find of course all the
software Parts um on the main repository
with some documentation as well um also
the source code files of course um I use
a lot of libraries and uh locking things
um probably could do these things more
efficient but my part of the processing
actually just takes 3 milliseconds so
there's not much to gain
here so time for a short summary the
final prototype is a spatial visual to
haptic sensory substitution device
that's a long s complicated sentence so
what I generally do is I take depth
information and make it haptically
tangible haptically feelable
it is also a navigation Aid system with
low complexity and low entry barriers
why low complexity I only have 3x3
Motors so you can can actually just see
that there is an object you couldn't see
uh Which object is there or which
specific properties it has um however it
allows you to learn the system more
quickly so you could um make it more
complex later on when you learned it for
example um um the final prototype has
some unique properties that I worked for
and that also uh could be proven to some
part in the study um first of all I am
using a 3D tough camera already said
that so instead of Grays scale cameras
that most uh studies used to that point
and that is just um because 3D images
are more accessible than standard
grayscale images because you want to
have the location of objects in the room
if you have a grayscale image you would
have to analyze light and Shadow
perspective paralu all these things to
actually extract the location data um
you just SK that by using a 3D
camera which hasn't been there of course
when they started to do sensory
substitution um studies now the second
point is um using the back of the hand
as in and output um offers a very high
degree of um possible interactions with
the data so um you can actually rotate
your hand and move your hand in three a
access uh axis so um as we've learned
interaction is key to the learning
process um so that makes the the first
steps
easier also important for the learning
process is of course that the feedback
is continuous and that it has a low
latency we spoke about that um and the
device has low costs on top so um that
is mainly because I use mass-produced
and non-critical technology so you can
actually put it on and also put it off
if you don't like it
there's no surgery needed or anything
like that um the components for the
Prototype would now be around 500 bucks
um I started the project 5 years ago
with technology that is available here
today you could build it for like
probably for about 100 100 bucks without
the tough camera um and last but not
least it is also variable already I mean
with that that you can use it outside of
the lab many projects have to stay in
the lab have to stay in a specific
system so so here we can go out in the
bright sunlight on the street with an 8h
hour battery
charge right and next is you just put
the unit onto the glove I already have
it here on the glove and you put on the
glove then you have this little strap
where you can place the battery you
place the battery by just turning it
once plug in the cable
and then the system boots up as I said
it's about 10 seconds so in a few
seconds I will be ready to go already
and um I can show you
now the data in a few more seconds when
also Wi-Fi is set
up
hey
okay I can't click
here um all right so now you see what I
feel actually
doesn't work again that's so sad um okay
you at least see the screenshot you saw
it for one second um I can go back to
the presentation where we have um one
picture
showing an
exemplary okay all right um an examplar
picture so um on the left side you see
the motor values um which range from 0
to
255 um blue actually is very close
purple is even closer green is about 1 M
and then after 3 m there's a cut off so
all the black things we ignore you can
Al this is just a setting you can set it
to off um outdoor mode or to indoor mode
and um on the left like this app is also
there for um for the study of course to
um see what this the participants saw
and also for getting some uh monitoring
data like um how warm the glove gets um
still doesn't work okay that's good back
um and right um that's actually the
depth image and as I said on the left
side you see the motor
values okay I will give you now a short
overview about the study I
conducted so first of
all I had six blind and eight cited but
blindfolded
subjects the central idea here is that
by including
the sighted subject it is possible to
compare how quickly devices can be
learned if you have to learn both
devices from scratch I had a complex
study design which also used mixed
models to actually make use of that
combination I had a controlled test
environment the obser course you can see
in the upper right image the numbering
system on the ground to place the
obstacles uh in a in a quick Manner and
on the bottom you can see one course set
up
I used qualitative and quantitative
assessments so first of all I had of
course the time you need to solve one
course and the result in collisions and
secondly I had questionnaires in
interviews collecting the feelings and
the needs and thoughts of the users
especially of course from the blind
users well then the subject had to do
some paperwork in the beginning of the
study and then they started to explore
the room with the glove so they could in
the training process ask all sorts of
questions um I had some semi-structured
training program where they could point
on objects then grab for them pass them
um all of that was basically to learn um
the very difficult part to understand
the relations between the felt patterns
and the reality so kind of the
adjustment in the
beginning and then in the test runs so
this is videos of the test runs um they
were on their own trying to solve a
predefined pseudo randomized set of
courses and as you can see they did it
with both devices um and they completed
um all layout all course layouts twice
without realizing it so I had one
version which is um the regular version
on the left for example and then on the
right side the same course but mirrored
um that is not just a clever statistical
move um that someone told me again but
also you we have now footage of the very
same same course actually done with the
two devices by the same person at the
same stage of training so what you can
see here is that the cane is faster
always and um you can also see that the
the glove Works um you can see that the
wave finding actually uh is nearly
perfect um so even better with the cane
sometimes um so kind of taking the
optimal path but slower and this is kind
of the main finding of the study as well
um that I actually expected before doing
the study so getting into the findings
and outlooks um the first and Main
finding is that the cited Group which
had to learn both devices um over
duration of the study uh itself so two
hours of wear time with uh the cane and
the glove over that time the cane is
faster and causes fewer
contacts so what does that mean whole
whole project is failed or uh what
What's the answer here so no that's not
the case uh that's science actually I
want to tell you why this study still
have some very interesting
outcomes um first of all I expected this
result as I said before the study um but
what I didn't expect actually is that
all subjects could solve all the courses
so in totally they did over two month
they did 800 runs and none of them
failed so um that is a proof actually
that subjects could learn to navigate
through this specific course by using
the device um secondly I often
experienced a very good performance
already um some some some minutes after
the first uh tutorial with the device so
just to note in many studies about ssds
they just let subjects um use the device
for dozens of hours in one case even 180
hours we had just one two hours of
testing of training before we tested so
just look at B again
Ben is here um walking 3 minutes after
he got the device on First and he kind
of chooses the optimal path already of
course he's slow you can clearly see
that um but no no 880 hours needed here
to get
started the third point is
um there are many examples of Highly
efficient wave finding I already told
that um so that means that even if
you're slow you kind of take the optimal
path and some of the users also reported
no most of the users reported uh deep
sensory
integration um meaning they said things
like I don't have to concentrate on the
vibration patterns anymore I just know
that there's an object I walk around
because that's what you do um and also
the last Point um I already said this as
well um there was a positive future
experience so people had actually fun to
learn the device and it was an
interesting new feel um of the world um
to have it with vibration
patterns all these things are summed up
in numbers and graphs in a scientific
paper that um appeared in 20 2020
2022 um again Open Access and I'm very
glad actually that it has been quoted
for a few times uh just because my
efforts always were that this um project
doesn't uh isn't there for just a few
pretty pictures and then end up in ends
up in a drawer but um that it would be
found on the internet maybe and people
take it reuse it and put it in their
project as well um that already seems to
work however to actually get proof for
some of the derivations um that I did
here um you would need more research um
it would be interesting to see how
people perform after 10 hours of
training after 20 hours maybe um but for
all that you would need funding you
would need uh institutions to work with
and and that is um a bit difficult with
such a niche topic um so that will be
kind of the summary the Outlook um of
the project but for now I'd like to
thank you for your attention sorry for
the messed up live demo it's always like
that I guess um if you have any
questions uh go to my website or just
ask your questions now I will be around
for a while and if you want to try out
the uh glove we can arrange a meeting
probably in the next days uh to test it
[Applause]
so you mentioned that there's a a deep
sensitive re integration and also that
you assume that this is in a ver visual
cortex sorry um so for the forly seeing
people that are no blind did they have
some sort of visual feedback in terms of
colors or or feelings or or how did you
come to this particular conclusion so
you mean how they described the actual
feeling they had um so the the point of
the deep sensory integration leads or um
points a bit to that direction because
they said they don't have to uh kind of
interpret what they feel but they kind
of know that there is an object and when
you ask them like do you see it then
they wouldn't reply like that and I
didn't ask it in the study because it
would be a suggestive question of course
um but afterwards um uh they describe it
a b differently because also the
information is so low complex probably
um I said that because there are um
studies about sensory substitution
devices they take a diff similar
approach and they could actually prove
that the in the MRI picture the visual
cortex get active um when you train a
lot with these devices so given from
your findings would you say that a
seeing person does perceive this
different than a non-seeing person yes
okay yeah blind people and also so
visually impair people have it have
easier time learning it probably Mike
three um thank you for your excellently
delivered speech I talk very clear um my
question is about uh placing the PCB on
the knuckles of the hand I see why you
would do that because the 3D camera must
be there but isn't it very fragile there
and uh fragile for uh bumping into
things um yeah probably I mean you would
have to build it it's not a um it's not
a product that you can use on the
streets actually it's a prototype I said
you can use it there because it's maybe
a good reason to test it on the streets
but it's not um also for many other
reasons uh not uh at all a replacement
for the cane for example which is very
versatile and reliable um so this is
just a prototype and you would need to
make protection for um destroying it of
course um I chose the region because uh
the sensor and the actus in my opinion
have to be very close to each other so
that the feedback loop is kind of set up
um and uh the position of the hand
having it like that um is a pretty
natural one so many people suggest doing
on the inner hand then you would walk
around like that which is a lot more
effort I would
say cool um there is a question from the
internet I have three actually but first
one is are you familiar with Professor
Thomas menzinger that there seems to be
similar Vue of your search I didn't
understand the name correctly but I
think I heard it the
singer um I'm bad with names probably
yes I mean I I basically read all the
sensor substitution device studies um so
probably
[Music]
yes go ahead the next
one okay let's go with three hello hi so
from what you said about further
research and the need for funding I I
gather uh you don't uh develop it
currently as a Comm commercial product
at the moment my question is have you
tried seeking this venue have you tried
looking for funding
for for making it into a commercial
product um and what were responses not
really because I wouldn't consider it as
being a commercial product or um a
navigation Aid system with all the
functions that you would need in such a
device um to me that is a research
project um that looks into how you can
interpret 3D data
um I already said it but for example um
is a very limited scope of information
that you get for example objects on the
ground are not included yet only if they
have a specific hate then they would be
in the picture so uh this is just a tool
to research into how you can integrate
3D Vision um with um vibrations actually
tactics me okay yeah sorry thank you for
the good speech and my question is um
have you experimented with different s
sensors radar for example and why did
you choose your specific time of flight
sensor in the end um in the beginning of
the project there haven't been that many
um tough sensors for example there's
very cheap ones out now um at least I
didn't find them and I wanted to be as
flexible as possible so I wanted to
build a platform where you can also do
different things with so we have now
Rasberry P which has a very low CPU load
so you could now for example also do
object recognition tasks um and other
things on the rasir pie if you would go
for an MCU with a smaller T sensor you
couldn't do that for example yeah okay
thanks M three uh I just have two
questions uh the first one um like as I
can see like from the arm for example if
you have like a hole in the ground and
it will be really hard to detect the
hole somehow then the blind people will
fill down the second question is like
why you didn't design your own BCB
somehow and and then you can make it to
to the SMD so you can create your own uh
mini computer specifically for this
tasks because then that will save the
power uh consumptions of the battery
yeah so to the first question um uh I
needed to reduce the comp the things
that I can research on so I researched
on first of all getting the 3D data how
to can we interpret it um and how well
would that work in a very limited way of
obstacle cross these boxes um so you
would need to inte integrate that I
think it's possible by scanning the FR
to differences and these kind of things
uh and there's also the possibility of
mixing the cane with such a device so
it's in the left hand the cane will be
in the right hand for most people and
you can switch it um and the second
question is kind of the answer of before
um uh of course if I could do it if I
would have the ability I could build
such a PCB um but for now it's also it's
a prototype so with this one I can also
just switch the the camera if it gets
broken and it did break in the study so
I'm very glad so that it's kind of
modular um and that I can put things on
or off if I want to go in a different
direction um because I didn't want to
put it put all the possible um uh
developments in in uh already by
choosing a PCB a specific PCB okay
thanks maybe one last question from CH
hi um nice project I saw you have nine
different vibration devices on the glove
yeah right Motors yeah yeah how did you
go about mapping the time of flight
information to vibration it's actually
pretty simple and algorithmic thing I do
a histogram and search for the closest
object that exceeds some threshold
because you have some grain and some um
noise that you want to filter out and
then um it's basically pretty pretty
quick task yeah okay
thanks okay any other question just
after the talk thank you
[Music]