Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/421 Thanks!
So my name is
in the Herald and
I will give you a little introduction to
Kay Kunstler, who is
from who's currently working in Japan
at, uh, Kiu,
and he is
going to give a talk about,
um, about eyewear computing.
And this is a really interesting topic
because our human brain is not
just limited for specific
visual processing, but it's actually
kind of, um, you could
say it is doing things
on its own. It is processing things on
its own. And its aim in
the end is to enhance
to maybe enhance the computing
we do by
specific devices, such as,
for example, eyewear.
And he's going to talk
about the beginnings of that.
And so give a
big round of applause for high concer.
Thank you.
Thank you very much, Ystem,
the mic is on.
Um, can I get my slides?
So for me, it's always.
You know, I give a lot of talks, but for
me, being here at the
Chaos Communication Congress is always
special. So I'm also a little bit nervous
and I hope that I won't be wasting
your time for the next 25 minutes
or so on. So the topic I want to talk
to you about today is where computing
augmenting the human mind, as mentioned,
and working at Kayoko Immediate Design
University in Yokohama, Japan.
And before I start to give you an
overview, I'll show you a quick demo,
because before and I had some problems
with Bluetooth in this room.
So I think now the demo works.
So what you see
here, I'm currently wearing
prototype from Ginns, from Japanese
glasses maker, and
it uses electrodes to measure my eye
movements. And you see my life blinks.
It also has an accelerometer inside.
So you can get also the motion
of the head.
So I'll go into details how this
thing works and what you can do with
it also.
But because the demo worked right
now, I thought it's good to start with
it.
And
so let me give you an overview over the
top. OK, so first I give you a
little bit of background about myself,
then I'll go into details why I think
I where computing is is interesting
or something to explore.
And then I'll talk about how we can
make it accessible so we can use
it every day. And normal people, not
only as geeks can use it, especially
enabling technologies like
electroencephalography.
And then I'll give a little bit
of an outlook on how we
can be able to measure
other cognitive states, something
like cognitive workload or so
on in real life.
But that's more or less in outlook.
And we just started with this work.
So as a background, I worked in wearable
computing.
And this means, you know, I used
mostly motion sensors and other sensors
that put them on users and tried to
figure out what they're doing.
My former supervisor, Paul, you
see him here and in this picture
called the call to the Christmas tree set
up because we put a lot of sensors on the
people and try to figure out what they're
doing and try to support
them during everyday tasks or doing
maintenance work, but also sports.
And a lot of my work was
trying to figure out how we can get from
these dedicated sensors towards
something you can wear with
your smartphone or sensors that are
embedded in everyday government.
And then since the last
two years, that got more and more
interested in what's going on in
our mind.
So can I also detect
cognitive state or cognitive activities?
In this case?
The eyes got more and more interesting.
You know, I started with Ichi and I
found, you know, I'm a little bit lazy,
so I found it too difficult
to make sense out of EEG signals.
And the eyes were another way
to observe the mind because,
you know, there are or
we use them every day.
We use them even while we are
sleeping, even when we have our eyes
closed, our eyes closed, they're
still moving.
And there's a lot of work in psychology
and cognitive science and other related
fields that link eye movement
to something like attention,
concentration, intention
and higher cognitive tasks.
But unfortunately, in most
of the eyes, today is used
for something like this or for
entertainment or
marketing. And now that we have mobile
eye trackers, you know, you also do this
in supermarkets and figure out
how your product must look like that.
People will pick it up.
And I think that's a waste.
And that's a little bit sad.
Um.
So last year,
I gave a talk about tracking
reading habits.
So in this case, we used
an mobile eye tracker from some
eye. This is also an optical system
using infrared lights.
You see on the top the view on the eyes
and infrared cameras underneath
your eyes to detect your
pupils. And with this, you can recognize
your gaze and you get suckered so
fast eye movements and fixations out.
And with this, we can recognize
reading and we can tell how many
words you read.
And not only this, but we can also
figure out what kind of document types
you're reading just by looking at the
visual behavior.
And that's important for Japanese
students. We can distinguish
science books versus mangas
and
and also we try or we slowly
try to assess how much she understands
or comprehension. But it's trickier or
more difficult.
That was last year.
And you could already build
some kind of Fitbit for the mind.
I'm not really sure if you want to have
this. You know, you could get your
reading count during the day.
How many words with what speed
you're reading, how many Mangus
versus science papers.
And but I'm wondering myself,
you know, what a healthy reading habits.
How can I improve my reading habits?
What happens if I copy
somebody is reading over
two or three years.
Will this make me an expert in a certain
field?
The biggest problem I see right now
with this approach using the technology
I introduced last year is
first of all, I mean, this just works for
reading.
So I would like also to have something
more channels. So focusing on
attention, concentration, focus
in in real life settings.
And the other thing is it should work
during every day.
And the biggest problem there is the
mobile eye tracker you see over here.
I mean, it's not something you would like
to wear and
maybe some people would like to wear it
first. And then the next problem is then
also the battery power or, you know,
the device is not.
So you can maybe record
four hours or five hours in one go.
And so we focused already on
tablets and smartphones.
But then the problem comes that, OK,
if you turn this tablet of
it's not working anymore, so you don't
have a coverage of the whole day.
So we try to implement eye tracking on
these devices. It also doesn't work so
well with a front facing camera.
The computer vision is quite complicated,
but we did also some work on on Google
Glass, although, you know, it's not
maybe socially acceptable everywhere.
There are more problems in Germany if
you walk around with Google Glass than
they are in Japan.
So it depends also a little bit on the
cultural background, but you can do some
stuff with it. And now I want to show a
demo a little
bit, an extension from last year's work
on glass.
So basically
Glass doesn't have an eye tracker, but it
has a small infrared sensor
over here that's
used to detect if you wear the device
or not. And also you
can have a long blink to take pictures,
which are fun. Yeah.
Uh, so what you see
right now is the picture I see on
the head mounted display in glass.
And, you know, you have this touchpad on
the side.
So I can see
all pictures I took.
I taped over the camera.
So if I take a picture right now.
You should see it just shows black and a
little white dot, because that's the
light sensor, otherwise I cannot see
anything on the display.
So what we did with it,
we have an open source class logger that
locks all of the sensors
from the device and we can
also monitor the infrared
sensor.
So this is something I showed
last year already.
And this is a little bit an advanced
version. And you can also recognize
that if I move my pupil and
don't blink. OK, some of the things that
I detected as eye blinks, but you can see
also the light, the infrared
sensor moves so you can detect
something like left and right movement of
the eye from the change of
the distance to glass.
And now.
A little bit more advanced demo, what can
you do with it?
For one, I can do some very limited
activity recognition with it.
So after maybe two or three seconds, it
should show what I'm currently doing.
This is just a two class segmentation,
uh, problem.
So currently it's talking.
And now
if I focus here.
So we're reading this just
uses my head
motion as well as
the eye blinks to detect,
to distinguish talking and reading,
talking works relatively stable
during the day. Reading is a little bit
troublesome with the infrared sensor
because of,
you know, if you're watching a movie or
so on, it might also see talking,
reading, because that's not working
so well.
But other things you can do with
hyperlinks, as I mentioned, one is
recognizing talking.
So this is simple because your
blink frequency will increase usually
double, and
there are also some correlation
with focus.
So usually if you focus on something,
you blink, frequency will decrease
and also with content.
So what I observed is
if you look at people reading from an
eating display, you recognize
that the blink when they turn the page.
And I would like to use also
the change blindness during blink.
I haven't really gotten a good idea what
I want to do with it, but you probably
know that if you have some kind of
stimulus or if you have some
if you blink and somebody changes,
maybe the color of the background
or changes some people in here, it
wouldn't recognize because I blinked in
between. And you could use this, for
example, for a display that changes
just when you blink.
So you wouldn't recognize that
something in your eye, in your view,
changed. But if you want to get the new
information, it's already there.
And some some ideas like this.
And I think it would also be cool for
horror games or interactive movies
or so on.
I think you can really scare somebody
with that and another
idea, something that's actually
also fairly simple is fatigue detection.
So just by the duration of blinks
and the blink frequency again, so if
it gets more, more, higher,
you can detect how tired somebody is
and how tired somebody will become.
And I
had the idea of equipping a batch
of students with just simple
Blinkx detectors for classes
and then have a ranking system of
lectures so you can get
the most boring lecture versus the second
boring lecture and so on.
I tried.
I discussed this with some professor,
says Osaka University,
the university was before hand and
didn't really like this idea too much.
I wonder why maybe we can do something
that came with this and then
other things you can do.
Blinkx, together with emotion, helps
you to distinguish closely related
activities.
In this case, we recorded reading,
watching a video solving Sudoku, puzzle
solving to have some
physical activity and also talking and
just blinking frequency you get already
quite far. You get up to 70 percent
and then if you
include the motions, you get up
to eighty two percent.
However, there are some issues also
with class and even more
with the eye tracking
hardware.
It's first of all power, but also
they don't really look so appealing.
So it's not something somebody will
wear during everyday
tasks. So there are new systems from some
eye and also taulbee
and they look
nicer. But you can already recognize
there are things built by engineers
and researchers for engineers and
researchers.
So for everyday people,
for for normal people, maybe,
uh, it's not possible.
So I was quite happy when Nami sensei
asked me to join for a project with
Jin, uh, Japanese classes
company and they make
prototypes of genes meme.
So the glasses I showed you in the
beginning and they have a very different
idea from Google Glass.
So this is not a full fledged
computer on your head.
It has no display, it has
no camera and
the device is just connected to your
phone. So currently it's just streaming
data to your phone.
One is the electromyography
and the other one is motion sensor.
So with one, you can detect eye
movements, a left, right, up, down
and also eye blinks.
And the other one, accelerometer
and Schriro.
We are directly working with them
in the research department.
And, uh, on one thing, I was also happy.
So on one side, I can work on smart
glasses. On the other side for some
promotional video, I could wear the
optical camouflage from the sensor.
So win win situation.
Uh, yeah. The demo in insert the
demo here.
Uh, so you already saw it.
And now I go into the principles,
how it works. So we use electromyography.
Basically your eye is a deep hole
and it has a positive charge in the
front, the negative in the back.
And if you place electrodes around your
eye, you can measure
a movement.
So the regular set up is like this.
You use the, um, electrode
on the top of your eye and
electrode below your eye to get the
vertical movement and one
left and right to get the horizontal
movement. And usually one reference, one
the advantage compared to
optical eye tracking.
Um, you can run a
very high sampling rates.
You have no problems or not so much
problems with battery power because you
don't need so much processing.
And some of the disadvantages
are, uh, it's just relative
eye movement and it can be
sometimes noisy.
So it depends also on the skin and so
on and on the electrode placement.
Of course, uh, there were some
other people who worked on this.
So you don't really have to use goggles.
You can also use headphones.
So monopolization from DoCoMo
used headphones for that, uh,
are spooling used.
Also this regular set up with
top and bottom for up and down and
left and right here for the
horizontal, um,
potential measurement, what the
chins mean now that, uh,
they integrated it into normal
eyeglasses. So they use a three point,
uh, Yochi here.
You see it in detail.
And if you can go to the GoPro, I can
also show it to you
on the.
Video.
No,
now, OK, so you see, these are the three
electrodes, so use the left and
the right one.
For if I'm not shaking
so much, I can actually also see
something on the left and the right ones
for the horizontal axis and the,
uh, these two then for
the vertical axis.
OK, back to the slides.
Right now, these are just first
prototypes and the battery runtime is
around eight hours streaming data.
If we do, On-Board, processing also will
get even better. So now you might wonder
what you can do with it.
Um, for one, we
can recognize the eye
movements as you see here.
So blinks and left
and right.
This was one of the earlier prototypes,
that's, by the way, ChawIa, he was here
last year. Unfortunately, he couldn't
troyen it this time.
And you know, what is the first thing you
try to implement?
If you have a binary
control system like I blinked
Flappy Bird, this is really, really,
really hard to play.
You also see it in the face of Shroyer
and I
can try it out for later and maybe you
can try it. But getting even a score of
one or two is hard.
What you can also do is, of course,
more interesting activity recognition.
In this case, we can detect reading
and just by time we assign a word
currently. So this is 15 to 20 percent
error. But of course we can analyze the
data further and get better
results. So reading and also the
talking or the head
motion change, as well as the blinking
frequency change easy to
detect. So you can get an overview over
your day how much physical
activity you did, because it also has
an excellent meter inside, but also
social interaction and how
much reading you did.
Now, I think right
now they mostly focus
for the product or so on.
They mostly focus on fatigue detection
and they want to release sometime next
year in September.
But if you're not into consumer products
and so on or don't want to wait till
September and want to build it yourself,
it's not so difficult.
So there are a couple of instructions
online. I was mostly astonished
by the eyeballed by
a Honduran teenager.
And, you know, his his setup really
works. So Musci, a
student of mine, also built one.
He took several of the Dowa
instruction sets and he just uses
two electrodes left and right to detect
the horizontal movement.
But as you see, it works
relatively well.
So even small
eye movements or so on can be detected
with a DIY set.
And now all
we have to hurry. So just five minutes
left before questions
and answers.
And now we are also trying
out other
Yochi set up the electrode
setups. What works best for reading
or for for other
cognitive tasks?
And you might wonder, can we use
Yochi also for
this tracking of reading habits and how
does it compare to the
optical eye tracker?
So in this case,
we use a medical yochi with
active electrodes.
So the one I have equipped
there and just looking at
the horizontal
component, again from the potential,
uh, that's a graph you see here
for me reading.
And I just use a simple peak
detection algorithm to detect that the
line breaks.
So these long peaks are the line breaks.
And interestingly enough, you can also
detect very small line breaks.
So kind of two or three word things.
So these are the not so
deep peaks.
And that's not possible with the optical
eye tracker because the sampling
rate is not high enough, so that's quite
nice. Oh, and by the way, this is
not published yet, but as soon as
it gets hopefully published,
also share the code also on and
share how you can
kind of filter. Also on the single.
And then for the last part,
for the last five minutes, I want to talk
a little bit about how we
can recognize more
general cognitive states.
In this case.
We try to track cognitive
load or brain sensing.
So we used near
the near infrared spectroscopy device
from Ishimatsu Leibniz's and it's
estimates the globin
change in the Blatche, in
this case in the prefrontal cortex.
So the oxygen change
in the blood we used the
Lebanese the genes meme and
an optical eye tracker, stationary
eye tracker.
Device, what do you get from the nurse is
brain activation, so the
red means high activation, blue
or green is lower activation?
Um, strange interface, but
yeah, and this
is the end, I guess, synched
with the brain
activation. So in this case, the reading
task.
Uh, but what I'm really interested
in, we didn't only record reading,
but also calculation and
so memory games.
So in this case, you see a memory task
and back, uh, that's
a common task to assess memory.
And one back is relatively easy
to back. It's already quite difficult
if you're not trained through your back
is more difficult and fall-back
nearly impossible if you have not
trained. And this is a recording from one
user, uh, with the
workload, you know, as I said, one was
easy for him.
Looking at the effort as
too was a little bit more difficult.
Three was most difficult
and for the person gave up.
So and I'm
most interested in this state,
if it's possible to detect the state,
not with the Lebanese device, but with
something like blinks,
I guess, features or some other,
uh, cheap sensors,
we can implement something
that keeps us challenged while
learning and not frustrated.
And that's something I would really
like to go. We just recorded the data.
I'm not sure if there's something there.
I saw some correlations, pupil diameter
or something, but that's not something I
can get over.
The Yochi Blink frequency is also
interesting, but it's not good enough to
detect the state yet.
Also predicted, because what I really
want to do is I want to predict that
if I increase the difficulty for a task,
the person will give up.
And that brings me more or less already
to the summary. So I believe that
if you look at the last century is the
biggest scientific breakthroughs where
about our physical limitations
so we can now travel faster,
we can build higher, we live more
comfortable and more along the longer
lives. And I believe the future
will be about overcoming all the biggest
scientific breakthroughs, will be about
overcoming our cognitive limitations.
And let's work together with
open tools and open, uh, a
code to to achieve this.
And that brings me also to a quick
thank you slide just one to name two
people. So especially Oliver, I'm often
the cost of Cuba who enables
part of the talk because of some hardware
I got from him and a special thanks
to the people who actually did
the work.
So.
So
you got to mind
and now if you have questions,
remarks or violent dissent
and also if there are people interested
in open eyewear platform or
so on, please come and talk to me later.
Thank you for the nice talk we
have.
We have five minutes for questions
you can line about the microphones.
We will start with microphone four.
Could you explain the blink again?
How do you capture that?
So for the blink,
I'm sorry for the interruption.
Please move out quietly.
No talking because there is a
poll.
You said it's a deep also it moves down
when you blink and blink is actually
muscle activity you get also.
So what you see here is.
Blink is relatively
nice.
So kind of this this signal
in the end and you get it over
the muscle movement. So this is not the
deposal.
So the eye does not move.
In this case, you move your muscles and
that's also something you can register
over the electrodes.
That's actually how do you say noise
signal if you want to recognize
Yochi.
So you have one kind of sensor capturing
muscles and the whole movement
of the eye.
Yeah. And you also see, I mean, you also
have to do some filtering.
So if I move my head and so on, of course
you will get this also on the Yoji
because of the small banks.
Yeah. One question from the signal,
Angel.
Uh, the question
was what sampling rate
does the Yoji operate
with for the line lengths?
So for sampling rates, this
depends on. So the urchins
mean currently samples at around one
hundred hertz.
Uh, it can go, I think, up to two
hundred hertz.
This has nothing to do with the
electrodes but with a chip inside
and for the line lengths, the data
you saw beforehand, that's a medical
Yochi. So this is five hundred hertz.
So here you need some some higher
sampling rates.
So for this you need five hundred hertz.
But I think maybe with two hundred or one
hundred fifty, you would also get
reasonable results for for line break
detection.
Thank you.
Microphone number three,
I was wondering if there's any ethical
reflection included in your studies
and in your work because.
Well, you don't have to have too much
fantasy just to think how this
thing, which is funny here, can be used
as a perfect tool of control.
Yeah, I agree.
And I think there's already
there's already a lot of work.
That's why I had this advertised in
Slate. I think we are already behind.
You know, some companies that work
in this field and have more knowledge
about the relation between
brain and movement
and so on, and also kind of how
you get this whole their whole study
is on how to place products or
how to design a supermarket or
something like that that are not out
there. And for me, the important
part is that the data stays
with you. You know, I don't want Google
or Apple or somebody like
this developing
glasses and then getting
all of our data, because I'm already
scared from the data,
you know, just the location data and
other data. I'm giving it away for free.
So there's definitely an issue.
I think, you know, one thing we can do is
just try to
open this research up and show people
what is possible to
hopefully prevent somebody
misusing this.
But, yeah, that's that's definitely an
issue. Thanks.
My phone number two.
Yep, yep.
My question is, are you doing any
research to connect your technology
with virtual reality?
For example, in the field of architecture
or in arts or.
I'm personally not
so much, but there are some people
that came through, especially not me,
Sensi, who who worked in virtual
reality and also thinks about this
that I had not.
So I think for especially for virtual
reality, I think optical
eye tracking is nice or nicer because
you can get the relation of where you
are, where where you're, where you're
seeing. And you don't have this problem.
If you're already wearing an Oculus Rift
or a similar device, you don't
have this problem of,
you know, having something lightweight
or so on to carry Yuchi.
I'm not so sure.
I think, you know, this Blinkx
thing could be something.
You know, the change Blanton's doing
blinks could be interesting, but
yeah.
One last question from NO2.
And I wanted to ask you about
whether you have you thought about
capturing brain waves
from the frontal cortex?
Uh, because it seems like, you know,
something that is around,
uh, so I tried
each year beforehand, so I played with
the motive. I have also the open BCI.
So the dot com thing, not the
thing.
And I was interested in the talks
yesterday, but unfortunately, I couldn't
get in and I haven't really met the
people from from my experience.
As I said, Ichi is relatively noisy
and it depends highly on users.
So for me, surprisingly, it works very
well. For a lot of my students, it
doesn't really work well because
of the placement of the electrodes and so
on. So I'm interested and look
into it.
But for me, being
lazy, I guess seems to be easier to
come by and and use.
Also, I find it funny and it's also
interesting to some part because we
saw now that maybe two
or three channels could be enough to
detect something.
So it definitely look into it.
But, uh, yeah.
So so far it seems to be
for me, more difficult.
Then using, I guess, things
one last round of applause for Kydd.