Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/761 Thanks!
I'm inviting you to a little
imagination activity.
Imagine that you can take
all of your experiences that you
have made at this Congress, take
them home with you and share them with
your friends and your family, everything.
The first time that you tried,
Marta?
The moment, endless moments
of going through the labyrinth of this
Congress hall and getting lost
for the hundredth time,
trying to enter the
club downstairs and
feeling like you can cut the smoke
with a knife.
Or actually being very
surprised and seeing Snowden appear
on stage, so all
of these wonderful memories, I imagine
you can record them and you can take them
home with you today.
This kind of recording activity
possibilities actually possible
to make this technology
that could theoretically
record everything 24/7,
Chi Conesa will
give us an overview of sensual
and actual actuation technology,
and he will tell us what the future holds
for a AR and VR.
He has brought with him some
experience. Some projects are actually
happening right now and will give us
an overview of how we are already
pushing the boundaries today
of humans sharing experience.
Kai is a researcher in variable
computing in a R
and VR and really
cool. He's also a founding member
of the Japanese Super Human
Sports Society.
Thanks.
The zone and without
further ado, Concer Ingi
is the mic on, yeah.
OK, thanks.
Thanks a lot for the great introduction
and thanks a lot also for
waking up so early in the morning
to come and join to talk.
It's always a special occasion
for me to talk about the chaos
communication Congress.
And I hope then for the next
30 minutes or so on, I won't be wasting
your time.
So the talk will be about, you know, kind
of from superhuman sports to amplifying
human senses.
And as the introduction
said already kind of we are
in the year zero of
consumer available VR devices.
So Oculus, the
wife and also
the Sony VR system and a cheaper
cardboard variants like Google Cardboard
and so on.
And we are on the brink of getting
consumer level 8R
and, you know, kind of already my
grandparents have holo lens.
So yeah,
in this case I always like to include
pictures of my grandparents because
they are awesome.
They always try out any type of
technology I give them and
they provide valuable feedback.
However, I have to start with a big
disclaimer and I will disappoint
you, especially regarding the
great introduction I got for the talk.
Please don't trust me,
especially about predictions.
I was working 15 years now and the
variable computing field and around
2004 I thought, you
know, today we would run around like
this.
And I see a lot of you kind of wearing
your head mounted displays and having
your one handed keyboard.
Nobody.
Oh, damn it.
So I'm really, really bad with
predictions. And kind of that was me at
the time.
One handed keyboard.
KUBIC This is a belt integrated computer
from from stage at
a time. You have to remember we didn't
have smartphones, so we had to carry
our computing somewhere else.
And then, you know, kind of seriously, a
couple of years back, I also thought this
would be a thing.
So so Google Glass and this type of
stuff. So don't trust me
about predictions and
don't trust me on what's coming next.
However, for this talk, you might wonder,
why did you get up so early in the
morning then?
What I will do today is I will talk
a little bit about a hot topic in
Japan right now, and that's
super human sports.
The idea for superhuman sports is to use
technology to enhance human
abilities over,
you know, kind of what we usually can
do. And because, you know, we are
in Japan, we, of course,
founded the society.
So since one and a half years, we have
the Superhuman Sports Society disclaimer.
As I said in the introduction, I'm
one of the society members
and also one of the.
So originally the idea came from
Inamine Sensei and
Kimoto Sensei and
Nakamura. I usually don't like
namedropping, but especially if you're
interested in human augmentation
and human computer interaction, check
out all the works from Inamine
Kimoto and also follow them closely
because they're two of my personal
heroes.
And then we have the society now
we're trying to expand it
internationally.
So since a couple of months we're working
with T.U. Delft, the Sports
Engineering Institute, and I don't
want to bore you longer.
However, I just want to highlight
one person in the advisory board
and that's whole Petrina, because warbird
kind of saved twenty sixteen
for me because, you know, it started all
with Boycie and
so on and then Brexit and Trump.
But Robert organized the Cybertron
in Italy.
I just wondered how many of you
heard about the Cybertron.
Oh OK. Couple.
Well nice it.
Has anybody been there.
Oh yeah. Even great.
Cool. My kind of audience.
Awesome. So in this case
this was an international competition for
disabled competitors
using bionic assistive technology,
so exoskeletons and so on.
And kind of that was just
great. And it's in the
uh in the terms of what I want to talk
to you about today, but what I want to
talk to you about today is then
superhuman sports.
And I kind of
selected these type of topics.
So augmenting our body armor
and mounting the playing field,
augmenting, training and cheering.
And I will show you a couple of examples
from researchers in
the society.
So first, going over to augmenting
our body.
There's a lot of work also
mechanically that's not so interesting
for us computer scientists.
However, it looks cool, so I included it.
One is a company called
Skeleton X.
They manufacture these
exoskeletons.
They are completely moved by muscle
movement. So nothing it
here, but still cool.
And then another thing is
the bubble jumper.
That's an idea that came out of a
hackathon we organized
at scale and we started kind
of with first sketches
and so on. But the important part for
also for our hackathon is you have
to demo. You have to implement.
It should be playing oh, yeah,
so you see the combination
of bubble ball and
sky runner and to
make a very safe and
nice sport for.
The coexistence of safety and force,
I'm not so sure about the safety
of.
You know, getting
over this to something a little bit more
serious, spider vision,
this is now work from Minami sensei.
And in this case, this is really about
extending the human
field of view.
And it's very, very simple.
They use the DKA one and they have
two cameras, one in front, one in the
back, and they overlay
the two video feeds
on top of each other, like
you will see right here.
And the interesting part is at the
beginning, it looks really terrible.
But after two or three minutes using
this, you can really separate the two
images and you can even read or
do other things. So you kind of then
you can be environmentally
aware.
The only trouble is and that's something
you see here, because you're wearing
a hat mounted display.
Still, we are said,
you know, there's an offset from the
camera to your actual eyes.
So it's very hard to interact with your
environment.
But that's kind of initial work where you
want to get at.
The next work I want to show you
is the synesthesia suit.
That's a full body haptic feedback suit
made by Mizuguchi and
Saba sensei, two of my colleagues at
Comedia Design.
And this was first done to promote dress.
That's a video game for
the PlayStation.
And what it gives you, you see it here on
the right side is it has
a haptic feedback, vibrators,
uh, 24 on the whole body.
So you can get a touch feeling on
your whole body and the idea
of Mizuguchi sensei wants to do with this
is he wants to give you simultaneously
visual as well as auditory
and haptic and touch feedback to
put you in some kind of trance state.
And if you try to rest with
this suit, it's quite an experience,
actually. So you get, you know, visual,
auditory and touch feedback
at the same time.
Then moving over to
augmenting the sports
field here, I want to
highlight a work
from Akimoto sensei
Aqua Kafe.
You see a water tank you can swim
in that swim are actually
worth 3-D shattered glasses
and you have a 3-D display around
the water tank.
So you can then really dove
in the Great Barrier
near the Great Barrier Reef, or you can
do competitions with sharks
and similar things.
Then augmented
training.
Here I will show a little bit
of old work.
Maybe some of you will know, but I just
find it cool enough to still
include it. And that's galvanic
vestibular stimulation by MIDA
and also Inamine sensor.
And what they were doing in 2004
is they apply a tiny
current behind your ear.
You have a.. And catalyst behind your
ear. And what happens if you apply this
tiny current is you messing
with your equilibrium, with your sense
of equilibrium.
So you will tend towards the nanotubes
if you do that and responds by.
And then.
Now, the reason why I show
this video is because, you know, this
would be impossible to do for a
researcher in Europe or us.
You see a student wearing this
device. You see the professor
with the control unit
and.
Can steer the person just left and right,
and I have to say I
tried it.
It's kind of it's not safe
to try out.
We don't know about long term effects,
but it's an interesting and it's
also user dependent.
So it kind of depending on the user and
depending on your skin, you might
need more or less of this
to the electricity.
But it's an interesting thing to think
about, especially for sports that require
equilibrium or two things
and then not work.
This is work from one of my students,
Takura. He was working on peripheral
vision glasses and his first idea
was to just give you notifications
in a way that other people don't realize.
So, you know, I could look still into
your eyes and you don't realize that I
might get some notification on my
peripheral vision.
I can still keep eye contact because so
far, you know, if you look at your hendee
or if you look at your watch, the other
person will realize that you're
you're looking there. So you started with
this idea.
And kind of the boring thing is, yes,
it works for notifications,
so you can have up to eight
different types of notifications.
I don't know why the video stopped, but
that's actually not so important.
You can have eight different types of
notifications. The other person cannot
see that you you're getting them.
However, what he really wants to do and
here we get to the interesting part is
he wants to try to influence
movement over your peripheral
vision.
And there's some other research,
some interesting research from
Furukawa at all.
Also, that human conference did
use some patterns on the floor
to guide pedestrians to
influence the movements of pedestrians.
And then Tukaram
wondered if it's possible using
then also peripheral vision glasses.
And this is the first test.
This is partly unpublished, but I
thought, I want to share it with you.
So in this case, what he's using is he
is using some lines that move along
this way and he lets people
walk. And guess what happens if
you let the lines move faster?
Yeah, I heard so people will move
slower, so walk slower,
so it's already possible to maybe
use it to influence speed and the next
thing he wants to try is also try
to influence direction.
So the next thing then is augmenting
watch and cheer.
So in this case,
I want to show effective wear
that glasses that
can detect your facial expressions
and in a very cheap or simple
way, that's my saying.
You see kind of the faces he makes.
He wears his own self-made
glasses. And how this works
is he uses just photo reflective sensors
in the glasses frame and
he detects the distance to the
skin. And with this, you can
get used to dependent very well
what type of expressions a person
is doing.
So you see it, and I like
the training for it, so you hear you see
the photo reflective sensors and the
distance measures between the sensors
and then kind of for a start,
he has to then
calibrate the sensors once
and you see the principle.
So, for example, for smiling,
you know, kind of your cheeks move up.
So it's easy to recognize that
you're smiling and even laughter.
So aid, facial expressions are
recognizable.
And yeah. And I like just
and making faces. That's the
normalization phase, then the training
phase and then the recognition phase.
That's user dependent, but it's very
cheap. And if you want to build this
especially smiling, it's easy.
You just need one for the reflector here
above the cheekbone and then
one maybe for reference and then it's
easy to implement.
So, of course, you know, kind
of what we can do with this is we can put
it into a VR headset
and then your character in VR
can have the same facial expressions.
You can think about remote participants
in a in a
in an event also on and then also showing
the mood or so on of the remote
participants.
It's not playing.
Yeah, but I think you get the idea
I don't know why the video stopped
and kind of also more interesting
then is you can also
kind of maybe use this for the science or
if you want to design a very, very scary
dungeon, you can kind of detect
how people know it's,
uh, how people are reacting
to some elements in your VR
environment and then, uh, do
the correct adjustments.
And also, you know, you could wear it
throughout the day and just get an
overview over your daily activities.
So, you know, kind of how much smiling
versus frowning are you doing also
related to some of the activities you
have?
Um.
That's that's interesting
and that's in line with some work also
from Rick Emoto sensei, who did
the happiness counter.
So he had the idea he's he used a smile
as an interaction modality.
So, you know, in the morning when you
wake up and your alarm clock rings,
you don't have to push a button, but you
have to smile to your alarm clock.
That's a little bit scary or creepy.
And because we are with this this
next one is a little bit of a joke, you
know, kind of if you if you recognize
that you don't smile enough,
we can also make you smile.
So this is now
22 smilers now activated over
the electrodes.
As I said, this is a little bit of joke,
I don't recommend anybody.
I hope nobody will make these type of
technologies in future.
But here the interesting part is people
students were playing with
MS and we recognized
one of the main most of
the nerves to control
our face actually come over
from this side of the face.
So what you can do is if you put
electrodes on different parts
here, you can actually activate different
muscles.
The students wanted to get smiling,
working, but first when they tried it on
me was my eye was blinking and other
things you can also do.
I blink.
Um, yeah.
And then, I mean, this is a little bit
on the joke side.
As I said before, however, if you want
to have references for more serious,
uh, electric muscle stimulation work,
I refer you to Max Fifer and
also Peter Lopez.
They also have a kid
out. So two references
to your health.
If you want to build your own or if
you want to build your own EMS and also
their source code for all of their papers
is, I think, out there on BitBucket
and GetUp!
And this is also just cool, you know,
cruise control for pedestrians and also
muscle plotter. So, um,
using your body as output
for a computer.
Um, however, now getting back to
the, um,
superhuman sports.
So the idea also for the Superhuman
Sports Society is to walk to
work towards Olympic
2020.
And we want to create
actually an augmented game culture
in Japan.
So not only inventing new sports
and using new sports to
pursue amplifying human senses,
but also creating this culture to
make these sports. So we are organizing
workshops. This is an example from
from Wykeham beginning
of this year.
Uh, so, you know, kind of you have a lot
of devices.
You get together researchers,
artists, designers, but also,
uh, local people that are just interested
in participating.
We do one day of ideation
and trying out different
types of technologies and
and research outputs.
So this is, for example, to check in
one of the researchers.
Yeah. You see all of the four people.
You know, you have a view of all four
people, I think,
anyways, and then, you know, at the end
trying to design these games together.
And so these are
some more.
Pictures from some of the workshops
and here the idea is,
first of all, as I said before,
we want to create a proof of concept
by making and playing
sports and that sport.
So not just the ideation
sketches or so on, but really
demonstrations demo more die
and then also getting fit ourselves.
So far I didn't really work for me, but
maybe in future and then also creating
these augmented
game creators for
the time being.
And now you might wonder after these
slides. Um hmm.
What's the scientific impact?
I mean, it's a lot of fun to do this,
but, you know, why am I focusing
on this?
And if you heard
my previous talk two years ago, you might
know that I'm interested in recognizing
and improving cognitive activities.
So concentration, attention,
comprehension and
I focus a lot on either computing.
That's my old talk,
a little bit of background about myself.
I was working with genes, Japanese
classes, company on sensing
glasses that have accelerometer and also
some low level eye tracking
in them.
And now, since December,
I have some good news.
I have support from the Japanese
government, especially as or GST,
to sponsor collective open eyewear
platform.
And I'm also looking for collaborators.
So to work
on this, because I think this is crucial
to start now
because, you know, I don't want to
end up in a situation that we have
today with our mobile phones that we have
just more or less or that most people
just have two providers, two big
companies that tell you
what you are able to run on your mobile
phones. And I think eyewear for me, you
know, believe it or not, because my track
record is not so good predicting the
future.
However, I think eyewear is the next
stepping stone is the next
thing. And I really want to have an
open eyewear platform out there.
So that's the background for myself.
And I think for me, the superhuman sports
is a great testing ground
for these type of technologies.
And then really not only doing sensing,
but also really trying to amplify
human senses.
In this case, I have to hurry a bit
because I'm behind time and I want to
still show a demo.
However, here, the idea is that sensors,
digital sensors are in some parts
already better than human sensors.
So, you know, you can have a higher
frame rates, you can have
a broader spectrum.
So you could see infrared and so on.
However, what is lacking from our
side is the interface to it.
So I wonder really, can
we create new
and amplified sensors
based on digital technologies
and just going with the
digital camera system analogy, I
just want to give you a small example at
the end of the talk now, and
that is going to assume so
if I would be standing here and there's a
sign in the back and I want to read the
sign and I cannot do it with my normal
eyesight, I will squint.
And, you know, wouldn't it be nice if I
could wear some kind of glasses that
automatically if they recognize that I
try to read this and squint, which
is to me then, and I don't even realize
that I'm using technology.
And unfortunately, I
wanted to show you a demo on
the whole, so I have one here.
However, I didn't manage to get
this working on the whole balance,
and I'm happy that I have
a student, George, who could
set up a demo on.
On a laptop that gives you
a feeling how this works.
So can you switch?
Yeah, so in this case,
this is unfortunately just on the laptop,
but you can get an idea
in this case I'm using a Tobii tracker
to track my I guess I have
a camera and just a simple open kvi
to track my
face.
And then if I now go
ahead and squint, it zooms
in. I can show you also this
on a on a document.
So if I squint, it zooms
in.
If I don't, it stops.
And that's actually a very, very simple
type of interaction you could do.
I hope that could show it to you on an
on in our system, but that's maybe
something for for next year.
Here again, as I said, you know, kind of
the desktop eye tracker and
I can show you a little bit how it works.
So this is just an open KVI system.
I think George uses the
eyebrows and the relation to the nose
and the size of my eyes.
So if I squint down, this
relation changes and it's
zooms in and if I stop, it
zooms out.
And so this just
should give you a small
idea of what I'm trying to
create or what I want to create
things, technology that's easy
to use, that
that you don't even realize that you're
using similar to the analog
glasses.
So on and also not
only for sensing, but hearing in other
senses as well.
So this brings me to the end of
my talk. I just have some obligatory
thank you slide. So these are all people
I am working with and that that
contributed to the research you've been
seeing.
And I have a special
thank you slide for the
students that actually did the work, and
especially George, who
made the demo you just saw,
but also musci for the
effective wear and the
the rest of the crew.
And now this
is the end of my talk.
And I think there's a time for
a couple of questions, remarks or
violent dissent.
So thanks a lot for your attention.
Wow, really cool stuff
that was so cool.
You have a chance to talk to Kai right
now to ask him questions, there are four
microphones in the room,
so find yourself behind one of
those microphones if you have a question.
Also, the Internet is always able to
ask questions if they are awake
by now. Does the Internet have
anything?
Yes, one question from the Internet,
yes,
is that code for eye tracking available,
which was the which
one?
So for eye tracking, I can recommend
you, uh, pupil apps.
So.
The eye tracking code right now, I'm just
using Toback's, so that's
kind of Taulbee proprietary,
they have an API.
Uh, that's, um, it's
just a desktop tracker.
However, if you want to have a mobile eye
tracker, I would recommend you pupal
apps. That's an open source eye tracker.
They just keep the latest revision
of their hardware close source
and they have all of their code up
on GitHub. So it's very, very easy to
to adjust. And they also have
settings for VR headsets.
So you can easily use it.
You can just kind of use build
your own cameras.
That's easy. Use their code.
If you kind of you have budget
constraints that Tobii X is
OK.
But I don't like Tobii,
as am I so much because
they use clotheshorse APIs
and it's hard to get to the actual images
because, you know, you want to have, for
example, pupil dilation
because that gives you information about
stress and other things.
And that's not so easy.
Right. Um, actually, let's go
to microphone number six,
please.
So I have just
a quick question about
how open you are to be
able to collaborate with
different universities from around the
world to use your technology and
research
I work in and
the effect of sciences department and the
University of Geneva.
And we actually use a lot of motion
tracking this.
I am tracking Bee super interesting
for a fact of science research,
especially as well our facial
expressions. We use that a lot.
So I would also be interesting.
So you've discussed how
to use it for like opening up
possibilities in the real world and
actually enhancing everyone as a general
population. But how open
are you to using it for research as well?
Yeah, we're very open.
I mean, that's also the reason why I give
the talk. I mean, now finally, I get also
a little bit of money to funders, uh,
from from the Japan side.
And I really want to find people that
want to collaborate on on this side.
And I think, you know, kind of especially
seeing how
Silicon Valley now also invests in
eyewear. And I think the next thing we
will see is eye trackers in VR
and so on and how close those
systems are.
I would really like to have this
collaboration. So just try to catch me
afterwards or just send me also email
would would love to to to keep in touch
things.
And microphone number two, please.
Well thank you. Amazing talk.
And how important do you think will
be the combination of external signals
like squinting or looking at something
and internal signals or like
directly measuring the brainwaves or
with an EKG? Do you think that is
important to combine those or
it's important to combine those?
I actually I started looking into a lot
of the EEG signals and so
on when I started this work, but
it's very hard to get into your skull.
So EEG, uh,
also brain computer interfaces, I think
will take a while, uh, to to
work. And you also have this trouble of
the signal processing is relatively hard.
And because I'm lazy, I started with the
eyes and the eyes give you actually
already a lot of information you want to
get from the brain.
I like to think about combinations.
So for some things, we are already
working with, uh, also EEG.
And I think it's it's crucial to find the
right type of sensing so that it works
throughout the everyday life.
I think one example was also jeans.
And now with the with the the, uh,
just project, what we really
trying to do is find simple sensors
to give you some information about your
your your brain activities and so on.
I think it's really a combination of
multiple sensors.
And I think, as I said, the AI is very,
very interesting.
Yeah, I think we have time for one more
brief. Question number four,
please.
When I was wearing an
eyeglasses, uh, my brain,
the distance two objects so I
couldn't grasp anything.
It just wasn't there where I expect
it. Yeah. Is there any chance this
problem will go away?
Uh, just depends heavily on the lens.
So there's still this problem.
It's also in the I mean, VR has this
trouble with the with the
the the the perception.
Uh, you I think you will have
always this issue.
I found it.
So the whole lens quite good for
this. But here the problem is also you
cannot grasp something because
the the distance to the object is too
far. So I'm still wondering I haven't
seen something that's really,
you know, kind of graspable, graspable in
the way you have the right distances
or so on. I think either your brain has
to get used to it, which I think it does.
After a while, it gets the distance
measurements correct, or we have to
employ some other tricks, uh,
to to do that.
What you can do for that, I'm not so
sure.
And with that, we're at the end of
the talk. Thank you so much.
Can you please help me think you,
Vincent?