Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/422 Thanks!
So we're giving a talk
about you want to fight for long term
something.
Now we are giving a talk and not
even that like sorry, we're giving a talk
on subtitling,
on subtitling videos
that are being recorded back there.
For example,
we are presenting work from a student
project at Hamburg University and the
computer science department.
We tried to convince our students
to present, but
they were afraid
this would be too late or too many
people or something.
So there's just one sitting of their
Floriane, hey, as long as
they're.
So Floriane is
studying computer science towards his
bachelor degree and
together with five
other students, five of the students
they are working on,
on producing
subtitles or, well, producing
subtitles semi automatically
for C three videos.
That's the database that we selected for
this task because it's freely available.
Thank you very much to that.
But of course, the what we're going
to do also works similarly well
for other videos.
Hopefully we haven't tried it yet.
Um, so what we did was
we looked at last year's videos
which were transcribed, most of them, or
many of them at least.
Um.
And so we have the transcriptions.
And for subtitling you need more than
just the transcription.
That is more than just the text.
You also want the timing information
for that text.
So you want to be able in your
favorite video player to Loden, for
example, SRT file, that seems to be a
data format for that.
And use and
click that in your favorite video
player and then you see the subtitles
or maybe on a C on
a on a brighter display.
If you if you can't see or visually
if you can't hear.
Um, so
I'm kind of
kind of lost now and
then.
So we got the transcripts and
so, so why do we want transcripts and why
not just use speech recognition.
It turns out that speech recognition
is not very good.
Hey.
Uh oh.
And and not just
now. I'm very glad that I will hand over
to honor for all the difficult parts of
the talk, just as waiting for the
orders anyway.
Right.
That's good enough.
I mean, don't break it.
I don't.
It's the other way round, please.
That maybe do whatever.
Um, so, um,
I think we already had this
switching not doesn't work.
That's funny.
It, it's funny isn't
it.
This computer has a demo prepared.
We are looking into whether that's going
to work or not.
So it's unlikely
and please place we
had at least some slides.
OK. Uh.
OK, that's. Yeah.
OK. All right.
So what's on it.
So I um.
Yeah. So this is a group
I think you've already seen Timo.
This is Wolfgang.
That is me. And
you're not on powerlessly.
I'm Tim and is currently on the
photo because I think you had to
move on that day.
Um, so
we had I wasn't
able to listen. What did you already say.
OK, so, um,
what we want to do is, um,
generate subtitles from
transcription.
We need to align them, track them into
subtitles and export
them into whatever format we want to
have afterwards.
Um, and our secondary goal
is acquiring speech data
because as a university we
usually don't have access to a huge
amount of data.
Um, that's the main
difference between one huge
global players like Google
who can just tape everything you say
into your phone.
Um, for example, if you
if you see something like that
and they can actually do that to
enhance their performance and we don't
have access to that, unfortunately.
Um, so free data is necessary
to, uh, actually, um,
have free software speech programs,
speech recognition programs.
So, um, yeah.
Uh, you already went into that
a bit. Um,
if you try to do,
um, speech recognition, you
really need to do a lot of stuff.
Um, you need to know about
the world as a whole.
Um, you need to know about
what is the context
of what I'm talking about
right now.
And you can all do that in your
brain because it's so powerful
and optimized for that.
But the computer cannot really
do all that right now.
They usually, um,
base their computation on what
sequence probabilities.
That means we have a large, large
copies of data which says,
well, um,
this has been said, uh,
I don't know, one million times.
And so this is much more likely than
something else that has only
been observed once.
And the problem here is that
all of you use weird words and weird
contexts. And then,
uh, the speech recognition just
gives up and says, well, I don't know.
Um, and to show
you, that is a really hard problem.
Maybe you've been to the talk yesterday
where Jacob Appelbaum
said,
oh, no, I have to remember that, uh,
new PJI has
safety faults.
And what
was transcribed live
by human was GnuPG
has safety faults.
And what he intended to say was
new p.g has safe defaults.
And that makes a difference.
And.
That shows that even for
humans, when there are no time
to actually type stuff in, they
will miss things, for example,
that he said before, that
it is good to use GnuPG.
Um.
And therefore,
we see that speech technology
is bad at understanding what
humans want to say and Timo, we'll talk
about that in detail
and I can sit down.
Yeah, you can sit down.
So, um.
Oh, no, there is no privacy anymore.
I was wondering why you were standing
here.
So, as I said, speech
technology technology is bad
at understanding what humans want to say.
So that is a problem.
So what we tried
or what we figured out was that humans
are at least relatively good at
transcribing speech apart from
that example that Ana just presented.
But then going from these
transcriptions to subtitles is still
a major issue.
So this is a screenshot I
took earlier today from the Emara
transcription interface, which is really
good at the job. But
the complexity of the interfaces is
somehow clear.
I think that it's not really easy to
figure out what to do.
You really have to figure out
how to do that. And it's really hard to
build a very good interface
for this task of
putting the timings onto
what? Like putting the timings on the
text, figuring out what was spoken when.
So it's really hard to build a good
interface. And I think it's hard
to build an interface because it's such
an unnatural task to humans.
We never do this in our everyday lives.
We never figure out when something
was spoken.
We just know what was broken.
So so even if we tried to improve
user experience for
this emara interface, that wouldn't solve
the underlying problem that humans aren't
made for this. Humans are made of
understanding, maybe for notetaking,
but certainly not for for
subtitling.
So our idea here was to
have the computer solve the unnatural
part of the task and the humans
the natural part of the task.
So the
yeah, I got all that idea,
the idea.
OK, and
so what
happens is a kind of pipeline that
is very similar to what speech
recognition does.
We go from the transcribed text and
we have to normalize things because
as people transcribe, they naturally
type things like 431
before Christ or might
even say onto Dominie if we're unlucky.
But they would certainly not say for
three one B period, C period,
unlikely at least
then we have to go from the normalized
that text that we've got and to more
like a sequence of speech.
Sounds like a or
what did I say. Four hundred or
something like that.
Right. A sequence.
And of course I hear most
likely pronunciation that maybe multiple
variants that we want to track
and then we want to figure out, of course
when when we have this sequence of speech
sounds, how does this
align with the acoustic
waveform that we've got up here?
That doesn't happen magically, but it's
a it's a learned model.
It's a learned from lots of data, as
it was implying earlier data
that is severely lacking
when you're working at a university
that we need in order
to have good models of how
the speech is being spoken of, what
speech sounds actually sound like.
And if we have that, then we get a
pretty decent timing information
for every word.
Hopefully that's our goal.
However, if we just follow
this standard approach
of figuring out things one after the
other and then doing
one big search over that, this
isn't robust because
there's so much that we have to take
on with like all the different timing
variations that could come
up. It's just too large.
So what you do in speech recognition in
general is you don't do a full, complete
search, but you do a beam search.
And in our case, what
we notice is that the correct solution
always falls off the beam.
So if you're lucky, it may work
on very short stretches like one or two
utterances in a row.
It may also work if you have very
distinctly clearly spoken
text. So what we tried
in a different experiment was looking at
spoken Wikipedia data while we
have the article and the text and people
really try hard to speak exactly
those words that are in the article.
But as we noticed earlier, transcriptions
at are just they have to
be done in almost real time so
it doesn't work as well.
So the approach that we use,
it's not our own.
We just know used it
is to stick with the fact that people say
more or less what's in the transcription.
There are some stretches of speech that
are not transcribed. This actually
happens a lot for
for data T-3.
It it appears, and also that
transcribed words that are not spoken.
So people just type anything and it just
happens not to be in the talk.
It's quite funny to look at the data.
We all believe it's really nice, but
it's nice, but it's not perfect.
So what we do is
we actually do this weird sequence
modeling that Ana was talking about
earlier, but not on like
random or huge amounts
of text, but only on the full
transcription that we got.
And then we the speech recognition on
the full thing and try to find
landmarks.
Well, we have a correspondence between
the recognition
result that we get with this kind
of like very specific speech recognizer
and and the original
transcription that we have.
And then we iterate over that
and divide and conquer approach.
So. So after that, we look at the
stretches between landmarks, trying to
narrow it down
as much as possible so that only
those few bits and pieces remain, that we
can't align to each other.
There is one problem that
it's not very fast and
it's only running at about one eighth of
real time.
So if you've got a one hour Akino,
that will take overnight,
which is quite long in my opinion.
So it's not real time anyway.
So this gives us
this this step will give us alignment.
So we know for every word
the the timing where it's been
and the talk.
However, we want subtitles, not
word by word alignments.
And with that, I hand off to Ana for the
next bit of talk.
OK, so how do
we get from those aligned words to
actual subtitles?
If you look at subtitles,
that's maybe more complex
than you might think because
you have limited space on the screen.
So you cannot say, well, I just
put one sentence there because
sentences tend to be really long,
especially in talks like
this talk.
So people really go on and go on
and go on. And then they say
and and and and then
you have a really long sentence.
The reading performance,
um, of the person who's
reading the subtitle
later on is also limited.
So you have to take into account
what can be read.
Um.
And so we need to break up those
sentences into shorter chunks
and they must be
somehow meaningful.
And
we can see that simple heuristics
don't really work for that.
Um, if you
know, this is from the.
No. Nine, just.
Sorry, no video.
So that this is not the time
for the video. So sorry.
So if you want to see the keynote from
last year again, you have
to do it online later on.
But this is from the keynote
last year. And this is
what a simple mistake actually did.
Um, you see the
subtitle here and it says, well, it
kind of got to me in a telephone
and so in a telephone.
What?
So and
that doesn't make sense.
But if you look at the
whole sentence, you'll see that
it occurred to me in a telephone call
and that makes much more sense.
So you need to actually
get the syntactic information on that
telephone and call
strongly related.
And you don't.
It's a bad idea to
have a trunk boundary between those two
words.
Therefore, we perform a full
syntactic analysis of each sentences
that looks like this,
um, this, um,
shows the dependance of
each word to
another word and then you get a
dependency tree, um,
that's done by some nice
program that does
this and it doesn't always
get it right.
So that means we have to take into
account that it doesn't
get it right all the time, especially,
um, because the
sentences can be really long and
that can be syntactic errors and so on.
And then, um,
we just search for the best
chunking that is for the best
split points in
the sentence, um, by defining
two cost functions, first,
by the splitting cost,
um, at each point,
which says, well, if you split here that
will cost you two hundred points
and if you split here that's
only one point.
And um the other
cost function is the resulting
trunk length.
So if you um have a
trunk length of only two words,
that's extremely bad.
But if you have a trunk length of
200 words that's also bad.
You want to be somewhere in the middle
and with this we actually get
better. Um,
subtitles
Um, yeah, our
results show Dybbuk Interface.
When you switch to that computer,
who does that?
He's in charge of that. Should I press a
button that.
No, no, no, no, I did not.
OK, so what you can see
here is, um, just the
debug output, um,
for, um,
our system and
it shows the
dependency tree.
I think the um.
Well, the tree is not really that
visible. So the gray
like part's over
there.
Those this is the dependency
tree. It's perfectly clear from
your sorry.
Um and then you can see
the different chunks um
and on the right, uh.
It's not there.
You can see all the information
we added, so, for example,
when does this word so
the first year, the year,
when does the world start?
It's the world number 29
and the syntactic pound
is what number 30 and so on
and so on and so on.
And so we we can we use
all this and we can
actually jump between
points and see
why the system actually does
what it does. And that's important
because we don't have a clear gold
standard where we can say, well, this is
wrong, this is right.
We can only have a look at it
as humans and say, well, this looks good.
And if it doesn't look good, well,
let's see why
this might be the case.
Um, yeah.
Uh, and usually you
would hear sound and I don't know what,
but not today.
So the sound is there.
It's just not very loud.
It's OK.
Uh, if you want to switch
back, that would be great.
OK, um,
I think it's
and we actually process,
I don't know,
several videos with it
and it mostly works and we have to
further work on this.
Um, and I think we will go
to this. Yes.
Um, is this your think
it would be with you.
Yeah. Yeah.
So, um, so to
sum up those results, um,
so some some statistics
are Laskey last year's keynote, we have
processed more videos than just this
one. So is this the five minutes before
the questions.
OK, good. Thank you.
Um, so this
is some statistic, last year's keynote,
we get alignments for roughly 90 percent
of the transcribed words.
So that's not all the words.
Unfortunately, we have some
ideas that further improve it.
Um, actually, we have a huge
problem with applause, which is very
disruptive and
laughter.
So we have the problem with applause
that the first words after applause, so
thank you very much.
The first few words after applause are
usually not correctly transcribed.
We know that from the data.
We don't have any good applause model in
our acoustic model because so far
we didn't get applause.
Now we've got no thanks,
no need, no money, no.
Thank you so
so far, we didn't have any applause data,
so we couldn't have it in our acoustic
models. Now we have data with
applause. We haven't really figured out
where the applause is. We'll have to do
some manual work on that.
But this will help us to build
better acoustic models that contain
applause and to then
further improve the transcriptions and
subtitles, of course.
So our current ideas are to
kind of estimate the duration of
those words that we don't know about.
Again, for the statistics, most of the
time it's very precise.
Sometimes it's just totally odd.
We don't have any precise numbers on
that.
I'm telling the students all the time
that we need evaluation because
we're doing scientific work.
I can talk all day about that,
but the semester isn't over yet.
We have more ideas.
We still need to improve chunking areas.
Also, we want to go to two
line subtitles,
which sounds like, oh, but
it does change the game a little bit
because now you have two different kinds
of, like, chunk endings
which need to be integrated.
And then there are all sorts of practical
issues.
We haven't integrated with Emara yet and
a suitable way.
We had promised originally
that we wanted to type
study one C three talks.
We haven't done that. I apologize for
that, but we're working on that.
And some people actually need credit for
this class. So it will happen, I promise.
Um, so the other direction
we were going for.
Yeah. So
um. Oh now I'm off.
But whatever.
Um so we also
want to use
this data that we got better.
Reminds me I got now
at language changes also a difficulty and
transcriptions.
Um so um.
The cable behind the back.
OK, now, better now it's better, OK?
And so
we also want to use the data
as it's very freely licensed to improve
the open source speech recognition models
that are out there.
You need lots of data because all those
models are usually machine learned.
It's it's estimations of large
amounts of data.
And as Ana said earlier, that's
a huge problem so far and it's much
improved already.
And we hope to get another, like, what
is it, 30 hours or so of
speech data, of
talk data into those acoustic
models which are freely available at
Oxford.
And it will be German.
It will mostly be Germans,
it will be English, but mostly German
speaking English. So that will also be an
advantage afterwards for the English
models, for German speakers.
Yeah, so that's one of the goals that we
have where we think this
is actually a perfect example,
where two sides actually gain something,
actually three sides, because the
students are learning something,
C three is getting transcriptions and
we are getting data.
So to wrap up an assignment is simple
for computers, hard for humans.
And this is exactly the opposite
for transcription, which is easy for
humans, hard for computers.
We combine the two for best results.
Then subtitling is more than just
alignment because
the splitting is actually very difficult.
Yeah, thank you very much.
Oh, no, no, thank you.
Not very much yet.
So we have plenty of ideas for future
work, which also at least partly
line up with the
C threes ideas about subtitling.
I really like the idea of having very low
delay subtitles.
So go from a few hours
delay to a few seconds.
That would be marvelous.
And in that we could use
the typing speed information
because these transcripts are actually
made on isopods.
We have like Kibaki
timing information and right now we're
throwing the survation all away.
We're just saying, OK, could have been
transcribed whenever, but usually
people transcribe and sequence and
mostly during the talks.
Thank you very much. Whoever is
transcribing it might be.
Yeah.
And we're throwing half of your
information away for the record,
but we intend to change that and use
this typing speed information
for the transcription process,
for the alignment process.
Sorry about that.
Of course, if we didn't have talks, but
more fantastic videos,
we could do video analysis and
all sorts of other things, which would be
really nice to have
kind of built information retrieval
content search facility, where you go
to the video directly when you search
for, I don't know what something say
privacy or something.
All right. Thank you very much.
Thank you very much.
And we're almost completely
out of time, and I wanted to ask whether
the signal Angela has one really pressing
question from the Internet,
because all the other people that would
be so great if you could just ask
directly so we can free the
room again for the next talk
would be so great.
OK, then let's have one question from the
Internet.
Did you test it with a normal movie
and some transcription extracted
out of some subtitles available?
No, we have not done that yet.
We intend to do that.
But as I said, not all
like our evaluation
scripts aren't really done yet.
So testing it on those data would
be hard.
I doubt that we would excel
of like commercial
services, but we're working in that
direction, at least as
you.
Well, please don't speak as other people
are still here.
And this was pretty fast.
So we can do one other question.
There's one microphone, three.
I, um.
OK, so you fleetingly mentioned that
the subtitles had to be readable,
which means that they have to be
displayed for a long enough period
that people can actually have time
to read them. And in classical
subtitles softwares,
what happens is you map subtitles
yourself and the software
calculates for you and allows you to use
a determined, fixed number of characters.
Are you going to do something like that
or do you just plan on going on
some sort of gut feeling of what is
readable and what is not?
And if you are to going to do that,
are you going to have to actually adapt
your transcriptions, which happens in
translations? You can't actually keep the
whole sentence.
You actually have to focus on
some part of it or not.
And
yeah, basically so.
So first question, whether
we intend to do more than karaoke.
Yes, we do.
Right now, we try to do to
get as a as precise timings as we
can, but then we have to think
about the usability issue that we have
to show. Is subtitle for a sufficient
amount of time.
Exactly. I haven't mentioned that.
We haven't mentioned that. But yes, we we
do intend to do that.
It's like an additional constraint.
Now to your second idea of
changing subtitles.
So that perfectly makes sense for
translations, I think, and is vital
there.
But we are trying to remain
truthful to what the speaker said or
at least to what the transcriber
transcribed, which may differ in those
cases.
But so far, I think
that's beyond the scope of of this
current little one semester
project, because we would need
exactly the kind of understanding of what
was going on in that sentence that we
don't have in order to be able to
shortall things.
Thank you so much, Ana and Tim.
I think you will be over there
on the outside to answer more
and more questions.
Thank you so much for that.
And this is your applause.