Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2650 Thanks!
[Music]
welcome back on the n x stage for the
next talk uh this talk will be in
English unfortunately there will be no
translations so but you will be able to
follow I guess if you came here I hope
you have all nice places and I welcome
uh Jurgen pble who will build uh talk
about building a cloud-free digital
voice assistant with free and open
source software please give him a warm
welcome and enjoy the
talk thank
you okay so uh first slide I guess is
always about uh who's talking up there
um I'm from cin cologne I work in
Security Management and um I build stuff
mostly for my kids and I Break Stuff uh
whatever needs to be
broken so
um first we're going to talk about um
what are voice assistants actually
composed of um then we're going to do a
demo um you're probably not going to be
able to see what's happening on the uh
display but um uh we'll get audio so um
you'll be able to hear what the uh hell
has to say um then uh I'm going to talk
about how what is it uh that actually
prizes this um um this uh voice
assistant and then there's a what's left
to do for release 1.0 and what's on uh
on the agenda for beyond
that and at the end there'll be a
Q&A so um when you're talking about
voice assist there's really five um
components that make up the core logic
there's the Wake word detector and uh
you presum we know this because uh it's
okay Google Alexa zi whatever that's the
wward detector
because you don't want um whatever
voices you have to always just trying to
make sense of what's being spoken and
interpret that as commands so the Wake
detector activates the uh the assistant
then there's um speech to text so it's
recording whatever someone has to say
translates that into text
and the next part is the intent poer um
so trying to make sense of what's been
um said skills that's the execution of
whatever the uh person requested and
then text to speech for the feedback so
you know that didn't work or
whatever so let's look at each of these
um just briefly the Wake word detector
is a presumably easy task you can do
sort of like pattern matching on the
acoustic signal Etc but um the issue
here is that you want a really really
high um um
um uh or a very very low false positive
rate because otherwise uh uh the uh
assistant will be going like oh yeah
what do you want from me every like
couple minutes and that's very annoying
um and of course the um environment is
uh you know as it is uh we don't know
and um I don't know how it'll behave on
here haven't tested it so uh we'll see
um and of course uh everybody has a
different voice so
um that task isn't really that easy um
so as far as uh free open source
implementation there's Microsoft preze
and um the uh true and tried uh pocket
things um and there's others you know
I'm not trying to list everything here
uh and the selections is really more or
less what I um played with in in my uh
my project so the next part is the uh
Speech to
Text so um there's really two parts
there's the um the software that uh
actually um tries to make sense of
whatever the audio signal is and then
there's the language model you need a
different language model for each
language and and um uh essentially the
language model contains the words that
are known um to be able to be uh
translated so um the critical parameter
here just like with the weight we
detector the uh false positive rate here
it's the word error rate um we'll get
back to that later on and again um two
open source implementations most deep
speech somewhat recent and CI that is
really fairly old by by today's means
but still a good uh
system so the next is the intent parser
um so um independent of uh whether I I
tell them turn yourself off execute
system shutdown or go to run level zero
you know the intent is the same so
intent parser derive the meaning um from
what's being said so in this case it
would map to you know the skill system
shutdown with a default parameter now
because I didn't tell him
otherwise on to the skills and um that's
really completely um what implementation
specific um there are some um that uh
have already a lot of skills included
some come at a bare minimum um and
there's pretty much always some API or
SDK that you um have to implement
additional skills
so texes to speech um again um language
model training data um so uh of course
the uh the function is to give the user
feedback on what has happened and um two
of the rather older ones uh the eite and
Merit TTS and very recent I think last
year was U microf mimic
3
so with that in mind we'll do the
demo
um I'm going to um just leave this right
here I have a USB cable that's just
power so I'm plugging it in right now
and
um it'll take a while um what's uh
coming now oh yeah disclaimer I have uh
taken some audio from the uh the movie
so uh that's being used here that's not
part of the project because obviously
you can distribute
that uh according to German law what I
do is uh legal so um yeah so um it's
booting up for um uh the Linux system
boot up is roughly 30 seconds and uh in
the uh uh time in between it's showing
some
animations um I have the animation on
the slide right
now um
and um what I did is essentially just
rip um artwork from from two different
projects um and uh you know constructed
something that would sort of make sense
for the boot up
phase and um once it booted up um
there'll be the the red eye popping up
and um H should say um that I'm fully
functional and uh I haven't heard him
but um
we can um I guess switch over to um to
the hell now let me just take a
look seems to be yep seem so um two
buttons or roty encoders on top one is
for the uh volume and turning volume
up I don't think anybody can see it
there's an um like a volume indicator
just like a Tacho meter on a car
so um the uh the Wake word is uh okay
hell that didn't seem to work let me
come
around okay
how okay
how Okay demos or demos
um so um I can trigger this uh manually
oh probably easier from down there um
the one button is the volume control
it's and the other button is an onscreen
menu and I can I have a menu voice
command when I push the button uh the
wood detector triggered so what doing
now is um is actually loading the
language model right now so the first
request is like a 10sec
delay um and I want to move that into
the bootup phase so it doesn't have to
be um you know on the first spoken
request hello
Dave hello Dave okay um now it should uh
um work right away so let's try that
again what date is
it and you know it's processing now um
and rendering the uh the speech output
and maybe it'll work I'm sorry Dave I'm
afraid I can't do that
okay
well I guess it's still a good demo
right
um so um I can um Power this down from
uh telling part uh you know turn
yourself offer Etc um uh What uh
um uh what happens then is you know it
shuts down the system uh shows the uh
animation and you probably can't see it
right here but it'll be in the uh the
recording so it's the uh the live
animations live lines that go flat and
one it's gone flat uh you know it's
dead so um now I'll plug it off um but I
actually have one
more um demo to show so um I'm going to
plug it back
in and that's uh the part well how do
you configure that thing um so uh it it
has a web interface actually um that I'm
going to connect to in a second and um
what I can do there is uh essentially
configure the whole thing I can
configure uh commands uh that it should
listen to um what what actions to take
Etc so um
we'll see
uh whether or not that'll work in a
minute okay let's boot
it okay so uh up to 10 well so maybe you
can see it right here um just barely uh
of course it has to be in the style of
of uh the movie um and so the the first
button is uh uh orders so uh we did uh
try uh what date is today uh just
earlier so that's this order so it uh
listens to the prompts what is the date
what is today's date what is today date
yeah and why that is uh exactly uh the
way it is right there is something we'll
talk about later on
so um let's leave it at that for the
demo
um and get back to the
slides oh there it is
um uh right so what are the um the key
takeaways from the demo except for the
demo Gods weren't that uh uh good to me
um there is the speech inter face uh of
course that's what the the main purpose
of the uh the voice assist is um but
there's the physical interface you know
turning the knobs getting feedback on
the display um anything you know um and
uh there's also a lot of little things
that need um attention so the eye candy
you know these boot animations uh other
people did the work I just you know
cuted them everything together um but
you know that needs to be be done so it
is actually something where you know
what is the devic's state so you can um
expect what what is it going to uh to do
when you interact with it um the
configuration um and uh of course error
handling and many many other things why
am I listing these because that's the
core of my project the actual voice part
so wake word detector
um um text to speech intent parsing Etc
is uh something that I uh that that um
I'm using other software for um but uh
as always you know you fiddle with it
and then you adapt it a little bit so
that part is also a little bit of a part
of my
product um so what is the um essentially
the uh the idea the goal uh I want to um
voice ass system that is actually usable
so response times you know boot up time
okay um processing of commands is
important so the user doesn't get bored
while waiting for uh the response error
rates you want it to to just work
right and uh uh also important um
everything uh should be done on the
device no uh connectivity required
except for maybe skill executions
then uh of course or at least in this
setting is uh you know open source uh
and uh it should be as easy as possible
to uh build um and um there's a couple
of things like 3D printed enclosures uh
they were designed so that the print uh
models are as easy to print as possible
so in the entire model there's just one
overhang well actually no a second one
is in there but um um
so I uh essentially try to make it as uh
easy to to build as possible and also
use components that are rather readily
available um and yes pies are uh hard to
get nowadays but it's the situation is
getting better so
um would hope that uh they'll become
available again um especially not a
project goal was the price so the uh
components
uh in here uh run up to roughly € 120 to
€200 depending on what you put in there
so I put also of ID card reader in there
you know you don't need it um then uh
you don't need
that
so what's in
there
um we have uh the uh pi02 um and that's
where the the Linux system is running on
and that's the the greenish
thing up here um below it is the uh
display with a
microcontroller and below that is a
essentially a sound sound board um it
has two microphones um and a speaker
connector um and then we have a uh
gpio um extender so um more uh pins to
connect to and in in my part of the
project also I I have a requirement for
RFID Card because my kids have Rd cards
they put in a reader and the um the um
you know the the Peppa Pig or whatever
starts playing so I wanted that to be
part of this project also um but uh
that's just the bare components how do
we need to uh connect them so the first
part is we connect the uh uh Pi uh Z
with the uh the Hat
and um that's
actually if you
could get the camera right here that's
actually a cable in here is is that in
the picture okay and uh it just runs
down and connects the Hat that's uh
that's down here be behind the the wire
mesh
um so those are connected now um then we
have uh next the connection from the pi
Z to the uh display with the micro
controller that's just a regular USB
cable uh and that provides power and a
data channel to the uh
microcontroller then uh i2c connection
from the microcontroller to the um GP
board uh because maybe you can also just
uh get the idea that these displays with
microcontrollers they don't have too
many
pinouts so then um ITC connection uh um
from the uh GPO extender to my uh RFID
uh
reader and then um of course and I
already mentioned this we have the
microphones on the uh the board uh then
we have a
speaker and uh then we need power and
now we're pretty much almost ready to go
uh except we didn't talk about the
rotary encoders as the user interface so
uh they wire up to the uh this GPO
extender and um you need five
connections per um per rotary encoder
has a push button so that's 10 cables
and then um we have uh quite a lot of
cables in a very constrained uh area so
that's getting quite bit fiddly at times
um but wait there's more there's uh ITC
polling um it2 is a um command response
protocol so um there's no way that a um
a device uh that has an event so the rad
when you move it it uh changes the um
the states on the the pins ITC has no
way to say okay hey uh here's something
that's happening so um you need to pull
this the microcontroller PS the states
of the gpios uh every I think 10 Micro
uh 10 milliseconds uh and when something
changes then it uh you know knows okay
the button has been pushed or whatever
um and obviously that's
not such a nice solution but uh what are
the uh what is the alternative well this
um gpri extender from uh adaf fruit has
uh um two pins that no uh that you can
actually use to
notify um the system that uh something
has happened some pin has changed uh and
you connect that down into the uh the
um the sound board um it has an uh extra
uh an additional uh pin out board it's a
growth port and um then at least you
could detect right there okay something
has happened what has happened we don't
know we need to pull that uh using ITC
uh and then you uh also connect the um
Aid card reader with that and uh have
then also a means to get events from the
card
reader I stopped there I didn't do this
yes it'll be the nicer solution but uh
you know with with this small space and
that many cables in there already uh I
was like okay no let's leave it at it
Toc poing and no uh uh out um outside
channel that that notifies that there is
an event just because of already the uh
the amount of stuff in
there
um so one thing oh no I'll get to that
later um that we uh also uh have to
consider is
um when we were to connect these cables
the um the notification lines so to say
uh that uh some it Toc event would be uh
or some GP uh change has occurred so
that can be ped um we need to talk about
the voltages because the Raspberry Pi
zeros are not 5vt
tolerant so um the uh i2c powering
system and ITC is both 33 or 5 volt uh
capable um um
or complin so um you would have to pick
a microcontroller with a display that
runs uh on a 33
uh system for for the ITC otherwise you
fry your Pi Z or you need to do a level
shifter which adds even more stuff to
that package so
um I left it at that
minimal uh setup so to
say okay um this is the uh the current
state um uh there have been some um uh
components that were swapped out so
before the pi02 came out I did have a pi
3B in there um which of course uh filled
up that uh that um um that case even
more so um I was glad that the pi 2 uh
Pi 02 uh does have enough uh Power at
least with some overclocking to to bring
usable performance um there uh before
the uh displays were available with
embedded uh controllers on the backside
from you know just regular shops where
you buy that stuff uh I did have a um uh
just regular LCD in there with an um you
know a display controller and uh also
just happy that I got rid of that stuff
and the last change really is um I put
in a bigger speaker in the bottom
because the uh the speaker I did have in
there first was rather uh not so loud so
Hardware was really really stable um and
now we get to the MCU options so
essentially the um the display what you
see um is behind the Dome so this is the
the component um and again for
printability and for uh easy build um
you print this you print this and this
is the the designed um uh uh casing for
for this microcontroller that I have in
here you have a different
microcontrollers one of the three that I
have um modelings for you just Swap this
and uh put it in your all Goods to go so
the uh the first one that I started with
um is the uh Roundy P um that's a um a
board with round uh display and that's
actually the one that's built in right
there um and it has an RP uh 2040 in
there that's uh the what's uh on the the
board uh known as the uh Pi Pico uh it's
has six pinouts uh works with 3.3 volts
so that uh that's perfect for us has
four gpos two of those uh are power and
ground and two of those are ITC capable
so it works just oh wait no power and
ground I did already have up there we
have two additional G power lines but we
have i2c and that's all we need um
that's what it um looks like uh it even
has a SD card uh on the back side uh I
don't make use of that um the next one
is a uh M5 stack Core 2 um and that's um
probably the most easiest um accessible
um option to to build because it is um
uh one of those monitor package thingies
uh that they have with their M5 stack
line um and also has a a growth
connector so um that um you know was
just easy to connect
um uh there's one um issue with this um
this board has a uh CH 90102 chip for
the USB connection for the uh
communication and uh it reports always
connected the serial CDC ACM um uh
connection is reported as always status
connected so there's no way for the um
Linux syst system to uh conclude in what
state the uh processor is it might not
even have um um uh a working firmware on
it hey I'm connected um
so I uh I like this physically but
that's that uh that chip behavior is
really uh not so nice uh and then
there's a wave share also an
rp240 um it uh does also have that um
ch9 102 chip same behavior um the nice
thing about it is it has 30 pin out and
um they are uh that's what's what's in
here and they are really really tiny
because they are half the size of a
regular
gpio so uh instead of um um 2. 54 mm
they're in 1.7 uh 2.7 mm so that's
really really tiny so let's uh get to
the uh software side and I need to hurry
up a little bit uh on the Linux side we
have the um the um speech processing
software uh I used the cop um and why
that's the case we'll get to later on um
that's for essentially all the voice
interaction um and then there's uh the
uh physical um uh interaction with the
the buttons and the display um that's uh
uh uh essentially what uh has been U um
the core of my work for this project and
the last part is uh commit up so um
that's a package that's readily
available in uh Debian um essenti for
network
configuration so um all the um um
application part components they are uh
in Python they run in um uh micro WGI
application server with the Python 3 pre
plugin and if uh you have never uh seen
that application server uh software I
highly recommend it it is one of the
most ex um
extensively um implemented application
server uh softwares I've ever
seen so um on the aduino the
microcontroller um aduino is the um the
framework that I use there um and
essentially what the what that thing
does is just IO Handler
it uh um just regularly PS it to see is
there uh something that uh that the user
triggered for me it also displays uh
whatever needs to be displayed on uh
shown on the display um and um
essentially as long as whatever device
you have as long as it can signal its
state to uh just uh some pins uh on um
on the uh GPO extender then you're all
good you can uh uh use it so the
firmware is in C++ because the um
animations uh all the graphics that that
are in there take up quite a lot of um
uh space
and
um the um RP 2040 only has 264 kiloby of
ram so that's uh rather constrained the
esp32 has uh loads and loads of memory
so that wouldn't be an issue but with
the
rp240 python microp python circuit
python wouldn't have been an option due
to memory
constraint so um the uh microcontroller
is really um stateless once in a up and
running
State and
um i2c Pauling um we'll get to this just
on the next slide um because that's also
where software history oh um let me skip
over over
this okay I moved my slides
um software history
um mcraft uh did build uh and we've seen
these um mcraft uh components for weight
word detector and uh Speech deck I think
on earth size um they uh wanted to do
everything on device also uh they uh put
everything they they made open source uh
except for the speech to text because
they did rely on cloud services for that
as well um so um microft uh was the
initial um um voice assistance software
that I tried but I then switch over to
caliop uh for the WW detectors um the
open source detectors they have a too
high false positive rate to be usable uh
in uh you know just everyday situations
um right now I actually use the pcoo
porcupine 2 which is not open source but
which is free uh uh in terms of uh money
you can get a personal license for free
so um that's not bad also and it works
quite nicely um and then um we have the
uh Speech to Text um uh topic so uh um
the um open source information have
usually a word error rate of around 10%
or more than that and that's really
unusable for interacting with a device
where you just expect to have uh you
know you just want it to work um so uh
the the reason why this is so high is
because the vocabulary is so big uh you
know they have 50,000 words uh in that
language model and um they don't know
what to expect so if we we could reduce
the vocabulary then uh we could uh
improve on the word error rate and
that's what uh cop uh uh allowed us to
do because the biggest drawback of kop
is that it does no intent parsing it
does essentially string matching so you
have to configure all variations of the
commands that you wanted to uh react
to and that's rather uh bothersome but
because we do that we have a non
vocabulary so we can reduce the
vocabulary for the uh Speech or text and
um that um has the effect that uh we
have a a word error rate of way way way
less than
1% I have to really try to um you know
under better conditions than this uh to
get it to misunderstand me so uh cop
also has rest API and
um um all these components talk to each
other essentially mqtt for uh the
different uh processes uh USB I already
set for talking to the uino controller
um the web interface for configuration
is done using uh uh flat def so it talks
to the flat demon that's IPC and the
browser of course uh talks to the uh the
front end that uh that is the uh the
console so um talking to the
microcontroller we have around 15
commands uh that need to be sent to the
device the uh device needs to send a
response um the device uh needs to send
events when some IO event has been
detected and also uh just locks this
lock or debug event uh from the mic
controller to the host so um various uh
parameter types so I decided to use Json
as the uh protocol and essentially it's
uh Json array the first element is the
command second element is uh um a
dictionary of uh key values for the
parameters so um due to timing I'm going
to skip over this real quick it's uh
essentially uh booting up configuring I
need to configure where the uh gopo
extender is on the ITC bus and then I
tell it okay start what is why did I put
this in here because um the ITC polling
is done on the second core of the RP uh
2040 and also on the second core of the
esp32 because they both have two cores
we can just move the it Toc polling
issue to the second core and not worry
about that anymore end of configuration
and then it says I'm running and then
the uh um the um Linux system says Okay
show the idle screen which is just the
current time um when an IO happens uh
there's an event from the
microcontroller to the host uh which
essentally just say okay I have it on
this pin you told me this pin has the
name volume and it's an event that you
configured as Delta + one so um I get
consumable uh events for the uh
applications aside on Linux system and
then uh the response is okay well then
show the overlay for the volume
control let me skip over that
um so anecdotes
um let me um talk about the uh USB here
first I did plug in the um the hell into
my laptop and um at home I have a uh you
know the um a docking station with a USB
hub and once I put this project aside
for a couple weeks and then uh uh you
know pull it out of the drawer start
working on again I plug it into USB hub
and I hid every now and then errors that
that I'm unable to explain and uh then
it takes a while and then it's like oh
man I forgot the USB hub doesn't have
enough power for the uh the thing to
operate correctly so it's really a power
issue and I run I've run into this many
times and then it's like okay I did
forget about that again
um and
um I'm going to skip over a couple of
these the Dome that um does really
nicely
um um I can't get it apart uh um you
know it sits on the display that does
the the the the lighting uh does very
cool uh reflection effects that was the
hardest part to Source I bought like 20
or 30 uh uh um different um domes before
I found one that was really uh a good
fit for this
project um and U micraft uh they just uh
went bankrupt I think uh at the
beginning of this year at least they
closed their operations so uh their
source code is still um online um there
have been a couple of commits to it but
um you know it's probably more of a done
deal now so release 1.0 I need to finish
uh quite a few bits here and there um
and um one important part is when you uh
just you know put in the uh install
image on the SD card uh you need to
configure the network card so the idea
is to show a QR code on the display that
you scan with your mobile phone that has
the connection for the Wi-Fi you connect
to the AP that's running on there that's
the commit up um uh software uh then you
configure it and then you're all running
so oh yeah and I want to do
documentation so Hardware
um I really have no idea if uh if there
is anything else to add to it um so
Hardware is probably stable uh unless
some Pi 03 comes out um software um I
want to um uh take a look at other open-
source uh wake wood detector
implementations and uh the other part is
calop uh needs in some areas quite a bit
of cod refactoring so that's what's down
the road for me on that and that's it so
thank you for all the organizers and
angels for um enabling us to be here and
um now we have time for
questions so do we have
questions hello thank you um one uh
thing I by myself and others usually
observe with um the um these devices by
the usual
know vendors with cloud and anything is
kind of hey play this and that music and
then you have a Spotify or YouTube music
account in the background um do you have
any thoughts about solving this
multimedia stuff and casting stuff to
TVs or is this basically something where
you have to go by the big windows closed
and we say we don't want that okay
um what I do uh already have implemented
that's really the other Pro Pro why
that's the the RFID card reader is in
there um I actually have um uh um a
python implementation for uh that's
working for um connecting to the Spotify
API and playing that um and uh I don't
think I've released that yet uh really
just TR code changes but um the last
time I checked a year ago um that didn't
work um so um that's main use case so to
say for for this um in terms of um
putting that into this um this whole
ecosystem of software that's running on
there it'll probably be a cop plugin
later on so expect that sometime down
the
road I think there was another question
over here
somewhere how is the micro of perform uh
the the microphone performance of the
system yeah so uh on Camp settings not
so good um when you're in a um you know
uh uh room uh in you know like your your
home um it's quite good um so uh 3 m is
um the distance where you can reliably
um talk to it so more than 3 m that's uh
iffy but um 3 m or closer um is really
is quite good um and um I would say if
you're like 3 m away you probably get a
word error rate of I don't know by just
by feeling I would say maybe
5% when you're like up close because um
uh you know it's right on the wall next
to you or so then you know way below
1% hi um how does your work relate to
other existing solutions that do the
whole thing in one like Snips or Jasper
open
Assistant
um um okay open assistance is really
like a home automation um thing that has
a voice interface if I
no not home assistant open Assistant
that's a oh okay um like Snips or jper
um open system I'm actually not aware of
um Snips um
um I'm I can't play Snips right now I
I've heard I have a looked at it but I
can't say right now what it what it was
so um maybe we we can talk
afterwards so all this work that you
have done do you have published it or is
this still a plan um well there's um uh
like a right now A not really working uh
State on GitHub um I have uh around uh I
would say 50 to 70 commits that I
haven't pushed to GitHub yet um so but
I'll do that uh rather soonish so
hopefully next week um so
um probably in a month I would say the
uh the state on the GitHub um oh and I
didn't put in the the URLs for the
GitHub um
well I mean you can Google it um but uh
probably in a month there uh that's uh I
expect to have a a a working state in
GitHub that you can just download and uh
uh install script Etc so um ra hopefully
by then it'll be rather easy easy to
deploy right now it's a lot of manual
fiddling
yeah there
you have the mic yeah so the word I
didn't get is uh what's the easiest way
uh I mean how easy it is to uh create
new commands uh so for example I want to
uh is there a web UI for that or I have
reild the whole thing no no that that
was the webbi that I shown okay so um
this right here MH um it's not so
visible but um you can edit the commands
right here but uh how does that uh
interact with the model which you said
you have a fixed model of words that
areed so um let's say you um you add a a
new order so and introduce new words
right um what it does it uh it uses the
calop rest API to sell okay there's a
new order that's the term for it with uh
the given uh prompts uh the words um and
um that is then used by the uh Speech to
Text uh um a plugin uh on request so
when when a speech to text uh um um
request is made it looks at the working
set of the words in the model uh in the
copi uh prompt model so um it's always
up to
date is it o off offline is uh that part
as well yeah okay the uh unless you have
a skill that needs uh some sort of
network connectivity this this thing
wouldn't need
anything
so well maybe this last
question um do you have implemented any
Multi-Device management so I trigger the
hot word and the device in the kitchen
and in the living room can hear it do
they speak to each other hey who will
handle the job of us um that's it's
actually um somewhat already implemented
you need to configure it because the cop
has HTTP rest and mqt um uh um um skills
so to say so uh you could interact with
whatever bus your home automation uh
device Pro or system has like home
assistant open Hub whatever but you know
it needs to be configured but it
technically will
work
okay
so unfortunately we have reached the end
of our time but also most of the
questions were answered so I guess you
are all ready to build your own hell
now and with that I think I think we
thank the speaker
again and
yeah thank
you