Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2634 Thanks!
[Music]
hello everybody Welcome to the Millway
stage how are you doing are you having a
good
evening do you like the milway stage do
you think it's great do you think we
should support
milliways that's great because I have a
the perfect way to do it in my pocket it
is a milway
coin and you can get one in the um
village behind me for a donation of €30
that they are using to build beautiful
stages like this and uh prepare the the
camp and everything around it um they
also have vintage coins from former
camps that you can look at if you have
one that you particularly like so this
one for example is from this year so it
has the logo of this year but there are
also even more coins this one with
resource exhaustion on it so maybe just
check it out after the talk
and now without further
Ado I have to check in with my beautiful
angels is the camera
ready all right we're going live and I'm
very very very
happy that we have Nicholas here
Nicholas really really likes
microcontrollers and he's going to tell
us all about them please give a hand for
Nicholas so uh thank you for the
introduction um thanks to Milly ways uh
for the for the stage here and thanks to
everyone for this amazing Camp um my
name is Nicholas I really like
microcontrollers uh that's the short
introduction the longer introduction is
I studied computer science uh at the
rwth a university some time ago and by
studying I mean I was building
autonomous robots at the robot club aen
uh since 2010 and out of this project
fell the um cus plus Library um that is
known as modem. which is a cbus plus 23
library that supports
3700 cortex M devices which I co-
maintain and then I got bored and I
started working at arm uh on arm v8m
sandboxing before I got bored of that
and returned to the university to study
for my Master's Degree and by studying I
mean digitizing the railway signaling
lab in the transportation engineering
department and designing a 32 second
scale uh modular signaling systems out
of pcbs and 3D prints because why not
and so I finally finished my Master's
Degree um and now I start I started
working at aarion which uh mostly
debugging the open source PX4 autopilot
for commercial drones uh so you know in
in summary my name is Nicholas and I
like
microcontrollers and for my day job I
debug the the sky Noe autopilot that you
can see here on the right uh it contains
a very fast stm32 H7 running at 480 MHz
and a bunch of sensors and the rest of
the board is U not my job it runs Linux
and communication probably not important
and the autopilot runs PX4 which is
actually open source um it runs the ntic
real-time operating system which is
fairly complex and it has some small
subtle bugs here and there and my job is
just
debug it's not actually embedded
software engineer that's a common
misconception it's just debug and it's
quite difficult because the the code
base is quite large and the processor uh
is quite fast and today I want to share
how I debug uh and some of the tools
that I wrote to help me
debug so what are microcontrollers
microcontrollers when I talk about them
I specifically specifically mean arm
cortex M
microcontrollers um they contain a
microprocessor from arm here cortex M7
in light green on the left which is
connected to a bus system to nonvolatile
Memories like flash and volatile
memories like SRAM here in yellow as
well as a bunch of special purpose
peripherals and these architectures can
be quite complex you can see here the
CPU uh is connected to several internal
buses uh several different peripherals
and different clock domains you can also
see the direct memory access the dma
peripherals in dark green uh so there's
a lot of stuff going on inside this
that's not just controlled by the CPU
and these peripherals can be internal
like the random number generator or they
can be external like the ethernet uh
peripheral which connects uh to the FI
uh via the microcontroller pins and uh
the memories here are typically in the
kilobytes uh so you don't really have uh
uh the opportunity to run Linux which is
why they use custom purpose real-time
operating systems like ntic and the
interesting part here is on the left the
CPU also has external signals uh for
debugging and that's what we're going to
talk about in this talk so if you want
to debug a microcontroller you first
need to connect uh to the serial wire
debug pins uh and there's a stlink V3
which costs about €1 it's the cheapest
and most capable debug probe that I've
ever seen it comes with a built-in
serial it's very fast I highly recommend
it I'm not sponsored
not by St um the debug probe then
communicates over USB to the driver
software which is typically open OCD uh
which stands for onchip debug not the
other thing and it implements the GDB
server protocol and you can then connect
GDB over TCP uh via and and then debug
the device via the command line
interface but this is typically then
also wrapped via the GDB machine
interface inside your your IDE and this
machine interface is mostly for
um exchanging state which is useful for
a user interface and of course this has
a lot of latency right uh you have to
walk several Stacks networking Stacks
USB Stacks uh it's definitely not real
time and it's particularly bad if you
connect it over
GDB so what is GDB GDB is the debugger
that comes with your arm non ABI tool
chain um arm provides you with an
officially tested and compiled
pre-compiled tool chain please use that
for compilation I've seen so many bugs
from weird tool chains but the GDB
unfortunately is not compiled with
Python 3 support so you need to install
another um another tool chain but just
Sim link the the GDB don't compile with
that tool chain and here's quickly how
you launch a GDB session um you first
launch the driver software which is open
OCD with the correct Target
configuration and initialization and
then you launch GDB with the elf file
which contains all of the debug symbols
that's very important and then you
connect to this GDB server which is
created by op
OCD and then there are a bunch of like
super basic commands there's control C
for halting the CPU and then the CPU
isn't running anymore so if you have a
drone that's flying in the sky and you
start the GDB it tends to stop doing
that so be careful uh particularly if
you have spinny things connected to your
microcontroller you can of course single
step through your code uh using these
commands you can use use back Trace to
show where you are um there are many
better tutorials on the internet just
consult them um GDB has its own
scripting language but it's really quite
limited it's not that great so instead
of like fixing that they just
implemented um a python API and you can
Source Python scripts using the source
command inside GDB and then you can
import GDB and this module really only
exists inside GDB it's it's not python
it is actually a implementation in C
directly inside GDB so that's fun to
debug and there are some limitations uh
the python API cannot write variables
you must use the workaround by using the
calling the GDB scripting API from
inside the python API bit weird um you
cannot access some of the debug
information like the C PR processor and
it's Lang it's a language independent
API so pointer semantics and CNC pluss
plus are some sometimes a bit weird I'm
not going to show you a lot of code I'm
going to talk mostly about Concepts um
let's start with something simple I
started working outon about seven months
ago uh and the PX4 autopilot project has
4 million lines of code which is a lot
um so I needed to get an overview and
the simplest thing is just to start your
debugger and see a couple of back traces
to understand what function is being
called by what and the easy way to do
this is to set a breakpoint on a
function and then do a back trace and
you can do this fast um automatically
with the commands command which adds
commands to the breakpoint um and then
just continues immediately uh and then
you can lock this into a file and then
postprocess it with python to get the
actual call graph uh I've used um graph
Vis it looks like this um quite complex
uh there were a couple of issues with an
SD card um some there was some grounding
of drones don't look it up and um
uh there was an issue inside this this
code and so I needed to know what is
actually talking to this SD card what
peripheral is responsible for this so I
created this call graph um and you can
see this is the resulting call graph
it's here it's a bit closer you can see
it's split into three parts so you this
is the SD MMC peripheral that actually
does talk to the SD card it's the lowest
level thing and you can see the three
marked functions down there read write
and modify registers um and it only
really has two entry points at the top
left and and right um the read and WR
data and this interface then talks to
the um fat file system implementation
which is available here in yellow that's
a lot of code great for debugging um and
finally above here you can see the
standard lip C functions on the left you
can see fsync uh and on the right you
can see open of course you can't see it
because it's too small you're just going
to have to take my word for
it um the yellow Tas is than the actual
logger code so there's a lot going on in
this thing and in a way PX4 is kind of
the perfect laboratory for all of these
tools because it's got everything uh
implemented in it it's really quite
impressive in you know a negative way
and a positive way and the great thing
about this is that you don't even need
to instrument your code you can just do
this in GDB you don't need to modify
your firmware um it's very slow so again
don't do it in the sky or
in any system that uh requires safety
but it's very helpful to get to know a
code
base and if you want to know how this is
implemented you can look this up in
detail in the open source embedded debug
tools uh it's a Python 3 Library it's
BSD licensed it's fully open source on
GitHub with lots of documentation it's
highly modular so you can reuse some of
the parts that are not useful for you
because it is of course specific for um
stm32 PX4 ntic
uh and it's actively maintained so feel
free to contribute or just use it as a
reference or just ask questions um and
the reference at the bottom right of the
slide refers to the python module so if
you want to look something up quickly
okay let's look at some more GDB tools
natic is a real-time operating system
with preemptive
threading and so PX4 uses a lot of
threats uh we start I think 1,700 mostly
because of historical reasons and um
here I've created the first GDB python
tool which is PX4 tasks and it lists all
of the threads with their P ID name CPU
stack usage and most importantly what
it's waiting for because again my job is
just debug and the most common source of
problems is semaphor I I hate semaphor
so much uh they are the bane of my
existence and being able to see them and
see their address pointer where they
live is great for my sanity um so this
is very useful uh how does it work work
this is implemented as a custom GDB user
command um GDB can look up symbols in
various Scopes here we're looking at a
global symbol but you can also have a
function symbol or a file symbol um and
natics like many of the OS out there
uses something called a ready list of
tasks that are ready to run uh and we
can take this linked list and just
iterate over each task and then look at
simple things it's usually a struct uh
task control block TCB that's what it's
called um and uh typically it contains
all of the relevant information but some
of the stuff is more indirect like if we
want to look at the stack usage we need
to do a binary search to figure out
where the watermark is or we need to
look at some timers to figure out the
CPU time so you can scale this up or
down how you want I mean it's python you
know import anti-gravity and so um
here's another example ntic implements
its own Dynamic interrupt dispatcher in
assembly because why not uh therefore
every interrupt Handler inside the
official envic table interrupt table is
the same function which is of course not
particularly useful um so I wrote this
small tool that finds this dispatch
tables and just renders it and you can
see they all have the same priority
because nutx doesn't support nested
interrupts uh it doesn't need to it's
fine um and you can see whether the
interrupt is enabled pending or active
and here you can see the irq 59 for
example is currently active so we we
halted by coin Ence inside this very
short interrupt which is currently
servicing the dma you can also see the
argument to the private dma buffer over
there um and you can see that IR q65 is
pending so it'll be executed next right
after this interrupt um and this is
super useful to figure out what function
to actually put the braak point on okay
that's enough nutex uh you're all
damaged
now you're welcome um let's look at some
stm32 two specific tools so stm32 is the
actual microcontroller on there it's
Hardware it cannot be changed um I often
need to know what the actual state of
the microcontroller pins are so this
tool just shows it to you you can see
the pin name the configuration the input
and output State um you can see the
alternate function you can see the
signal names and functions that are of
course specific to our um um
autopilot so for example the pin A6 and
A8 are inputs with pull-ups and they're
currently High which you can see by the
Caray uh if they're low they have the
the lower bar again I'm limited to asy I
think maybe Unicode I'm not entirely
sure hm that would be interesting H and
then there's also pin a13 and A14 which
are this serial wire debug connection
that we're currently using to debug this
chip and we can see that they're
internally connected to the alternate
function five and zero and you can also
see that the data connection H is
configured for a very high rate very
high data rate which is plus VH so this
is a compact uh
description how does this work well the
debugger cannot only access the internal
SRAM but it can actually access the
entire bus system so it first goes
through the 64-bit bus and then it goes
down here through another 32-bit bus
because why not and then it finally
accesses the GPO register files um I
don't know what port K did maybe it was
naughty but it's separate from the other
ports who knows
St and for this to work I need to know
the exact address of the G peripheral
which I can find in the stm32 H7
reference manual uh which is thousands
of pages long uh it's a giant PDF which
is very fun to copy stuff out of uh so
not great to work with uh but I need to
know so this is the address I need to
know what the the bit Fields inside the
register mean so I can also look this up
here and then I can write this into a
short python script to uh read iterate
over each pin and then condense this
into this table but of course I cannot
write a custom parser for every
peripheral on the stm32 because it's got
dozens of peripherals and I'm not
getting paid to do that so the wouldn't
it be nice if there were a machine
readable data source of this information
and yes there is people have already
thought about this it's called the
system view description files uh which
is standardized in the cmis standard um
and it's an XML format that describes
the entire register map so you don't
have to copy it out of the uh PDFs
anymore um you can use this machine
readable data source unfortunately it's
not super accurate maybe like 90% there
are some issues because T and so um
these SVD files are a little bit broken
but there are some manual patches
available and uh I also wrote my master
Theses about improving this and there is
a great GDB plugin already from
pangy um which does all of the heavy
lifting for you it loads the SVD file it
tells the debug probe where to load the
memory and how to interpret and then
render it in a structured form and our
or or My embedded debug tools just wrap
this Tool uh and also provides you a
difference viewer so this is what it
looks like right so again it's a custom
GDB user command where you tell the
peripheral dot the register that you
want to look at and then you get this
view the raw binary in this case we're
looking at the dma uh two stream zero
configuration register s0c worlds of the
tongue and you can see the raw register
value here and then below the raw
register value all of the bit Fields
with the name and the description all of
this information comes from the SVD file
and this is extremely helpful for just
quickly checking the configuration of a
peripheral without manually fiddling
around with bits and then doing mistakes
which I've done plenty of times you
still of course need to read the
reference manual to understand the
entire context but just to quickly uh
understand what this means it's it's
very helpful there's also a um there's
the possibility to combine this with
Hardware watch points so inside the uh
STM inside the cortex M you can declare
a memory region that should be watched
for writes and reads and then it
triggers a breakpoint and uh if you keep
the previous state of the register you
can render this a difference you can see
in Gray what it was before and then in
black plus what it's now and this is of
course even more helpful because you can
step literally step through the rights
you're not stepping through lines of
codes you're stepping through register
rights which is very helpful
particularly when you combine it with a
a back trace and maybe you can see it in
the front row but this is again SD card
stuff which uh sets up the dma transfer
to transfer the file to the SD card um
so very
helpful finally the last GDB tool that I
want to show you is core dumping support
uh which is not really natively
implemented in GDB because cortex M are
a bit weird and they're very specific so
there is no generic implementation but
it's actually relatively simple you only
need to read out all of the volatile
memories like SRAM and all of the
peripherals and then store that in a
file the non-volatile memory from flash
that already comes from the L file so um
the SVD files here are used to not have
to read out the whole 32-bit address
space which would be 4 gigabytes it
would take forever only I'm only reading
about a megabyte here because I can only
read the actual ual parts of the memory
map that are used which I know from the
SVD file and then you would write a GDB
server that just pretends to be
connected to a device instead of
actually being connected to the device
and GDB cannot actually tell the
difference and this tool is already of
course you know other smart people have
already thought about this and one of
these smart people is Adam green who
implemented the crash debug utility
which does exactly this as you can see
here I can strongly recommend that it's
extremely useful for if you're tired
just taking a core dump and then
shutting the device down throwing out it
out the window you know setting the
office on fire not that I would do that
um and uh you can archive these buggy
devices in their current state and then
you can either try again later or you
can send it to another engineer who's
maybe more tolerant of this
[ __ ] and then um finally all of
these tools except this one are running
while the CPU is halted this this one is
running when the CPU is on fire because
I set it on fire um but there are
certain things where you cannot actually
halt the CPU like if the Drone is
currently flying around so here you need
to profile your application and the
simplest profiling method is
logging it's usually on a
microcontroller done over the serial
link um it's very low cost uh it's built
in It's very effective unfortunately
everyone loves it everyone uses it uh
and of course it's a necessary tool
particularly if you log uh the uh the
actual output here not over the serial
device but onto for example an SD card
which is great if because then you have
a crash log unless of course the bug is
inside the SD card Handler then you
don't have a crash log um but it's of
course way too slow for our processor uh
we're running at 480 megahertz uh it
produces several million events per
second uh you're not going to push that
over uh the microcontroller serial link
there are lots of events so we need a
higher band with
connection and there is a simple idea
just log everything to a ring buffer
inside SRAM can be as large as you want
and then let the debug probe do the
transfer because the debug probe is
attached um it can access the SRAM as
we've shown before and it can just
periodically pull asynchronously in the
background without disturbing the the
CPU at a low priority and this is the
entire idea behind sagar's system view
which provides a library for you to
serialize artos events and then also to
timestamp them at a microsc resolution
but it's all done in software so the
serialization uh the the the threading
the scheduling the semop force the
interrupt all of done is all of that is
done in software so it's actually quite
a large
overhead uh and of course it's
proprietary and it costs money and
unfortunately there is no nutx support
um so I thought well I can do
that please hard can it be wouldn't it
be great if instead of doing everything
in software there was some Hardware
peripheral that did that for you and you
don't have to look any further but the
um instrumentation Trace macro cell and
the data watch point and Trace
peripheral it really rolls off the
tongue arm is really good at
abbreviations I love it and here you
have 32 channels that you can write 8 16
or 32bit values to and it also logs the
exception entry and exits without any
code modification just does it in the
background and this whole thing is
implemented in Hardware including the
time stamping at cycle accuracy so it's
in every cortex M except the cortex M
zeros but we don't talk about that
anyway uh the whole thing um you really
only need to add the single uh line
which adds the
which writes the value that you want to
log to the port in this case it's Port
three so the fourth
channel so this is then serialized and M
demultiplexed into a multiplexed into a
single pin which is called the serial
wire output which is basically a very
fast
uart uh and the stlink V3 the $12 probe
that I showed you at the beginning can
read this up to 2 megabytes per second
which is fast right we can write maybe
300 kiloby to the SD card because of the
overhead right but 2 megabytes a second
that's good so this is enough to uh
output all of the scheduling information
but unfortunately it is a fairly compact
binary format and we need to pars it and
this is where the orb code project comes
in because those are the people that
thought well we can also do what Seer
does but we can do it better because we
can do it open source and then other
people can
contribute and they've implemented a
part bser for this bite stream which can
demultiplex this uh and then convert it
uh into something useful it's a cap API
uh and I have written the orb orto tool
which is a C++ tool that then takes this
data and converts it into F trce packets
which is the standard or well de facto
standard way of uh um storing Trace
information on Linux and it's called
orto because we I use the perfetto UI to
visualize this so orb code and peretto
orto are you getting it hey h and then
this is visualized by perfetto which
originally as I said was um used by
Android and Linux traces but you know
why not use it it's great and here at
the top uh you can see the
CPU which is uh shared by all of these
different threads and you can also in
the line just below it see all of the
interrupts you can see a very long
interrupt there in the middle the SVC
call and this is not real uh this is
only because the in exception entry and
exits have a lower priority than the
other packets so there's often a packet
drop and sometimes the exit gets lost
and so that's why you have these very
long it's not a problem in in real life
you just need to know it
um on the left here you can see all of
the threads with their names and P IDs
uh NX um has uh PX4 uses a lot of NX
threads so this is a long long list if
you scroll down there you can try this
yourself there is a file you can drag
this into your web browser and it just
works and happy scrolling around and um
by the way this is this is just every
tick up there is 100 micros so this is
very very fast and you can see each of
the threads here for example um we have
a bunch of work cues that run all of
these sensors they periodically pull
them and read out the values and you can
see that the work has started and then
the threat was interrupted and then the
threat returns and the work Q concludes
so just having the threading information
is not enough you need to know what the
actual work CU is is doing here um so
this is incredibly useful and also just
incredibly educational to understand how
a real-time operating system works
because you can see the scheduling
happening when for example there is an
interrupt you can correlate the edges to
the interrupts and this is exctly how
the artor works right an event arrives
and oh uh this interrupt happened I need
to reschedule immediately another thread
uh so just scrolling around this in your
free own time is very educational and
very fun but it actually becomes much
more interesting if we zoom out because
we can suddenly see some patterns you
can see here this as I said before the
work cues they pull the sensors every
couple of milliseconds you can see every
tick here is 1 millisecond and we pull
it approximately
every 2 milliseconds so 500 Herz I think
the update rate is 400 Herz or something
um and this is a nice visual way of just
seeing if there are any timing issues
because if we zoom out further here you
can see 100 millisecond tick we suddenly
see a hiccup the it toc1 task has become
irregular the work cues have become very
long several hundred uh several dozen
milliseconds long and the reason for
this is was there was a power glitch a
brown out on the external sensor bus
because somebody plugged in too many
sensors it was me and uh so the sensors
had to be reinitialized again my job is
just debug I don't know how power works
so this kind of visual
debugging with your
eyes I mean how else would you do it but
is incredibly fast to spot timing issues
right you can just look at this thing
and be like yeah regular regular regular
oh it's not regular anymore there's the
problem uh and it also makes you look
like a wizard
because it's cool you can zoom around
and in it and other people are like wow
so unfortunately the serial wire output
is still Limited in bandwidth uh 2
megabytes per second is not enough so
arm decided you know arm is a hardware
company so they decided to solve the
problem with even more Hardware who
would have guessed so there is parallel
tracing which gives you a 4bit white bus
up to 1 gabit per second
in theory in practice of course it
depends on the signal quality but you
know it's more than enough a couple of
dozen megabytes should be enough per a
second and in addition to all of the
previous functionality of the itm and
the DWT um there's also a compress
compressed instruction Trace that allows
you to reconstruct the entire program
flow off device uh it has to be
compressed obviously because you cannot
log every single in struction because at
480 MHz it would be a lot of data um so
it only actually lcks the branching cuz
the rest is known right you just follow
along and uh there this is very fast it
requires custom Hardware usually in fpga
with a USB 3 Connection to really get
the 1 gabit per second in the standard
but uh the J Trace which is sort of the
the cheapest de facto standard for this
interface costs a lot of money and I got
one for work and it was not as well
documented as I had hoped um and the orb
Trace instead again from the orb code uh
Community is actually open source and
you can buy it and it's 10 times cheaper
it's not you know fully working yet but
uh it's definitely more documented and I
could get it to work faster so it's
working for me which is important uh so
there is a very helpful Community around
it uh there are some smart people that
work on it and if you want to hack
around with this this is like the
ultimate thing it can do everything that
I've previously said including swo uh
input and debugging of course and I
think it even has a Serial link um the
question is what do you do with this
data because now you get several hundred
megabytes of uh instructions and what do
you do with this data and the most
useful thing that I found is a research
paper called micro AFL which is stands
for micro American fuzzy Lop which does
fuzzing on the device using this
branching information because that's
exactly what American fuzzy Lop needs
branches um and it found a lot of bugs
inside the stm32 hardware drivers the QP
drivers um and it would be very
interesting to use this uh perhaps in an
open source setting um to test our
autopilot before it takes off right that
would be a great idea so I'm looking for
interns to work on this right now
because apparently I need to do jur
tickets it's my job now
and um yeah so this is the end of the
line there is no more Hardware there's
no more tooling we're out of stuff to do
uh so in conclusion debug all the things
right use more GDB use more debug
Hardware uh check out my cool open
source BSD licensed project maybe just
for inspiration maybe just you're
curious want to see how it works and uh
the orb code project is always looking
for competent embedded people so if you
want to contribute there have fun uh
maybe liberate all of the debug tools
from the commercial hands would be very
nice and uh thank you and do you have
questions so thank you so much Nicholas
uh for your talk um over there you see
an angel this Angel is willing to take
questions please signal him your
wish thank you for
your um one
moment ah yeah thank you for your talk
uh you mentioned at the beginning that
uh peripheral like spin things wouldn't
react too good to hoing the CPU or
changing the timing yes you have
experiences which kind of peripher like
memory displays other chips work good or
not so good can you share experiences
with that so as always it depends
because of course St also thought about
this and they have um the debug MCU
peripheral which controls how other
peripherals are behaving while the CP
use halted so you can actually not halt
timers for example if they control a
motor so you can halt the CPU but not
the timers that control the motors um
but in
general in
practice the entire system is not as
autonomous that it would keep running
with only the timer right so the Drone
needs to read out the sensors and then
does the control Loop so it's not
actually that useful it's only useful
for stuff like motor control um things
that don't work well also are
communication protocols because usually
there is another participant that waits
for packets and then timeouts happen and
then you have weird states where the
microcontroller still thinks it's
communicating but it's actually not the
other partner has already left so
unfortunately this halting stuff is
really the the using GDB is really only
useful for a bench setup uh and that's
really how I use it on in my work um we
we never ever attach a debugger to a
drone because they cost a lot of money
um and again my job is just
debug um to anyone uh watching us from
The Ether it's also still possible to
ask questions to our signal
Angel okay the stream is down oh no okay
the stream is down so sadly not but
there is a question here um I have a
question uh do you also use GDB for uh
regression test or integrated in your uh
testing
environment not yet but I am looking for
an
intern to do exactly that because of
course there there's some more stuff
that I didn't talk about which is
scripting GDB from the outside right all
of these were user commands from the
inside but the machine interface you can
use to call GDB from the outside and
orchestrate
other things like logic analyzers or
waveform generators and then actually do
uh the inspection from the outside and
the inside of the device uh but of
course the problem is do you program
this manually like what is there a a
standard definition of an i2c protocol
uh that you can then fuzz around with
right the stuff doesn't really exist
like there are Manual implementations of
this but there's no formal stuff that
you can just put into a
constraint solver or something and play
around with this so those are
unfortunately still quite a lot of
research questions the tooling is kind
of in place like you can certainly see
where this is going but the specifics
are not there yet but yes very good
idea any more questions please um show
your hand or stand up if you have
another question I will uh look at
you very intense ly and count in my head
to
three so thank you so much Nicholas for
being here and uh bringing us into the
beautiful world of microcontrollers
please a warm Applause
[Applause]
yay thank you our next talk is going to
start at 1 p.m. it's going to be
Electronics prototyping is way too easy
and there's no reason you can't do it so
I hope to see everyone of you here um if
not have a really nice Camp anyway so
goodbye thanks um one one more thing
while I have you here it's not related
to the talk I still have old badges from
EMF Camp from uh sha and from G so if
you want to have unused free badges
please don't throw them away cuz I don't
have anything to do to do uh with them
yes excellent all right thank
you