*rC3 preroll music*
ysf: Hello and welcome to the infrastructure review of the rC3 this
year, 2020. What the hell happened? How could it happen? I'm not alone this year.
With me is lindworm who will help me with the slides and everything else I'm going
to say before. And this is going to be a great fuck up like last year, maybe. We
have more teams, more people, more streams, more of everything. And the first
team and lindworm who I'm going to introduce is the SHOC. Are you there with me?
Lindworm: Oh, yeah, so I got to go to the SHOC. Yeah, it's kind of a stress this
year. We only had about 18 heralds for the main talks rC1 and rC2. And we have
introduced about 51 talks with that. Everybody from this home setup, which was
a very, very hard struggle. So we all had a metric ton of adrenaline and excitement
without… within us. So here you can see what you have seen, how a herald looks
from the front. And so it does look in the background. Oof. That was hard, really
hard for us. So you see all our different set ups here, do we have? And we are very,
very pleased to also have set up a completely new operation center: the
Herald News Show, which I really, really like you to review on YouTube. This was
such a struggle. And we have about, oh, wait a second, so as we said, we're a
little bit unprepared here, I need to have my notes up. There were 20 members that
formed a new team on the first day. They made 23 shows, 10 hours of video
recording, 20 times the pizza man rung at the door. And 23 mate bottles had been
drunk during the preps because all of those people needed to be online the
complete time. So I really applaud to them. That was really awesome, what they
brought over the team and what they brought over the stream. And this is an
awesome team I hope we see more of. ysf, would you take it over? *ysf is muted*
Oh, no. My, my bad. So is the heaven ready? We need to go to the heaven and
would have an infrastructure review of the heaven.
raziel: OK. Du hörst mich noch? Ja, hallo? Ich bin der raziel aus dem Heaven und ehm…
Yeah, heaven is ready, so welcome, everybody. I'm raziel from heaven, and I
will present you the infrastructure review from the heaven team. We had some angel
statistics scrapped out a few hours ago. And on this year, we have not so much
angels like last year, because we had a remote event, but we had a total of 1487
total angels from which 710 arrived and even more of 300 angels that at least
still did one shift. And in total the recorded work done to that point was
roughly 17 and 75 weeks of done working hours, and for the rC3 world we also
prepared a few goodies so people could come visit us. And so we provided them a
few badges there. And every angel that, for example, found our extinguished…
expired extinguisher and also extinguished fire in heaven. The first batch was
achieved from 232 of our angels and even less. But still a good number of 125
angels accomplished to help us and extinguish the fire that broke out during
an event. And with that numbers checked, we also will jump into our heaven. So I
would like to show you some expressions and impressions from it. We had quite the
team working to do exactly what the heaven could do: manage its people so we needed
our heaven office. And we also did this with respect to your privacy, so. We
painted our color… our clouds white as ever, so we cannot see your nicknames, and
you could do your angel work but not be bothered with us asking for your names.
And also, we had prepared some secret passage to our back office. And every time
on the real event, it would happen that some adventurers would find their way into
our back office. And so we needed to provide that opportunity as well, as you
can see here. And let me say that some adventurers tried to find the way in our
sacred digital back office, but only a few were successful. So we hope everyone found
its way back into the real world from our labyrinth. And we also did not spare any
expenses to do some additional update for our angels as well. As you can see, we
tried to do some multi-instance support. So some of our angels also accomplished to
split up and serve more than one angel at a time. And that was quite awesome. And so
we tried to provide the same things we would do on Congress, but now from our
remote offices. And one last thing that doesn't… normally doesn't need to be said.
But I think in this year and with this different kind of event, I think it's
necessary that the heaven as a representative, mostly for people trying
to help make this event awesome. And I think it's time to say the things we do
take for granted. And that is thank you for all your help. Thank you for all the
entities, all the teams, all the participants that achieved the goal to
bring our real Congress that many, many entities missed this year into a new
stage. We tried that online. It had its ups and downs. But I still think it was an
awesome adventure for everyone. And from the Heaven team I can only say thank you
and I hope to see you all again in the future on a real event. Bye! And have a
nice New Year. lindworm: Hello, hello, back again. So we
now are switching over to the Signal Angels. Are the signal angels ready?
Hello! trilader: Yeah, hello, uhm, welcome to the
infrastructure review for the Signal Angels, I have prepared some stuff for
you. This was for us… slides, please? This was for us the first time running a fully
remote Q&A session set, I guess? We had some experience with DiVOC and had gotten
some help from there on how to do this, but just to compare, our usual procedure
is to have a signal angel in the room. They collect the question on their laptop
there and they communicate with the Herald on stage and they have a microphone like
I'm wearing a headset. But in there we have a studio microphone and we speak
questions into it. Yeah, but remotely we really can't do that. Next slide. Because,
well, it would be quite a lot of hassle for everyone to set up good audio setups.
So we needed a new remote procedure. So we figured out that with a signal Angel and
the Herald could communicate via a pad and we could also collect the
question in there. And the Herald will read the question to the speaker and
collect feedback and stuff. So we had 175. No, 157 shifts, and sadly we couldn't fill
five of them in the beginning because there was not enough people already there.
And also technically it was more than five unfilled shifts because for some reasons
there were DJ sets and other things that aren't talks and also don't have Q&A. We
had 61 angels coordinated by four supporters, so me and three other people,
and we had a 60 additional angels that in theory wanted to do signal angel work
but didn't show up to the introduction meeting. Next! For, as I've said for each
session, each talk, we created a pad where we put in the questions from IRC,
Mastodon, and Twitter and. Well, we have a bit more pads than talks we actually
handled, and I have some statistics about an estimated number of questions per talk.
What we usually assume is that there's a question per line, but some questions are
really long and have to split over multiple lines. There are some structured
questions with headings and paragraphs some heralds or signal angels removed
questions after they were done. And also there were some chat and other
communication in there. So next slide, we took a Python script, download all the pad
contents, read them, counted the number of lines, remove the size of the static
header. And in the end we had 179 pads and 1,627 lines if we discount the static
header of nine lines per pad. So that in theory leads to about nine questions in
quotation marks because it's not really questions but lines. But it's an estimate,
per talk. Thank you. ysf: ... talk and what I've learned is
never miss the introduction. So the next in line are the line producers ha ha ha ha
stb are you there? stb: I am here, in fact, so *singing*. So
the people a bit older might recognize this melody badly sung by yours truly and
other members of the line producers team, and I'll get to why that is relevant to
what we've been doing at this particular event. So what does, what do line
producers do? What does an, Aufnahmeleitung actually perform? It's
basically communication between everybody who's involved in the production, the
people behind the camera and also in front of the camera. And so our work started
really early, basically at the beginning of November, taking on like prepping
speakers in a technical setup and rehearsing with them a little bit and then
enabling the studios to allow them to actually do the production coordination on
an organizational side. The technical side was handled by the VOC, and we'll get to
hear about that in a minute. But getting all these people synced up and working
together well, that was quite a challenge. And that took a lot of Mumbles with a lot
of people in them. We only worked on the two main channels. There's quite a few
more channels that are run independently of kind of the central organization. And
again, we'll get to hear about the details of that in a minute. And so we provided
information. We tried to fill wiki pages with relevant information for everybody
involved. So that was our main task. So what does that mean specifically, the
production set up? We had 25 studios, mainly in Germany, also one in
Switzerland. These did produce recordings ahead of time for some speakers, and many
did live set ups for their own channels and also for the two main channels. And
I've listed everybody involved in the live production here. And there were 19
channels in total. So a lot of stuff happening. 25 studios, 19 channels that
broadcast content produced by these studios. So that's kind of the Eurovision
kind of thing, where you have different studios producing content and trying to
mix it all together. Again, the VOC took care of the technical side of things very
admirably, but getting everybody on the same page to actually do this was not
easy. For the talk program, we had over 350 talks in total, 53 in the main channels
And so handling all that, making sure everybody has the speaker information
they need and all these organizational stuff, that was a lot of work. So we
didn't have a studio for the main channels, the 25 studios or the nine, the
live channels, the 12, they actually did provide the production facilities for the
speakers so we can look at the next slide. There's a couple more numbers and of
course, a couple pictures from us working basically from today. We had 53 channel...
53 talks in the main channel. 18 of them were prerecorded and played out. We had 3
where people were actually on location in a studio and gave their talk from there.
And we had 32 that were streamed live like I am speaking to you now with various
technical bits that again the VOC will go into in a minute. And we did a lot of
Q&As, I don't have the numbers how many talks actually had Q&As, but most of them
did, and those were always like. We had a total of 63 speakers we did prepare, at
least the live Q&A session for and helped them set up, we helped them record their
talks if they wanted to prerecord them. So we spent anywhere between one and two
hours with every speaker to make sure they would appear correctly and in good quality
on the screen. And then during the four days, we, of course, helped coordinate
between the master control room and the twelve live studios to make sure that the
speakers were where they were supposed to be and any technical glitches could be
worked out and decide on the spot. If, for example, the line producers made a mistake
and a talk couldn't happen as we had planned because we forgot something. So we
rescheduled and found a new spot for the speakers. So apologies again for that. And
thank you for your understanding and helping us bring you on screen on day two
and not day one. But I'm very glad that that we could work that out. And that's
pretty much it from the line producers, I think. Next up is the VOC.
ysf: Thank you stb. Yes, you're right, the next are the VOC and kunsi and
JW2CAlex are waiting for us. Franzi: ... is Franzi from the VOC. 2020
was the year... Hm? Hi, this is Franzi from the... from VOC. 2020 was the year of
distributed conferences. We had 2 DiVOCs and the FrOSCon to learn how we are going
to produce remote talks. We learned a lot of stuff on organization, Big Blue Button
and Jitsi recording. We had a lot of other events which was just streaming like
business as usual. So for rC3, we extended the streaming CDN with two new locations,
now 7 in total, with a total bandwidth of about 80 gigabits per second. We have two
new mirrors for media.ccc.de and are now also distributing the front end. We got
two new transcoder machines, Erfas and Enhanced cir setup we now have 10 Erfas
with own productions on media.ccc.de. So the question is, will it scale? On the
next slide... Alex: Yeah, next slide.
Franzi: ... we will see that it did scale. We did produce content for 25
studios and 19 channels, so we got lots of lots of recordings which will be published
on media.ccc.de in the next days and weeks. Some have already been published,
so there's a lot of content for you to watch. And now Alex will tell us something
about the technical part. Alex: My name is Alex, Pronouns it/its. I
will now tell you the technical part first, but more of the organization. I was
between the VOC and the line producing team. And now a bit how it worked. So we
had those two main channels, rc-one and rc-two. Those channels have been produced
by the various studios distributed around the whole country. And those streams,
this is now the upper path in the picture, went to our ingest relay, to the FEM, to
the master control room. In Ilmenau there were a team of people adding the
translations, making the mix, making the mixdown, making records and then
publishing it back to the streaming relays. All the other studios produced to
channels. Those channels took the also the signals from different studios, make a
mixdown, etc. publish to our CDN and relays and we publish to the studio
channels. As you can see, this is not the typical setup we had in the last year in
the presence. So, our next slide, we can see where this leads: Lots of
communication. We had the line producing team, we had some production in Ilmenau
that has to be coordinated. We have the studios, we have the local studio helping
Angels. We have some Mumbles there, some RocketChat here, some CDN people some web
where something happens. We have some documentation that should be. And then we
started to plot down the communication paths. Next slide, please. If you plotted
all of them, it really looks like the world, but this is actually the world, but
sometimes it feels like they're just getting lost in different paths. Who you
have to ask, who do you have to call? Where are you? What's the shortest path to
communicate? But let's have a look at the studios. First going to ChaosWest.
Franzi: Yes, on the next slide, you will see the studio set up at ChaosWest TV. So
thank you, ChaosWest for producing your channel.
Alex: At the next slide, you see the Wikipaka television and fernseh-streamen
(WTF) who have the internal motto: "Absolut nicht sendefähig - chaos of
recording". But even then, at some studios, you look more like studios, so
this time at the next slide at the hacc. Franzi: Yeah, at hacc, you will also see
some of the bloopers we had to deal with. So, for example, here you can see there
was a cat in the camera view, so, yeah. And Alex, tell us about the open
infrastructure orbit. Alex: The open infrastructure orbit
showed. In this picture, you can see it's really hard to see how you can make a
studio look really nice, even if you're alone, feeling a bit comfier, more
hackish. But you have also those normal productions as in the next slide. The
Chaosstudio Hamburg Franzi: Yeah, at Chaosstudio Hamburg, we
had two regular work cases like, you know, from all the other conferences, and they
were producing, onsite in a regular studio set up. And last but not least, we got
some impressions from ChaosZone TV. Alex: As you can see here, also quite
regular studio setup, quite regular. No. There was some Corona virus ongoing, and
this is we had a lot of distancing, wearing mask and all the stuff that
everyone is safe but c3yellow (c3gelb) will tell you some facts about it. But
let's look at the nice things. For example, the minor issue: On the second
day, we were sitting there looking at our nice Grafana. Oh, we got a lot of more
connections. The server load's increasing. The first question was: Have we enabled
our cache?". We don't know. But the number of connections is growing that people are
watching our streams, the interest goes up. And we were, well, at least the people
are watching the streams. If there is a website, who cares, the interest works.
But then we suddenly get the relations. Well, something did not really scale that
good. And then using the next slide, this view. This switched pretty fast from after
looking at this traffic graph. "Well, that's interesting" into "Well, we should
investigate". We get thousands of messages on Twitter DMs. We got thousands of
messages in RocketChat, IRC, and suddenly we had a lot of connections to handle; a
lot of inquiries to handle, a lot of phone calls, etc. to handle. And we have to
prioritize for us the hardware then the communication, because otherwise the
information won't stop. On the next slide you can see what our minor issue was. So
at first, we get a lot of connections to our streaming web pages, then to load
balancers, and finally to our DNS servers. A lot of them were quite malformed. It
looked like a storm. But the more important thing we had to deal was all
those passive aggressive messages from, from different persons who said: "Well,
you can't even handle streaming. What are you doing here?" And we worked together
with the c3infra team, thanks for that, how to scale and decentralize a bit more just to
provide the people the connection power they need. So I think in the last years,
we don't need to use more bandwith. We showed we can provide even more bandwith
if we need it. And then, noting everything down…
Franzi: So is it time to shut everything down? No, we won't shut everything down.
The studios can keep their endpoints, can continue to stream on their endpoints as
they wish. We want to keep in touch with you and the studios, produce content with
you, improve our software stack, improve other things like the ISDN, the Internet
Streaming Digital Node, the project for small camera recording setups for sending
to speakers needs developers for the software. Also, KEVIN needs developers and
testers. What's KEVIN? Oh, we have prepared another slide or the next slide.
KEVIN is short for Killer Experimental Video Internet Noise, because we initially
wanted to use OBS.Ninja, but there are a couple of licensing issues. There is not
everything in OBS.Ninja is open source like we wanted, so we decided to code our
own OBS.Ninja-style software. So if you are interested in doing so, please get
into contact with us or visit the wiki. So that's all from the VOC. And we are now
heading over to c3lingo. ysf: Exactly. c3lingo oskar should be
waiting Studio 2, aren't you?
oskar: Yeah, hallo. Hi, yeah, I'm oskar
from c3lingo. We will jump straight into the stats on our slides. As you can see
here, we translated 138 talks this time, as you can see, it's also way less
languages than in the other chaos events that we had since our second languages
team that does everything that is not English and German was only five people
strong this time. So we only managed to do five talks into French and three talks
into Brazilian Portuguese. And then on the next slide… We are looking at our coverage
for the talks and we can see that on the main talks we managed to cover all talks
that were happening from English to German and German to English, depending on what
the source language was. And then, on the other languages track, we only managed to
do 15 percent of the talks from the main channels. And then on the further
channels, which is a couple of others that also were provided to us in the
translation team, we managed to do 68% of the talks, but none of them were
translated into other languages than English and German. On the next slide,
some global stats. We have 36 interpreters, which in total managed to
translate 106 hours and 7 minutes of talks into another language simultaneously. And
the maximum number of hours one person did was 16 hours and the minimum number of
hours, the average number of hours people did was around 3 hours of translation
across the entire event. All right. Then I also have some anecdotes to tell and some
some mentions I want to do. We had two new interpreters that we want to say "hi" to,
and we had a couple of issues with the digital thing that didn't have before with
regular events where people were present. For example, the issue of sometimes when
two people are translating one person's starts interpreting something on wrong
stream. Maybe they were watching the wrong one. And then the partner just thinks they
have more delay or something. Or, for example, a partner having a smaller delay
and then thinking the partner can suddenly read minds because they can translate
faster than the other person is actually seeing the stream. Those are issues that
we usually didn't have with the regular stream, but only with the regular events,
not with remote events. And yeah, some hurdles to overcome. Another thing was,
for example, when on the r3s stage, the audio cut out sometimes for us and but
because one of our translators had also already translated the talk twice, at
least partially to because and it was already canceled after those, they
basically knew most of the content and could basically do a Powerpoint Karaoke
translation and was able to do most of the talk just from the slides without any
audio. Yeah, and then there also was... The last thing I want to say is actually I
wanted to say, give a big shout out to the two of our team members that weren't able
to interpret with us this time because they put their heart and soul into this
event happening. And that's stb and katti, and that's basically everything from
c3lingo. Thanks.
ysf: *muted*
Hello, c3subtitles is it now. td will show the right text to his slides you already
saw a minute ago. td: OK. OK, hi, so I'm td from the
c3subtitles team. And next slide, please. So just to quickly let you know how we get
from the recorded talks to the released subtitles. Well we take the recording
videos and apply speech recognition software to get a raw transcript. And then
Angels work on that transcript to correct all the mistakes that the speech
recognition software makes. And we again apply some autotiming magic to to get some
raw subtitles. And then again Angels do quality control on these tracks to get
released subtitles. Next slide, please. So as you can see, we have various subtitle
tracks in different stages of completion. And these are seconds of material that we
have can see all the numbers are going up and to the right as they should be. So
next slide, please. In total, we had 68 distinct angels that worked 4 shifts on
average. 83 percent of our angels returned for a second shift. 10 percent of our
angels worked 12 or more shifts. And in sum we had 382 hours of angel work for 47
hours of material. So far we've had two releases for rc3 and hopefully more yet to
come, and 37 releases for all the congresses, mostly on the first few days
where we didn't have many recordings. We have 41 hours still in the transcribing
stage of material, 26 hours of material in the timing stage and 51 hours material in
the quality control stage. So there's still lots of work to be done. Next slide,
please. When you have transcripts, you can do fun stuff with them. For example, you
can see that important to people in this talk are "people". We are working on other
cool features that are yet to come. Stay tuned for that. Next slide, please. So to
keep track of all these tasks, we've been using a state-of-the-art high-performance
lock-free NoSQL columnar data store, a.k.a. a kanban board in the previous
years. And because we don't have any windows in the CCL building anymore, we
had to virtualize that. So we're using kanban software now. At this point, I
would like to thank all our hard-working angels for the work. And next slide
please. If you're feeling bored between congresses then you can work on some
transcripts. Just go to c3subtitles.de. If you're interested in our work, follow us
on Twitter. And there's also a link to the release subtitles here. So that's all.
Thank you. ysf: Thank you, td. And before we go into
the POC, where Drake is waiting, I'm sure everyone is asking why are those guys
saying "next slide"? So wait. In the end, we have the infrastructure review
of the infrastructure review meeting going on. So be patient. Now, Drake, are you
ready in Studio 1?
Drake: OK. Hello, I'm Drake from the Phone Operations Center, and
I like to present to you our numbers and maybe some
anecdotes at the end of our part. So please switch to the next slide. Let's get
into the numbers first. So first off, first off, you registered about 1950 ...
5195 sip extensions, which is about 500 more than you registered on the last
congress. Also, you did about 21 000 calls, a little bit less than on the last
congress. But, yeah, we are still quite proud of what you have used our system
with. And yeah, it ran quite stable. And as you may notice on the bottom, we also
had about 23 DECT antennas at the congress or at this event. So please switch to the
next slide. And this is our new feature, it's called the... next slide ..., it
is called the eventphone decentralized DECT infrastructure, which we especially
prepared for this event, the EPDDI. So we had about 23 RFPs online throughout
Germany with 68 DECT telephones of which is up to it. But it's not only the the
German part that we covered. We actually had one mobile station walking out through
Austria, through Passau, I think. So indeed we had an European Eventphone DECT
decentralized infrastructure. Next slide please. We also have some anecdotes, so
maybe some of you have noticed that we had a public phone, a working public phone in
the RC World where you could call other people on the SIP telephone system and
also other people started to play with our system. And I think about yesterday
someone started to introduce c3fire so you could actually control a flame thrower
through our telephone system. And I like to present here a video. Next slide
please. Maybe you can play it. I have quite a delay in waiting for the video to
play. So what you can see here is the c3fire system actually controlled by a
DECT telephone somewhere in Germany. So next slide please. We also provided you
with SSTV servers via the phone number 229, where you could receive some
pictures from event phone, like a postcard basically. So basically you could call the
number and receive a picture or some other pictures, some more pictures. And next
slide please. Yeah basically, that's all from the Eventphone and with that we say
thank you all for the nice and awesome event and yeah, bye from the first
certified assembly POC. Bye. ysf: Thank you, POC, and hello GSM Lynxes
is waiting for us.
lynxes: Yeah, hallo, I'm lynxes, I'm from
the GSM team. This year was quite different as you can imagine. However,
next slide please. So but we managed to get a small network running and also a
couple of SIM cards registering, so where are we now. So next slide please. As you can
see, we are just there in the red dot. There's not even a single line for our
five extensions but we managed 130 calls over five extensions. And next slide
please. So we got, so we got five extensions registered with four SIM cards
and three locations with mixed technologies also two users so far sadly.
And one network with more or less zero problems. And so let's take a look on the
coverage. So next slide please. So we are quite lucky that we managed to get an
international network running. So we got two base stations in Berlin. One in the
hackerspace AfRA, another one north of Berlin. And yeah one of our members is
currently in Mexico. And he's providing the remote chaos networks there. Yes, so
that's basically our network. So before we going to the next slide, we have what we
have done so far is, we are just two people instead of 10 to 20 and had some
fun with improving our network and preparing for the next congress. And next
slide please. And yeah, now I'm closing with the EDGE computing. We improved our
EDGE capabilities and yeah, I wish you a hopefully better year and yeah maybe see
you next year remote or in person. Have fun.
ysf: Thanks and I give a hand to Iindworm for doing the "slide DJ" all the time, and
he now switch to the Haecksen who are next and they bring an image and melzai is
waiting for us in Studio 3.
melzai: Hello, what's phones without people?
So I'm giving now an introduction over here. How many people we needed to
run the whole Haecksen assembly. We had around 20 organizing haecksen and we had
around 20 speakers in our events. And we had in total around 40 events, but I'm
pretty sure that I even don`t know all of these. As you realize, the world is pretty
large. So we needed around seven million pixels to display the whole Haecksen
world. And that needed around 400 commits at our github corner of the internet.
Around 130 people receive the fireplace badge in our case. And around 100 people
tested our swimming pool and received that badge. So great a year for non ???. Also
around 49 people showed some very deep dedication and checked on all memorials at
our Haecksen assembly. Congratulations for that. There were quite a many of these
ones. Our events are run on our BigBlueButton from the Congress and so we had
starting from day 0 no lags and we're able to host up to 133 people in one session.
And that was quite stable. We also introduced four new members around 13 new
Haecksen joinded just for the Congress. And we increased about to the size of 440
Haecksen overall. Also somewhat, we got new Twitter accounts supporting us, so we have
added over 200 more Twitter accounts. And so, you know, our messages are getting
heard. But besides the ritual, we also did some quite physical things. First of all,
we distributed over 50 physical goodie bags to the people with microcontrollers
and self-sewed masks in it, as you can see on the picture. And also sadly, we shopped
so many rC3 Haecksen-themed trunks that they are now out of stock. But they will
be back in January. Thank you. ysf: No, thank you. And I'm going to send
thanks to the Choaspatinnen… Chaospat*innen… who are waiting in Studio
One. Mike: Hi, all this is Mike from the
Chaospat*innen team. We've been welcoming new attendees and underrepresented
minorities to the chaos community for over eight years. We match up our mentees with
experienced chaos mentors. These mentors help their mentees navigate our world of
chaos events. DiVOC was our first remote event and it was a good proof of concept
for rc3. This year, we had 65 amazing mentees and mentors, two in-world
mentee/mentor matchup sessions, one great assembly event hosted by two of our new
mentees, and a wonderful world map assembly built with more than 1337
kilograms of multicolor pixels. Next slide, please. And here's a small part of
our assembly with our signature propeller hat tables. And thank you to the amazing
Chaospat*innen team: fragilant, jali, azriel and lilafish. And to our great
mentees and mentors. We're looking forward to meeting all of the new mentees at the
next chaos event.
lindworm: Yeah, I think that was my call.
So next up, we'll have the, let me see, the c3adventure! Are you ready?
Roang: Hello, my name is Roang Mewp: and I'm Mewp
Roang: and we will talk about the c3adventure, the 2D world, and what we did
to bring it all online. Next slide please. OK, so when we started out, we looked into
how we could bring a Congress-like adventure to the remote experience. And on
October we started with the development and we had some trouble in that we had
multiple upstream merges that gave us some problems. And also due to just Congress
being Congress, or remote experience being a remote experience, we needed to
introduce features a bit late or add features on the first day. So auth was
merged just 4:40 AM in the first day. And on the second day, we finally fixed the
instance jumps – you know, when you walk from one map to the next – we had some
problems there. But on the second day it all went up. And I hope you have all
enjoyed the badges that have finally been updated and brought into the world today.
What does that all mean? Since we started implementing, there have been 400 git
commits in our repository all-in-all, including the upstream merges. But I think
the more interesting stuff is what has been done since the whole thing went live.
We had 200 additional commits, fixing stuff and making the experience better for
you. Next slide. In order to bring this all online, we not only had to think about
the product itself, not only think about the world itself, but we also had to think
about the deployment. The first commit on the deployer, it's a background service
that brings the experience to you, has been done on 26th of November. We started
the first instance, the first clone of the work adventure through this deployer on
8th of December and a couple of days beforehand, I was getting a bit swamped. I
couldn't do all of the work anymore, because I had to coordinate both of the
projects. And so my colleague took over for me, and helped me out a lot. So I'll
give over to him to explain what he did. Mewp: Yeah. So imagine that on Day -5 I
get a message from a friend that, "Hey, help is needed!" So I say, "OK, let's do
it." And Roang tells me that, "OK, so we can spawn a instance and to scale it
somehow and do that." And I spawned the deployer and my music stops. I streamed
music from the internet, and I wondered why did it stop? And I have noticed that,
oh, there are a lot of logs now. Like, a lot. And I have finally Day -4 noticed
that the deployer was spawning copies of itself each few seconds in the log. So
that was the state back then. Since Day -4 until Day 1, we have basically written the
thing. And that's, well… Day 1 we were ready. Well, almost ready. I mean, we have
like 14 instances deployed. And I forgot to mention that, when we were about to
deploy 200 ones at once, it wouldn't work because all of the things would time out.
So we patched things quickly, and 13 o'clock we had our first deployment. This
worked, and everything was fine, and… wait… Why is everybody on one instance?
So, it turns out that we had a bug, not in the deployer, in the app that would move
you from the lobby to the lobby on a different map. So during the first day, we
have we've had a lot of issues of people not seeing each other because they were on
different instances of the lobby. So we are working hard, and… next slide, please,
so we can see that… we are working hard to reconfigure that to bring you together in
the assembly. I think we have succeeded. You can see the population graph on this
slide. The first day was our almost most popular one. And the next day it would
seem, that's OK, not as popular, but we have hit the peak of 1600 users that day.
What else about this? The most popular instance was lobby, of course. The second
most popular instance was hardware hacking area for a while. Then the third, I think.
Next slide please. We have counted, well, first of all, we've had in total about 205
assemblies. The number was increase day- by-day, because people, through the whole
congress, they were working on their maps. For a while, CERT had over a thousand maps
active in their assembly. Which led to the map server crashing. Some of you might
have noticed that. It stopped working quite a few times during Day 3. And they
have reduced the number of maps to 255. And that was fine. At the end of Day 3, I
have counted for about 628 maps, and this is less than the, if, than was available
in reality, because it was the middle of the night (as always), and it was it
wasn't trivial to count them. But in the maps I have found, we have found over two
million used tiles. So that's something you can really explore. I wish I could
have, but deploying this was also fun. Next slide, please. And what… Yeah?
Roang: Just a quick interject. I really want to thank everyone that has put work
into their maps and made this whole experience work. We, we provided the
infrastructure, but you provided the fun. And so I really want to thank everyone.
Mewp: Yeah, the more things happen on the infrastructure, the more fun we have. We
especially don't like to sleep. So we didn't. I basically exchanged with Roang
the way that I slept five hours and during the night and he slept five hours in the
day. And the rest of the time, we were up. The record, though, is incorrect. Roang is
now 30 hours up straight, because the budgets were too important to bring to you
to go to sleep. The thing you see on this graph is undeployed instances. We were
redeploying things constantly. Usually in the form of redeploying half of the
infrastructure at any given time. The way it was developed, you wouldn't have
noticed that. You wouldn't be kicked off your instances, but for a brief period of
time you wouldn't be able to enter any one. But… Next slide. I have been joking
for a few days at the Congress that they have been implementing a sort of
Kubernetes thing, because it's automatically deploy things, and manage
things, and so on. And I have noticed by Day 3 that I have achieved true
enlightenment and true automation, because we have decided to deploy everything at
once at some point. The reason was that we are being DDOSed, and we had to change
something to mitigate that. And so we did that, and everything was fine. But we
made a typo. We made a typo and the deployment failed. And one the deployment
failed, it deleted all the servers. So, yeah, 405 servers got deleted by what I'm
remembering was a single line. So it was brought out automatically, and that wasn't
a problem. It was all fine, but well, to err is human, to automate mistakes is
devops. Next slide? What's important is that these 405 servers were provided by
Hetzner. We couldn't have done that without their infrastructure, without
their cloud. The reason we got up so quickly after this was that the servers
were deleted, but they could have been reprovisioned almost instantly. So the
whole thing took like 10 minutes to get it back up. And, next slide. That's all.
Thank you all for testing our infrastructure, and see you next year.
ysf: Thank you, c3adventure! So this was clearly the first conference that didn't
clap for falling mate bottles! If that's not the thing, maybe we try next year? The
Lounge. And I know I have to ask for the next slide too. The rc3 Lounge artists.
And I was asked to read every country where someone is in, because everyone had
to make the Lounge what it was: an awesome experience. So there were: Berlin, Mexico City
Honduras, London, Zürich, Stockholm, Amsterdam, Rostock, Glasgow, Leipzig,
Santiago de Chile, Prag, Hamburg, Mallorca, Krakow, Tokyo, Philadelphia.
Frankfurt am Main, Köln, Moscow, Taipei Taiwan, Hannover, Shanghai, Seoul… Seoul,
I think, sorry. Vienna, Hong Kong, Karlsruhe and Guatamala. Thank you guys
for making the Lounge. So the next is the Hub and they should be waiting in
Studio Two. *audible echo*
XXX: …software is based on Django. And
it's intended to be used for the next event. The problem is it was a new
software. We had to do a lot of integrations, yeah, live during Day 0.
Well, OK. No. OK, yeah, hi. I'm presenting the Hub, which is a software we wrote for
this conference. Yeah. It's based on different components, all of them are
based on Django. It's intended to be used on future events as well. Our main problem
was it's a new software. We wrote it and, yeah, a lot of the integrations were only
possible on Day 0 or Day 1. And yeah. So even still today on Day 4, we did a lot of
updates, commits to the repository, and even that numbers on the screens are
already outdated again. But yeah, as you could possibly see, we have a lot of
commits all day, night, or all night long. Only a small digit, 6 AM. I am sorry for
that. Next slide, please. And yeah, because the numbers you're quite busy
using the platform, some of these numbers on the screen are already outdated again.
Out of the 360 assemblies which were registered, only 300 got accepted. Most of
them were, yeah, event or people wanting to do a workshop and trying to register an
assembly. Or, duplicates. So, please organize yourself. Events, currently we have over
940 in the system. You're still clicking events, nice. Thanks for that. The events
are coordinating with the studios, so we are integrating all of the events of all
the studios, and the individual ones, and the self organized sessions. All of them. A new
feature, the badges. Currently you have created 411. And, yeah, from these badges
redeemed, we have 9269 achievements and 19 000 stickers. Documentation, sadly, was
a 404, because yeah. We were really busy doing stuff. Some documentation has
already been written, but yeah. More documentation is, will become available
later. We will open source the whole thing of course, but right now we're still in
production and cleaning up things. And yeah. Finally, for some numbers. Total
requests per second were about 400. In the night, when the world was redeploying,
then we only had about 50 requests per second, but it maxed up to 700 requests
per second. And the authentication for the world, for the 2D adventure, it was about
220 requests per second. More or less stable due to some bugs and due to some
heavy usage. So, yeah, we appreciate that you used the platform, used the new Hub,
and hope to see you on the next event. Thanks.
ysf: Hello Hub. Thank you Hub. And the next is betalars waiting for us. He's from
the c3auti team, and he will tell us what he does and his team did this year.
betalars: Hi, I'm betalars from c3auti, and we've been really busy this year as
you can probably see by the numbers on my next slide. We have 37 confirmed Auti-Angles
and today we surpassed the 200 hours mark. We have 10 Orga Mumbles
leading up to the event and there are almost five million unique pixels in our
repository. I'm pretty convinced we've managed to create the smallest Fairydust
of rC3, provided by an actual space engineer. And the Tree of Solitude is not
the only thing we've managed to create, contribute to this wonderful experience.
On our next slide, you can see that we also contributed six panel sessions for
autistic creatures to discuss their experiences and five Play sessions for
them to socialize. We helped to contribute a talk, a podcast, and an external panel
to the big streams. And on our own panels, we've had up to 80 participants that need
to be split up to five breakout rooms so they could all have a meaningful
discussion. And all their ideas and thoughts were anonymized and stored on more than 1000
lines of markdown documentation that you can find on the Internet. But 1000 lines of
markdown wouldn't be enough for me to express the gratitude I have towards all
the amazing creatures that helped us make this experience happen and for all the
amazing teams that worked with us. I'm so happy to see you again soon, but now I
think I will need some solitude for myself.
ysf: Thank you betalars. So, lindworm, are you ready? The next one is the video, as
far as I know. It's from the C3 Inclusion Operation Center. I don't know the short
name; C3IOC? And it's counting down three two one go.
*video without audio*
So, video is like a very difficult thing to play in those days, because we only used to do
stuff live. Live means a lot of pixels and traffic is done from this here, from this
glass, to all the wires and cables and back to the glass of your screen. And this
is like magic to me, somehow. Although, I. am only. being. a robot. to talk.
synchronistically. with all the.... It's been around enough time, I think, to
switch back to Lindy with the video. I tell you what we are you going to…
*video without audio*
nwng: Hello everyone, I'm nwng from the new C3 Inclusion Operation Center. This
year, we've been working on accessibility guides that help the organizing teams and
assemblies improve the event for everyone, and especially people with disabilities.
We have also worked with other teams individually to figure out what can still
be improved in their specific range of functions - but there are still a lot to
catch up on! Additionally, we have published a completely free and accessible
CSS design template that features dark mode and an accessible font selection. And
it still looks good without Javascript. 100 Internet points for that! For you
visitors, we have been collecting your feedback through mail or twitter – and
won't stop after the Congress! If you stumbled across some barriers, please get
in touch via c3ioc.de or @c3inclusion on twitter to tell us about your findings!
Thanks a lot for having us. ysf: Thank you for the video. Finally,
technical's working! We should… does someone know computers? Maybe? Kritis is
one of them, and he is waiting in Studio One to tell us something about C3 Yellow
or c3gelb wie wir hier sagen.
Kritis: Yeah, welcome. I'm still looking
at this hard drive. Maybe you remember this from the very beginning? It has to be
disinfected really thoroughly, and I guess I can take it out by the end of the event.
And for… the next slide with the words, please. We did found roughly 0777 hands
wash options and 0x3FF waste disposal possibilities. We checked the correct date
on almost all of the 175 disinfectant options you had around here. And because
at a certain point of time, people from CERT were not reachable in the CERT room
because they were running around everywhere else in this great 2D world. We
had the chance to bypass and channel all the information because there were two
digital cats on a digital tree. And so we got the right help to the right option.
Next slide, please. We have a couple of options ongoing. A lot of work had been
done before. We had all the studios with all the corona things going on before, but
now we think we should really watch into an angel disinfectant swimming basin for
the next time, to have there the maximum option of cleanliness. And we will talk
with the BOC. If we can maybe achieve to use this Globuli maxi-cubes for the
Tschunk in the upcoming time. Apart from that, in order to get more Bachblüten and
everything else, we need someone who is able to help us with the Potenzieren
for homoeopathic substances. So if you feel welcome with that, please just drop
us a line to: info@c3gelb.de. Thank you very much and good luck.
ysf: Thank you Kritis. Finally happy to hear your voice. I only know you from
Twitter, where we treat our stuff together, or I yours and you, mine, don't.
Maybe you're going to change it… please? And, talking about messages. Chaos Post
was here too, and trilader, whom we already heard earlier, has more to say.
trilader: OK, welcome. It's me again. I've changed outfits a bit. I'm not here for
the Signal Angels anymore, but for Chaos Post. So, yeah. We had an online office
this year again, as we had with the DiVOCs before. And I've got some mail numbers for
you that should be on the screen right now. If it's not, if it's on the title
page, please switch to the first one where it lists a lot of numbers. We had 576
messages delivered total. This is numbers from around half to six. And 12 of them we
weren't able to deliver because, well, non-existent mailboxes or full mailboxes
mostly. We delivered mail to 43 TLDs, the most going to Germany, to .de domains,
followed by .com, .org, .net, and to Austria with .at; We had a couple of
motifs you could choose from, the most popular one was "Fairydust at Sunset", 95
people selected that. Next slide. About our service quality. We had a minimum
delay from the message coming in, us checking it, and it going out for about a
bit more than four seconds. The maximum delay was about seven hours. That was
overnight, when no agents were ready, or they were all asleep, or having… being
busy with, I don't know, the Lounge or something? And on average a message took
you, took us 33 minutes from you putting it into our mailbox to it getting out.
Some fun facts: We had issues delivering to T-Online at the first two days, but we
managed to get that fixed. A different mail provider refused our mail because it
contained the string c3world, the domain in the mail text. And apparently new
domains are scary, and you can't trust them or something. We had created a ticket
with them, they fixed it, and it was super fast, super nice service. Yeah. Also, some
people tried to sent digital postcards to Mastodon accounts because they looked like
email addresses or something. Another thing that's not on a slide is we had
another new feature this time. That was our named recipients. So you could, for
example, send mail to CERT without knowing their address. And they also have a really
nice postcard wall, where you can see all the postcards you sent them. The link for
that is on our Twitter. Thank you. ysf: Thank you Chaos Post. lindworm, are
you there? lindworm: Ja, ja. Ich bin da, Ich bin da.
Hallo, you're hearing me? ysf: I hear you.
lindworm: So I have to switch some more. It's kind of stressy for me, really.
ysf: You're doing an awesome job. Thank you for doing it. So just out of
curiosity, and did you have a problem accepting any cookies or so?
lindworm: No, not really. ysf: I heard somewhere. That some really
smart people had problems using the site because of cookies.
lindworm: Oh, no, that was not my problem. I only couldn't use the site because of
overcrowding. That was often one of my my little problems. And please, I hope you
don't see what I'm doing right now in the background with starting our pets and so
on. And what I wanted to say to all of you, this was the first Congress where we
have so many women and so many non-cis people running that show and being up
front the camera and making everything up. I would really thank you all. Thank you,
that you made that possible. And thank you that we get more and more diverse, year by
year. ysf: I can only second that. And now we
are switching to C3 Infrastructure. lindworm: Yeah, we need to.
ysf: I'm sure a lot of questions will be answered by them.
lindworm: And I try to make up the slides for that, but I do not find them right
now. patrick: Look mom, I'm on TV.
thies: Yeah. Welcome to the infrastructure review of the Team Infrastructure. I'm not
quite sure if we have the newest revision of the slides, cause my version of the
stream isn't loading right now. So maybe lindworm, is it possible to press
control-R? And you're seeing a burning computer, then we have the actual slides.
Patrick: Let's just Powerpoint Karaoke without the background music.
thies: Yeah, and without the PowerPoint presentation in realtime. Now I'm seeing
me. Let's wait a few seconds until we see a slide.
Patrick: We want to wait the entire stream delay.
thies: It's just about 30 to one minute. Patrick: Well done.
thies: Yeah, I'm thies and I'm waiting. And this is Patrick, and he's waiting too.
Yeah, but that's in the middle of the slides. Can we go… OK. Yeah. I'm now
seeing something in the middle of the slides, but it seems fine. OK, yeah. We
are the team C3 Infra. rC3 Infra. We are creating the infrastructure. Next slide.
We had about nine terabytes of RAM and 1,700 CPU cores. The whole event there's
only one dead SSD that died because everything's broken. We had five dead RAID
controllers, and didn't bother to replace the RAID controllers, just replaced them
with new servers. And 100 percent uptime. Next slide. We looked about 42 hours on
starting screens of enterprise servers. 20 minutes max is what HP delivered. And we
are now certified enterprise observers. We had only 27%-ish of visitors using IPv6.
So that's even less than Google publishes. And even though we had almost full IPv6
coverage – except some really, really shady out-of-band management networks – we're
still not at the IPv6 coverage that we are hoping for. I'm not quite sure if that's
the right slides. But I'm not quite sure where we are in the text. Yeah, Patrick.
Patrick: Yeah, so before the Congress there was one prediction: there's no way
it cannot be not DNS. And while it was DNS at least once, so we checked that box. And
let's go over to the next topic, OS. We provisioned about 300 nodes, and it was an
Ansible-powered madness. So, yeah, there was full disk encryption on all nodes. No
IP logged in the access logs, we took extra care of that. And we configured
minimal logging wherever possible, so the case of some problems we only had WARNINGs
available. And there are no INFO logs, no DEBUG logs; just the minimal logging
configuration. And with some software, we had to pipe logs to /dev/null because the
software just wouldn't stop logging IP's, and we didn't want that. So no personal
data in logs, so no GDPR headache, and your data is safe with us. The Ansible
madness I've talked about was a magical deployment that deep bootstrapped into the
live system and assimilated into the rC3 infrastructure while it's still running.
So if you didn't boot your machine then what? They're just running. When a OS
deployment was broken, it was almost always due to a network or routing. At
least the OS team claims that, and this claim is disputed by the network team of
course. One time, the deployment broke because of a trigger happy infra angel.
But let's not talk about that. Of course, at this point, we want to announce our
great cooperation with our gold sponsor ddos24.net, who provided an excellent
service of handcrafted request to our infrastructure. That was a great demand or
great public demand, with a million requests per second for a while. But even
during the highest or peak demand, we were able to serve most of these services. We
provide the infrastructure to the VOC, and they quickly made use of the provided
infrastructure deployed there. Overall, an amazing time to market. We had six
locations, and those six locations where some wildly different, special snowflakes
overall. So we had Düsseldorf, 816 CPU cores there, two terabytes of RAM, and we
had 10 gigabits per second interconnect. There was also a 1 terabit per second
Infiniband available, but sadly, we couldn't use that. It would have been
nice. The machines that had a weird and ancient IPMI, which made it hard to deploy
there. And the admin on location never deployed bare metal hardware to a
datacenter, so there were also some learning experience there. Fun fact about
Düsseldorf, this was the data center with the maximum heat. One server, seven units,
over 9000 watts of power. 11.6 to be exact. Which is why they had some to take
some creative heat management solutions. Next was Frankfurt, there we had 620
gigabits of total uplink capacity, and we actually only used 22 gigabit during peak
demand. Again, by our premium sponsor: ddos24.net. There was zero network
congestion and 1.5 gigabits per second were IPv6. So there was no real traffic
challenge. For the network engineers of you, it was a full Layer 3 architecture
with MPLS between the WAN routers. And there was a night shift on the 26the and
27th for more servers, because some shipments didn't arrive yet. The fun fact
about this datacenter was the maximum bandwidth. Some servers there had 50
gigabit uplink on the server configured. It was the data center with the maximum
manual intervention. Of course, we had the most infrastructure there and it wasn't
oversubscribed at any point. We had some hardware in Stuttgart, which was basically
the easiest deployment. There were also some night shifts, but the thanks to
neuner and team this was a really easy deployment. It was also the most silent
DC, so no incident from Day -5 until now. So if you're currently watching from
Stuttgart now, you can create some issues because now we said it. Wolfsberg was the
smallest DC. We only had three servers and we managed to kill one hardware RAID
controller, so we only could use two servers there. So, yeah. And then Hamburg
was the data center with the minimum uptime. We never could deploy to this data
center because there was a broken netboot and we couldn't provision anything there.
And of course, the sixth data center was the Hetzler Cloud, where we deployed it on
all locations. Deployment fun facts: we received a covid warning from the data
center. Luckily, it didn't affect us. It was at another location. But thanks for
the heads-up and the warning. The team leader of a sponsor needed to install
Proxmox in a DC with no knowledge, without any clue what they were doing. We
installed Proxmox in the Hamburg DC, and no server actually wanted to talk to us,
so we had to give up on that. And there had to be a lorry relocated before we
could deploy other servers. So that's that was standing in the way there. Now, let's
get to Jitsi. Our peak count were 1,105 users at the same time, on the same
cluster. I don't know if it was at the same time as the peak user count, but the
peak conference count was 204 conferences.
I hope we can still beat that today, but this is data from
yesterday. The peak conference size was 94 participants in a single conference. And
let me give condolences to your computer, because that must have been hard on it.
Our peak outgoing video traffic on the Jitsi video bridges was 1.3 gigabits per
second. And we had about three quarters of the participants were streaming video and
one quarter of them had video disabled. Interesting ratio. Our Jitsi deployment
was completely automated with Ansible, so it was zero to Jitsi in 15 minutes. We
broke up the Jitsi cluster into four shards to have better scalability and
resilience. So if one shard went down, it would only affect part of the conferences
and not all of them. Because there are some infrastructure components that you
can't really scale or cluster, so we went with with the sharding route. Our Jitsi
video bridges were at about 42% peak usage – excluding our smallest video bridge,
which was only eight cores and eight gigabytes, which we added in the beginning
to test some stuff out, and it remained in there. And yes, we overprovisioned a bit.
There will also be a blog post on our Jitsi Meet deployment coming in the
future. And for the next time we, for the upcoming days, we will enable 4K streaming
on there. So why not use that? And we want to say thanks to the FFMEET Projekt, who
contacted us after our initial load test and gave us some tips to handle load
effectively and so on. We also tried making DECT call-out working. Spent 48
hours trying to get it to work, but there were some troubles there. So sadly, no
adding DECT participants to your Jitsi conferences for now. jitsi.rc3.world will
be running over New Year. So you can use that to get together with your friends and
so on over the New Year. Stay separate, don't visit each other please. Don't
contribute to covid-19 spread. You've got the alternative there. Now let's go over
to monitoring. thies. thies: Yeah, thanks. First of all, it's
really funny how you edit this page, but reveal.js doesn't work that way until
lindworm reloads the page, which hopefully doesn't do right now. Everything's fine,
so you can leave it to be. Yeah, monitoring. We had to Prometheus and
Alertmanager set up completely driven out of our solemnly one and only source of
truth: our Netbox. We received about 34 858 critical alerts. It's – looking at
my mobile phone – it's definitely more right now. And about 13,070 warnings. Also
definitely more right now. And we tended about 100 of them. The rest was kind of
useless. Next slide, please. As it's important to have an abuse hotline and an
abuse contact, we received two network abuse messages, both from Hetzner – one of
our providers – letting us know that someone doesn't like our infrastructure as
much as we do. Props to ddos24.net. And we got one call it our abuse hotline, and it
was one person who wanted to buy a ticket from us – Sadly, we were out of tickets.
Next slide, please. Some other stuff. We got a premium Ansible deployment brought
to you by turing-complete YAML. That sounds scary. And we had about 130k DNS updates
thanks to the World team. At this point they're really stressing our DNS API with
the re-deployments. And also our DNS, Prometheus, and Grafana are deployed on
and by NixOS thanks to flüpke and head over to flüpkes interweb thingy. He wrote
some blog posts about how to deploy stuff with his NixOS. And the next slide,
please. And the last slide from the team is the list of our sponsors. Huge thanks
to all of them. It won't be possible to create such a huge event and such loads of
infrastructure without them. And that's everything we have.
ysf: Amazing. Thank you for all you've done. Truly incredible, and showing
everything to the public. So I promised that there will be a kind of behind the
scenes look of this infrastructure talk or review. And I really have nothing to do
with it. Everything was done by completely different people. I'm only a Herald,
somehow lost and tumbled into this stream. And so I'm just going to say switch to
wherever. Show us the magic. Karlsruhe: Three hours ago, I got the
call… Hello and welcome from the last part of the infrastructure review and greetings
from Karlsruhe. So three hours ago, I got a call from lindworm and he asked me, how
is it with this last talk we have? It may be a bit complicated. And he told me, OK,
we have a speaker. I'm the Herald. Oh, that's always so. And then we realized,
yeah, we don't have only one speaker, we have 24. And so that's why we called
ChaosWest and built up an infrastructure which dampfkatze will explain you now in a
short minute. I think so. dampfkatze: Thank you. Yes. Oh, I lost the
sticker. OK, after we called ChaosWest, we came up with this monstrosity of the video
cluster. And we start here. The teams streamed via OBS.Ninja onto three
ChoasWest studios. They were brought together via RTMP on our Mix1 local
studio, and then we pumped that into Mix2, which pumped it further to the VOC. The
slides were brought in via another OBS.Ninja directly onto Mix2. They came
from lindworm. Also, the closing you will see shortly hopefully will also come from
there. And ysf and lindworm were directly connected via OBS.Ninja onto our Mix1
computer. And Mix2 also has the studio camera you're watching right now. And for
the background communication, we had a Mumble connected with our audio matrix.
And lindworm, ysf, and the teams, and we in the studio locally could all talk
together. And now back to the closing with… No, to the Herald News Show, I
think. lindworm will introduce it to you. lindworm is live.
lindworm: Is ysf still there? Or do you come with me? So it will take a second or
billions of years. So thank you very much for this review. It was as chaotic as the
Congress.
*postroll music*
Subtitles created by c3subtitles.de in the year 2021. Join, and help us!