*36c3 preroll music*
Herald: So, hey, we're finally ready to start, we have Volker Krause here with a
privacy by design travel assistant and it's going to be about building Open
Source travel assistants, I think, and this talk will be in English. And if you
want translations, wenn ihr eine deutsche Übersetzung haben wollt, haben wir hier
hinten auch ganz tolle Übersetzer in unserer Kabine, da könnt ihr auf
c3lingo.org mal reinhören, wie die alles live mitreden. Genau. Now. Let's have
a warm welcome for Volker here and have fun with his talk.
*Applause*
Volker Krause: Thank you. OK, so what is this about? You probably know those
features in, most prominently Google Mail, but I think TripIt was the one that
pioneered this. So GMail reads your email and then detects any kind of booking
information in there, like your boarding passes, your train tickets, your hotel
bookings and so on. And it can integrate that into your calendar and can present
you a unified itinerary for your entire trip and monitor that for changes. And
all of that doesn't cost you anything. Maybe apart from a bit of your privacy.
Well, not too bad, you might think. But if you look at what kind of data is actually
involved in just your travel. Right. The obvious things that come to
mind, your name, your birthday, your credit card number, your passport number,
that kind of information. Right. But that isn't even the worst part on this,
because those operators don't just get to see your specific data for one trip,
right? They get to see every… everyone's trip. And now if you combine that
information, that actually uncovers a lot of information about... relations
between people, your interests, who you work for, where you live and all of
that. Right. So pretty much everyone here traveled to Leipzig for the last four days
in the year. If that happens for two of us, once, right, that might be
coincidence. If that happens two or three years in a row, that is some kind of
information. But yeah, what to do about that, right? The easy solution
is, just not use those services. It's like first world luxury stuff anyway. That
works until you end up in a foreign country where you don't speak any of the
local languages and then get introduced to their counterpart of Schienenersatzverkehr
or Tarifzonenrandgebiet. And at that point, you might be interested in actually
understanding what's happening on your your trip in in some form that you
actually understand and that you are familiar with, ideally without installing
15 different vendor applications for whatever you actually might be
traveling, right? So we need something better. And that obviously leads us to,
let's do it ourselves. Then, we can at least design this for privacy right from
the start. Build it on top of Free Software and Open Data. Well, of course,
we need to... at least it's not entirely obvious that this will actually work,
right? The Google and Apple, they have a total different amount of resources
available for this. So, can we actually build this ourselves? So let's have a look
at what those services actually need to function. And it turns out it's primarily
about data, not so much about code. There are some difficult parts
in terms of code involved as well, like the image processing and a PDF to detect a
barcode in your boarding pass. But all of that exists as ready-made building blocks.
So you basically just need to put this nicely together. So let's look at the
data. That's the more interesting part. And in general, that breaks down to
three different categories. The first one is what I call personal data here. So
that's basically booking information, documents or tickets, boarding passes,
specific for you. So there at least you don't have a problem with access because
that is sent to you and you need to have access to that. But it comes in all kinds
of forms and shapes. So there are the challenges to actually extract that . The
second kind of data is what I would call static data. So, for example, the location
of an airport. Now, you could argue that that could change and there is rumors
that some people apparently managed to build new airports. I live in Berlin, so I
don't believe this. Jokes aside, so, "static" refers to within, static within
the release cycle of the software. So several weeks or a few months. So this is
stuff that we can ship as offline databases. And offline, of course, helps
us with privacy because then you're not observable from the outside. And the third
category is dynamic data. So stuff that is very, very short lived, such as delay
information. There is no way we can do that offline. If we want that kind of
information, we will always need some kind of online querying. Then let's look
through those three categories in a bit more detail. For the booking data, google
was faced with the same problem, so they used their monopoly and defined a standard
in which operators should ideally have machine readable annotations on their
booking information. And that's awesome, because we can just use the same, the same
system. That's what nowadays became schema.org, which I think Lucas mentioned
in the morning as well. At least in the US and Europe, you'll find
that in about 30 to 50% of booking emails you get from hotels, airlines or event
brokers. So that's a good start. But then there's the rest, which is basically
unstructured data, random PDF files or HTML emails we have to work with. There's
Apple wallet boarding passes. They are somewhat semi structured and most
widespread for flight tickets. Well, that's somewhat usable. And barcodes, so
that's what you, again, see on boarding passes or train tickets. I could
probably fill an entire talk just with the various details on the different
barcode systems, the one for boarding passes, I think, Karsten Nohl had to talk
at Congress a few years back, where he showed how they work and what you can do
with them. Instagram #boardingpass is a very nice source of test data. The one
that you find on, on German railway tickets is also pretty much researched
already. The ones we actually had to break ourselves were the one for Italy. I
think to my knowledge, we are the first ones to publish the content of those
binary barcodes. And we are currently working on the VDV Kernapplikation
E-Ticket, which is the standard for German local transportation tickets. That
actually has some crypto that you need to get around to actually see the content. So
there is, if you're interested in that kind of stuff, there is quite some
interesting detail to be found in this. But let's continue with the
static data. There, of course, we have Wikidata. That has almost everything we
need. And we are making heavy use of that. And that's also why I'm here today on the
Wikimedia stage. One thing that Wikidata doesn't do perfectly is timezone
information. That's why we're using the open street map data for this. There's in
Wikidata, three different time zone… or ways of specifying the time zone. UTC
offsets, some kind, of coarse, human readable naming like Central European
Summer Time, and then the actual IANA time zone specifications like Europe/Berlin.
And that's the one we actually need because they contain daylight saving time
transitions. And that is actually crucial for travel assistance, because you
can have a flight from, say, the US to Europe, at the night where there is
daylight saving time transition on one end. And if we get that wrong, right, we
are off by one hour. And that could mean you miss your flight. So that we
need to get absolutely right. And Wikidata there mixes the three timezone
variations. So that's why we fall back to OpenStreetMap there. Another area
that still needs work is vendor specific station identifiers. So there's a number
of train companies that have their own numeric identifier, or alphanumeric
identifiers, which you find, for example, in barcodes of tickets. So that's our
way to actually find out where people are traveling. So that's something we are
trying to feed into Wikidata as we get our hands on those identifiers. For airports,
that's easy because they are internationally standardized. For train
stations, that's a bit more messy. And finally, the dynamic data. That's again,
an area where we benefit from Google using their monopoly. They wanted to have local
public transportation information in Google Maps. So they defined the GTFS
format, which is a way for local transport operators to send their schedules to
Google. But most of the time, that is done in a way that they basically publish this
as Open Data. And that way, all of us get access to it. And then there's Navitia,
which is a Free Software implementation of like a routing and journey query service
that consumes all of those Open Data schedule information. And that then in
turn, we can use again to, yeah, find our departure schedules, delays and that
kind of live information. Apple Wallet also has some kind of live updating
polling mechanism. But that is somewhat dangerous because it leaks personal
identifiable information. So the, basically, a unique identifier for your
pass is sent out with the API request to to pull an update. So that is basically
just a last resort mechanism if you have nothing else. And then there's a bunch of
vendor specific, more or less proprietary APIs that we could use. They are
unfortunately not often compatible with Free Software and Open Source, because,
they might require API keys that you're not allowed to share, or they have terms
and conditions that are simply incompatible with what we are trying to
do. So for some, this works, but there's still some room for improvement in those
vendors' understanding the value of proper Open Data access. OK, so that's the
theory, let's have a look at what we have actually built for this. So there's two,
ya, backend components, so to say there is the extraction library that implements the
schema.org data model for flights, for trains, for hotels, for restaurants
and for events. It can do the structured data extraction. That might sound easy at
first, but it turns out that for some of the operators, doing proper JSON array
encoding is somewhat hard. So, I mean, you need to do a... need to have a comma in
between two objects and brackets around it. Some of them struggle with that. So we
have to have lots of workarounds in, in parsing the data we receive. Then we have
an unstructured extraction system that's basically small scripts per provider
or per operator that then, yeah, use regular expressions or XPATH queries
depending on the input and turn that into our data model. We currently, I think,
have 50, slightly more than 50 of those. I know that Apple has about 600, so that is
still one order of magnitude more. But it's not impossible. Right. So I think we,
we have the means there with Free Software to come to a similar result than
people that have an Apple or Google scale budget for this. The service coverage is
actually quite different. So, for Apple, I've seen their custom extractor.
So they have a lot of like US car rental services. We have somewhat more important
stuff like CCC tickets. So the Congress ticket is actually recognized and I
managed to get in with the app. What the expection engine also does is it
augments whatever we find in the input documents by information we have on
Wikidata. So we usually have time zones, countries, geo coordinates, all that
useful stuff for then offering assistance features on top. And input formats is
basically everything I mentioned. The usual stuff you're getting in an email
from a transport operator or any kind of booking document. The second piece on
like, on backend components is the public transportation library. That's basically
our client API for Navitia mainly, but also for some of the proprietary
widespread backends like HAFAS. That's the stuff Deutsche Bahn is using. And it can
aggregate the results from multiple backends. And if you're using Open Data in
the backend - *interference noise* - it propagates the attribution information
correctly. So. And just a few days ago, it also gained support for querying train and
platform layouts or "Wagenstandsanzeiger" in German so we can have all of that in
the app. And now of course there's the KDE Itinerary app itself. So it has, oh… it's
very hard to read here. It's basically a timeline with the various booking
information you have grouped together by trip. It can insert the live weather
information. Again, that's online access, so it's optional, but yeah, it's kind of
useful. And this is… you probably can't read that. But that's my train to Leipzig
this morning and that's actually the Congress entry ticket. And the box at the
top is the collapsible group for my trip to Leipzig for Congress. And it can
show the actual tickets and barcodes, including Apple Wallet passes. So, if you
sometimes have a, like a manual inspection at an airport where they don't scan your
boarding pass, but look at it, apparently that looks reasonable enough that you can
board an aircraft with it. At least, I wasn't arrested so far. And then we have
one of my favorite features, also powered by Wikidata. It's the power plug
incompatibility warning. *interference noise* - So, I mean, if you're traveling
to, say, the US, or UK, you're probably aware that they have like incompatible
power plugs. But there are some countries where this isn't – at least to
me, isn't that obvious, like Switzerland or Italy, where only half of my power
plugs work. So this is the Italy example. It tells me that my Schuko plugs won't
work, only my Europlugs and. *interference noise* - And the right one is, I
think for the U.K., where nothing is compatible. If you occasionally forget
your power plug convertor while traveling, that is super useful. And then, of course,
we have the integration with real time data. So we can show the delay
information and platform changes. The part in the middle is the alternative
connection selection for trains. So if you have a, like a train ticket that
isn't bound to a specific connection, right, then the app lets you pick the one
you actually want to take. Or if you're missing a connection, you need to move to
a different train, you can do that right in the app as well. The screenshot on the
right hand side is the, like your overall travel statistics. So if you're interested
in, like, seeing the carbon impact off of all your trips and the year over year
changes, right, the app shows that to you. And I wasn't really successful, but that's
largely because the old data is incomplete. So if you're interested in
that, right, since we have all the data, that can help you see if you're
actually on the right track there. And then to get data into that, we also have a
plugin for email clients. This one is for for KMail. So it basically then runs the
extraction on the email you're currently looking at and it shows you a
summary of what's in there. In this case, my train to Leipzig this morning,
including the option to add that to the calendar or send it to the app on the
phone. We also have the browser extension. So this is the website of the yearly KDE
conference, which has the schema.org annotations on it. And the browser
extension recognizes that. And again, offers me to to add that either to my
calendar or to the itinerary app. And that also works on many restaurant websites or
event websites. They have those annotations on the website for the Google
search. So again, we benefit a bit from the, Google *incomprehensible*. OK, then
we get to the more experimental stuff that basically just was finished in the last
couple of days, that we haven't shown anywhere else publicly yet. The first one
is, and that's a bit better to read, at least, if you saw the timeline earlier,
right, it had my train booking to Leipzig and then the Congress ticket. But that
still leaves two gaps, right. I need to get from home to the station in Berlin,
and I need to get from the station in Leipzig to Congress. And what we have now
is a way for the app to automatically recognize those gaps and fill them with
suggestions on what kind of local transport you could take. So here the one
for Leipzig to Congress is expanded and shows the tram. That still needs
some work to do live tracking so that it accounts for delays and changes your
alarm clock in the morning if there's delays on that trip. But we have
all the building blocks to make the whole thing much more smart in this
area now. And that, I think was literally done yesterday. So that's why the graphics
still are very basic. That's the train layout, coach layout display for
your trip. So that you know where your reserved seat on the train can actually be
found. Then, I only showed the KMail plugin so far. We also have a work-in-
progress Thunderbird integration, which is probably the much more widespread email
client. Featurewise, more or less the same I showed for KMail, so it scans the email
and displays your summary and offers you to put that into the app or, possibly
later on also into the calendar. This one is even more experimental. I can
only show you a screenshot of Web Inspector proving that it managed to
extract something. That's the integration with Nextcloud. I hope we'll have an
actual working prototype for this in January then. Those two things are, of
course, important for you to even get to the data, the booking data, that then
the app or other tools you built on top can consume. OK, so where to get this
from? There's the wiki link up there. The app is currently not yet in the Play Store
or in the F-Droid master repository. We have an F-Droid nightly build repository.
I hope that within the next month we'll get actual official releases in the easier
to reach stores than what we have right now. If you are interested in
helping with that, there's some stuff in Wikidata where improvement on the data
directly benefits this work, and that is specifically around train stations. I
think in Germany, last time I checked, we still had a few hundred train stations
that didn't have geo coordinates or even a human readable label. So that's something
to look at. Vendor-specific or even the more or less standard train station
identifiers is something to look at. So UIC or IBNR codes for train stations,
that helps a lot. Yeah. And then, we kind of need test data for the extractions. So,
forget everything I said about privacy. If you have any kind of booking documents or
emails you want to donate to support this and get the providers you're using
supported in in the extraction engine, talk to me. That would be extremely
useful. Yeah, that's it. Thank you.
*Applause*
Herald: Hello, hello? Yeah. That's a very impressive project, I think, do we have
questions then I'll hand you my microphone. Yes.
Q: Would it be possible to extract platform lift data for train stations?
A: Sorry? Platform…. Q: Platform lift data.
A: Oh, I think Deutsche Bahn has an Open Data API for the live status of lifts.
That would, of course, in theory be possible. What we are trying to do is to
be generic enough so that this might not be applicable in just one country,
although it is very European focused because most of the team is there. But
lifts is something that is easy enough to generalize in a data model, right? Its
location on the platform, and, are they working or not? So, yeah, that that would
be a nice addition. That goes into the entire direction of, ya, indoor navigation
or navigation around larger train stations and airports. So that's probably something
where we could use a better overall display with the OpenStreetMap data and
then augment that with, like the, where exactly is your train stopping and in
which coach is your seat, and then have the lift data so we can basically guide
you to the right place in a better way. Yeah.
Herald: Any more questions? Yes. Q: It's the mobile app written in Qt as
well? A: Yes, most of this is C++ code, because
that's what we use at KDE. The mobile client as well. There's a bit of Java for
platform integration with android. I don't think anyone has ever tried to build it on
iOS, but of course it works on Linux based mobile platforms as well, thanks to Qt and
C++, yeah. Q: So you mostly talked about the mobile
app so far, which is understandable, but as it's a QML application does it also run
on desktop? And, a second question, how do, how do all the plugins and the
different instances of the app share their data?
A: So, yes, the app runs on desktop. I was trying to see if I can actually start it
here. I'm not sure on which screen it will end up. That's where we do most of the
development. Let me see if I can move it over. Oh, thank you. And I need to find my
mouse cursor on the two screens. Uh. I think I need to end the presentation
first, but, yeah, short answer, of course. There we go. And let me switch to… to…
yeah, so that's it, running on desktop. It has a mobile UI there. That
could, of course, be extended to be more useful on the desktop as well. And in
terms of storage, that is currently internal to the app, there is no second
process accessing the actual data storage. That would just unnecessarily complicate
it for now. But if there is a use for that, yeah, we'll need to see.
Q: Yeah, but, but, but there was an option, in the e-mail plugin, for example,
to send it to the app. Can I then only send it to my local app and not to the
mobile app? A: Oh, the central app, that's using
KDE Connect. That's an integration software that allows you to remote control
your phone from the desktop. So that's basically bundling up all the information
and sends it to the app on the phone. And… or it can import it locally, so.
Herald: OK, do we have other questions? No, we don't have time? So then, thank you
very much, Volker, maybe you can tell people where they can find you if they
have anything more they want to talk about. But….
A: Yeah, I mean, there's my email address and otherwise I'll be around all
day, all four days. Herald: Around where?
Volker Krause: Probably somewhere. So it just is a bit tricky.
Herald: …catch him before he runs away, then! All right. So give a round of
applause again and thank you, Volker!
*Applause*
*postroll music*
Subtitles created by c3subtitles.de in the year 2021. Join, and help us!