Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1143 Thanks!
Hello, everyone, hello to the Stream and
the worldwide Internet, welcome
to Wikipedia, Asima.
And now I want to give you all
I want to ask you to give a warm
round of applause for me who will explain
to you how artificial intelligence and
Wikipedia can come together.
Hello, everyone, my name is
let's start
with that.
So my name is Ameristar with
any I've been a developer at
Wikimedia and Wikipedia for 12 more
than 12 years now.
And I've been in that staff of Wikimedia
Deutschland for
two years almost.
And then I live in Berlin and
I will explain about my work and
my place a little bit more.
So I work at a place called Wikimedia
Deutschland, which is the German chapter
of Wikimedia Foundation.
Wikimedia Foundation is the place that
runs Wikipedia and all of its sister
projects, which includes Wikimedia
Commons and the KeyData and all sorts
of things.
The total number of engineers
who run Wikipedia is 300,
which is incredibly crazy,
low amount compared to other companies
and because it's a nonprofit.
And so it's
a really it's an honor, more of
an honor than a job to be in this among
300 engineers
to do this type of work.
So I'm in a
team called the Wikimedia Scoring
Platform Team, which is sort
of a team
that seems like Wikimedia Foundation
among there's another
employee of the committee, Dutch Lander,
and there's two other
media foundation who lives in the United
States. So it's basically lots of
different times and coordination
and our responsibilities
to build services that are
immersion learning and an idea
that empowers and helps users.
I get to I get to that
in more details later.
And
so what we build is
something called Orris.
And
I want to talk about why or why or what
is Auris and what is needed.
And it is of with something called
eddied quality.
So you know about this problem of
Wikipedia, that's there's like a lot
of Handelsman finding this amount of
vandal's among all sorts
of ideas is really hard.
So there were there was some ehi
that it's built to help
it and we built this and then
we saw like maybe we should just make
it a platform for EHI in general and
we move forward. So the first project
was edited quality and then it moved
forward to other things.
So let's talk about why it's
needed. So this is planet Earth.
And by the way, this data is from
March of 2012 because I was too lazy to
update it. And then
so it's seven billion people and
a number of monthly Wikipedia readers
is 500 million.
So basically,
if one person in ten person uses it
at least once a month
and all of these people, 500
million people, only 10000
of only 100000 of them,
edit Wikipedia, which means
out of these five hundred people
for each 10000 viewers
and readers, just one editor,
how many of you are consider yourself
editor of Wikipedia
and that explains it.
Uh.
So what how Wikipedia
is built is Wikipedia completely all of
its content is volunteer work, and
if you get some money basically to run
the servers and pay for these 300
engineers and all sorts of other
things.
So all of this content
is really huge.
It's five million articles and
it's just a number, right?
Five million articles.
It's really hard to grasp it.
So I start with an example.
There's an article in Wikipedia called
Lists of Lists of Lists.
And if you click on the first one,
there's a list of ancient kings
and then you click on the next one.
It's there's like other lists,
which is lists of ancient kings, and
there's a list of pharaohs.
And if you click on that and then you get
to the list of errors and then
if you click on the first one, you get
the first real article
and not a list.
It's like three orders, three
layers of leasing until you get to the
real article and
how it works.
Wikipedia is powered by something called
Mediaweek.
How many of you know about media wiki?
That's really good. That's good.
So the media, the key of it is really
ugly and it's
really old piece of software
that has been written back.
And it's basically
empowers anything that
it's a sort of revision content
management.
So and it's really good to do it in scale
and end in a mentality
of this thing is like you first
make an edit, it goes live
and then it gets reviews.
And that's that's
the whole mentality and design of this
software. And also it has a
saying that you have assured
authorship. So it's really hard to say
I own this article.
There's no way to say that.
And there's like a policies against that
that you cannot say. I own this article
even if you started or you write most
of it. And so how quality
control in Wikipedia works, it's
a really complex situation because there
are several lines of defense in
Wikipedia, how to prevent vandalism.
And so there's one fully automated
but called Tanji, which is
built on a neural network and
it's ChiX edits and just review them
really quickly and revert them
to things are really
bad. And there are that
those like tools called Huggler
or some other things that people
have and they're like laptops.
The way that they actually contribute to
Wikipedia is not that they don't they
don't write articles or improve, they
don't improve articles.
They just in their work
and they're like bored and they just
check the recent changes and check, oh,
these are expensive. And just this is
some way to contribute to Wikipedia.
And it is,
for example, in case of Hugo, which is a
software and it basically
shows you the one that's most likely to
be vandalism first.
And then you can move forward to the ones
that are least likely to be vandalism.
And there are admins.
So I think there's 3000 admins in
English, Wikipedia and another because
it's smaller and it's completely
community elected.
It's completely volunteer and they
can ban people or protect pages and
several levels.
So here comes the machine classifier and
the machine classifier or I.
It's like machine learning is a part
of A.I. And
if you want to describe what machine
learning is, is basically in
really simple terms, is like in
ordinary cases, you have some sort of
algorithms that you say
it is edit is vandalism or not.
So you say it is edited, contains this
bad word is vandalism and there's
no other way.
But what we do in here is actually
say look at we
have this at this hour, we know it.
That's our vandalism and we know this.
It is that are not vandalism.
So just go and figure it out at
what is vandalism and then you can use
this pattern. And for future cases
or like in another case, it's really
simple. It's like you show ten pictures
of orange and then you showed them
pictures of Apple and then you say,
just go figure out how like an apple
look like.
And
so what we do is that we basically
try to describe
and edit in features
or turn them to like a bunch of numbers
or in, mathematically speaking, a vector.
So we say if the edit is made
by a user that's not registered because
there tend to be more likely to be
vandalism, how many characters are
spinet? How many characters have been
removed? Because, for example, if a
person removes like
lots of page
or how many repeated character has been
added, so, for example, there are lots of
vandalism that's actually they just
add a lot of discrimination marks
everywhere or
longest token or a number of bad words.
Edit finding bad words is actually really
funny. I use it in l.p system
to build this and it
goes. Through history of Wikipedia and
based on that, he just looks
at this as are most likely to be in
edits that are reverted, but they are not
happening. Other types of edits,
it's it has some funny
results. For example, for Norwegian,
Scientology came up as a bad word.
I don't know why a
so and then we say and it's a black
I don't know how it finds its patterns.
It just gives me a number between
zero and one and that's all.
So how it's useful.
And so basically in English,
Wikipedia, there's one hundred sixty
thousand edits per day.
That's really vast amounts of edits
and machine classifier just gives
a number to all of them from zero to one.
And what it does, we put
a threshold that has the highest
recall. It doesn't have really
good precision.
It means like if we get this
10 percent edits, there might be a lot of
it is a very good.
But still, we want to collect
all the things that are might be
vandalism.
So we just remove all of this 90 percent
edits and we say just review.
It is 10 percent and we give this to
humans to just review that.
And so we
that machine prediction, it used to be
like thirty three people that worked
eight hours a day just to review this
for this English Wikipedia
and now it's for people.
And then
lots of machine classifiers
have been written for Wikipedia is
like sticky, like a Claverton
John Hagel, but they are written by
people who are computer scientists and
has a lot of experience skills
and machine learning and see through the
system because it's a really large
system.
And so what we did was
they built just an API.
We say we give you that number,
we give you this machine learning number,
and you just go figure
that and build anything that you want on
top of that. And I want to talk about
like values cases we had for this.
So in this example,
there is in the top end,
it is a really good edit that changes
if it fixes a link in Wikipedia
and it says with probability
of 94 percent, it's
not a vandalism.
And then someone just removes the
references, says it almost grows on
trees. I don't know.
And the
true for this is like 92 percent.
This is vandalism if somebody needs to
review it so we don't do any sorts of
edits on our own, we just provide
this API. It's sort of like and
I would call your machine learning as
a service.
So then it
makes it really easy for people who
doesn't have much knowledge about
machine learning or about
software engineering to just build a tool
on top of our tools.
And I'm going to talk about it like.
So one thing that happened was,
um.
So first of all, do you know how many
of you do know about especially recent
changes?
OK, so I'll explain it about a
little bit further. And so if you
the way that Wikipedia finds out
which these are vandals, vandalism or
not, like a lot of people find out, is
there's a if you make an edit, it just
immediately goes to a special page called
especially since you just automatically
generated.
And then anyone who checks
is page, you can say, oh, OK, this is
probably needs to be checked.
So they just go there and look
it up. And there was a product team
in Wikimedia Foundation.
They said, oh, this is really nice that
we have this API.
So we built this tool that just basically
highlights the edits that are likely
to be vandalism for anyone who checks the
recent changes.
And then you can say, oh, OK, I will
review this or I revert this.
So it's up
to here. And there was like other things
that there are like
Bossert automatic reverse things that are
made on top of ORUs or other things.
But now we realize, OK, we
have this machine learning service,
that we can do all sorts of things on top
of that of why not just just
stick it if it is vandalism or
not, you just can do all sorts of other
things. So we move forward
with, too.
There are lots of other classification
and machine learning models that we
built, but I really like this too.
First is article quality.
It basically says
what? Or it basically quantifies
a quality of an article, says so
in English, Wikipedia.
There's five levels of quality for an
article. It's called Feature Article,
good article.
And then like an
article B, C,
start and stop.
It's like this spectrum.
And so we turn this into
number. So we give the people a number
saying, oh, this article seems
to be a see article.
And this is one thing about this.
Like, so there's five.
Million articles, and they change all the
time, so how are you going to assess the
quality of these things?
So there's people who go there and
say, oh, this is a C-level article, and
then five years pass by and no
one reads this article.
But the article has improved a lot.
So I think that we did was actually
going through all of these assessments
and saying, OK, it seems
that these articles have improved a lot.
You might need to assess this
thing that you did.
So this was
I will talk about this more often.
And there's another one is that called
draft topic.
It basically uses a word to make
an LP system to say
if this article without knowing anything,
if this article belongs to this category
of space or
physics or chemistry
or like, it's about culture.
So it's basically just gets
the content of the article and gives you
what type of subject it is.
And it's really useful for a
huge amount of new articles that's
being added to Wikipedia and people don't
know what topic they are.
So I want to so
I know a little bit about physics so I
can review these type of articles, but
I don't know. I don't want to see
anything about like art because I don't
know how to I can say if it's a good
article or not.
So one thing that someone did, it's
called an ostrich dashboard.
So it's basically education program.
This is like a foundation
that works closely
to the community foundation.
And they go to the students and they give
them articles and they say, like
how to improve Wikipedia.
They turn a students, university
students, especially to Wikipedia
editors. And they built
like it's sort of a dashboard in that
you can give an article and it's
a suggestion of what type of other
articles similar to that you can improve.
And it gives a really high percentage
to things that needs to be completed.
And this completeness, the score that I
can I don't know if you can see it here
in here. This completeness score is
comes from ORUs.
So it gives this sort of priority
to things that or is things
it's not completed.
And this is one of my favorites.
So I do
you know this woman, Carlana
or Emily Temple.
So you really should
know her as she was a Wikipedian a
year for two years
ago, three years ago.
And she was a person who did
a lot of editing for about
articles of woman scientist, and she got
lots of hate mail. So one day she
said, that's enough.
I vow to
write an article about a woman scientist
every time I get the hate hate mail.
And she did a lots of folks
and she had lots of works
to improve the articles of woman
scientists.
We published the paper and
we were able to measure her impact.
This is like I really love this graph.
This is amazing.
So we sort of
got all of the quality
of all of Wikipedia over the time.
And then we got the average of quality
and then we looked at the quality and
say, OK, this is
the average, and then got another
average, which is average of woman
scientist.
And we realized, OK, starting from 2000
to 2004,
they didn't get much attention.
So the quality was below the average
of Wikipedia.
And then it comes to 2012.
And then Achelen, our Emily
Tembo started working on articles
and then she started making
a wiki project and doing all sorts
of outreach to universities.
And then it got improved until 2014.
That is actually get bypassed
the average quality of
all of Wikipedia.
And now it's way better than
the average Coltons going improving
exponentially.
And there's another use case.
And so
sorry, how many of you know about Wiki
data?
That's nice.
So it's basically a repository
that has the
basic information for everything, so
Google uses it,
Siri uses it everything,
and it has a structured
way of handling the data.
But because it's a wiki, it can be
vandalism happening in it, too.
And it was this famous case that if
you would ask, Siri was the national
anthem of Algeria, it would say is this
is this Esposito?
Because it's someone vandalized this
item in the data and saying,
like, change the national anthem of
Algeria to discuss it.
And so I think that after
that, Amazon used Auris
and looked it up and says
when they are like sort of gathering
our data from the data, they look at
Orissa's scores and they see if they or
as scores for that item is
dropping really low.
And it seems that probably evangelism
is happening there.
They just don't put them inside their
internal system and.
And there's another thing about this is
that ethical layer is really
important here, so
everything about Orris is open, all
of its data, all of it.
Of course, everything about is completely
open. And then this is
an API.
You just can flip everything.
So one thing is in Facebook,
you usually get things that Facebook
thinks is the most important to you, but
you cannot flip the switch and check out.
OK, I want to see things that are
least important to me.
But you can do this in Wikipedia.
You can say, oh, no, no, I don't want to
see this that are likely to be vandalism.
I want to see the best efforts.
You can just do it and just change the
order and you get the results.
And there's another problem that we used
to have was after
there was a robot that basically was
reversing all of this and
it was really good.
But the problem was they was too good
because there was lots of new editors who
would do a good faith mistake.
They wanted to fix something, but because
Wikipedia is super complex, you cannot
get everything right in the first edit
and a lot of cases it just randomly
gets rejected by a bunch and
then people just run away.
And there was studies that
prove that that's changing.
And this robot
to a human would change the course of
things. And
like there was like how it was proving.
It was like this robot was broken for a
couple of days because
some reasons it was less of a problems.
But at the end,
user retention for that month went up.
So one thing that we did here was
that we built two systems saying, like,
it's just not determine if it's a
vandalism or not, just determine if it's
if it is damaging, it is
made in a good faith or not.
And we don't let people to
we tell people not to worry about edits
that are made in good faith by
robots.
And we also can a lot of people
use this and they go to like sort
of health pages and find these
type of edits using Auris and they go
to the user and say, oh, thank you, that
you come here.
But this edit has this small problems,
please take care.
And it would give it like more socialized
and more welcoming space to them.
So it's we are
working on that to make it better.
And so now I want to talk about
Auris components and how it's
built together.
Any questions up to here?
Nothing.
OK,
so
the first thing is something called the
killable, the source code, because
everything about us is public.
So the source code for that is in
GitHub, Wikimedia slash cannibals.
And it's you can try it out
in the labels of dabbling meth labs
dot org. And it's basically
a people can go there and
get their work set, like usually
volunteers. And they say if
there it is, if you scroll down
and you see an edit here and
they say if it's damaging or if it's a
good face and you can choose if it's
good faith or not, and
it would collect all of its data and
improves its models
or make this models for new things,
Rescoring is the basically the
scientific part behind Auris.
It's a basically a wrapper around this
Python library called Psyche's Learn.
And it's just something
that if I can I want to show it.
You can build this tool and you build
a model, just load it up from pixels
and then you give it an
API endpoint and you say,
OK, this revision, this is a revision
ID one, two, three, four, five,
six, seven, eight, nine.
And then you say, OK, give me
probability if it's a vandalism or not,
it just it gets all of the futures
and the futures and tell
you if it's a good idea or
bad.
But and also so the thing is, like
we build this all of these models, but
how we're going to gather the data, how
we are going to build this model and how
we are going to ship these models to the
production, it's
has sort of built in
a place called Model Repositories.
So for the one that detects if
it's a this is a vandalism or not, it's
in its quality.
And if it's the one that's actually
assessed quality of articles, it's
called article quality and so
on. And Orest
itself, it just acquires everything.
A basically and also it has
a like it's a flask app and it
has a nice caching around it.
And so you can have this discussion so
you don't get a lot of requests
so it can just run in scale
and is accessible for everyone
in the community.
So I want to just show you quickly one of
the like results.
So this is a
little something probably.
Can you read it now or
more?
Mark,
I OK, so this
is just output of
arrests that the commuters, especially
three scores and
some random number, that is a revision
idea of edit
an article of Milkovich.
So we say, OK, we support this model.
These are versions of these models
and this is Ajka quality.
And it thinks it's a B level
article, also a B level.
It's less than gay and
F A, but it's top of C
start and stop and it gives
us some numbers of how confident
it is. And so it says
if it's the last major, this article
is damaging or not.
It says with ninety three percent
confidence, it's not a damaging edit
and a giraffe quality says if it's a spam
vandalism or attack article
and says, no, it's OK.
And drafts are a draft topic says
it's more probably stem as
subcategory of space.
And so, like we have this
all of these categories here,
the the the the and
the last on this, uh, this article
is made on a good face with a really high
confidence.
So basically it's like a really simple
this is just Jason Prettify.
So if you want to see it in reality,
it's like this just you can just pass it
from Jason and everything.
Everything will be fine.
So this is
just the output. That's the only thing
that we provide. We don't do anything
after that. And people build tools on top
of this.
OK, let's see.
OK.
Yes.
Um,
so, uh, any
questions is
basically all of it.
Yeah.
Mother weighed support,
according to labor today, to.
Now, I can't repeat the question.
So the question.
You can see how there
is like and it's good, though, for
me to do this more than
you could in. It a
way statements by the.
So one thing about this is that
we usually crowdsource like
all of the labels, so
individual labels, if we can go there and
look at what
people actually editors, how many
edits and we can even check who edited.
But so it's a sort of really open
that everyone can see what you did.
But we never given a score to anyone, so
we basically give it to them more of a
that's how we can
say it. It's more like objective
because Orison
or in Orissa's actually stands for
objective. So you want to be more
objective and don't give sort of
values to more people.
But at the end, if there is evangelism in
there like labels and some people just
go to mislabeling everything,
we just can remove or remove everything
from the dedicated user and just remove
them from the database.
And that would be it.
Yeah.
Another question.
What I really liked about the systems
that you talked about is that they
they don't that's always a human in the
loop, right? They don't really try to
replace anyone's work.
They just try to focus it better.
So that's that's something I really like
about that. To what extent is that like
a philosophical principle that you have
or what you also if you if you could have
I don't know if you
could have about that.
That perfectly divides good from from
bad edits or whatever
what you want to deploy it or what you
still keep humans in the loop.
Hmm. And so there are two
parts of it. So one thing about is that
Wikimedia Foundation engineers usually
try to build a platform and they don't
want to an infrastructure.
They don't want to a intervene
in any things in the community.
And anything that's related to content
is the community's work.
And it's really our principle
to just not meddle with that.
And so we build tools, we make a
life for them easier.
So like, for example, in
this climate, change was only
existed in English, Wikipedia, but after
Auris came to life.
So lots of people who didn't know much
about machine learning was able to use
that. So this sort of reversed but
actually came into life in a
Spanish Wikipedia wiki data program,
Wikipedia and some other Vickie's.
So it
removed the barrier of doing that.
But it's like a catalyst.
But we are never going to do
anything on behalf of the community.
Would it answer your question?
Sorry if this question was already
answered, but can I query
or as with any text that I like
only with Wikipedia articles, so
can I do topic modeling for any text or
so?
Can you be more specific?
What do you mean? Like anything?
Because like so in Wikipedia we also
like or a specific data to one
items of data.
What do you mean. Something is specific
in Wikipedia itself.
Like or something like outside if
I can.
Hello. OK, um
I was a little late so maybe I didn't get
all of it. So does this have some
kind of text input field where I can
basically put in my own text
and that get topic labeling for that
specific text, something
that you can do and some people
actually do it is that you want to get
a score for a revision that made this
happen in this article.
But also you can tell Orris
just change this feature.
And like, for example, if you want to
access codes, you have an article.
And so it
goes and looks up at how many headings it
has, how many sections it has for how
many progress has.
And then A,
you can say, just imagine
it doesn't have like ten paragraphs.
Imagine it has 12 paragraphs
and give me another number so you can
make it or as to sort of give you
just dedicated future to.
And for that,
it's actually someone uses this ad to
give us some sort of recommender system
that allows rich people.
So they go to the students and
say, so this article looks like a B
level article. And if you want to make it
like a good article, just add 10 more
references for this article so they
would know. OK, so if I add just one
paragraph and ten more articles, it will
level up this article so
it can be used this way.
But anything we say that, for example,
we don't do any sort of users or
anything and also
for security reasons, we don't give
it to get text or like
data sources for that from users
Vlady to get it from
Wikipedia.
So it goes through Wikipedia's API.
But you can change futures.
I'm about a bit interested about
the differences
between Wikipedia and Wiki data,
which is easier to classify, is easier
classify wiki data edits
like I assume there are lots of truth and
kind of eternally true, like the national
anthem of a country would change very
rarely.
Yeah.
So the question
to answer is there's two types of
so there's one pro and one can I
can like sort of simplify to
this. And one thing about
the wiki data is accuracy is really
high because you can just make it and
easily say, OK, this item is about
a living person or a day item,
like really common patterns
like vandalism where items are can
be easily added as a
feature because it's like a structured
data.
So it's really accurate.
It's advertised like add much, feature
it just really quickly.
It was able to pick up everything and
but there are two things.
So lots of vandalism comes from
editing descriptions.
So and these are free from text and
it's a little bit hard to detect that.
And there's another part is that ninety
nine point two percent of these
ads are really good apps.
But so the percentage of vandalism
in Wikipedia is seven percent.
Seven percent of it is vandalism,
but the data is zero point one
percent or something like it's really,
really smart. So finding a really, really
good and not classifier,
but a really good dataset
to label was really hard.
I think we basically got around
half a million revisions and just
expected all of the features and say,
yeah, maybe we should just label this
one. So it was really because it's sort
of like a but dominant
wiki.
It's not a more like English Wikipedia.
It's just more human than what.
Did you think about the case
that someone trained his own eye
against yours and then hides
his vandalism
and a cloud of words, which
did you think about that all the time?
I think so.
And so there are two things.
So one good thing about this is being
really public is that people can come and
say problems that it has.
So, for example, someone came and said
it has some bias towards newcomers
because newcomers tend to be more
like vandalizing.
So we also have problems.
But the good thing about being public is
that people can just come to us and say
these problems exist.
And but also, on the other
hand, if we are not trying
to catch sneaky vandalism, this is
some sort of thing called sneaky
vandalism. And there are like other lines
of defense for that.
And so it's
OK, let me assure you.
Hmm.
So there's a
there's another line of defense we
want to do this, we catch this type of
vandalism, which is according
to research, is actually made only by
a 14 year old high school kids that just
are bored. And so it's not
trying to catch those sophisticated
attacks against Wikipedia.
I can say more questions
if you want to reach out to us.
So these are my emails.
The first one in here is within
the media. That is my work account,
Allard's, Grubert, Gmail is my personal
account. Don't ask why you chose this
username. I was told when I chose
it, and this is
my username, Wikipedia Islamicized within
the DMD, because this is the law
and this is my personal account.
And in Twitter you can find me in those
accounts.
The second the first one is in plain
English. The first one is mostly Persian.
And if you want to
say one thing about our team is
screenplay's from team is some people
are living in the San Francisco.
Some of it live in Minneapolis.
I live in Berlin and people live
all around the world. So we have sort of
an online
office, which is a
channel called a Wikimedia I
Freenode. And you can just come there
and just ask questions and say, hey, I
saw this presentation.
I'm asking. I have discussions and I
think just.
Thank you.