(moderator) The next talk is by Anders Sandholm
on Wikidata fact annotation for Wikipedia across languages.
- Thank you. - Thanks.
I wanted to start with a small confession.
Wow! I'm blown away by the momentum of Wikidata
and the engagement of the community.
I am really excited about being here
and getting a chance to talk about work that we've been doing.
This is doing work with Michael, who's also here in the third row.
But before I dive more into this,
this wouldn't be a Google presentation without an ad,
so you get that up front.
This is what I'll be talking about, our project, the SLING project.
It is an open source project and it's using Wikidata a lot.
You can go check it out on GitHub when you get a chance
if you feel excited about it after the presentation.
And really, what I wanted to talk about-- the title is admittedly a little bit long,
it's even shorter than it was in the original program.
But what it comes down to, what the project comes down to
is trying to answer this one very exciting question.
If you want, in the beginning, there were just two files,
some of you may recognize them,
they're essentially the dump files from Wikidata and Wikipedia,
and the question we're trying to figure out or answer is really,
can we dramatically improve how good machines are
at understanding human language just by using these files?
And of course, you're entitled to ask
whether that's an interesting question to answer.
If you're a company that [inaudible] is to be able to take search queries
and try to answer them in the best possible way,
obviously, understanding natural language comes in as a very handy thing.
But even if you look at Wikidata,
in the previous data quality panel earlier today,
there was a question that came up about verification, or verifiability of facts.
So let's say you actually do understand natural language.
If you have a fact and there's a source, you could go to the source and analyze it,
and you can figure out whether it actually confirms the fact
that is claiming to have this as a source.
And if you could do it, you could even go beyond that
and you could read articles and annotate them, come up with facts,
and actually look for existing facts that may need sources
and add these articles as sources.
Or, you know, in the wildest, craziest possible of all worlds,
if you get really, really good at it you could read articles
and maybe even annotate with new facts that you could then suggest as facts
that you could potentially add to Wikidata.
But there's a whole world of applications of natural language understanding.
One of the things that's really hard when you do natural language understanding--
these days, that also means deep learning or machine learning,
and one of the things that's really hard is getting enough training data.
And historically, that's meant having a lot of text
that you need human annotators to then first process
and then you can do training.
And part of the question here is also really to say:
Can we use Wikidata and the way in which it's interlinked with Wikipedia
for training data,
and will that be enough to train that model?
So hopefully, we'll get closer to answering this question
in the next 15 to 20 minutes.
We don't quite know the answer yet but we have some exciting results
that are pointing in the right direction, if you want.
Just take a step back in terms of the development we've seen,
machine learning and deep learning has revolutionized a lot of areas
and this is just one example of a particular image recognition task
that if you look at what happened between 2010 and 2015,
in that five-year period, we went from machines doing pretty poorly
to, in the end, actually performing at the same level of humans
or in some cases even better albeit for a very specific task.
So we've seen really a lot of things improving dramatically.
And so you can ask
why don't we just throw deep learning at natural language processing
and natural language understanding and be done with it?
And the answer is kind of we've sort of done to a certain extent,
but what it turns out is that
natural language understanding is actually still a bit of a challenge
and one of the situations where a lot of us interact with machines
that are trying to behave like they understand what we're saying
is in these chat bots.
So this is not to pick on anyone in particular
but just, I think, an experience that a lot of us have had.
In this case, it's a user saying I want to stay in this place.
The chat bot says: "OK, got it, when will you be checking in and out?
For example, November 17th to 23rd."
And the user says: "Well, I don't have any dates yet."
And then the response is:
"Sorry, there are no hotels available for the dates you've requested.
Would you like to start a new search?"
So there's still some way to go
to get machines to really understand human language.
But machine learning or deep learning
has been applied already to this discipline.
Like, one of the examples is a recent...
a more successful example is BERT
where they're using transformers to solve NLP or NLU tasks.
And it's dramatically improved the performance but, as we've seen,
there is still some way to go.
One thing that's shared among most of these approaches
is that you look at the text itself
and you depend on having a lot of it so you can train your model on the text,
but everything is based on just looking at the text
and understanding the text.
So the learning is really just representation learning.
What we wanted to do is actually understand and annotate the text
in terms of items or entities in the real world.
And in general, if we take a step back,
why is natural language processing or understanding so hard?
There are a number of reasons why it's really hard, but at the core,
one of the important reasons is that somehow,
the machine needs to have knowledge of the world
in order to understand human language.
And you think about that for a little while.
What better place to look for knowledge about the world than Wikidata?
So in essence, that's the approach.
And the question is can you leverage it,
can you use this wonderful knowledge
of the world that we already have
in a way that you can help to train and bootstrap your model.
So the alternative here is really understanding the text
not just in terms of other texts or how this text is similar to other texts
but in terms of the existing knowledge that we have about the world.
And what makes me really excited
or at least makes me have a good gut feeling about this
is that in some ways
it seems closer to how we interact as humans.
So if we were having a conversation
and you were bringing up the Bundeskanzler and Angela Merkel,
I would have an internal representation of Q567 and it would light up.
And in our continued conversation,
mentioning other things related to Angela Merkel,
I would have an easier time associating with that
or figuring out what you were actually talking about.
And so, in essence, that's at the heart of this approach,
that we really believe Wikidata is a key component
in unlocking this better understanding of natural language.
And so how are we planning to do it?
Essentially, there are five steps we're going through,
or have been going through.
I'll go over each of the steps briefly in turn
but essentially, there are five steps.
First, we need to start with the dump files that I showed you
to begin with--
understanding what's in them, parsing them,
having an efficient internal representation in memory
that allows us to do quick processing on this.
And then, we're leveraging some of the annotations
that are already in Wikipedia, linking it to items in Wikidata.
I'll briefly show you what I mean by that.
We can use that to then generate more advanced annotations
where we have much more text annotated.
But still, with annotations being items or facts in Wikidata,
we can then train a model based on the silver data
and get a reasonably good model
that will allow us to read a Wikipedia document
and understand what the actual content is in terms of Wikidata,
but only for facts that are already in Wikidata.
And so that's where kind of the hard part of this begins.
In order to go beyond that we need to have a plausibility model,
so a model that can tell us,
given a lot of facts about an item and an additional fact,
whether the additional fact is plausible.
If we can build that,
we can then use a more "hyper modern" reinforcement learning aspect
of deep learning and machine learning to fine-tune the model
and hopefully go beyond what we've been able to so far.
So real quick,
the first step is essentially getting the dump files parsed,
understanding the contents, and linking up Wikidata and Wikipedia information,
and then utilizing some of the annotations that are already there.
And so this is essentially what's happening.
Trust me, Michael built all of this, it's working great.
But essentially, we're starting with the two files you can see on the top,
the Wikidata dump and the Wikipedia dump.
The Wikidata dump gets processed and we end up with a knowledge base,
a KB at the bottom.
That's essentially a store we can hold in memory
that has essentially all of Wikidata in it
and we can quickly access all the properties and facts and so on
and do analysis there.
Similarly, for the documents,
they get processed and we end up with documents
that have been processed.
We know all the mentions
and some of the things that are already in the documents.
And then in the middle,
we have an important part which is a phrase table
that allows us to basically see for any phrase
what is the frequency distribution,
what's the most likely item that we're referring to
when we're using this phrase.
So we're using that later on to build the silver annotations.
So let's say we've run this and then we also want to make sure
we utilize annotations that are already there.
So an important part of a Wikipedia article
is that it's not just plain text,
it's actually already pre-annotated with a few things.
So a template is one example, links is another example.
So if we take here the English article for Angela Merkel,
there is one example of a link here which is to her party.
If you look at the bottom,
that's a link to a specific Wikipedia article,
and I guess for people here, it's no surprise that, in essence,
that is then, if you look at the associated Wikidata item,
that's essentially an annotation saying
this is the QID I am talking about when I'm talking about this party,
the Christian Democratic Union.
So we're using this to already have a good start
in terms of understanding what text means.
All of these links,
we know exactly what the author means with the phrase
in the cases where there are links to QIDs.
We can use this and the phrase table to then try and take a Wikipedia document
and fully annotate it with everything we know about already from Wikidata.
And we can use this to train the first iteration of our model.
(coughs) Excuse me.
So this is exactly the same article,
but now, after we've annotated it with silver annotations,
and essentially, you can see all of the squares
are places where we've been able to annotate with QIDs or with facts.
This is just a screenshot of the viewer on the data,
so you can have access to all of this information
and see what's come out of the silver annotation.
And it's important to say that there's no machine learning
or anything involved here.
All we've done, is sort of mechanically, with a few tricks,
basically pushed information we already have from Wikidata
onto the Wikipedia article.
And so here, if you hover over "Chancellor of Germany"
that is itself a Wikidata, that's referring to a Wikidata item,
has a number of properties like "subclass of: Chancellor",
"country: Germany", that again referring to subtext.
And here, it also has the property "officeholder"
which happens to be Angela Dorothea Merkel,
which is also mentioned in the text.
So there's really a full annotation linking up the contents here.
But again, there is an important and unfortunate point
about what we are able to and not able to do here.
So what we are doing is pushing information we already have in Wikidata,
so what we can't annotate here are things that are not in Wikidata.
So for instance, here,
she was at some point appointed Federal Minister for Women and Youth
and that alias or that phrase is not in Wikidata,
so we're not able to make that annotation here in our silver annotations.
That said, it's still... at least for me,
it's was pretty surprising to see how much you can actually annotate
and how much information is already there
when you combine Wikidata with a Wikipedia article.
So what you can do is, once you have this, you know, millions of documents,
you can train your parser based on the annotations that are there.
And that's essentially a parser that has a number of components.
Essentially, the text is coming in at the bottom and at the top,
we have a transition-based frame semantic parser
that then generates the annotations or these facts or references to the items.
We built this and run on more classical corpora
like [inaudible], which are more classical NLP corpora,
but we want to be able to run this on the full Wikipedia corpora.
So Michael has been rewriting this in C++
and we're able to really scale up performance
of the parser trainer here.
So it will be exciting to see exactly
the results that are going to come out of that.
So once that's in place,
we have a pretty good model that's able to at least
predict facts that are already known in Wikidata,
but ideally, we want to move beyond that,
and for that we need this plausibility model
which in essence, you can think of it as a black box
where you supply it with all of the known facts you have
about a particular item and then you provide an additional item.
And by magic,
the black box tells you how plausible is the additional fact that you're providing
and how plausible is it that this particular item is fact.
And...
I don't know if it's fair to say that it was much to our surprise,
but at least, you can actually--
In order to train a model, you need,
like we've seen earlier, you need a lot of training data
and essentially, you can use Wikidata as training data.
You serve it basically all the facts for a given item
and then you mask or hold off one fact
and then you provide that as a fact that it's supposed to predict.
And just using this as training data,
you can get a really really good plausibility model, actually,
to the extent that I was hoping one day to maybe be able to even use it
for discovering what you could call accidental vandalism in Wikidata
like a fact that's been added by accident and really doesn't look like it's...
It doesn't fit with the normal topology
of facts or knowledge in Wikidata, if you want.
But in this particular setup, we need it for something else,
namely for doing reinforcement learning
so we can fine-tune the Wiki parser,
and basically using the plausibility model as a reward function.
So when you do the training, you try to pass a Wikipedia document
[inaudible] in Wikipedia comes up with a fact
and we check the fact on the plausibility model
and use that as feedback or as a reward function
in training the model.
And the big question here is then can we learn to predict facts
that are not already in Wikidata.
And we hope and believe we can but it's still not clear.
So this is essentially what we have been and are planning to do.
There's been some surprisingly good results
in terms of how far you can get with silver annotations
and a plausibility model.
But in terms of how far we are, if you want,
we sort of have the infrastructure in place
to do the processing and have everything efficiently in memory.
We have first instances of silver annotations
and have a parser trainer in place for the supervised learning
and an initial plausibility model.
But we're still pushing on those fronts and very much looking forward
to see what comes out of the very last bit.
And those were my words.
I'm very excited to see what comes out of it
and it's been pure joy to work with Wikidata.
It's been fun to see
how some of the things you come across seemed wrong and then the next day,
you look, things are fixed
and it's really been amazing to see the momentum there.
Like I said, the URL, all the source code is on GitHub.
Our email addresses were on the first slide,
so please do reach out if you have questions or are interested
and I think we have time for a couple questions now in case...
(applause)
Thanks.
(woman 1) Thank you for your presentation. I do have a concern however.
The Wikipedia corpus is known to be with bias.
There's a very strong bias-- for example, fewer women, more men,
all sorts of other aspects in there.
So isn't this actually also tainting the knowledge
that you are taking out of the Wikipedia?
Well, there are two aspects of the question.
There's both in the model that we are then training,
you could ask how... let's just...
If you make it really simple and say like:
Does it mean that the model will then be worse
at predicting facts about women than men, say,
or some other set of groups?
To begin with, if you just look at the raw data,
it will reflect whatever is the bias in the training data, so that's...
People work on this to try and address that in the best possible way.
But normally, when you're training a model,
it will reflect whatever data you're training it on.
So that's something to account for when doing the work, yeah.
(man 2) Hi, this is [Marco].
I am a natural language processing practitioner.
I was curious about how you model your facts.
So I heard you set frame semantics,
Right.
(Mike) could you maybe give some more details on that, please.
Yes, so it's frame semantics, we're using frame semantics,
and basically,
all of the facts in Wikidata, they're modeled as frames.
And so that's an essential part of the set up
and how we make this work.
That's essentially how we try to address the...
How can I make all the knowledge that I have in Wikidata
available in a context where I can annotate and train my model
when I am annotating or passing text.
Is that existing data in Wikidata is modeled as frames.
So the store that we have,
the knowledge base with all of the knowledge is a frame store,
and this is the same frame store that we are building on top of
when we're then passing the text.
(Marco) So you're converting the Wikidata data model into some frame.
Yes, we are converting the Wikidata model
into one large frame store if you want, yeah.
(man 3) Thanks. Is Pluto a planet?
(audience laughing)
Can I get the question...
(man 3) I like the bootstrapping thing that you are doing,
I mean the way that you're training your model
by picking out the known facts about things that are verified,
and then training the plausibility prediction
by trying to teach the architecture of the system
to recognize that actually, that fact fits.
So that will work for large classes, but it will really...
It doesn't sound like it will learn about surprises
and especially not in small classes of items, right.
So if you train your model in...
When did Pluto disappear, I forgot...
As a planet, you mean.
(man 3) Yeah, it used to be a member of the solar system
and we had how many, nine observations there.
- Yeah. - (man 3) It's slightly problematic.
So everyone, the kids think that Pluto is not a planet,
I still think it's a planet, but never mind.
So the fact that it suddenly stopped being a planet,
which was supported in the period before, I don't know, hundreds of years, right?
That's crazy, how would you go for figuring out that thing?
For example, the new claim is not plausible for that thing.
Sure. So there are two things.
So there's both like how precise is a plausibility model.
So what it distinguishes between is random facts
and facts that are plausible.
And there's also the question of whether Pluto is a planet
and that's back to whether...
I was in another session
where someone brought up the example of the earth being flat,
- whether that is a fact or not. - (man 3) That makes sense.
So it is a fact in a sense that you can put it in,
I guess you could put it in Wikidata
with sources that are claiming that that's the thing.
So again, you would not necessarily want to train the model in a way
where if you read someone saying the planet Pluto, bla, bla, bla,
then it should be fine for it
to then say that an annotation for this text
is that Pluto is a planet.
That doesn't mean, you know...
The model won't be able to tell what "in the end" is the truth,
I don't think any of us here will be able to either, so...
(man 3) I just want to say
it's not a hard accusation against the approach
because even people cannot be sure whether that's a fact,
a new fact is plausible at that moment.
But that's always...
I just maybe reiterated a question that I am posing all the time
to myself and my work; I always ask.
We do the statistical learning thing, it's amazing nowadays
we can do billions of things, but we cannot learn about surprises,
and they are very, very important in fact, right?
- (man 4) But, just to refute... - (man 3) Thank you.
(man 4) The plausibility model is combined with kind of two extra roles.
First of all, if it's in Wikidata, it's true.
We just give you the benefit of the doubt, so please make it good.
The second thing is if it's not allowed by the schema it's false;
it's all the things in between we're looking at.
So if it's a planet according to Wikidata, it will be a true fact.
But it won't predict surprises but what is important here
is that there's kind of no manual human work involved,
so there's nothing that prevents you from...
Well, now, if we're successful with the approach,
there's nothing that prevents him from continuously updating the model
with changes happening in Wikidata and Wikipedia and so on.
So in theory, you should be able to quickly learn new surprises.
(moderator) One last question.
- (man 4) Maybe we're biased by Wikidata. - Yeah.
(man 4) You are our bias. Whatever you annotate is what we believe.
So if you make it good, if you make it balanced,
we can hopefully be balanced.
With the gender thing, there's actually an interesting thing.
We are actually getting more training facts
about women than men
because "she" is a much less ambiguous pronoun in the text,
so we actually get a lot more true facts about women.
So we are biased, but on the women's side.
(woman 2) No, I want to see the data on that.
(audience laughing)
We should bring that along next time.
(man 4) You get had decision [inaudible].
(man 3) Yes, hard decision.
(man 5) It says SLING is... parser across many languages
- and you showed us English. - Yes!
(man 5) Can you something about the number of languages that you are--
Yes! Thank you for asking.
I had told myself to say that up front on the first page
because otherwise, I would forget, and I did.
So right now,
we're not actually looking at two files, we're looking at 13 files.
So Wikipedia dumps from 12 different languages
that we're processing,
and none of this is dependent on the language being English.
So we're processing this for all of the 12 languages.
Yeah.
For now,
they share the property of, I think, being the Latin alphabet, and so on.
Mostly for us to be able to make sure
that what we are doing still make sense and works.
But there's nothing fundamental about the approach
that prevents it from being used in very different languages
from those being spoken around this area.
(woman 3) Leila from Wikimedia Foundation.
I may have missed this when you presented this.
Do you make an attempt to bring any references from Wikipedia articles
back to the property and statements you're making in Wikidata?
So I briefly mentioned this as a potential application.
So for now, what we're trying to do is just to get this to work,
but let's say we did get it to work with a high level of quality,
that would be an obvious thing to try to do, so when you...
Let's let's say you were willing to...
I know there's some controversy around using Wikipedia as a source for Wikidata,
that you can't have circular references and so on,
so you need to have properly sourced facts.
So let's say you were coming up with new facts,
and obviously, you could look at the cover of news media and so on
and process these and try to annotate these.
And then, that way, find sources for facts,
new facts that you come up with.
Or you could even take existing...
There are a lot of facts in Wikidata that either have no sources
or only have Wikipedia as a source, so you can start processing these
and try to find sources for those automatically.
(Leila) Or even within the articles that you're taking this information from
just using the sources from there because they may contain...
- Yeah. Yeah. - Yeah. Thanks.
- (moderator) Thanks Anders. - Cool. Thanks.
(applause)