*Music*
Herald: Hello everybody, we are ready to get started we have Lucas and Amir here
and they want to give us a quick introduction of a project from the
wikimedia foundation called "cloud services" and how it might be may be
useful to all of us. So let's give a round of welcoming applause to Lucas and Amir.
*Applause*
Lucas: Thanks! yea, hello. So "wikimedia cloud services" is basically this big
collection of all kinds of different things which are useful if you want to do
taking your things in the wikimedia universe like with wikipedia or other
projects and you get them free of charge or you can just use them and the only
requirement is that you use them for something that's a kind of relevant to the
mission of wikimedia of promoting free knowledge and that kind of stuff and it's
kind of split into the things that you can do with your regular wikimedia account
which any registered user can do and then there's also things you need a special
account for on a different system called wiki tech and Amir is going to talk more
about those later but first let's just look into some of the things you can do
with your regular wikimedia account. And if you want to follow any of these links
there's a shortcut here. I was about to switch the next tab, so let's just stay
here for a few seconds yeah. So the first thing is the API sandbox which is if you
want to use the MediaWiki API to figure out what you have on a page or to make
edits or any kind of stuff. The API sandbox is a special page that's really
useful to find out how to use the API for example here's all the different actions I
can use that say query is the kind of general catch-all action that's here and
then I get down here a list of all the parameters I can use with queries such as:
I want to have all the user info and what kind of user info do I want? I want
options, blablabla. I would like to have some different format versions. So it
gives you all these nice inputs for figuring out exactly how to use the API
what's valid what's not valid and then you can make the API request and there you get
a response and we can't read anything because it's zoomed in way too much. But
it's very helpful when trying to use the API and then in the end here you can see
what you need to do in your own code to make the same API request. And for
anything that you can't do with the normal API - so if you want to do some kind of
more expensive analysis - you can often do that with Quarry, which is a tool that
lets you write SQL queries against databases that are almost like the ones in
production like you don't have user passwords and stuff but you'll have all
the database tables with page metadata and connections between them and the logs and
all kinds of stuff and you can just write your SQL here send it and you get the
results for example here's the number of lexemes published a days so it's some kind
of selecting from the page where the namespace is the lexeme namespace and
grouping that by the date and then we get something like all the way down to
September which is apparently when I ran this query there were here there were 116
lexemes created in this day. Or here someone had a list of edits to JavaScript
and CSS pages on Hungarian Wikipedia so you can run these queries against any Wiki
you like, like this here in wikipedia one. And if you can't get by with just SQL what
you also have is this thing called Paws, which gives you a Jupiter(?) instance if
you've heard of that you can basically write your own Python code here and do it
in a very convenient way because there's all kinds of auto-completion and helpful
things. So i can just try to copy this and run the code (then I needed a new cell
below it… there we go, Thanks!) and if I type item I should get helpful hints what
I can do with the item (if it's not hanging or something or the tab control
space no oh there we go yeah) and it's also a very useful way to work with py-
wiki-bot or you can also directly get normal shell here. And one thing (oops did
I click and wrong thing? I would like to have oh no I don't want a bash notebook I
want a new terminal that's what I want). And here you have for example database
dumps in (where was it?) public/dumps/ something public again… So if you want to
do some kind of analysis here on the data dumps you can get them here and then have
all the computing that you want I guess to analyze the wiki more thoroughly and all
of this is hosted in the Wikimedia Cloud for you and you don't need your own server
or anything. Oh yeah I had two more examples of that, for example here: I use
that too so there were a lot of items on Wikipedia where there was some encoding
error, this should be an apostrophe like down here and instead it was this kind of
I with an accent and I hacked together some ugly Java/Python code to make all of
these edits and it was already logged in as well I didn't need to worry about
logging in or having a password or anything. So it's a very convenient way to
make edits as well. Or you can build something nicer here you can insert like
markdown cells to explain what you're doing and how the code works and build
nice notebooks like that, which are almost self-explanatory. And those are some of
the things you can do just with your Wikimedia account and now Amir is going to
talk about some other things. Amir: Thanks Lucas! So the thing that we
can do is that maybe some of you like me think that doing thing in browser is for
kids I need to do things in terminal I need to do connected system and then you
can access for a wiki tech account which you can just make a wiki tech account in
this place called wiki tech. (where is the li… no no but I do'… the main thing, the
main list. yeah okay) And so in here so and then you make a wiki tech account and
it gets approved quickly and then you get the shell and then you can just quickly go
there (where is yer…) and you can go to this shell and just log in and then you
have access to day a big set of nodes in the cloud and you can just do whatever you
want. Also you have access to the core dumps and you have access to the replica
database. Let me show it to you. [mumbling] So for example you can go to LS
/public/dumps/public/wikidatawiki/ and then you get - oh there's like all sorts
of time and everything that you want to, but if you also… you can do something else
is that you can just do SQL wikidatawiki and then you go inside the wikidata's
database, I mean it does you don't have the rights you can you cannot write to
their replica because it's a replica and also it's sanitized so it doesn't have
their like hash of user password and stuff like that but still you can do just select
varies from recent changes limits five and yeah and then you get all of the things
that you want you cannot even describe anything you want to directly into their
system and then there is also we have something called the job grid so you can
just put a crown and anything that you want to or just it's something run
something directly and you goes to the a big note of cloud kubernetes and then just
runs everything that you want to in its here there's a more information about it
in here there's a like a long help that it says like oh I used to run this job and
then job of what it does and you can get this so you just need to it's a bash
command you can run any bash command and send it okay return me this output to this
place and the other places one thing that you can do is also there's a web server
that you can access everything directly so you can just put a PHP file there and into
the Apache and then yet for example this is this is an example that we built
together I think two two Christmases ago but this was like you can just see this is
a piece before the source code is available and you just copy pasted that
source code into like a directory and it was there and every time we click on it
and you get most of the edits that happen on description wiki data that might be
vandalism and we can fix it also a this is not just the only thing that you can do
with this is that you can also put a Python flask application is this the file
implants and then this can be just a Python application and you can just have
the file there and also know JSON Java there's so many of them also you can have
own database like I have something that has its own database for example quick
categories in here has jobs that are here this is this tool for its own built-in
database inside our select cloud services and its uses it just fine you can do that
as well and also there's a cloud VPS that it doesn't do any kubernetes it just you
can make a VPS of your own and then do whatever you want with it so for example
and you get a project and you get the quota it's a slightly more limited but
also you have access to the whole VPS you have sudo rights on it you can do whatever
you feel like about it so we have like for example this project in here and it's
called tools and then there's proxies and you can for example go into that instance
and reboot it and do whatever you want and you can make new instance and look at your
culture and look at everything else there and also you can also make it even a wiki
on one of those cloud VPS systems which is for example we did in here in here if you
look at it it's just a wiki and the difference is that for other ones for
example for the vandalism dashboard you have tools that wmf labs org and then
slash WD w VD which is the tool itself but in here we get our own subdomain so which
will be wiki data - like seam that flew out the wmf labs org and you can even put
all sorts of add subdomains for the wmf labs or as long it's not taken so you can
build a media week instance instance or you can just complete a new software
anything you can put a word processor who cares and then you can use it it's very
simple your own thing and you can help lots of experience. Anything else?
Lucas: I don't think so. Most
important I would say is tool Forge to run your websites or if that's not enough for
you cloud VPS and then you get your own VMware you can do absolutely anything you
want as long as it matches those rules and stuff and I think that's it are there any questions?
Herald: Hello thank you very much for the talk that was very quick so maybe
anybody has a question here I'll give you my microphone to ask it. I don't see any
hands nope okay I don't think we have questions but if you're just too shy to
ask I think these guys always hanging around here around the wikipaka wiki so
if you have anything you want to talk about you'll find them later okay then
give a round of applause again for Lucas and Amir.
*Applause*
*Music*