Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/1654 Thanks!
Hello, everyone, and welcome
back to our
cows first clean.
Our next speaker is Hendley,
Hoyer Hensley Hoyer
is a researcher at the University
of Theme and in the Institute
of Information Management in Berlin
and in his talk Rage
Against the Machine Learning.
He will explain why
audits are a useful method
to ensure the machine learning systems
operate in the interest of
the public.
Welcome, Hendley.
I'm Dr. Henry, and welcome to
rage against the machine learning.
Auditing YouTube and others
in this talk, I will explain why audits
are useful method to ensure that machine
learning systems operate in the interest
of the public.
My goal is to empower civic hackers,
and I'm going to do that by releasing the
scripts that I use to audit
YouTube. And I'm also going to explain
to you how to use these scripts.
Why would it be interesting to
audit YouTube or other machine
learning based curation system?
Well, YouTube has more than two billion
users per month, and 70
percent of the videos watched on YouTube
are recommended by a machine learning
based curation system.
This is remarkable because every
fourth person worldwide relies on YouTube
as a news source, that percentage
is even higher for younger people
there. Every third 18 to
24 year old consumes his or
her news on YouTube.
This means that YouTube's machine
learning based system plays an important
role in what billions of people watch and
how they see the world.
Why do we need machine learning in these
systems? Well, there are eighty
two point two years of video uploaded
to YouTube every day, so that's
500 hours of video uploaded
per minute for a team of human
experts. It would be impossible to review
and categorize this user generated
content.
Now, YouTube markets its recommender
system as a sophisticated algorithm
to match each viewer to the videos
than most likely to watch and enjoy.
In this talk, I will show that popular,
unrelated content is king.
So who am I?
My name is Dr. Henry Hoyer, and I'm a
researcher at the University of Priem
and the Institute for Information
Management.
And this talk is based on my doctoral
thesis focused on auditing machine
learning.
And in this talk, I would explore audits
as a way of making sense of complex
and proprietary machine learning systems
used by YouTube, as well as others.
And this is based on a research project
I conducted together with Hendley Cool,
and we applied to anti-amnesty
harvest.
And this talk is based on my doctoral
thesis called uses and machine learning
based curation systems and
using the link, you can download the
thesis for free at the library
of the University of Bremen,
as many of you may know.
But she noted based curation systems a
special type of artificial intelligence.
The definition of AI, according to
Hansen, is that it's an umbrella
term for computer systems that are,
quote, able to perform tasks
normally requiring human intelligence,
and I prefer the term machine learning
because it's a bit more precise.
And also because many of the successes in
AI that we've seen in the recent years
have been obtained through what is called
statistical machine learning.
You might even have heard about the term
deep learning, which is an even smaller
subset.
So what am I referring to when I say
machine learning at a certain
kind of artificial intelligence that
infers decisions from data
formerly Mitchell define machine learning
as follows A computer
program is set to learn from experience
with respect to some class of tasks
and performance measures if
it's performance at past A.
as measured by P improves with
experience.
And machine learning enabled many of
the recent advances in artificial
intelligence, it is used to
recognize handwritten digits
to recognize people and objects and
images to translate
from one language to another to
drive cars.
And that's the focus here to recommend
postings, photos and videos on
platforms like Facebook and YouTube.
And in my research, I focus on
Facebook and YouTube because they are
two of the most visited websites
worldwide.
And they use a metal based system to
create the content for billions of users.
Now, recommender systems like ones
you find on Facebook and YouTube
have a long history, famous early
examples include Loon from 1958,
the information lands by Malone at all
the tapes for iMessage Ring System by
going back at all and group plans
by Resnick at all.
Looking at the research out there, you
find that Facebook received a lot of
attention regarding algorithmic
awareness, user believes about
the system, how its system
works and the biases
that the system infects.
Meanwhile, especially when I started my
research, it was comparatively little
on YouTube. Despite its importance
and the many people who use YouTube.
And what motivated my research was a
study by Islamiya from
2015.
They found that sixty two point five
percent of Facebook users were not aware
of the existence of Facebook's News Feed
algorithm.
They also showed that users are upset
when posts by close friends or family
are not shown.
And users mistakenly believe
that their friends intentionally chose
not to show them this post.
Aslam, you wrote in the extreme
case, it may be that whenever
a software developer in Menlo Park
adjusts the perimeter, someone
somewhere wrongly starts to believe
themselves to be unloved.
And there's a lot of research that has
pointed out the political,
social and cultural importance
of machine learning based curation
systems.
Zeynep Tufekci wrote
in The New York Times that YouTube may
be one of the most powerful radicalizing
instruments of the 21st century.
And challenges like fake news,
bias predictions and filter bubbles
make an understanding of email based
curation systems an important
and timely concern.
Journalists and researchers have accused
M.O. Based curation systems of
enabling the spread of fake news
or conspiracy theories in general,
and these accusations make sense because
such systems can shape, uses media
consumption and influence their actions.
I would talk a lot about bias in this
talk, so I want to operationalize
what I understand as bias
in the context of my thesis and also
this talk.
I operationalized bias as an inclination,
prejudice or overrepresentation
for or against one person
group. Topic idea or content,
especially in a way considered to be
unfair.
And they are famous examples for pious
predictions.
Epstein and Robertson, for instance,
found that biased search engine
results can shift the voting preference
of undecided voters by 20 percent
or more.
So considering this prior work,
I started to believe that it is important
to understand whether users are aware of
the email based systems they're
interacting with and whether users
understand how such systems work.
Otherwise, users might believe
that they're presented with an objective
reality, even though the news they're
seeing is the result of a co-production
between their actions as a user and
a machine learning system's ability to
infer their interest.
Now considered this example, if we
have a recommender system, it can easily
lead to a virtuous circle, so
you end up watching a video related to
human rights.
And then you learn about the treatment of
asylum seekers at the borders,
and that leads you to develop an interest
in the decriminalization of Civil
Sea Rescue.
However, it can also lead to a vicious
circle where you watch a video about
a crime committed by a foreigner
and then you see many videos about crimes
committed by foreigners because the
system just infers
that's what he's interested in.
That's what he likes.
So it's just giving you what you like,
and you may end up with a distorted
view of reality, changed political
views and even xenophobia.
This poses the question How does
machine learning influence people?
And you might have heard about the
potential dangers of so-called online
radicalization and algorithmic
rabbit holes where people end
up in this loop that I just described.
But they have one topic and then
see more and more related to that
particular topic.
And there was one incident that really
motivated me to understand what's
going on on YouTube and what kind of
recommendation the system is provide.
So most of you probably would remember
that in Canada, the stabbing of a citizen
spawned street demonstrations and
rioting.
And the New York Times wrote an article
called As German Sikh News
YouTube Delivers Far Right to Rights.
So, according to the Times, people
who try to inform themselves on YouTube
were shown increasingly radical far
right videos about the incident, which
allegedly radicalized them
and which fueled the protests.
And this motivated us to perform
audits to see whether YouTube
is actually systematically
recommending more and more
radical content.
Why is this important?
Well, video recommendations of political
topics and news have special
requirements, especially in Germany.
We have laws that force
broadcasters to
provide fair and balanced reporting.
We have also laws that make sure
that minorities are protected.
And Germany, one of these laws is the
so-called one thing starts photog
the interstate broadcasting equipment,
and it's a law that enforces that
broadcasting services report in
a fair and balanced manner that takes
minority views into account.
So motivated by this, we
performed the 2019 YouTube Cannon's
audit, and we
found that YouTube is not pushing
users towards politically extreme
content by consistently suggesting
more extreme videos.
YouTube is also not letting users down
rabbit hole by zooming in
on specific political topics.
What we found is that YouTube is
pushing increasingly more popular content
as measured by the views and likes.
The sentence evoked by the videos
decreased while the happiness
increased.
Now, let's take one step back,
because this is only part of a
much larger puzzle to thoroughly
understand radicalization on YouTube
and how YouTube influences the behavior
of users, research would have to
show that YouTube
is presenting users with increasingly
extreme content that this extreme
content negatively affects
users attitudes, that this
affects their intentions, and
that this changes their behavior.
With this talk, I really only can
talk about the first point that is
whether YouTube is presenting users
with increasingly extreme content.
But I strongly invite other researchers
to look at all these different aspects.
And these audits can really be a voice
of the voiceless.
The dictionary defines the word
ordered as a systematic
review or assessment of something.
And in this talk, I will show you
that audits can enable researchers
and civic hackers to uncover
the potential hidden agendas of social
networking sites.
Or it's especially interesting
because they're immediately meaningful to
users, as newspaper reports
by Smith at all suggests.
So unlike explainable AI techniques,
which might require a deeper
understanding of statistics, these orders
can be interpreted by anybody.
So why perform audits?
Because they enable individuals and
society at large to monitor
and control the recommendations
of machine learning systems?
And I found that orders
are a very useful way to identify
potential biases enacted by
these systems.
Some think it all distinguished five
different kinds of algorithmic ordered
studies.
Court ordered noninvasive user
audits, scraping audits, sock
puppet audits and crowdsourced
audits. And I'm going to explain
each one of them step by step
in the following.
So if a code audit, you
obtain a copy of a relevant algorithm
and then you study the instructions in a
programing language.
And this is challenging since the code
is considered valuable intellectual
property and the code
is commonly conceived using trade secret
protection understanding systems
through code audits.
It's also challenging because algorithms
depend on personal data.
That is, they need to be audited with
real data to be understood.
Machine learning algorithms are also
quite trivial, so the data
is the most important thing.
And to illustrate this, I have the quote
example here on the right.
And that's actually a fully functioning
machine learning system that can detect
spam. It was called it using the
Python Library Typekit Learn,
which makes it quite easy to train
machine learning systems, hiding a lot of
the complexity.
And what you can see here is
that the things that are specific
to the spam filtering use case are
just the two things that are highlighted.
It's the file with the data,
it's the emails that see us fee,
as well as the dimensionality of the
data. Now, if we
would change the data from
images that CSP to constant
CSP, we could easily turn the system
into a car recommendations.
We could also swap the file image
sources for a file called cancer
that sees fit and turn this into
a breast cancer detection system.
It all goes to show that the
algorithm and studying the algorithm
is not sufficient and not really
helpful for our use case.
So we really have to look at the output.
The second type of order that some
forget or recognize the so-called
noninvasive user audits.
There you asked us questions
using a survey format.
However, this comes with serious sampling
problems because how do you actually
reach the users that you want to reach?
This also comes with important validity
problems, for instance, due to cognitive
biases, because people might just
remember things wrongly and
might not be good about explaining
why they did certain things.
The third kind of audits are the
so-called scraping audits, and
there you have a script that interacts
with a platform, for instance, by
querying a particular URL.
And this allows researchers to obtain
a large number of relevant data
points, a more sophisticated version
of these audits of the so-called sock
puppet or.
He is really impersonating
a user and creating
programmatically constructed traffic.
And this is what I will be focusing on,
and I'm going to explain it in more
detail and the next step.
So the other potential way of
performing an audit are the so-called
crowdsourced audit.
And then you recruit a large number of
users to use
a particular platform.
So it's quite similar to the sock puppet
audit, but it's doing it with real
people.
However, this is challenging because
you need to find a large number of people
that can either be done through Amazon
Mechanical Turk or through
inviting volunteers, but
that can be quite challenging.
So I performed the so-called sock puppet
audit, where I wrote a script
that is remote, controlling a browser
and impersonating a
real user just to remind us what we
were trying to do.
We were motivated by the candidates
incidents, and we wanted to know whether
YouTube is actually
showing increasingly radical, far
right videos for a variety of political
topics, as the New York Times has
claimed.
So using a Firefox based bot
that I'm going to release with this talk,
we performed 150
random walks that always followed the
same procedure.
We randomly picked one of nine political
topics from Germany.
Then we entered the topics in German
into the YouTube search bar.
Then we randomly picked one of the top
10 search results.
And then we save
the video page and watched it for random
number of seconds.
Then we randomly chose one of the top
10 video recommendations displayed
in the right sidebar next to the video.
And then we repeated this 10 times.
And we looked at both quantitative
metrics like the number of likes and
views, as well as qualitative
metrics.
And for this qualitative investigation,
we performed an in-depth analysis
for which we randomly select three videos
per topic and coded three videos
per random walk.
We called it the initial video,
the fifth video and the tenth video,
and this coding was performed by three
independent raters one male to female,
all in their twenties to thirties, who
did not know about the research question.
And they really watched the videos
for five minutes or more.
And then they assessed how closely
related the videos are to political
topics. They also rated whether
the videos evoked sadness or happiness
on an 11 point scale from these
zero to most 10.
So we simulated a regular
web browser by Remote Control, a browser,
and we collected between 12 and 25
random walks per topic.
So for each random walk and each topic,
we started a new browser instance
and cleared all cookies.
All random walks were collected in May
2019 with the same laptop
on the same network.
So the decision to select the fifth
and the 10th recommendation for
the in-depth analysis was made at the
beginning of the study.
That is, before reviewing any of the
material and before we performed any
kind of analysis, the raters
reviewed all videos in the same
randomized order.
We computed hit dogs over
to understand how strong our
integrator agreement is,
and we found substantial agreement
regarding how similar the videos were
to the topics in our investigation.
That said, point seven six five
and the sentence evoked by the videos,
which is set point six one three.
We also have moderate agreement for the
happiness, which is that point four
for one.
So you might wonder what are the topics
that we chose?
So we took nine political topics
from a representative telephone poll
conducted on behalf of the very best
Deutschlandfunk.
You find two topics here on the slide.
I won't read them out, but they were what
people at the time thought were the most
pressing issues.
And we use the keyboard just like they
were in the telephone pole, and
our audit revealed that recommendations
become significantly more popular,
measured by views and likes.
You can see a steep increase from the
initial videos to the recommendations.
Note that we operationalize popularity
as the number of views and likes
we included, both because views are an
implicit measure of popularity, while
likes on explicit measure of popularity
regarding views.
It also remains unclear how many
seconds of video must be watched before
it's counted. We have a table here which
provides the median and mean numbers
of views and likes, comparing
the initial videos and the fifth
recommendations.
A substantial increase in views and
likes can be observed, especially
between the initial videos and the fifth
recommendation.
While the initial videos have a
median of nine thousand five hundred
views, the first recommendations
have a median of around 200000
views.
After following a chain of 10
recommendations, the views have a median
of almost 300000 views.
The number of likes increases
significantly to the initial
videos have a median of one hundred and
seventy likes, while the fifth
recommendations have a median of one
thousand four hundred four likes.
We performed two men, Whitney
Youth Test, which support the finding
that the number of views and the like
change between the initial videos and the
recommendation.
The audience also revealed that
recommendations become significantly
less related to political topics.
The median topic similarity
rating of the initial videos was
eight.
This decreased dramatically to zero
point eight three after following only
five recommendations.
The similarity remains very
low for the 10th recommendations, with a
median of one to
two. New tests indicate that
the topics in the videos changed between
the initial videos and the fifth
recommendations and between the initial
videos and the tenth recommendation.
So all these results indicate a strong
topic drift.
We also found that the happiness in the
video increased while the sadness
decreased, so the happiness
changes from a median of zero
for the initial videos to a median of
two for the fifth and tenth
recommendation.
So while seventy five percent of
the initial videos have a happiness
rating between zero and two,
more than half of the fifth and tenth
recommendations have a happiness rating
higher than two.
Regarding the sadness evoked by the
videos, the trend is opposite
the median ratings in the box plot in
the figure move from one point sixty
seven for the initial videos, down to
zero point zero for the fifth and
zero point three three for the tenth
recommendation.
So while more than half of the initial
videos have a centeredness rating, higher
than one point six seven seventy five
percent of the tenth recommendations
have a rating smaller than one
overall, in contrast to what the New York
Times reported.
Our findings suggest that the dangers
of online radicalization may
be exaggerated.
Now, taking a step back and taking the
power back, I want you to understand
that scraping audits and sock puppet
audits on my opinion the most
promising method to investigate complex
machine learning systems because these
audits can be used to identify
popularity biases like the one that I
showed you. But it can also be used to
see whether a system is enacting a
gender bias, or if a system has
a tendency to discriminate against
or towards a particular ethnic group.
So from your experience reading
the news, you know that controversial
political topics require a balanced
presentation of all arguments in
a way that weighs the pros and cons.
However, the audit suggests
that YouTube's recommendation system
is not suited to help users inform
themself about complex political
issues. Popularity is measured
by likes and views was the defining
factor for selecting recommendation.
And if this is the case, then minority
views are not adequately taken into
account by the system.
So the popular recommendation that you
see here are of course, attractive
for the majority, and this could
be motivated by financial incentives
that try to optimize the watch time
for broad majority.
Our audit corroborates Smith
at all, who also performed random
books. Smith, at old random walks
were criticized as artificial because
they relied on YouTube's API.
In contrast to that, we remote control
Firefox browser from a university
network.
So in a way, the audit is a prime
example for the recentering
of public engagement around the
complementary interest of the majority
and profitability.
And this connects to Harper's
investigation of the so-called big
data public and its problem
because, as I said, from a platform
perspective, we're longer watch times was
more shown ETS, which leads to
more money.
It makes a lot of sense to target the
majority.
One important limitation of my approach
is that we cannot rule out that a rabbit
hole effect exists for a particular
subset of users and topics, especially
for those where users actively looking
for fringe content.
And that's why it's important to
understand the users and their
understanding of the system, because
YouTube is a complex socio technical
system with human and non-human
actors who all influence how
information is accessed and understood.
And we also wrote a paper
on this in more detail where we examined
how middle age users without a background
in technology think YouTube works.
That is, we asked them why they
think they see the recommendations they
see.
So we found four big user beliefs.
One is related to the current user's
previous actions and how they influenced
recommendations.
The second is related to social
media. That is how other users
actions influence the recommendations.
The third is related to the algorithm
and what the algorithm regards
as similar who is similar, what
the similar and also what kind
of context the algorithm actually
takes into account.
And the fourth user belief relates
to the organization and the company
policy.
And it's interesting because especially
company policy is connected to
a lot of negative beliefs where
people think that YouTube is
actually selling the recommendations
and they think that YouTube has
psychological experts that just try
to keep them watching and watching.
If you want more, read the paper.
It's written by us, llevado
me, one of the best
fighter and katwe that there,
and it was published at the CCW
conference this year.
Now, here's my call to action.
I want you to collect YouTube search
results, video recommendations
and advertisements for different topics.
And I want you to do this without user
accounts and with user accounts.
And my goal is to systematically
analyze the recommendations by
YouTube's machine learning system.
And the next step after that would be to
design and implement and evaluate
algorithmic transparency tools
that help users understand and influence
their recommendation.
And in the following, I will show you the
script that I wrote, and I'm also
going to point out where it can be
adapted to not only study YouTube
and to not only study YouTube across
countries, languages and topics, but also
to study other platforms like Instagram,
like Tik Tok and a variety of more.
The audits, in my opinion, could be a
powerful tool to surveil
surveillance capitalism.
With the orders that I described, it
would be possible to investigate
how the content is targeted to
individual users, so the
audits could be used to explore
how advertisements are targeting
specific users, and this directly
relates to the dangers of the so-called
surveillance capitalism.
Shoshana Zuboff describe surveillance
capitalism as human experience,
which is used as raw material for
translation into behavioral data,
which are declared as a proprietary
behavior. So plus fed into
advanced manufacturing processes
known as so-called machine intelligence
and fabricated into prediction products
that anticipate what you will do now.
Considering these political and economic
forces, it's vital to investigate
how it are targeted to use
it. So these orders that are presented
could be a tool to investigate
personalization as well as
the user profiles of contemporary
surveillance capitalism.
I also believe that a foundation
for machine learning based systems
is needed.
And in the thesis I describe in detail
two different models that could be used.
One is following the German Association
for Technical Inspection the truth?
The other one is following the German
Foundation for product testing.
This all quite stiff compound and
both approaches could be used to make
sure that machine learning systems act in
the interest of society at
large.
To have institutions, for instance,
evaluate each car in Germany every
year to ensure that a car's street
legal.
The purpose of the German Foundation for
product testing is to compare goods
and services in an unbiased way.
So the truth ensures that something
complies with a certain norm, commonly
making binary decisions, whether
something is permitted or not.
The shift to bottom test usually develops
a catalog of criteria used to
compare different instances of a specific
kind of product or service.
So an expert consortium defines these
criteria for specific products or
services and a particular context.
Now, a foundation for machine learning
based systems could adopt these schema
and iteratively develop criteria
for the control of them based curation
systems. Audits could then be used to
make sure that the system is not
affecting a popularity bias, or
that the system is not discriminating
against ethnic minorities
or certain gender identity.
And I really hope that this talk will
inspire other researchers to examine
use this understanding of machine
learning based creation system or other
machine learning systems, and to motivate
them to design and develop novel ways
of explaining and auditing such
system.
But until these bigger
things are established, it's kind of up
to you and me.
So here's my call to civic hackers.
Use the script to investigate the
recommendations and the ads on YouTube.
And he has some ideas you could look at
fake news and pseudoscience related
to climate change or
the COVID 19 pandemic vaccination
in general.
The moon landing conspiracy or
the so-called flat earth theory.
So in the repository, there are two
scripts that I'm providing.
One is called cruel YouTube,
and the other one is called extract data
from downloaded videos
that both Python scripts.
So let's consider the first one called
Cruel YouTube.
As I told you, the goal
is to remote control a web browser
and be using the web testing
library selenium for that.
Selenium is also available in other
programing languages, but I'm using it
here by a python.
And this is based on a Chrome browser.
You can use different problems.
There's also an extension for Firefox
and others.
And in the script, you have different
parameters that you can set.
So here's the number of paths
to collect per keyboard, and I set that
to 20.
Then the number of search
results to consider, which is set to 10,
and the number of related videos to
consider which is set to 10, and the
number of related videos to visit that.
And that's the number of recommendations
we're collecting, and the is a bit weird.
Here are the different keywords
that we're entering into YouTube
to download the recommendations.
And it's quite easy for you to add your
own keywords, so you could just
say.
And then saved the fire.
And that would be sufficient.
I'm going to remove it for now.
If you just want to replicate the same
approach that I showed you in the
paper, then.
And that would be sufficient.
So when running the script, we randomized
to order of the keywords
and then we have the main loop here
where we select for each of the keywords,
we select the number of recommendations
that we specified and the parameters.
And for that, we start a new Chrome
instance and clear all the cookies.
That's what we're doing here, and
then we're taking the keywords
and we're entering them as a search
query to YouTube.
We are opening up the link,
the URL.
And we're waiting a bit.
And the reason for that is because
YouTube is dynamically loading a lot of
the videos.
So when the web browser
finished loading, they're still loading
going on in the background where a lot of
data is retrieved and that's what we're
waiting for here.
And I just basically wait
until the browser knows
there's an element called comments,
and he knows that by the it's an element
of the ID comments.
And then we're preparing a file name
because we can't just save up to the
file system.
And if
if we haven't already visited that
website, we're going to write that down.
We're writing the entire source of
the website.
And then we're collecting the top end
recommendations.
And then we're selecting a random video
from these recommendations, and then
we're following the recommendations up to
a certain depth.
And it's always the same procedure
will open the website awaiting
for a random amount of time.
We're waiting until we can see the
comments and then we're saving
the path and we're finding
one of the recommendations and selecting
one of the recommendations.
And it's quite nice because we can just
use the CSC classes
to find certain elements in the
website based on the idea
and based on the class.
And after that, we're not only saving
each of the individual videos that we're
visiting, but also the
path.
So which video led to
another and we're saving that to a fire
cord crawl under psychopaths?
So if you were to add that this could,
the easiest way would be
to edit your own keywords.
And, of course, to change the parameters.
But you can easily adapt this code also
to visit other websites like Instagram,
like Telegram and
then collect data through this
mechanism.
So the system selected education
policies, and it's now downloading.
The different videos.
And I'm stopping it here to show
you the downloaded videos.
And I do that with typing control.
See?
Let's have a look at the source code
of one of the videos that we downloaded.
And you can see that this is really the
whole HTML document,
including all the CIUSSS
and the JavaScript, and you can.
I find a variety of things
in the in the data.
For instance, if we look for the video
title, we find success for that
video title.
But we also find the actual video total.
And that's how our education system
is embarrassing itself.
And that's really what we're doing
programmatically, right, I and we have
the HTML and then we use
that to extract
certain information and also provide
a script to help you with that.
And that's a script extract data
from downloaded.
Videos.
So here we're using a Python library
called Beautiful Soup.
That allows you to pass HTML
and to be able to search the extent
efficiently.
So what we're doing here is we're looping
over all the videos that we downloaded
and then we're parsing the HTML
that we downloaded.
So what you can see here is with
selecting the number of views
of a video based on the
ID info text class
you can't.
So how do I know where the views are?
Well, it's quite simple because I just
looked at the source code.
So if we find a video that we're
interested in.
Like this wonderful talk by Dave Keyser,
we just look at the CSS
selectors by right clicking,
clicking inspect and cruel, but it's
the same in all the browsers.
So we have a span with
the class view count and
that's within a diff that's
called in full text.
And based on that?
Now going back to the source.
We're selecting the count, and we're
taking the first ones because there's
usually just one and we do
this for all the different things, for
instance, the date on which the video was
posted. The name of the channel.
The number of subscribers.
And as you can also see here,
it's a bit more tricky to get the
likes and dislikes.
But you can have a look at the code on
your own to figure out it's not really
rocket science, either.
So here are the references of the
paper.
I mentioned quite a large number
of papers.
So just quickly going to scroll through
them and giving you a chance to
stop the video to look at them.
And I again, invite you to have a look at
my doctoral thesis uses
machine learning based curation systems.
Thank you very much for your attention.
OK. Thank you for your interesting
talk, Hendrik.
And now
you have the option
to tell us some question
used to FC
three, CW
or hashtag on social media
or to your featured
to do so, and
there already are some question
and wish.
And the issue is, can
you please provide the link to
the slides?
Sure, yeah, I can upload them, I just put
them in the repository, I think there's a
link to the repository in
the video, and I can definitely provide
the slides.
Excellent.
OK, then the first question
how would you
want a platform like YouTube to
make minority views
more attractive without
also advert
advertising seminars, small
extremist views?
Very, very good question.
I think my main the main idea
behind the talk and also behind a
lot of the other research that I'm doing
is to give people more control.
And it kind of starts even just with the
knowledge that these recommendations
are selected by a machine learning
system, and they're selected
with a particular rule, right?
And that's the first step.
So everybody knows I'm seeing the
recommendations because there's a system
that's actually like Amazon trying
to find things that are similar to what
I've done in the past and just trying to
like, show me stuff that's similar to
what I've done in the past.
And I think that understanding is the the
first and maybe the most important
step. But the second step would then be
also to give people control over
what they're seeing.
And that would be
to just give them more tools
and more configuration settings
to decide what recommendations
they want to see and in what context they
want to see them.
And I think the the second question
of the more extremist as in like
far right extremist, I think that's
a that's a different issue in a way
that's kind of policing what's uploaded.
And that's a bit like
it's orthogonal to what I'm talking
about, but I think this is more related
to actually making sure that people
like them conscious. Apple thinks this is
not it shouldn't be on the platform in
the first place.
Right? So that's not a recommendation
system issue policy, but very
good question. Thank you.
OK. Thanks.
So next question,
what do you think about the recent
cat made in Germany
initiative that aim
to track the little responsible?
To be honest, I don't know much about it,
so I really can't comment on it.
I think the idea is in like having
like responsibility. I am all for that.
But it's something that's really, really
hard to do.
And I think a lot of people are working
on this actively.
But yeah, so the more the merrier.
So very much welcome that.
But I can't comment on that particular
because I don't know what that one.
OK.
And do you
also run audits
in which you choose to videos
of the same topic before and
picking then the rest of
randomly?
I didn't get that fully.
What do you mean?
Like, uh, you
you you talked about
how you do, you do the audits?
And the question is, did you
also run audits in which
you choose to reduce of the same
topic topic before
picking the rest of them randomly?
No, I haven't
done that yet, but I think it's something
that's worth doing.
I think the most important thing that I
really want to do in terms of what with
the audience is understanding
personalization.
So it fun.
I'm not sure when I created
my YouTube or Google account, but it's
been years that ten years or something,
right? So they really know a lot about
me, and I think
nobody really understands yet
how that influences the recommendations.
And I think it would be really
interesting if people with their very
old accounts, not just accounts they
created a week ago, but thinks they've
used for years and years start to do
these audits relating to the topics that
I presented here, but also related to
more urgent topics.
For instance, you can just go to the Ivy
Deutschland fund, which every
now and then is asking people in Germany
what's interesting? It's Representative
Paul by the idea of the video,
and you can then use these topics to see
what what people are or might be
googling or might be looking for on
YouTube.
OK. Yeah.
While doing that, aren't there any
limits?
YouTube imposes your
own holding.
Yeah, I mean, that's the thing.
I mean, there's there's different ways of
doing this.
And I think like I commented
on this, if it all works, which
we're using, the YouTube API
and the YouTube API has very clear
limitations on what you can do and what
you can't do. But I edit them
remote controlling the browser, so I
would. So there are no natural limits.
Of course, you will be blocked if you're
too eager, let's say.
So you should be responsible and you
should have delays every now and then
and take some time.
But there's no technical
limit, right? But it would be nice
and don't overload
the service.
OK, now we have the question,
gentlemen.
And Ithink a cookies lesson
that tarnished Krishnan unfun
column fund regulation appear excessive.
It seems like an awesome place, an IP
range. Even so, is Dennis
even suggesting its request,
even though is that in his request
relevant? What are these folks?
Let's just repeat it for the other
people. You must be on any speakers.
So the question is what about cookies?
And what about so like
that? Just deleting the cookies is not
sufficient because you're coming from the
same IP address and there's a lot of
things that are like really
limiting you here.
And it's just one limitations I have to
live with in this particular order.
I did not do that, and I did that in
May 20 and 19 quite
recently after the cabinets
incident. But that's also kind
of my idea of releasing the script
because there's so many things that
have an influence on the recommendations,
at least have a potential influence that
we really need a lot of people to
do audit studies to really get an idea.
And in a way, that's why I want other
people to be able to do these kind of
things because I wholeheartedly agree
with the comment and I think that the IP
makes a difference. I mean, I know kind
of scientifically said, OK, these are
the limitations of our approach.
This is what we know.
This is what we controlled for.
But but yeah, probably it has an
influence, but we don't know.
So I think we just need a lot of that.
If people kind of Wikipedia style to
to to go about these problems,
about this problem.
OK, thanks.
Then
the are a lot of question, how
how you did this,
like did you or
how you two acts in doing
this and maybe did
you run into any counter measurements
against automation
on YouTube?
No, actually not.
I think that's in a way why this is such
such a nice hack in
that like we just use selenium and a lot
of people use selenium as part of
the web testing.
It's a very, very widely
used tool, right?
It can be part of your continuous
integration workflow, just making sure
that there are certain things about your
website's work.
And it's quite hard to block in a way
because it's an extra Firefox.
It's it's not just trying to be Firefox,
it is Firefox.
And of course, you can like if you
look at the behavior of the user and
it's quite artificial in that way and you
probably can detect it.
But when we did this, we didn't see
anything. I mean, you definitely need to
look at the legal situation and make sure
that you don't the and obey with all the
laws. But
for this academic purposes that we've
been doing this, we didn't see any
problems.
Next question, you two might be
using several different animal
Rothman and keep
changing their tech.
How would your research
address this?
Again, this is limited in a way.
I mean, we did this at this particular
time and with this particular
purpose. But that's also one source
also open sourcing it, right?
I want other people to look at this, and
it's known that there is a lot of AB test
and there's probably dozens
of versions of YouTube running at the
same time targeting different people.
But in a way that's just showing how
important this kind of research is
because again, like it's billions of
people using this, 70 percent
of the videos watched recommended by the
algorithms, and we don't know shit about
it, to be blunt, anyway.
So, yeah, I think just take the script
and then have many people do this.
Yes. Yeah.
Thank you. And.
Yeah, there are.
Oh, that's a new question and
why it's crawling with chrome,
no problem, there's
no money.
Surfing with a Tor browser is
a constant.
It's constantly
blocked.
It's Tobruk.
What's the what's the question?
Surfing with a Chrome browser is no
problem, but using Tor browser
is most of the time
problem because there are captures
and verify and
it's constantly blocked.
I have no idea, to be honest.
I know that you can use selenium of Tor
and also know that it can be interesting
because you can get different endpoints,
of course, so that can be quite useful.
But what actually YouTube is
doing to prevent people from Tor and why?
I don't know.
Yeah.
Then there are some people in
the it's
telling you that it's a
really nice talk.
Thank you very much for that.
You're welcome.
Um, yeah,
are there any question
for this Q&A session
left?
You can go to the sea
chat.
It's linked in the streaming
media khedive
page or do it on
social media.
And I put the slides and the repository,
I can just look at it and people can find
it. And yeah, especially if you it's a
media science students and the like,
you see scripts and then do exciting
stuff because there's a lot of
interesting research on Twitter,
for instance, because it's quite easy to
do this kind of research.
And I really hope in a way with releasing
the scripts that
I make researching YouTube a bit easier
and hopefully also instrumental, because
then in principle, it's really the same
in a way, you just toss in a while
and and look for the HTML
elements like I showed.
You need to know manner between Python
a bit, but you can only do so much.
OK, a new question
up yet doesn't
randomly choosing a recommended
video has its own bias
in. This may prevent
the other angle
from learning a user preference
and on a rabbit hole.
Yeah, very good question.
And definitely it has an effect.
I reflect on that in the thesis, and
in a way it's a conscious decision,
right? I mean, you can do a lot of
different things, and
that's just the one way I try to, which
I thought is the most interesting way
that I can do now.
But it definitely has an effect and
definitely is also quite different from
a human being on the website,
right?
But again, that's I think why this is
only one small
piece of the puzzle, and we definitely
need more than that.
Another question.
Did you investigate on the effect
on having German IP
address or the
browser language in German?
Mm hmm.
I had had a university
IP address.
The browser was actually set to English.
My whole system was actually English at
the time.
I'm acknowledging it again.
And in a way, this is like this puzzle
piece which has this that this different
these different settings, but we don't
know what difference that would make,
right? Do we get different
recommendations?
And it's about it.
OK.
Then thank you
very much for your awesome talk.
And here.
Thank you.
Yeah, and tough, beautiful
as we speak.
Same to you and happy hacking to
everybody, and let's hope we
can do a bit surveillance of surveillance
capitalism.
OK, bye bye.
Bye.