[Music]
foreign
whose passions include art and code and
computer music
combined these ideas
in his PhD thesis on authoring
interactive media
and he is now an interact an independent
researcher and is with the authier io
non-profit where he is a lead developer
on OCR the open software system for
interactive applications where tries to
bring some of the results of his
research to the broad masses of
interactive artists out there
in his talk he will show us that
interactive media can be so much more
than just a movie with three different
pre-recorded endings you can choose from
yes
hello and welcome to Jean Mikhail celery
hey thanks and this was I think the best
intuition I ever got so I try to
remember it for the next time so hello
everyone so um
um I'll be mainly representing and doing
a small demonstration of OCS score which
is a free software that we've been
developing with
um the OCR that I will profit for
something like well we are not far from
10 years now which is a kind of free
software sequencer to combine various
kind of media we call that interimedia
so for instance video sound but also
controls with Hardware devices and
inputs that people may not expect at
first to be part of interactive artworks
um so it's the screen share
um
okay so
um for the first few minutes I'll
quickly explain how the store looks and
for something like 20 minutes I'll go
and try to build some scenes from
scratch to show what are the
capabilities of the software
um then we'll be able to discuss and if
you have any question I'll be super
happy to answer them
so V software is a sequencer so if any
of you do music you may know the simple
Paradigm of
putting a sound on timeline hitting play
and
[Music]
so um this is the most simple thing one
can do but
what is less common in this kind of
software is also the ability to put not
only sound but also video content
this for instance
[Music]
so I can also come and take some video
effects
and say okay you will play at the same
time than my sound
and
of course with various levels of
controls on these systems so this is the
the very basic the the very gift of it
and now I'll try to to build something
that
well bring some sound so first the first
thing that one has to do when using
score and if anyone wants to follow the
same steps while I'm doing the
presentation uh the software is uh
freely downloadable at uh
the
um.io website so feel free to to to to
strum to roam along and try to do stuff
at the same time
always good to have people trying it and
so the first thing that one may want to
do is to First Define which kind of
peripherals we are going to use for our
interactive artwork so for instance here
on my desk I have a joystick I have a
keyboard I have this little thing which
is called a lead motion which is a kind
of camera for hand gestures and all
these things have starters that we are
going to be able to use as part of the
performance so for instance if I add my
lead motion here and
you can see some tables of number moving
and that is the position of my hand for
instance things like that
um
this I also want to use my trusty midi
keyboard
and I will learn some characters so
I'm hitting the parameters that I want
to use for the controls
[Music]
okay here so this gives me an array of
things that I can use to map to to sound
effects things like that and what else
do I want to use
um
let's see how I want to use a joystick
okay so this is basically the raw
material for the interactivity of the
work
oh
okay and I'm getting a fantastic demo
effect that is
great
where are we okay
this and
[Music]
and
[Music]
so I'm mostly using Linux and the
software is mainly developer neural
Linux but if you want to try on other
systems of course it also works on Mac
and windows and we are especially in
need of people who um try it on a metal
like raspberry pies and things like that
and if you want to try on value somebody
important report then please tell us any
super useful things to know
so to get started I'll just put a very
simple drum Loop that I
will be a modifying over time so
here we have all the things that
we can use we call them processes and
those are things that will we'll just
basically
trigger things at various
times so for instance if I want
to have some sound okay I'll just do a
piece
um
[Music]
okay so this is a bit too fast
so this is very very simple
um
something and now what if I want to
control
this with some effect for instance if I
want to apply some reverberation
or maybe some
bit crushing it's much more fun I think
okay
so now there is something running and
now what if I want to for instance
control this parador with say my major
stick
then I can just go inside here
and take okay
I have my joysticks access so for
instance when I rotate it like that
and they can just drag it here and
I could want also to apply some effects
on the
River operators
say this on the witness of my River
[Music]
okay and this gives us a very simple
basis of work now what I want to do is
to of course build something a little
bit more elaborate on this
um so I'll start by adding some drum
Loop
[Music]
because this is going to be very boring
very quickly
so
[Music]
um
so at any point one can just press play
here
[Music]
and if I do this it will play only one
time so what I want to do is say okay I
want you to to keep running forever and
do that
[Music]
and I'm getting some
code from images
[Music]
okay
all right now
um another thing that I may want to do
is to play with live instruments it's a
for instance
[Music]
on that
so what I need to do is to add
an audio device
and on this audio device so I know that
my best guitar inputs
is
um one two
[Music]
like that
okay
and I will add a
Looper here so I can do for instance
some level of live looping
and I start my looping thing only when
I'm ready
um
a whole bar and
play folder please and we'll start at a
very low volume
that's
[Music]
okay
so now and here it's waiting for me to
start recording things
[Music]
foreign
[Music]
okay and now we can start
playing some effects for instance
[Music]
okay now we can also
try to add some
[Music]
foreign
so for instance I have a set of patterns
so
I want to do another pattern
[Music]
um
[Music]
this is a very simple
pattern
um
synthesizer very smart
[Music]
so I want to focus on anything in this
sound
[Music]
um
[Music]
foreign
[Music]
thank you
[Music]
okay
and it seems that when I have a
not set the right pitch but it's okay
[Music]
your place
foreign
[Music]
things like that
yeah
so something like this
[Music]
awesome
[Music]
now I can just take here
[Music]
thank you
[Music]
I can start to have two things that will
leverage various kinds of hardware and
build some kind of live media systems so
this is the first example let's do a
second example quick and this one so
maybe some of you have recognized this
musical team and if not I will just say
that it is from Robert Prince from the
90s
um
all right so we'll just leave that
uh so doing this means that this part
will never never play and it won't even
use CPU resources and now uh what I want
to do is to use not only
um
my best as an input but also
um
something else
so I think this
so I'll name it game
and it will use my three and four inputs
okay you don't care about you uh
and we want to add a new scenario
so we are going to do our second work in
this Ang slate here okay and here what
we want is to
um
have some video input so I have some
Scripts
to Launch
a
um
you for instance
yes
yes all right and
this
okay
[Music]
let's see you
I'll go here you will go here all right
perfect
okay and um
this
and now we want to use not only audio
content but also video content so I'm
going to add some video inputs to my
system okay
right here and what I can do is say
um okay
as soon as as soon as this score runs I
want to play my audio input
okay I'll just put again effect and as
input I'll say
game and let's see
[Music]
um I will also add a fuel filter here
[Music]
for instance so you can use here and
stuff like that
um all right
yeah I can start
[Music]
okay
[Music]
um
other things as we go into it and
process them
for instance
[Music]
would you for instance if I want to uh
add some effects from The Sound output
[Music]
and if I want I can even use
[Music]
like visible matters
say Okay I want my audio inputs to go
and control for instance
[Music]
that if I
enter some zombie things
yeah as you can see
awesome
video control and stuff
[Music]
so yeah so
um basically score is a software which
helps to combine all these kind of
[Music]
media systems and things like that and
put them in a timeline and score them
over time and we'll do um
do a kind of live situation with various
kind of media then well make scores of
it basically
so that's the gift of it I'd say so well
if there are questions I think I can
take them now
thank you for this interesting short
introduction into some things that
OCR can do for Aussie a score can do
remember for the viewers out there you
can ask questions in the IRSC on Twitter
Mastodon
and yeah one one thing I noticed you had
just this presentation was mostly audio
and video because that is that worked on
a online conference yeah but if I
understand correctly the intermediate
aspect of Austria encompasses far more
than that yes so basically
um so we have this abstraction of our
protocols so and so in the system you
have all these set of protocols so uh
for this uh pressure thing I didn't use
any kind of for instance Network
protocols but it's super common in
intermediate systems to use for instance
the osc protocol which is a standard
protocol across media apps to uh to to
send my chat messages to each other for
instance one can have a pure data
program or a processing program and
communicate
and Discord or send control message to
to us here there is also built-in
support for various kind of others so
joysticks remote so leave Motions like
this and well we are working on the
immerse so you can also directly use
arduinos and things like that so if you
have an Arduino you can just plug it and
start exchange exchanging messages on
the serial Port without having to go
through some sort of bridge
uh which well is useful when you do
stuff like kinetic installations things
like that and uh
and we are looking forward at adding
more protocols so for instance there has
been more and more demand for iot
protocols for instance to like mqtt
things like that
um because people don't only want to do
art but they also apparently want to
control their house and turn on the
lights at the right moments and things
like that and for this and qtt and
things like that are super useful
protocols to have and then it's an open
system and as soon as there is something
useful that we hear of then it's going
on the world map as yeah it could be
cool to have that new way to control
things and
um yeah yeah uh well there's also
support for um light
um you know DMX for stage lights things
like that so you can directly automate
for instance let's say
um
um if you have projector you can
directly control it and send the DMX
data through artnet or relevant protocol
so we try to make it usable for any kind
of art show and things like that museum
installations so yeah
and regarding all those features is it
can you say that osseus score could be
something like OBS Studio on asset
hmm good question well
um for me it's more complimentary I I
use OBS quite a quite a bit um for
streaming things like that and for
instance from osia you can so OBS has
this uh uh osc over websocket add-on and
you can edit nausea and control for
instance these scenes of OBS so the idea
is more to integrate with that kind of
software and if for instance Jose isn't
going to have streaming capabilities I
think to Twitch or things like that like
OBS has and it makes more sense for me
to
score OBS if that's needed but leave to
OBS what OBS does Best But then if
someone wants to come and have nice pull
requests then I will be happy to merge
them
so it's more like a stream Jack on asset
where you control OBS if you want yeah
with everything in your environment but
well that's that's today at least and I
don't know what we feature has in store
so
[Music]
another question more related to the
music part
um can you read classical Street Music
and do you think that classical music
education is useful or necessary to
produce music oh in general or in this
context in general I'll say yes
um personally I didn't have a classical
music formation and actually I regret it
a bit and some things will have been
much easier if I had taken a few classes
on Harmony and things like that but for
using score it's not necessary so in
score you can import MIDI files for
instance
let's say I should have some okay so a
very simple midi files like single
chords
or um do we have some midi here
um okay well you you can write anymore
if you play it because then we get
breakdown notices oh yeah yeah that's a
very good point
um so yeah so there isn't
um direct shipmentation however there is
this project which I think was developed
at uhm
or which is a gido I'm not sure if it
was developed at that no it was
developed at Gram and it's an open
source system for rendering notation
it's used for instance in the inscore
software and well it's been a requested
for a long time to add the good delivery
to have some level of rendering of sheet
music but
um personally I am barely able to read
sheet music so it's not a super high
priority for me my needs are covered
with simple MIDI files and things like
that but
for instance we are working a lot with
um
let's say composers from
non-classical PCS and the
music and the Contemporary asthmatic
music concrete things like that and
we've been working into for a partition
[Music]
um
score
score rating system for contemporary
music which is called a cosmoscripts so
I don't have the demonstration here but
it's a separate way of rendering scores
adapted to contemporary music and it was
relatively easy to integrate it's it was
done by a few students during the
student project this year and something
similar for Sheet Music will be
meaningful I guess if someone wants to
do it so again I'll be super happy to
accept any people requests which
adds some hit music rendering to MIDI in
score and things like that
yeah
talking about scores for songs can you
arrange a complete score in a complete
song in Austria with some interactive
elements and then for example some
actions at specific points yep so for me
so that that's my goal so I come from
the um back in my proprietary lifetime a
decade ago I was using a lot I built a
knife to make music and
I really want to reach the point where
what you can do in Ableton you can also
do it in score maybe not as easily
because the not everything is the same
of course and both Authority things
differently in some ways but at some
point I want to be able here to write a
whole song in it what's missing as of
today is a good say mixing user
interface for instance to easily have
like in all traditional sponsors there
is the idea of tracks and things like
that that um
is always present and in score series
not really well there are things similar
to tracks like you can say okay
you will
you for instance will be amazing this
and then you can adjust the volume in
here but
that's not enough I think this and also
we lack and that's the focus version all
the uh recording things like right now
you can't record audio or media during
things current I want this to be labeled
because for me that's the final frontier
before starting to actually make songs
so it's on the roadmap uh I'd say that
today you can do it if you are really
dedicated and if you say user audacity
or something like that next to it to
record uh songs and at some point I want
score to be a fully legit digital
workstation but it's not there yet and
working on it
and just to give maybe to the viewers an
idea of what are the things that can be
done with score I can maybe quickly show
the gallery on the website which shows
various PCS that were in for instance
there are
modular PCS there are live performance
over the Internet for instance
there are in museum installations there
are plays stage plays things like that
um
yeah various kind of things but there
isn't a single song in all of those so
it's always there's always often some
musical elements but I don't think
anyone wrote an entire
no there isn't there is piano chronix
number three from Illuminati which is
actually a piano piece which was
composed in this course so there was one
so yeah
score uses a graph editing feature which
is probably very easy to get started do
you ever get lost in complicated graphs
and hope for S Expressions ah so very
good question and uh
um so
um if you are used to say pure data
Maxim SP things like that for me so as
you can see it's graphs but what you'll
notice is that for instance the graphs
that you have in score the objects are
much bigger than the objects in say
create our Max MSP and the idea is to
have it's closer to you know devices and
other software so you don't have for
instance something that just does a
multiplication or something like that
you really have
um high level components which do a lot
of things and which are I think more
disabled by composers and musicians and
if you want to on one hand if you want
to get down to the um
so let's make a new document see if you
want to get down to uh more precise
level you can just drop your data patch
and you can actually edit your data
directly in score and uh
so this one is a
light noise patch for instance
and at the other end of the spectrum if
you really like coding then you can use
for instance right now A JavaScript
so you can have some
some GS script which will well generate
things
and if you are really motivated but here
I'm not sure it will work on that
computer
not confident no it didn't work but
um you can use also C plus plus and at
some point there were since the last two
months I had three or four requests to
introduce some
scheme dialects I I've been it's been on
my roadmap to have some level of
yeah having an embedded lisp
um
but then I'm not a big list person I I
really like the language and I always
think of the xkcd comic where you see
this Bliss of
people being in God being in Bliss about
the universe being made of s Expressions
but I'm more often imperative and
object-oriented person but I recognize
we need for it and I think at some point
there will be a a way to to also put
lisp in here like you can put GS and you
can put your data and it will be another
language that you can use
sorry
um another question is if can you use
Aussie adjust for example for the video
and then link it somehow to Ableton or
bit wick for the music performance
um absolutely so for instance in um so
if you want synchronization on Linux and
for now it's only on Linux we support
Jack transport so you can say okay that
score will be say a easier client of
Master but well you you want something
to be the master in that case because
you want to drive to have a single
timeline for your whole system and then
there are a few things that allow for
instance to well for instance if you
need to play back video
um well you you just do it and if also
what case we had recently was wanting to
use score with touch designer and we
support protocols for sending and
receiving videos for instance for the
Doom screen I was doing it with foreign
there is also experiential ndi support
to be able to send video for Network
then in France we've been working on a
public well kind of publicly funded
research project to
allow how to say mainly it's been about
making networked artworks and basically
artwork Square you have three peoples in
three different cities because well
covered and all that led to the need for
such things and as part of this there is
of course research on
all that we needed in terms of you know
protocols for exchanging various kinds
of media between software so at some
point it should be possible relatively
easy to say Okay I want to maybe
generate one sound at one point in score
and I have most of the sound being done
in Live or between and things like that
so
theoretically possible I haven't done it
much
but I mean I know that there's people
like in this Gallery page here there are
some stuff that use both score and live
in the same in the same performance for
instance so it's not unheard of
there's another remark from the chat in
this same direction not quite a question
but he says that yeah integration is the
the answer you've showed all those
protocols that RCS supports and midi
alone can assume many of these things
yeah yeah well midi just has this small
resolution problem but
um yet at first when the first version
of the software it didn't even handle
um
audio or video first it was only a
controller for other software
um but then at some point we had a lot
of pencil who were like okay but I just
wanted to play a sound at five seconds
and I don't want to open save pure data
and to make some player and then we
added sound and then people say the same
thing with videos we also added the
video and now for simple things well you
can do already a lot in score but if you
have a super complex uh video or audio
workflow in live between touch designer
or whatever software you are using then
by all means use it because at first the
software is a controller for other so
far so it's it's original
yes okay
[Music]
[Applause]
[Music]
[Applause]
[Music]
[Applause]