Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2126 Thanks!
All right. Many, many, many petabytes
of data is
what we're talking about this time today.
Excuse me, terabytes, I'm getting mixed
up here. Anyhow, here's James Wrigley,
who works for the data analysis team at
the European eel
that is the largest generation of light
sources, and
they have really big
loads of data easier to handle.
And he's going to tell us today how to do
it. And
actually, he wrote, one
of the things that fetches his cold,
cold heart is that each top
showing Al Gore is running at 100
percent.
This is what I suppose this is going to.
So here we go, James, it's yours.
All right.
Thank you very much.
It's my and strength, and
I want to talk about how to
drop or understand and experiment.
And if you
which basically means that I'm going to
try and give you the perspective
that I'm going to try and walk you
through and experiment from the
perspective of of the facility.
So I'll talk about the rate that
we have here, the data that it creates
and the systems that we've built to
manage all of that data.
So I will start off with a
little personal background.
I am originally from New Zealand.
I started working in the Data
Analysis Group at the European Expo
this year, and I'm
what's called an instrument contact.
So that means that I am assigned to one
of the six instruments at European
level to help them with their specific
analysis needs.
And so far, that has mostly
involved working on things related
to online analysis, which
is the real time analysis that's used to
guide and experiment.
But we'll get some more of that later
from day to day.
One of the most common languages I use
would be Python, along with
C++, and
with regards to tools, I can
generally go for Emacs with
Konza and the other.
So some of you may perhaps
not be familiar with what I
feel is so I'm going to give a quick,
quick, quick introduction to that.
So X stands for X-ray
free electron laser.
It's the latest generation of what are
called light sources, and it's
basically a massive machine that is
designed to create very
short, very intense pulses of
X-ray radiation, which is used
to study various physical processes.
There are other kinds of light sources,
such as synchrotrons, for example,
but some things that sit expels apart
from those.
For one, the very short pulse thing.
And this is often on the order
of 20 seconds or tens of femtoseconds.
And every one of these X-ray pulse
X-ray pulses is really intense.
So there's lots of photons in there.
The radiation is coherent.
That's very important for experiments,
and it feels also
capable of a really high repetition
repetition rate, which is
on the order of a mega-hits.
So that means we're able to create
a lot of pulses in a very short amount
of time.
And on the right hand side, I just have a
little picture here of one of the X-ray
detectors and X. So this is
the DFAC, and we'll get back to it in a
bit.
But here's just an example of the kind of
data that you could expect it collects
at an expo.
So that brings me to the European
Expo, which is the world's most
powerful X-ray free electron laser.
This is actually it is
actually located in happened,
and the whole thing is
about three point four kilometers long.
So this all starts at
the dizzy sites and barren fields,
and there's a machine here called an
electron gun.
What this does is it injects
electrons into a linear particle
particle accelerator, which is located in
this section.
And I have here a little picture just
showing the the inside of the tunnels.
This, the entire facility is is
underground.
And the electrons travel
through these superconducting
RF cavities, which accelerates
into very high velocities.
And after they've been accelerated,
they entered what we call
the undulate systems.
So this is a picture of
one of the insulators
and the electrons travel through this
little tube here, that little silver tube
and an undulated is basically an array
of magnets.
The magnetic field that these magnets
create, which on the top and bottom of
these metallic slabs, the magnetic
field that this creates causes the
electrons to wiggle, and that
motion for acceleration creates
X-ray radiation.
And that's the thing that we really want.
So after the radiation
is generated from the electrons, the
electrons attempts, we don't need them
anymore.
But the radiation can continue down
these tunnels into the experimental
hall at the second
external sites in Shenfeld.
And here I just have a picture of one of
the experimental hatches in the
experiment hole.
So there there are six different
instruments at the European level,
which focus on different kinds of
science, and each
of them have their own experiments, each
with their own detectors and equipment
and so on.
And here you can see an instrument
scientists looking into this little box.
This is the sample chamber, so that's
where they would place the samples.
The radiation would come in from tunnels
from the right hand side.
It would hit the sample in here and
then the result of that would be recorded
by an X-ray detector.
So that's the facility overview
of a very, very simplified version.
Let's get to some of the technical
details.
So the the
accelerator that we have is
capable of delivering laser
pulses at 27 kilohertz.
That means 27000 pulses
of radiation per se,
but they don't arrive continuously
there. They arrive in batches
and these are called trains.
And in this diagram, you can see
something about the structure of the
trains. So the trains arrive
at 10 10 times every second
and each. Each pulse of X-rays
is compressed with between within this
train, and
the repetition rate for the European zone
can be as high as four point five
megahertz.
And this is really unique and it's really
important for certain kinds of
experiments.
The Pulse then also very,
very short. So they're on the order of 10
to 20 seconds or even lower.
One thing that you probably
might have noticed is that this whole,
this whole machine is extremely
complicated, and
there are literally thousands
or tens of thousands of different
devices that produce data attacks.
So this can be
either hardware or software or devices.
But all of them are controlled
using in-house control systems.
It's called crabber, and it was developed
from scratch with an excel.
And on the right hand side, you can see a
little diagram of the
devices just for one instrument,
just for the MIT instrument.
Each of the notes here represents
a single device.
And the lines between nodes represent
some data that is being shared or sense
between devices
and the SIT.
And there are similar graphs you
could generate for the other five
instruments at exfil.
So it's it's a really
complicated system that produces a lot of
data that this is all necessary
to carry out the experiments that we
do.
So the experiments usually last about six
days, and they
can create anything
from hundreds of gigabytes to petabytes
of data.
In fact, recently we
actually broke our internal record for
the most data collected during a single
experiment.
And there was there
was some use of the media who recorded
data who recorded about three
point three petabytes of data in
just six days, which is really
insane.
But that does beg the question of
what exactly is generating
all this data. Right?
And the answer is that most of
it is coming from 2-D detectors.
So let's talk about that for a second.
There are a bunch of X-ray detectors.
We have an excel and I've listed some
of them up here.
On the right hand side, there's a picture
of the disk to see
this. Just this detector was
actually developed, especially, for
example, like also
the agent.
But there are some others that are
slightly more commercialized.
This is the detector is
one megapixel, but
in general, the resolutions can be
from. It can be anywhere from half.
Megapixel to four megapixels
and the most of the most common one
megapixel detectors like the DNC.
What we call pulse results.
So that means that they're able to record
data for every single pulse
within a train.
And this is really the high repetition
rate comes in because you
can imagine if you have lots of pulses
within a train, these trains are coming
ten times per second
and you're getting a megapixel data
for each one of those.
That's really a lot of data
for the DSS to see, for example.
It's capable.
It's capable of of holding
frames for eight hundred pulses
at a time, so it's got 800 memory cells.
And if you do the math,
this works out to create
about sixteen point seven
gigabytes of raw data
per second.
That's like, that's really, really
a lot.
Another way to think about it is that
it's about two and a half of bitcoin
blockchain per minute.
And keep in mind that this
data is just coming from a single
detector usually
attacks, so there will be multiple
experiments running simultaneously
between the different instruments.
So you could very well have situations
where there's three detectors that
are all spitting out data at
gigabytes or tens of gigabytes per
second.
So a big area of interest for us is
data reduction because this
this data with this kind of data is
really difficult to manage.
And in some cases, there are ways that we
can actually reduce it ahead of time, so
we don't need to store so much raw data.
And to give you an idea of
how much this is in proportion to
the rest of the data, I've got a little
table here which is
showing the rough grouping of
all the data and excel.
So each row here
vaguely corresponds to a
group of some kind of data.
So we've got data coming from the agents,
the DSC, LAPD and the young
pharaoh. These are all 20 detectives.
And then we've got this
other source in here called DEA.
And this basically
represents all of the non detected
data.
So that would be things like,
excuse me, that would be things like
murder positions,
various other sorts of information like
beam energy, that sort of thing.
And in total, we have something like
60 petabytes, a dead end zone.
And you can see in this column here,
which are showing the total storage
amount the nuns
detected. Data only takes up about
three petabytes, which is really
not much
so.
So that's why data reduction is it's
such a big topic for us at the moment.
And you can imagine that with these kind
of veterans like, you know, 16
gigabytes per second, the need to be some
special systems to be able to handle
that.
And to do that, we have
a special computer environment
and this is called the online cluster.
So the online cluster is
a cluster that is physically located
on site in Chernobyl,
and it's used for running all of the
systems that we need during
an experiment.
So of course, this would be the devices
which are run by the control system
or other, but it's also things
like online calibration and online
analysis.
So there's about a hundred and fifty
nodes in this cluster.
Most of them are CPU.
There are a couple of GPU units,
but those are mainly used for
the online calibration pipeline.
For the sake of security, it's
quiet for the sake of security,
access to it is quite restricted,
so you need special permissions to
access it remotely.
And there's also no direct access
to the outside internet.
When it comes to storage,
the main file system we have is
this, which is a proprietary distributed
file system.
And this is all running on systems
for performance because we need to write
a lot of data really quickly
and again for performance we use and
which is a different
kind of networking technology that's
designed for very high speed data rates.
And and here is
a grossly oversimplified
view of the different systems
that we have running on the online
cluster. So all of these would be active
during an experiment.
So something on the left
hand side, we have devices.
These are things that are running in the
control system and they produce data.
They could either be
representations of physical hardware.
So it may be a digitizer
or a camera or a detector,
but it could also be a purely software
device.
So let's say, for example, that
you want to do some image processing
on data from a camera in real time.
You might have a software device to do
that.
And the instrument scientists
who are operating the instrument
are able to select the data from
which device they want to save because,
you know, there can be hundreds of
devices per instrument, and
it's not usually necessary to just do all
of that.
So they will select a set of
those so that certain subset of devices
to save data for.
And this will be used by the DEA Q,
which is the data acquisition system.
And one of
the DEA main responsibilities
is aggregating the data from all
of those devices and then writing
it to the online class.
The storage here, which, as I said
before, is
which is which is using GPUs
running on top of these.
But feedback for the instrument sciences
is also really important during an
experiment, because getting beam
time at the European Excel is
extremely expensive.
If you do the math, I think it works out
to something like thousands of euros per
minute.
So it's very important that the
instrument scientists know exactly what's
happening at any given point.
And that's the point of online analysis,
and the online analysis usually
requires some data from
a 2D detector, for example,
and the raw data that
the detectors spit out isn't
really usable.
It needs to undergo some kind of
calibration, and that's
what the online calibration pipelines
for. So this thing takes the road to
check it out from the day you
recalibrate it, and then they can be
fed into whatever online analysis
tools we have running.
We've actually done a lot of work over
the last year and upgrading this for
better performance.
And the end result is that
the instrument scientists are able
to get some feedback from
from the instrument on the order
of a couple of seconds, which is
very useful.
Now, the last part that I want to
talk about with regards
to this diagram is this
process that we call migration.
So the storage on the online
cluster is not
meant for long term use.
It's really just there for the
data that's being saved right now during
an experiment.
There is a separate computing environment
called the Ofglen cluster, and
this is used for long term data storage
and for any kind of more
rigorous final analysis.
So there is a process called migration,
which will basically
copy all of the data from the online
cluster to the
to the off-line cluster.
So after some data has been recorded,
it'll just be copied and there will now
be accessible on the same cluster for
for scientists to analyze.
And now I
want to give a couple of examples of the
online analysis because I think that
can be quite interesting.
So the first example I want I want to
show is this thing called bracket
analysis.
So just for background,
when you when you fire
photons at some kind of crystalline
structure, they will reflect in a
certain direction and
on the detector.
This shows up as basically
it basically shows up as a blob.
And for certain kinds of experiments,
it's very useful to able.
It's very useful to be able to track
the features of this blob.
So by that, I mean things
like that position.
So where is that on the detector?
Often you do some curve fitting
to to attract people.
So in this in this particular case, we're
fitting a Gaussian through it and we're
getting this ventricle of the full with
that half mix, which is basically
the width of that.
Brad Pitt.
And then another thing is the intensity.
So how bright is this paper?
And on the right hand side, you can
see an example of some analysis of
a Brad Pitt that was done during an
experiment.
And what was really cool in
this screenshot that that I have is
that you can see how the position
of the Brad Pitt changes
as the instrument scientists remove a
particular motor.
And being able to view this kind of
data in real time is
extremely helpful for them to know that
they're actually doing the right thing.
So that's that's one kind of analysis,
another kind is this thing
called a musical integration.
So in the last
slide, I showed you just like a single
little blob on an unknown detector.
That's just a bracket.
But in some kinds of experiments,
they actually create lots
and lots of brackets, and this
ends up having
and the data from the detector ends
up having radial symmetry.
So these rings because scattering rings
and they're actually composed of lots and
lots of frag peaks, all at the
same radius from each other, and
that this kind of experiment, what's
really useful is to count, for example,
how many rings around the
radio position of the rings.
So how far is the ring away
from the center of the beam and also
the intensity of the rings?
And you do need to have good geometry
for this is an accurate geometry
because it's quite sensitive.
But as long as you have this circuit.
And so just to expand
a little bit more as a movement
integration, as I mentioned,
you're taking this this image that's
that's on the right hand side with some
radial symmetry and
you converted to polar coordinates.
So one way of
thinking about this transformation is
imagine you have a circle, then you
make a cut at one point in the circle
and then just unfold that circle into
a line.
That's kind of the transformation
that's happening here.
And what you then and what you what you
see happen is that the circles
that show up in the image with Cartesian
coordinates end up in straight lines
and probably coordinates.
So on and
on the bottom axis, you have the radius.
This is the distance from the center of
the beam and on
the vertical axis you have the angle.
This is the azimuth angle.
And what we want to do in this case is
just integrate along the
as neutral angle to get what's called
a 1D scattering curve.
That's something like this.
And you can see that
the bright lines in this image of the
polar coordinates basically correspond
to the link to the peaks of the visible
and the scattering curve.
And this is really what what
instrument scientists want to see.
So they want to know how many peaks
are out here. Like roughly speaking,
you could say this may be three big ones.
They're one of the they want to know
their position. So how far away are
they from the center of the beam?
And they wanted to know the intensity.
So how tall are the peaks?
And this can all be done online
and semi real time with a couple
of seconds of latency.
And I have a very
simple example here.
This is using a slightly different
dataset previously, and
this dataset you can see there's actually
many different scattering rings
in this dataset.
There's only one fairly broad
ring, and you can see that
that that's showing up quite clearly and
the one scattering curve.
And we also had tools for doing peak
funding automatically so they can find
the position of the speed,
like how far away is that from the center
of the beam?
And if they want, they can also do
some curve into it.
So in this example, they've
just put it a gauss into this curve and
you can see the parameters that it spits
out.
So that
is online analysis.
So this is stuff that happens
during an experiment while it's running.
I want to switch now and talk about
the different environment that we have
for doing offline analysis,
and that is the ultimate cluster.
So what I'm calling the the
offline cluster is actually the
Maxwell cluster, which is a computer
cluster that Visa operates
for its users.
And we like exfil
worked very closely with this.
We share a lot of the infrastructure,
and that's that's the reason why we're
using the Maxwell cluster here.
So we use it for long term storage and
for any final analysis,
primarily because it's just way bigger
and more powerful than the online
cluster.
So the Maxwell cluster has about 800
nodes, compared to about 150
for the online cluster and
about a quarter of those AGP units.
It is quite different from the online
cluster and that there's a real mixture
of sort of storage systems.
There's also a mixture of the physical
media that use to store the data.
So, of course you have ages,
but there is also magnetic tape
storage available.
And just like the online cluster, it
also uses InfiniBand for networking.
So this
cluster is what
this is, what we and these who
come here use for offline
analysis and to be
specific.
This is the analysis that happens after
an experiment, and it's
usually where you would prepare some
analysis for publication in a journal,
for example.
And the exact kinds
of analysis varies
very much from experiment to experiment.
There are certain fields with
techniques that have
existing tools for for
analyzing data.
So one example would be serial
crystallography.
I'm going to get to this in a second.
But zero crystallography is just
an experimental technique.
And there's a couple of existing tools
with both command line and graphical
interfaces that that
the users can use to analyze the data
they get here.
Another option would be using Jupyter
notebooks. So the Maxwell cluster
operates to have incidents,
and these are very popular.
So on the right hand side, I have a
little example of this.
Typically, they're written in button,
and you can see here just
a quick sample of someone
analyzing some data from a spectrometer.
And here they're just we're just using
that part to to display all this.
So this is it.
It's using fairly standard tools.
Coincidentally, the offline
calibration system also uses notebooks.
So remember, I said earlier that we have
an online calibration system.
And this is used for correcting
the detective data in real time,
but it's not entirely
rigorous, and we have another system
flute for doing this more rigorously,
and that's the off line calibration
system.
So this will actually use this trip
to notebooks as well, which
might seem like a bit of an odd choice.
But the advantage is that it's
very usable for detective scientists
to look at to play around
with different methods of correction, for
example.
And all of the data that we
create both from, you know, the raw
data and and the off line calibration
system.
This is all stored in HD
of five files
and we have a couple of libraries for
working with these files,
like extra data and extra GM
that I've mentioned.
Extra data is really from working
directly with.
It's more it's more generic.
It's meant for working with data from a
run. An extra GM is
used for working
with data from 2D detectors.
And I have a little example
here.
Showing the use of extra data,
so and the fixer upper the
top, you can see that we do a batch of
imports.
We import this classic with actually
extra data and
the Fed. So we open a specific
run from an experiment and
we look at what's in this run.
So in this particular
run, there were four thousand
trains and lasted for
about six and a half minutes.
And you can also see all of the devices
that this run has data for.
In this case, it's a spectrometer.
And and the final.
So you can see that we're actually
getting some data out, out of
out of out of this front.
So in this first line, for example,
we're getting a bunch of camera data
and the the data that's returned
here is actually just a regular and
empire.
So we make use of Python
and the regular,
uh, its Python
ecosystem number crunching libraries
quite a lot.
So there's a lot of use of nankai and
sci fi and X-ray
and that sort of thing.
And yet this is just a multidimensional
umpiring.
And now,
like the entire points
of offline analysis, is
to
do some rigorous
analysis that is ready for publication.
And just to conclude, I
want to give a couple of examples
of what kind of analysis users
do.
So there's two that I want to talk
about. And the first one
is related to this technique
called serial femtosecond
crystallography.
And this technique is used
to find out the structure of
biological samples like proteins.
For example, this particular
technique requires the very high
repetition rate and lots of versions.
So what you do?
Broadly speaking, is you get a bunch of
identical proteins that you want to
study.
You put them in a liquid solution
and then you fire it in a liquid right
into the path of X raping.
And the idea is that hopefully
there will be one one sample
and protein sample per X-ray
pulse.
You'll get some data on the other
changes and then later on
working backwards. From all that data,
you will be able to figure out the
structure of that protein or whether the
biological sample you have.
The problem is that you need a
lots of
you need a lot of hits, basically, you
need a lot of photons scattering off
a lot of different proteins for
this to work, like on the order
of tens of thousands, for example.
And this is why
the high repetition rate is so important
for this kind of experiment and because
you run the idea because ideally you've
only got one protein
per pulse. You need lots of photons
because the probability of
a photon actually hitting and scattering
off the detector is very low.
And that's why the intensity of
excel is so important.
The final example I have is
ultra fast dynamics,
and there was an experiment recently
where some scientists looked
into the radiation damage
from water molecules, and
they cut a long story short.
What they what they ended up discovering
was some information about how
a water molecule is destroyed from
X-ray radiation, and this process
is extremely fast.
It happens in femtoseconds,
and that's why the short pulsed
field is so important.
The pulse is sort of analogous to
the exposure time of a regular camera,
so you can imagine if you're trying to
take a regular photo of a
of a fast process, you need a very small
exposure time, and it's the same
thing with an field.
You need a very short, very
short pulse to capture
the dynamics of this really fast
processes.
And this is just two examples there.
There are many others that you can find.
And, yeah,
so real and tangible,
real next occurrence once
Andy Griffith and Rush ends up.
Well, how very charming.
Thank you very much.
I'm I'm flabbergasted.
And I would lie if I would say
that Andrew stood out, to be quite
honest.
Social science?
Sorry.
See?
Yeah, we're bit overdraft, but never
mind.
You don't have that many questionable and
okay, they're coming in by now.
Okay. If you want to ask questions.
Go to Bino.
Go to the info dropdown menu
and no, excuse me.
Go to the chat and you'll
find the hashtags and the Irish
Sea climbs we've tried we can boast of.
We have some here.
I'm just going to read it out to you
if you need a high amount of pulses per
second. Why don't you spin a few plates
with holes in front of a continuous
light source?
So the problem
is that actually, let me just go back
to slide number
two.
Four.
So this thing
uses an electron gun, and
the reason we have the sponge structure
is because the electron gun needs to be
recharged. Basically,
the thing that you're referring to with
having like a continuous stream of pulses
is actually possible, and I believe there
is one facility in
the states. I'm not exactly sure which
one. Maybe it's serious,
but they actually have a storage ring.
So I think what's happening is
that in front of their electron gun, they
have a circular structure.
This is storing all of the electrons and
then they're all fired off continuously.
But we don't have that.
So that's why you have the sponge
structure of all the pulses arriving in
trains.
OK. Well, is a thank you.
Here's the question how does your
lab considered or evaluated
Julia as an alternative to Jane?
And so Julia
is on my personal
radar, but I
don't think there's been any
rigorous study of it just yet,
though I can certainly see why it would
look attractive.
I believe there was like a brief
foray into using Julia,
and if I remember correctly
for for the use cases that
they were using it for
Io was the bottleneck rather than
computers. So Julia didn't offer
much of an advantage over Python.
Having said that, there's certainly other
there's other use cases where I think
Julia could make a lot of sense.
Typekit, thank you.
I'm.
This is a question that is critical
to sort of fundamentally probably
for your application, you consider futons
to be waves or particles.
I'm just going to pass the question, I
have no idea how to answer it, I'm sorry,
but I yeah, I've no
idea what I might
suggest. It's perhaps emailing
itself.
There's probably somebody who would be
other asked that question, but
I don't have the background for that.
Sorry, that I exposed
you there, did meet you.
We'll get to you use
event based cameras.
And
I'm not exactly sure what you mean
by event based cameras.
Perhaps the question is referring to
weather events trigger a camera.
And I think the answer to that
is no.
There is a very there's
very elaborate timing system at exfil,
which tries to synchronize all of all
of these things at the same time.
And so I don't
think there are any event based cameras,
but perhaps I misunderstood the question.
OK.
Currently, there's no further question
coming up, said we'll just wait a minute
or two.
Maybe
they'll meet the long way from Down
Under even, excuse me,
New Zealand.
Yeah, yeah.
How did you get here?
I am.
So after I
after I graduated from university,
I work for a little while
in the Netherlands for a
financial technology company and
then doing this kind of scientific
computing work.
So it's my goal,
I suppose.
So a couple of years
ago, when Israel published
the opening for the position that I got,
I applied for it and I'm
very lucky that they picked me.
So I didn't exactly move all the way from
New Zealand to to have it with
my from Hamburg to the Netherlands.
I'm sorry. New Zealand to the
Netherlands to Epic.
You had still
New Zealand to the Netherlands is
a.
Quite a jump in itself, isn't it?
Yeah, it is, yeah, yeah.
The flight was terrible.
I think it was about three hours
because it was no good.
I would imagine, no, I hate flying.
It was terrible.
In my younger days, I loved it, took me
to wild places.
Yeah, but it's it's sort of become a drag
and the pain of the.
Do you miss New Zealand?
And.
I have to say not, not particularly.
It's a great country, to be sure,
but I would be a great country.
Yeah, yeah.
But I have to say I, I really enjoy
the work that I do here.
So there's there's not many
days that I miss New Zealand,
and perhaps
I just haven't been away from it for
like maybe, maybe in a couple of years or
I'll feel homesick.
I would assume the weather is better.
It depends, it depends, so
I used to live in the capital city,
which is go to Wellington and it's
a it's a harbor city kind of
like kind of like camping
and the weather was actually not
so different.
I would say that on average it was maybe
a couple of degrees warmer
than it was.
It's always an extremely windy and rainy
city, so
it doesn't feel too different
from have given that it's not
true. Point taken?
Yeah.
Yeah, sure, it is
bogus. The people are still listening to
us because somebody
asked here is the software
compatible comparable between
the physics and the financial industries?
Did it take you a great reboot
freed from finance to physics
and good Christian?
Or is it just deja vu?
I would say it was actually very
different.
So.
I mean, for starters, most
of my work now is in Python, whereas
previously it was in C++.
We do use C++ and external for
like the very computer intensive
parts of the pipelines.
But the majority is really in Python.
Just trying to think of what it was, but
the other differences are
financial technologies is typically
or at least designed, at least for
the stuff that I was working on, it was a
lot simpler.
So X, well, it's really helpful to have
some background knowledge of physics.
And there was none of that in the
financial technology, and
I also find the work a lot more
interesting.
So that's and does exfil fulfill these
expectations so far?
So far it has. Yes, I'm very happy
to say what has worked.
Yeah.
Last question is on your
HPC cluster.
What schedule do you
uh?
So on the offering cluster, we use
left and.
I haven't actually used it that much
myself.
So, yeah,
I think that's the only sketch that we
use.
I think it is, but a-,
as far as I'm aware, we don't use
the term on the online cluster.
Can you hear me now?
Yeah. Yes, I'm sorry.
I'm awfully sorry.
I should have changed that hardware
before the
game.
I'm supposed to say hi to you
from Andrea Thorne.
I'm not sure he remembers me, but I work
in the same area and he also in Hamburg,
and I'm around three and world,
if you like to meet him.
So here you are.
All right. The message for you.
Thank you. Hey, Andrea,
there's another question that's come in.
What's the unique feature of
this?
As far as I remember these, it seems to
be happy with Avis.
This is a good question,
and
unfortunately, I'm not sure I have a good
answer for you because I'm not very
familiar with those two file systems.
Like, off the top of my head, I guess
one.
So I don't know anything about the
performance characteristics of those two
systems.
I guess one kind of nice thing about
this is that we get support
from IBM for it.
I can't really
answer that. Peter, I'm afraid.
Hey, that does it with the questions.
I was wondering if it was worth
the one thing to the breakout room
because the questions don't seem
to be so thick.
Actually, there is no no question around
anymore.
So if that's OK with you,
we can call. We're right on time.
It's perfect.
And so thank you again,
standing still.
I'm still flabbergasted by the magic
of whatever it is.
It's a lot.
It really is.
I thank you very much for having me.
It's great.
Thank you for coming.
Thank you for giving us some of your time
and have some happy
times in Hamburg.
Thank you. Remember when the fish starts
swimming by in about the height
of your eyes?
Yes, you may complain about the weather,
and it sounds good.
It sounds good.
OK.