Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/3192 Thanks!
[Music]
the upcoming talk will bring back
memories at least they brought back
memories to me when any of these uh
sounds produced by platforms like Super
Mario World Chrono Trigger Super Metroid
and some more are being played our
speaker will explain and share with us
how that actually works how that sound
synthesis works and give us a deeper
understanding how these most favorite
memories of most of our youth were
actually being created so please give a
big round of applause for kle
[Applause]
[Music]
[Applause]
fil
[Music]
[Applause]
[Music]
e
[Music]
hello and welcome everyone what you just
heard was the intro to Super Metroid and
part of the amazing atmosphere that this
intro has is of course the sound this is
the ultimate spc700 talk the talk about
the architecture behind the sound of the
Super Nintendo Entertainment System this
talk is part of a series of ultimate
talks where we try to convey as much as
possible about a certain architecture or
computer in 1 hour so how did I steal a
microphone well actually they gave it to
me I'm barely known for all of these
things the most important one for this
is that um I am the author of The spz
ASM assembler for this architecture
that's what got me interested so first
of all let's set the scene uh Super
Nintendo entertainment system came out
in 1990 and um this is the roughly how
it is laid out internally so the main
CPU has access to the cartridge RAM and
ROM and it runs all of the uh game logic
and all the graphics and the PPU the
picture processing unit is actually just
a very dumb graphics processor it needs
to be told very precisely what to do so
the CPU has to spend a lot of time
telling the PPU exactly what tiles to
draw and what Sprites to draw and so on
on the audio side that's actually
completely different because the ssmp is
a full general purpose um microprocessor
which talks to the synthesizer chip the
sdsp it has its own memory space with
ram and it doesn't have access to any of
the cartridge memory so for some reason
they decided that an audio core
processor was necessary that makes this
architecture very interesting you can
even see that in the boards here on the
right side you can see the um P two PPU
chips and the CPU chip on on the left
side the very top chip is the ssmp and
below that the D speed chip and even
later on when they combined multiple
chips into one in later board revisions
and they still kept the CPU and PPU chip
on the lower left large chip and the SAU
chip with DSP and S SMP on the upper
right this is a die shot of that it also
contains the memory you can see the ssmp
chip on the uh upper left and the memory
uh audio Ram right uh of that and the
lower section is the DSP
chip so this audio processor was very
much ahead of its time in a time where
most consumer console audio Hardware was
doing frequency uh modulation syn
synthesis FM and they had full 15 bit 32
KZ samples eight voices could play those
they had modern adsr envelopes pitch
modulation between voices allowing you
to do FM regardless um Echo and F
filters and also a simple noise
generator so this this talk will roughly
be split up into three parts first I'm
going to talk about the CPU architecture
then I'm going to talk about the
hardware sample format called brr and
finally I'm going to talk about the DSP
Hardware itself so the architecture is
often compared with the 6502 and while
it was clearly explicitly based on the
60502 by the Sony Engineers who designed
it um it came out very differently so
first of all all 256 op codes are
occupied this is a full op code map and
that's already very different from the
6502 which has a very small part of its
op code space actually occupied it
supports binary coded decimal arithmetic
uh has a lot of support for single bit
computations and 16bit instructions also
Hardware multiplication and division and
some very unusual branch and call
instructions in practice it runs at the
same speed as the 60502 even though it
runs at twice the clock rate all the
instructions take roughly twice the
amount of a clock Cycles so it's
practically the same speed it's an 8bit
processor it has a 16bit memory space so
64 kiloby and it has interrupt support
memory mapped IO timers and
communication ports for the main CPU
these are the registers the accumulator
is where all primary computations go
through the index registers X and Y um
are mainly used for special addressing
modes the program counter points to the
current instruction 16 bits and the
stack pointer actually only has eight
bits because it always lives in the one
page of memory so the upper uh eight
bits are fixed to hex 01 and the program
status word is the flag register it
contains normal processor Flags like the
negative flag the Overflow flag V the
half carry flag the zero flag and the
carry flag it also contains the P flag
and the two interrupt Flags so this is
what the memory map looks like memory is
split up into Pages um every page is 256
bytes large so a page can be identified
by the upper eight bits um that's how we
get the zero page with Hardware
registers and then the one page which
also contains the stack and there are
some addressing modes which um access
the so-called direct page um which is
either the zero page or the one page and
which page it is is determined by the P
flag here in the flags register um this
is very similar to the zero page
behavior of the 6502 and that's why I'm
going to say direct page in general
because it can either mean the zero page
or the one page depending on processor
State the all of the memory is uh
occupied by Audio Ram except for some of
the uh iio and RAM regions and bit
instruction addressing can only go up to
a very small part of audio RAM and
finally at the very top of audio Ram or
at the very top of the memory space we
have the 64 bytes of bootrom which can
also be switched out for more audio Ram
that's where the uh chip boots initially
when you start it up so let's talk about
addressing this is a very simple
instruction add with carry a with direct
page 51 which means that from the direct
page address Lo memory location 51 we
will uh load memory and now this depends
on whether uh you have the uh direct
page bit set one or zero it could be
either 01 51 or
0051 uh in terms of the address but I'm
going to assume here that it's the zero
page so uh we load the value 77 and add
that to the
accumulator more interesting is once we
introduce memory indexing so this is 5 Z
index with X so we take the immediate
address and add it to the X register
giving us the address 52 and now we load
from direct page address 52 and add that
to the accumulator um we can also use x
directly as um an an address so as a
pointer effectively so in this case
loading from memory address 02 this is
also possible with the Y register and
also you can directly increment the X
register after uh the operation allowing
you to step through memory more
efficiently and all of this allows you
to use X and Y as array indices
effectively which is very useful you can
also do direct addressing on the whole
memory space specifying a 16bit memory
address however that means that the
instruction will be three bytes instead
of two bytes one B for the op code and
two bytes for the memory address um and
also it will be slower so you should
always pref prer using direct page
addressing when
possible what is not possible on the
whole memory space is indirect
addressing so now we do indirect 07 plus
X so we take the immediate 07 added to
the X register and now what we load from
address 08 is not the actual value that
we want to add instead is it's a 16bit
pointer and because the spc700 is a
little Indian architecture we now load
the pointer 7f a full 16bit pointer and
now load from that address and um add it
to the
accumulator this is also possible with
the Y register instead here we
immediately load the pointer at the
direct page memory address and then add
uh y to the uh to the pointer allowing
us to step through the 16bit address
instead of just the direct page which is
arguably more useful so let's talk about
some actual
instructions
the many instructions share the move
nimonic these are all of the load
instructions which load uh some place in
memory into uh one of the three general
purpose registers the only addressing
mode we haven't seen here is the ones
prefixed with a hash those just load in
immediate instead of loading from memory
so for example move X hh20 will load
hx20 into the X
register the store instructions are the
exact opposite they also use the move
neonic they store one of the three
registers in to memory register
transfers also use the movement ammonic
and this is one of the only ways of
accessing the stack pointer directly by
transferring it to and from the X
register you can also load memory
directly you can load one direct page
address directly with a value of another
direct page memory location and you can
load immediates directly into the direct
page arithmetic instructions can use uh
fewer addressing modes all of these that
you you can see here um there is ADD
with carry Subtract with carry compare
and set Flags which effectively only
performs a subtraction but doesn't store
the result anywhere it just sets the
processor flags bitwise and bitwise or
and bitwise exclusive or and there's
also some single argument instructions
increment decrement left shift right
shift rotate left rotate right and the
shifts and rotations only perform one uh
operation in their respective
Direction now I told you that there
would be binary coded decimals support
and binary coded decimal is a way of um
supporting of storing hex of storing
decimal numbers in hexadecimal or binary
numbers in a way that it can be
displayed to a screen more easily
however this is an audio Cod processor
not a video code processor so like the
binary coded decimal support is entirely
pointless and I'm not going to explain
how it works and the only useful part
really is the exchange nibbles
instruction which will exchange the two
4bit nibbles of the
accumulator so there is some support for
16-bit arithmetic always with the w for
wide I guess of for word um and it
usually uses the combined ya register
which simply combines the uh y register
and the a register into one 16bit
register now I promised a hardware
divider and um here it is the multi
multiply instruction will multiply the Y
register with the a register and store
the combined 16bit result um into the
combined ya register so not losing any
sort of precision and the Divide
instruction will divide the combined
16bit ya register by the X register and
store the result into a and the uh
remainder into Y and this is by far the
most expensive CPU instruction it takes
12 Cycles but it's still vastly more
efficient than anything you can write in
software and this is actually quite a
useful feature because uh in audio you
have have to do a bit of uh
multiplications and divisions sometimes
so I imagine that's uh quite
useful for manipulating the stack there
is traditional push and Pop instructions
and you can directly push and pop the uh
Flags register as well now for bit
computation there is a whole host of
support you can set and clear a specific
bit in the direct page and these uh the
notation here is that the dot and then
the number afterwards signifies the bit
within the memory address there's also t
uh test and set and test and clear
instructions uh which perform the exact
operation as you would think given their
atomic primitive names uh except it
doesn't really make sense for Atomic
operations since you only have one
processor there is uh various logical
and move instructions and they use the
carry flag as a single bit um
accumulator effectively and also you can
invert the bit coming from memory before
uh performing the operation that's
signified with a slash in the assembl
notation here so there's quite a few
possibilities and because the carry flag
is used so much for single bit
computations there's also a whole host
of U manipulation instructions just for
that let's move on to control flow
normal Branch instructions um take an
8bit signed offet so they can jump up to
127 bytes forward and negative 128 bytes
backwards from the next instruction
after the branch and you can either
Branch unconditionally or conditionally
on most of the processor flanks you can
also Branch depending on whether a bit
in the direct page is set or
cleared and there's some weird branching
instructions the compare and Branch if
not equal instruction will compare the
value in the direct page to the
accumulator and only Branch if they are
not equal which saves you a compare and
the decrement and branch is not zero
instruction will decrement its argument
and then Branch to the uh offset if that
wasn't zero which allows you to write
Loops very efficiently so you can put
this instruction at the end of a loop
and jump back up to the loop and if you
initially set up the loop counter that
gives you a very efficient way of
writing
Loops for jumping to further away places
you can use the jump instruction which
jumps to a full 16bit address there's
also an indirect jump instruction which
allows you to use jump V tables in
memory for calls there is the normal
call instruction which will store the uh
program counter to the stack and restore
it on return there's also the PE call
instruction which can only call into the
uppermost page of memory and uh you can
put some very important uh commonly used
routines up there I guess it's faster
and smaller in terms of encoding and
then there's also the 16t call
instructions which take up quite a large
part of the op code encoding space but
uh they effectively call into a hardware
defined subroutine table starting at FFC
0 and going up there in like two by
increments um so if you have um you can
you can store like uh function pointers
there uh which allows you to call even
more important even more frequently used
um functions and yeah their locations up
there and you can return from all of
those with a return instruction because
they all push the program counter in the
same way there's some instructions for
manipulating the other flags importantly
of course you can clear and set the P
bit which allows you to control which
page is the uh direct page you can uh
there's a knob instruction and a Hal
instruction now I told you that there
would be interrupt support well external
interrupts are not connected on the
hardware software inter interrupts are
pretty much useless unless you maybe
have a debugger but I don't know so
support is mostly uh useless and pretty
much unused on uh the spc700 so I'm not
going to explain any of the interrupt
support more interesting are the
hardware registers 16 of those starting
from uh P from address 00 F0 to 00 FF
first two are control internal CPU
control registers the next two control
uh the address of the DSP that you want
to write to and the data that you want
to read to or write to a read from the
DSP so this is how you access the
hardware registers of the synthesizer
and we are later going to see uh quite a
few of those uh registers and where they
live there is four communication ports
for talking to the CPU there is two
auxiliary ports which connect to 16
wires on the external chip however they
are also not connected so they're also
pointless and then there's timer control
so the test register um pretty much
shouldn't be tou touched because it's
only uh intended for internal testing
purposes there's two ways to enable or
disable timers you can disable all
rights to Ram you can crash the CPU by
writing to to bit two if you ever wanted
to do that um and also you can add extra
weight states to ROM and IO accesses and
because audio Ram is shared by both the
synthesizer and the SP
spc700 uh they have to wait for each
other and the synthesizer always takes
priority so there's a very specific
cycle of weight States and you can add
extra weight States on top of that if
you really want to which is just going
to slow your program down so don't do it
I guess more useful is the control
register which allows you to enable or
disable all the timers allows you to
clear the ports in groups of two and
also allows you to set whether the top
of audio Ram contains the boot ROM or
extra um audio Ram because you usually
don't need the boot ROM after you've
booted so the timers they run at
different frequencies there's three of
them t0 and T1 run at 8 khz T2 runs at
64 khz and their respective uh division
register controls uh what frequency it's
divided uh down to and a division value
of zero actually means uh divide by 256
which allows you to go down to 31.25
Herz and the timers don't cause any
interrupts they instead count up as 4bit
counters and you can read the value out
from TN out that's the only way to
access their value so now let's switch
perspective a bit as I talked about
earlier this architecture um the the
spc700 has no way of accessing cartridge
memory so you need to upload cartridge
memory into audio Ram somehow like for
all the samples of the code of the other
data that you need um and uh that's why
the boot ROM exists it implements a very
simple I don't have time to go through
it but it implements a very simple
upload routine this is a
commented um commented disassembly that
you can also find on the internet um so
the way that this works from the CPUs
perspective is uh for starting the first
transfer we write the um Target address
of the uh of the memory that we want to
upload into Parts two and three we write
a non-zero value into part one and we
write zero into part uh we write the
magic value cc into Port zero so what is
important to remember here is that those
ports are full duplex so when I from the
perspective of a CPU write something
into part zero uh and read back that
part zero again it I'm not going to read
back what I just wrote I'm going to read
back what the ssmp last wrote into it so
it's like full duplex and I can't read
back what I wrote so it does make sense
to wait for the ssmp to respond with the
exact same value in the exact same port
because they are full duplex so now for
sending more data I can send the data
into uh Port one and increment the data
index into Port zero and wait for the
acknowledgement which is sending exact
uh the exact same index back into the
port zero and uh the data index can
increment and overflow past 256 so it
can send arbitrarily large blocks um the
ssmp doesn't care but if you want to
start a new Block in another memory
location you have to set up a new Target
address a non-zero value in Port one and
then anything except the last index plus
one which tells the ssmp to stop the
current transfer and start a new one and
for ending the uh all the transfers you
write another address into ports two and
three you now write a zero value into
Port one and anything except the last
index plus one in part zero and now this
address is interpreted as a jump address
where the SS SMP will jump execution to
so from that point on you can take
control over the system and do whatever
you
want okay that's basic basically all I
have to say about the architecture uh
let's move on to the audio side of
things to the brr sample
format
so uh first of all importantly there's a
difference between what you would call
sample in in music which is a small
snippet of audio that you're playing for
example a single note of an instrument
and what we call a sample in digital
signal processing which is uh a single
digital value of a digital signal uh so
I unfortunately I have to use both terms
interchangeably I hope it's not too
confusing anyways first of all I should
maybe give you a little bit of a crash
course on how digital audio actually
works I even have a YouTube video for
that I should link that maybe um so um
audio initially analog audio is just
very fast movement in air uh so it's
like the sine wave you saw initially we
can record that uh with a microphone and
then we have an analog representation of
that in electricity voltage over time
and for making that digital we first
have to sample it with the sample rate
like we saw here so at regular intervals
we take note of the current voltage and
the uh how often we have to do that uh
depends on a bunch of complicated math
but effectively we have to do it more
than 40,000 times per second so I sample
rate above 40 khz and now we have those
analog voltages um which we then need to
quantize into concrete integer numbers
like is happening here in this case
16bit numbers and then we can just uh
store those and reproduce the original
sound perfectly by doing the entire
process in Reverse later on and this way
of storing things is called pulse code
modulation and if you just store those
samples into a file you basically have
the we file format so um learn something
else important and useful here except
for old outdated architectures um so
what's what is this bit rate reduce
thing well the amount of data that comes
out of a raw PCM stream is way too much
to actually directly use in the spc700
with its four 46 64 kilobytes of memory
because that would mean you could only
store about 2 seconds of audio which is
absolutely nothing um so what you have
to do is reduce the bit rate and that's
where this name comes from but it's kind
of a misnomer more correctly we should
call it adaptive differential pulse code
modulation 8 PCM adaptive means that the
compression that we apply adapts to the
input differential means that we mainly
saw the difference between samples and
PCM is what I just explained to you so
the bits that come out are 15 bits uh
that's the bit depth um there is some
places where we use 16 bits um there is
various complicated rounding modes and
as yeah switching between those two bit
depths but effectively the sample are 15
bits and the playback rate is 3 2 khz of
the entire synthesizer system which is
exactly one sample per 32 CPU Cycles um
the synthesizer and uh ssmp run in a
lock step effectively they they share
clock rates everywhere so brr is split
up into blocks each blocks block has
nine bytes one bite is a header and
eight bytes contain two 4 bit samples
each sence samples so that's 16 uh
samples which can have a range of ne8 to
postive 7 and the header initially
contains the loop and end Flags which
determine whether the sample ends after
this block and also maybe whether we
want to loop back to an earlier point in
the sample so that we can play part of a
sample
repeatedly so the incoming nibbles are
initially shifted given the shift value
in the header the shift is offset by one
so actually a to have no shift at all it
means you have to use a shift value of
one and a shift of zero shifts one by to
the right which is pretty much pointless
um shifts above uh 11 or the value of 12
don't make sense because remember we
have four bits here and we can only go
up to 15 bits so we can only usefully
shift by uh 11 so that's why shift
values of 13 to 15 are a hardware glitch
it's consistent but it's technically a
glitch
and uh yeah don't don't use it and um
that this entire shift process of course
increases the range of our incoming
values from negative uh 78 to positive 7
to like much larger range but
importantly now we pass it through a
filter the filter is also specified in
the header and there is four of them and
the filter zero just uses the values
that we just got from shifting as is
retim as we would say um and this is
useful if you don't know what the
previous samples are for example at the
start of a brr audio sample uh or at the
start of a loop point the first filter
effectively adds the previous output
sample to this current input value that
we are getting in and this means that
this value that we're getting in is just
encoding the difference between the
previous sample and the next sample and
that's where the differential or Delta
coding comes in and because samples in
in waves are always pretty close to each
other this is actually already very
efficient the filter 2 um does something
similar except it uses uh plus two of
the previous sample and minus one of the
sample before that which if it was
actually plus two and minus one it would
be exactly a polinomial order to
predictor uh which means
that again I don't have details to go
into all of the math behind how this
works and why it works but effectively
we can use Parabola so pols order two to
approximate how the samples behave
locally because that actually matches up
because of tailor series and uh then
that allows us to predict where the next
sample is and this gives us even better
compression sometimes than filter one
now filter 3 is weird because it's
similar to Filter 2 except the values
are kind of off and I have no idea what
they were trying to do with that I guess
it exists and you can try compressing
with it but I have no explanation so
this entire process of taking previous
samples multiplying them by certain
factors and then adding the next input
value into it is called linear
predictive coding and it's still the
state-of-the-art compression method uh
for modern lossless compression
algorithms like uh the free lossless
audio codc and AAC so youve already
understand 50% of how those audio
formats work you're wel welome um
so um we still have kind of a storage
size problem because if we want to play
like musical instruments we would need
uh one separate sample for every single
pitch that would already mean that we
would need like 12 different samples for
just one octave and that would of course
again blow up our memory requirements
for samples so what the hardware can do
instead is take a single sample and play
it at different pitches so that we only
need a few samples for an entire whole
instrument so the way that this works is
that there is a per Channel pitch
counter which is incremented by the
16bit uh pitch Hardware register um so
the N here indicates the voice of course
there's one register per voice and
that's done for every single sample that
the DSP outputs and the pitch counter is
not really uh 16bit it's 4.12 bit fixed
point so the upper four bit indicate the
index within the current brr block of
course 16
samples and the lower 12 bit or at least
the upper 8 bit of that are used for
interpolating between brr samples um so
uh I have some examples for like the
pitch values the fixed Point pitch
values you can put in and what playback
rate that would roughly equate to um it
this clamed at around four times
playback rate so two
octaves and this is maybe a better
visualization of how that works um if
we're playing at half speed that
effectively means that we increment the
orange line which is our current
position our current pitch uh pitch
value the internal register um we yeah
increment that by half a sample every
time so we play each sample twice um the
3 Z for example would mean that we only
play every third sample and we skip two
samples in the middle but of course we
don't just do that we don't just repeat
samples and Skip samples because that
would sound absolutely horrible and
nobody would want to listen to that so
what we do instead is we interpolate
between samples and uh for various
technical reasons we only have access to
the next two and the previous two sample
and now we have this playback position
again in the orange vertical line and we
want to figure out how to yeah how to
sum up those samples interpolate between
them um to get a approximation of what
the sample would sound like if we
inserted more samples in between so we
don't just linearly interpolate that
would also sound kind of horrible it
turns out that using a gion curve or a
bell curve is actually uh a very good
idea signal processing people will know
why um but it's not important for our uh
purposes and um the way that this
interpolation is done is that we have a
hardware register a hardware lookup
table uh I have written it in Big N
which contains all of the uh bell curve
factors and it only contains half of a
bell curve because it's exactly mirrors
there is 512 entries and goes up uh to
hex
519 so um if you stare at this long
enough you can convince yourselves that
these indices actually make sense the I
is the offset within the current sample
it's the uh orange thing in in the
internal register here the 8bit
interpolation table index
and uh then we have to scale the entire
thing down at the end um but that's how
we can very nicely interpolate between
samples and it actually sounds pretty
good now the uh only thing that you have
to be careful with is that this will
cause a low pass filter to be applied to
the sample even if it is played at the
original 32 khz playback rate so one
sample uh per one input sample per
output sample um so if you want to avoid
that you have to apply a pre-emphasis
filter so you have to use the inverse
filter of this which is easy because
it's linear and invertible um and many
tools do that spsm does that BR tools
does it but there are some games where
you can tell that that wasn't actually
properly done so yeah something to look
out
for we have now explained everything
about how the sampler format works and
uh all of this decoding happens in
Hardware of course and we I'm going to
talk more about how the Hardware works
this is a full or pretty much full
slightly simplified block diagram of the
dsp's internal
pipeline um the orange Parts exist one
for each voice and the gray parts are
registers or memory where yeah the DSP
interacts with
those so first of all the part that
we've mostly already seen brr data comes
from Main memory is indexed by the
source number and the uh directory and
then goes into the brr decoder and the
brr decoder writes out the corresponding
end X flag for each voice when it sees a
loop or an end flag in its block and
stops playing or Loops back depending on
uh what the exact flag was so endex is
one of those many registers that
contains one bit for each voice and just
yeah signal something for for each voice
and you can see there's a various
registers where the DS P writes values
into for software to read back and be
informed about what the synthesizer is
doing and the inverse is obviously also
true so the BR decoder passes its
samples into the pitch calculation uh
thing with the gausian interpolation
that we saw and now
importantly wrong direction um
importantly it doesn't just use the
pitch register it actually does pitch
modulation I lied to you earlier so we
use the uh upper 11 bits from the output
of the previous voice that's the
slightly darker uh orange thing that
goes down below everything else that
comes from the previous voice and we use
that multiplying it with the pitch
register if a pitch modulation is
enabled for this voice and the way that
the multiplier works is that it can if
it has maximum output volume it's going
to multiply the pitch by two and if it's
uh at minimum output volume it's going
to multiply by zero and in the middle is
one so it's it's a bit nonlinear but you
can do some interesting modulation
things there I think because most games
don't use it but you could in theory the
output of the pitch calculation is um
the actual BR sample subsystem output
and we can now switch between that and
the noise generator which is global for
all of the voices via the noise on flag
and then we combine that output uh
signal with the envelope generator's
envelope volume the envelope generator
takes in the gain in idsr registers and
also the reset signal and outputs an
11-bit envelope factor which is also
written to a software readable
register and the output of land
multiplication is basically already the
output sample of this voice we get that
out into the VN out register for
software to read back at least the upper
eight bits and then we split the path
into stereo so here we get the left and
right channel volume for uh this voice
and I'm going only going to show the
left voice uh Left Channel now because
the right channel is exactly the same so
on the master path we add up all of the
different voices and we multiply by the
master volume corresponding to the
channel on the echo path we can first of
all enable or disable Echo for each
voice IND visually then we add up all of
the voices that do have Echo enabled and
then we enter wrong direction again then
we enter the uh Echo U the echo buffer
subsystem so here we can disable all
Echo wres via the echo enable register
and the echo buffer lives in memory in
audio RAM and is addressed by the echo
Source address and Echo delay
registers the output of the echo buffer
then passes through the F filter and is
multiplied with the main Echo volume and
also part of the echo signal can pass
back into itself creating a feedback
with the echo feedback Factor um which
actually allows you to repeat some uh
some snippet of audio multiple times
which actually makes this into an echo
effect and not just a delay effect so
here we can see those two paths combin
from the top the Echo and from the left
the main volume and now we enter the
analog domain with a digital to analog
converter and a small preamp and an
analog mixer now combines the
synthesizer output with the cartridge
analog audio in and the expansion Port
audio in so cartridges could have their
own uh sound processors on them and and
put output their own audio and also the
expansion ports could do that I think
the satella view does that sometimes uh
nobody's exactly sure somebody should
research that um but after the main
mixer we get the master mute uh switch
which is controlled by software and also
the post amplifier then passes that
signal into the uh various output
streams like the uh RF generator and the
RCA
jacks yeah so that's the entire Pipeline
and now I'm going to uh explain the
various parts that we haven't looked at
in detail yet first of all how are BR
samples actually found and played well
there's a sample directory which is a
single page in
memory which contains 64 entries four
byes each and uh which entry to use for
a voice is determined by The Source
number Source number Register
um and uh yeah so this this contains of
course the start address where we should
start reading brr blocks from when we
start playing the sample and a loop
address where we should loop back to uh
once we have finished uh playing and we
read a loop flag at the very final block
now the loop address doesn't have to be
within the sample itself it can be an
entirely different location in memory
the BR decoder really doesn't care you
can like have multiple samples that have
the same Loop section of audio if you
really want to uh I guess you can do
some interesting things there but this
actually means that the source numbers
that you can read in like a register
viewer um are pretty intuitive because
they actually specify uh logical sample
numbers and not just random
addresses now for envelopes this is the
way that we uh shape audio and determine
uh what what kind of yeah it's a very
important part of how uh the uh how a
certain note will actually sound so the
DSP can do adsr envelopes adsr stands
for attack Decay sustain release and
it's still the standard Modern Way of
modeling envelopes on Modern
synthesizers attack means how fast do we
reach maximum volume once we start
playing Decay means how fast do we drop
off from that maximum volume afterwards
sustain means what volume do we want to
hold effectively forever until we let go
of the uh of the noes and once we let go
or stop playing um release determines
how fast do we reach silence and this
mode is enabled by setting the seventh
bit of the uh VN adsr register one the
lower register to one and this is is the
way that it actually works on the DSP
chip so every attack rate step we uh
first of all we have to receive a key on
Signal of course that's again one bit
per uh per voice in a special Hardware
register and once that happens we
increment the envelope volume by 32
every attack rate step until we reach
effectively maximum volume after that we
decrement with a multiplication factor
of 0.996 every Decay rate step so this
gives us an exponential decrease slowing
down over time once we reach the user
defined sustain level we Again decrement
by 0.996 every sustain rate step and
once you reach a key off signal one bit
per voice um we reach the release phase
which is Hardware fixed to8 every one
step so what are those rates and what
are those steps um so the rates are
controlled by a rate table so the uh
input put rate is a symbolic 5 bit value
which determines how many samples we
wait until we take the next step and
this means if you put in zero there's
never going to be a step so we're never
going to change anything and that's how
we can achieve actual infinite sustain
by setting the sustain rate to zero and
then that means that the uh sustain step
is never
taken and you can go up to like one step
every single sample which is for example
what the release does
um so the way that this is actually
stored in the register is uh this and
the sustain level is basically just
started with a factor of hex one0
0 and uh the gain mode is a little bit
of a simpler envelope mode you enable
that by setting the seventh uh bit of
the adsr register to zero and you can
then either choose between a constant
envelope gain by setting the seventh bit
of the gain register to zero or a gain
curve a fixed gain curve which um you
can use by setting that 7th bit to one
so fixed gain volume is pretty much
self-explanatory it's always going to
stay at a certain uh gain or a certain
volume um now the gain curves are more
interesting those are the four if if you
look in the register it has four
different modes that's the gain curve
part bits five and six and those are the
four mod modes in order 0 to three um
the first mode is linear decrease which
decreases the volume by NE 38 every gain
rate step exponential decrease again
uses the infamous 0.996 Factor linear
increase uses the plus 32 uh step and
bend increase also uses plus 32 until we
reach a value of 600 after that it's
going to increase by only eight so you
get a slightly uh lower increase in the
upper volume
register and the gain rate again uses
the gain table which we saw before it's
the exact same
behavior so you might now wonder what
are these decrease modes like if we
start with a sample the um the envelope
is going to be at zero and if we then
enable a decrease mode like it can't go
below zero that doesn't make any sense
um that's true that's also not the way
that it's intended to be used because uh
this is intended to be used a very fine
grain control so by
switching between
different uh envelope modes during
during actual sample playback you can
like first switch to one of the
increased modes and then switch to a
fixed gain mode and then switch to a
decrease mode and maybe again to an
increase mode or whatever there's many
many things you can do with that by
controlling it very precisely from
software and that's really what uh this
is intended
for um so let's move on to the noise
generator Global per uh Global for all
voices it uses a linear feedback shift
register to create a random stream of
samples which the rate of which can
again be controlled by one of these rate
table entries it's in the flags register
in this case kind of weird like a global
control register which also contains the
mute flag um so the way that a linear
feedback shift register works is we take
the lower two bits exort them together
shift all the bits one right and take
that exort bit and put it into the top
this actually turns out to be a pretty
good pudar random number generator given
how easy it is to implement in hardware
and also if you put in a certain
initialization value which the hardware
does it will go through all of the
possible 15-bit numbers
eventually so echo in uh in in Hardware
with uh uh with a ram buffer works like
this initially we have to consider that
we have like some start address of the
echo buffer and some end address um but
that's that's not really how the
hardware sees it effectively after the
end address we just jump back to the
start address there is no real concept
of of start and end as far as the
hardware is concerned so we consider
this a ring buffer which is also used in
uh other areas and the entries are not
really bites but instead two 16bit
samples one for the Left Channel one for
the uh right channel so every entry is 4
bytes long and we have a certain pointer
to the next Echo input so uh we read out
that value here with negative Echo delay
and um we that is the output of the echo
buffer system and then we write back
into that exact memory location the
current input value that we were given
and then we move on to the next memory
address and of course wrap around at the
end and what that allows us to do is
that by the time we wrap around to that
initial um value that we just wrote that
value will have been delayed by the
amount of samples that fit into this
buffer and that allows us to perform
these delay effects or Echo effects if
we write things um into it multiple
times so the way that this is controlled
exactly in Hardware um we have the echo
s Source address register which controls
the page where the buffer starts and the
echo delay register controls the buffer
size in multiples of 512 samples or 248
bytes so if you really want to you can
set that to uh 32 kilobit almost which
will take up like half the audio Ram I
guess then you don't have any space for
for code or data or samples uh but that
allows you to have an echo time of
around about 240 milliseconds which is
pretty cool and the uh delay of zero and
like buffer size of four is just because
there's inter an internal buffer
register in the uh Echo subsystem so
that means you can never have a true
zero Echo uh zero delay Echo and the
yeah the other registers uh you should
already know about so the F filter is
the thing that's directly after the echo
buffer so fi stands for finite impulse
response well what does that mean um
this is the textbook definition from uh
my German signal processing textbook so
suffice it to say I'm not going to
explain to you uh what why it is called
firir but the cool thing about finite
impulse response filters is that you can
Implement them by multiplying certain
factors here uh s signed 8bit
coefficients with the previous output uh
samples of the uh Echo buffer and add
all of those together and this actually
allows you to do repeated echo of course
by uh outputting actual the the sample
again uh it allows you do low Pass High
Pass various uh delay not delay um
various Reverb effects and other EQ
effects which is pretty incredible for
just some factors I mean you have to
know what you're doing but it's
possible
so that's pretty much all the hardware
explained uh the rest of my time will be
spent on demoing uh what you can do with
all of this so let's start with
something that you're hopefully all
familiar with with and the Super Mario
World overw World
theme um on the right here you can see
all of the uh
various uh registers just scrolling by
and M there's some very nice labeling
there also if you want to uh look this
up for yourself for your favorite game
how how exactly the voices are used for
example you can tell that some of the
voices are only used for sound effects
and not for the music and for sound
effects um if you pay attention to the
uh Orange um uh here and what is that
the the parts yeah the CPU SMP ports you
can see those always changing briefly uh
just around the sound effects uh and
that's how most of the games actually do
it they um uh will send small signals
over the ports from the CPU to the SMP
whenever it needs to place certain sound
effects corresponding to gameplay okay
um then there's also people who say well
we have this nice audio core processor
um we don't care we're just going to
stream um S&P samples directly from the
main CPU to the um DSP and now you
really have to pay attention to the um
to the ports here uh all of the four
ports um once the actual music
starts you can see it is changing in the
Rhythm with the music and I actually
stepped through the code they basically
just use the S&P to read out the uh
register data uh that is coming from
Main CPU and write it directly to the
DSP chip um which I think is kind of
pointless but also very good because you
can uh spend more memory on samples and
less samples uh less memory on uh
playback data yeah um uh this is a theme
that I have heard way too
much um I just wanted to show you how
some of the music looks in an
oscilloscope view because that's what we
do um you can tell that channel 8 is
reserved for sound effects there's no
sound effects playing here of course and
also maybe Channel 7 is a good example
of how they do a vibrant by changing the
playback frequency very quickly with the
page registers that we saw
before this a fun Little tune and
finally just to
demo uh what you can do do if you
actually uh put in a bunch of work and
take up almost all of the um audio Ram
with just a single song and have access
to Modern trackers but it it actually
works on Hardware uh this is a cover of
a um older arcade game I think yeah from
1985 and blue electric covered that uh
recently um just to give you an
impression of how actually very amazing
this sound chip can
be
[Applause]
oh so no Channel off the sound
[Music]
effects
[Music]
[Applause]
[Music]
I would let it play out in its entirety
but unfortunately we don't have any more
time so I'm just going to uh go over to
the end you can uh watch all of his
covers on YouTube they're pretty great
and there's many other people that I
probably don't even know of who are
doing amazing things uh with this uh
architecture and uh yeah with this sound
system so these are all of the main
resources that I used in putting
together this talk and most importantly
the Fest manual contains all of the
hardware details that I couldn't even go
over here like rounding errors and
various ways of how the hardware
glitches out when you give it
interesting uh inputs um also the SP
spc700 video series by ISO freeze retr
game mechanics explained uh was a huge
inspiration for me and uh contains a lot
of cool
graphics and finally U my own manual at
the very bottom is in my opinion the
best assembly manual but I'm biased so
because I wrote
it um anyways if you want to get
involved um there is a bunch of
different trackers available H I like
furnace which is actually really new one
only about 2 years old but as far as I
can tell its Hardware emulation is very
accurate and uh you can make some
interesting songs with that you might
want to write your own sound engine if
you're interested in some assembly
programming also I would really love for
someone to reverse engineer those chips
because as you saw there is very high
resolution die shots available I think
you can do the whole um uh the whole
thing that the visual 6502 people did
with uh these chips so I would really
love some if someone could actually do
that who uh knew what they were doing
and of course uh if you want to get
involved with spasm my little assembler
for this architecture you can try that
in your browser there's developer help
available and uh just talk to me I have
a bunch of things that I want to do with
that so yeah thank you for
[Applause]
[Music]
listening although unexpected we do have
some time for a short Q&A so please line
up at the microphones and we start with
the signal angel please go ahead yes
such an advanced sound chip just bags to
be used for unintended things are you
aware of any games that used the chip
for non-sound related things that is a
good question I've asked myself that as
well no I'm not aware but also to be
fair I haven't really looked deeply into
many sound engines I've mainly looked
into the NBA and uh NBA Live 995 sound
and the superet sound engine so
unfortunately no I can't answer that
thank you okay microphone number one and
please keep in mind that it's a question
not a comment yeah so back in the day
when you wanted to develop a game did
you get any tools at all from Nintendo
or Sony or or the other um de yeah yeah
um Distributors of the
ships uh I'm I don't I don't actually
know uh that's that's a good question I
haven't found anything like that that
there is not that much information on
the chip available beyond the resources
that I showed so I don't know so so all
the people wrote their own trackers they
probably wrote their own trackers I'm
I'm pretty sure mostly thinking about
how they compressed BR samples did they
have like Sony software for that I I
don't know unfortunately okay thanks do
we have more question from the senene
ranger yes yes do you know if any
official software uses BCD instructions
um I haven't found any software that
uses BCD um The Exchange nibbles
instruction is kind of BCD support but
it's that's used more often in various
engines uh but no I'm not
aware okay microphone number one again
please hi um the pitch um modulation
that you mentioned uh it's looks it
looks like that you can do FM modulation
with it are you aware of
anything that uses it for that maybe
with even multiple um
operators channels for operators um yeah
I was I was I was looking around but I
couldn't so far couldn't find anything
there's various things that look like
they use pitch modulation especially
modern things but always when I looked
into the tracker commands there was
nothing about Pitch modulation so I'm
pretty sure it's
underutilized okay
thanks
okay I think we are coming up to the
last question do you have one
signal last question how did you get
interested into this topic like writing
an emulator writing games and such um no
I was I was really intrigued by the
video series from uh atro game mechanics
explained which I also showed here um
mainly because I come from the audio
side I've worked with various codecs and
written audio stacks and whatnot so I
was mainly interested in the audio side
um and not really the games or uh
assembler side at least initially and
then I started writing the assembler
which made me learn about the uh entire
architecture okay thanks a lot huge
round of applause for
[Applause]
[Music]
fil
[Music]
la