Audio Software

The 3DO system depends on three sources of software to control its audio hardware:

The Red Book playback code in ROM, which is used only for playing Red Book CDs without Portfolio boot files.
The Audio folio, which generates and processes sound. It plays back sound samples, synthesizes new sounds, and can modify both sampled and synthesized sounds.
The Music library, which imports and creates scores, and plays back large sampled sound files. Scores are ordered assemblies of sounds and other components that a task can play back later. A score can typically be music, collections of sound effects, drawing or animation functions, or combinations of different elements. A score can also be an imported MIDI score.

Designing Sound with the Audio Folio

Working with sound is a fairly abstract process. You cannot see the sound, and many of the modifications you make to sound in the digital domain do not have simple parallels in the real world. For example, think of changing the reverb time of an instrument. In the strictly acoustic world, the only parallel is moving to another room or, even less intuitive, rebuilding the room around you. To make these operations more intuitive, the Audio folio bases its sound generation and processing on the earliest and easiest to understand synthesizer: the modular analog synthesizer.

A modular synthesizer is a collection of modules and patch cords. Each module can produce an electric signal or modify an incoming electric signal. For example, one module produces a sawtooth wave signal. Another module accepts the sawtooth signal and then filters out all high frequencies. Yet another module accepts the filtered sawtooth signal along with three other incoming signals, merges them, and sends out a stereo pair of signals.

Analog synthesizer modules typically have input ports to accept signals, output ports to send signals, and controls (quite often knobs) that control the actions of the module. Combining the work of the modules creates rich signals and sends them out to the speaker system, where they're translated into sounds. The patch cables combine the modules. For example, the output of the sawtooth wave generator is connected by cable (it is "patched") to the low-pass filter, which filters out high frequencies. The resulting signal is patched into a ring modulator module that adds some grit to the signal, and then passes it on (via another cable) to a mixer that adds the signal to other signals. The mixer then passes the results out (via more cables) to the amplified speaker system. The output sound is the result of the combination of modules patched together by cables.

As a sound plays through a patched set of modules, the individual modules can be tweaked by changing their settings. This modifies the quality of the sound. For example, the module combination described above may include a knob on the low-pass filter module. When the knob is turned up or down, the filter cutoff point moves up or down, letting more or less high frequencies through. As the synthesizer user turns that knob up and down, the resultant sound gets more or less shrill as higher frequencies pass and are filtered out.

The Audio folio provides the equivalent of modules, controls, and cables with instruments, knobs, and attachments. Each instrument generates or modifies a digital signal; each knob on an instrument controls the way the instrument works; and each attachment connects the output of one instrument with the input or knob of another instrument. Audio folio calls work like the hands of a musician, making attachments between instruments and finally sending the output to the DACs, where it's converted to an analog signal and sent on to amplified speakers. Creative instrument connections and control make rich and variable audio possible.

Instruments

Each instrument used by the Audio folio is created externally and provided in a large library of instruments that comes with the 3DO Toolkit(TM). Those instruments can be combined to create new instruments using software development tools such as ARIA or Patch Demo. The new instruments can then be used alongside the standard instruments. It's important to note that the Audio folio does not create instruments internally.

An instrument is stored on a CD (or on the Macintosh(TM) in a development system) and, when needed, it is loaded into DRAM. The Audio folio allocates DSP resources to handle the instrument and makes the instrument available to tasks to play or modify sounds and to connect to other instruments.

Instruments can vary from simple to complex. Some are simply mixers or reverb units; others are completely self-contained sound synthesis environments, comparable to a commercial digital synthesizer. Each instrument can contain several types of elements:

Inputs, which accept a signal from another instrument.
Outputs, which send a signal from the instrument to another instrument or to the final audio output.
Knobs, which accept control input for the instrument.
Sampled-sound pointers, which point to a sampled sound waveform table that the instrument can play.

The number of inputs, outputs, knobs, and sampled sound pointers available to an instrument is limited only by the resources available in the DSP, but a successful instrument should be frugal. An instrument with too many inputs, outputs, knobs, and sampled-sound pointers is a resource hog and cannot coexist with other instruments on the DSP.

A task must use an instrument to play any kind of sound or note. For example, to play back a sampled sound a task must use a sampled sound instrument, which is a very simple instrument that simply points to the sampled sound and plays the sound whenever instructed. The task can then send the sampled sound through other instruments for modification, or it can send its output directly to the DACs. When a task uses a more complex instrument, such as a full-blown synthesizer, it often doesn't need to use additional instruments to fatten the sound; the synthesizer has enough internal capabilities to come up with the necessary sound.

Each instrument, no matter how simple or complex, is DSP code that runs during each DSP cycle to create an ordered set of sample values that represent sound. When the Audio folio allocates DSP resources to an instrument, it assigns an input DMA channel for each sampled sound input, an I memory location for each standard input and output, an output DMA channel for optional digital output, and an external input memory location for each instrument knob. Memory is also allocated for programs and DSP "ticks" or instructions cycles. These resources can be exhausted, thereby limiting how many instruments you can play at the same time. When multiple instruments play, their code is located in N memory, daisy-chained together in order of priority, and their other resources are allocated appropriately.

Knobs

Each instrument knob controls an instrument parameter. Knob parameters are defined when the instrument is defined by the audio workshop tool. Some knobs control simple parameters such as frequency or amplitude. Others control more abstract parameters such as the timbre used in synthesis. Each knob is identified by a name.

When an instrument is first loaded into the 3DO system, all knob-controlled parameters are set to default values. To change any of those parameters, a task must first grab the knob for the parameter it wants, by specifying the knob's name in the audio call GrabKnob(). The task then tweaks the knob to the setting it wants. Knob tweaking can be a one time event to create a new and steady setting, or it can be a continuous event to create a changing setting. For example, a task can tweak a filter knob up once to create more shrill sounds, or it can move the filter knob up and down to create a sound that audibly alternates between shrill and mellow.

If, at any time, a task needs to check on the setting of a knob, it can peek at the knob and get a return value of the knob setting. When the task is finished with the knob, it releases the knob, which retains its setting value at the time of its release.

Digital Audio Samples

Some instruments can play digital audio samples from RAM using one of the DMA/FIFO channels. Samples can be played by attaching them to instruments. A sample can be loaded from an AIFF file on disc, created from scratch, or come from any other possible 3DO source. Samples can also be included with the instrument definition. A sampled-sound instrument cannot play sampled sound files or tables directly from a compact disc.

Many instruments contain multiple sampled sound pointers so that the instruments can use a different sampled sound for different pitch ranges. For example, a sampled piano instrument can use one sampled sound for its lowest octave, another sampled sound for the next octave up, and so forth for each octave of its range. Multiple samples avoid stretching or shrinking the original sample too much for higher and lower pitches. This makes playback more realistic.

Playing Notes

Once an instrument is loaded and connected, it does nothing until instructed to play a note. The word "note" implies a pitched sound of a certain duration, part of a musical score, but that is not the case in the Audio folio. A note here can be any sound, from the plucking of a guitar string to the rumble of a race car. Notes are used to play music and also to provide sound effects. For example, a sampled sound instrument can be set to play a "boom" whenever an explosion appears on screen. When a task asks the instrument to play a note, the instrument sends a "boom" out through the speakers.

Note Stages

The quality of a note over time is set in an instrument definition. However it is shaped, a note must have three stages: a start, a release, and a stop. The instrument determines how the note sounds at each stage. For example, consider a trumpet note. When the note starts, there's a "puh" sound followed by the blare of the trumpet. That blare holds until the note releases. On release, the blare quickly diminishes until the note stops and there is no more sound. As another example, consider a sampled sound playback. When the note starts, playback starts and can loop over through a section of the waveform table. When the note releases, the waveform table plays through to the end, where the note stops.

The concept of start, release, and stop stages correspond to the stages of a key press on a piano or similar keyboard. When a key goes down, the note starts-the string vibrates, a pipe blows, or an oscillator starts. When the key comes up, the note is asked to finish-the string is damped, the pipe stops blowing, or the oscillator stops. Because most instruments do not finish vibrating immediately, the vibrations die out and the note stops some time after the key release.

In the Audio folio, start, release, and stop are commands to an instrument that mean the following:

Start asks an instrument to begin note execution, which can start sampled-sound playback, new sound generation, or any other event meant to occur during the note.
Release asks an instrument to finish note execution, which can mean playing a sampled sound table to the end, tapering off a synthesized sound, or immediately stopping the sound. What happens on release depends on how the instrument is designed to play the note.
Stop asks an instrument to stop note execution completely and immediately so the instrument no longer produces a sound. The difference between stop and release is that a stopped note stops sounding immediately and ignores any of the rest of the note. A released note can continue playing after the release request until the note is finished.

To play a note on an instrument, a task uses the audio calls StartInstrument(), ReleaseInstrument(), and StopInstrument() to tell an instrument to start, release, or stop a note. The instrument's response to each of the three commands depends on how the note is defined and shaped within the instrument. Some instruments can have a predefined note definition (such as a sampled-sound player), others can have knobs that let a task change the note definition (something like envelope controls on a standard synthesizer). As a note progresses after its start or release, any time-based process, such as an envelope, is timed by audio ticks. These ticks are generated 240 times per second by the DSP, and can be sped up to a maximum of around 1000 times per second.

Specifying Pitch and Loudness

When a task signals an instrument to start a note, it can optionally specify a frequency and amplitude for the note, which directly sets its pitch and loudness. It can also optionally specify pitch or velocity as given in MIDI scores. The velocity information determines amplitude, envelope shape, or other sound qualities that are set by the instrument's definition. (These qualities are usually those affected when you strike a music keyboard key harder or softer.) The pitch information is fed first to a tuning table, which contains a frequency for each of the standard half-step pitches in all audible octaves. The indexed frequency goes to the instrument, which uses the frequency to play the note. The tuning table by default offers 12-tone equal-tempered tuning, but allows many other tuning systems as well for music playback. If a task wants to avoid standard half-step tuning, it can send frequencies directly to the instrument, bypassing the tuning table.

Voices

The ability to play a single note at one time is called a voice. Each instrument has a single voice. To play more than one note at a time (for musical harmony or multilayered sound effects), a task must use more than one instrument at a time, and can use multiple instances of the same instrument to create a chorus of like-timbred voices.

To play notes simultaneously, a task simply asks for a new note to start on one instrument before other instruments have stopped playing their notes.

Using the Music Library

A task can use Audio folio calls directly to time and play notes on instruments, but it is simpler to use higher level Music library calls to take care of these jobs. Music library calls fall into these categories:

Sound spooler lets you continuously send audio data to the Audio folio for playback.
Sound file player plays large sampled sound files from disc without having to load the entire sample file into memory at one time. (It uses the sound spooler).
Juggler calls create and assemble scores, and then play those scores back using a scheduling mechanism called the Juggler.
MIDI file-parser calls read a format 0 or format 1 MIDI score file from disc, and translate the contents into a score playable by the Juggler.
MIDI interpreter calls create a virtual MIDI environment and then translate standard MIDI messages into Audio folio calls that produce the proper MIDI-specified results. These calls are usually used as a simple sound manager to simplify playing sound effects.

Spooling Sound Files

Audio folio functions allow a task to play back sampled sounds in RAM. Unfortunately, being RAM-resident limits the length of the sampled sound: a very long sample can easily use all available RAM and still need more. The Music library's sound-spooling functions allow a long sample to be stored on disc, safely out of RAM, and then spooled into RAM a small section at a time for continuous and uninterrupted playback.

To spool sounds, the Music library borrows an idea from double-buffered animation: it reads from one section of RAM while it writes to a different section of RAM. To do so, it creates several sound buffers in RAM, each capable of storing a section of the sample file, each capable of playback through a sampled-sound instrument. These sound buffers are linked together and are played in circular order: from the first through to the last, then repeated from first to last over and over, as long as requested.

While circular playback of the sound buffers occurs, other calls spool a disc-based sound file a section at a time into unplayed buffers, just as double-buffered animation draws into undisplayed graphics buffers. Writing spooled sample sections into sound buffers in circular order keeps a step ahead of reading from those buffers. The result is smooth playback of a large sample file on disc. Because the spooling process runs at high priority, it is very unlikely that playback will be interrupted by the activities of other tasks.

Playing Scores

At the heart of the Music library is a family of calls that support the Juggler, a scheduling mechanism that handles lists of timed events and executes them at the proper time. To work properly, the Music library supports an object-oriented programming environment where a task can create Juggler objects containing scores (which are collections of events) or other Juggler objects. These objects can be assembled into hierarchical structures and respond to method calls changing their properties.

Without getting into details (which are described in Creating and Playing Juggler Objects), the Juggler and its programming environment support these features:

Scores that can include musical events such as notes, or nonmusical events such as calls to draw figures on the screen, retrieve information from CD, and so on.
Objects that are a collection of subobjects. This allows one object to be a collection of scores that can be played back sequentially or in parallel.
Real-time object manipulation so that a score or score collection can change structure in response to a task's requests. A score or collection changed during playback changes the quality of playback. This is useful if a task wants to change the musical ambience during the course of its execution.
Completely abstract event timing so that a score can be played back using the strict time available from the audio timer (or similar time source), or using a task's own generated time, which can depend on changing conditions within the task.

Playing MIDI Score Files

Musical scores for computers are typically created using MIDI sequencing software available on development computer platforms. These scores can be saved as Format 0 or Format 1 MIDI files, and then stored on CD-ROM for playback using Music library functions. The Music library's MIDI-handling functions can import a Format 0 or Format 1 MIDI score from disc, translate it into a Juggler object, and then store it in RAM.

To play the MIDI commands stored in a MIDI Juggler object, other Music library calls can create a virtual MIDI environment in software. That MIDI environment uses data structures to specify audio instruments assigned to respond to different MIDI messages, to keep track of notes played and voices used, to keep track of overall volume and stereo mix, and to handle other parameters commonly changed during playback of a MIDI score.

Once a MIDI score is translated and stored as a Juggler object, and a MIDI environment is created to handle playback, the Juggler can execute the MIDI events stored in those objects. Each MIDI event contains a standard MIDI message, which a MIDI translation function translates. When a MIDI translation function translates a MIDI message, it makes the appropriate Audio folio calls to carry out the message. It also keeps track of its activity in the MIDI environment's data structure. For example, when the juggler reads a MIDI Note On message in a MIDI event, it calls a Note On translation function that first finds the appropriate audio instrument to play the note, and then starts the instrument and records the fact that it is playing in the appropriate data structure.

See Playing MIDI Scores for more detailed information about MIDI score playback.