How SoundManager 2 Works

SM2 = JavaScript, ExternalInterface and Flash

After a number of years of hacking away at this thing, I figured it'd make sense to detail some of the inner workings and design decisions behind SoundManager 2.

In brief, a Javascript API for sound is presented to users, and internally the JS calls Flash methods (and vice versa) via ExternalInterface to make the actual sound functionality work. Flash is running the audio, but passes the relevant status back to JavaScript-land for implementation by the user.

One would assume JS/Flash features would be trivial to implement, but the reality involves a fair bit of fiddling and troubleshooting. Creating the sound methods and logic in Flash is fairly simple, but extending it to JS-land is where things get interesting.

Building the JavaScript-side API

Flash 8, which introduced ExternalInterface, has an API for loading, playing and controlling sound. In exposing this to JS, numerous methods must be written and some state must be passed and maintained between JS and Flash.

Simply exposing Flash's API could be done, but SM2 wraps the exposed Flash methods and provides a more JavaScript-friendly API for sound including individual SMSound object instances with their own methods and properties, which live under the soundManager controller.

soundManager.createSound() maps to a parallel createSound() method on the Flash side, where a Sound() object is made and stored in an array. JavaScript also creates a similar sound object, an instance of soundManager.SMSound(), with methods and properties that effectively map to the Flash object. For example, calling a SMSound instance's play() method results in JS-side state changes to that object's playState and paused properties if applicable, and a subsequent call to an "exposed" JS/flash method on the Flash movie DOM element including several simple string or integer parameters. In the play() case, the JS/Flash method is flashMovie._start(soundID, loopCount, positionOffset);

The primary sound methods shared between JS and Flash for SoundManager 2 are as follows:

soundManager.createSound() -> flashMovie._createSound(soundID:String, url:String, onJustBeforeFinishTime:Number, usePeakData:Boolean, useWaveformData:Boolean, useEQData:Boolean, isMovieStar:Boolean, useVideo:Boolean, bufferTime:Number);

SMSound.play() -> flashMovie._start(soundID:String, loopCount:Integer, positionOffset:Number);

SMSound.stop() -> flashMovie._stop(soundID:String, stopAll:Boolean);

SMSound.setPosition() -> flashMovie.setPosition(soundID:String, positionOffset:Number, isPaused:Boolean);

SMSound.pause() -> flashMovie.pause(soundID:String);

SMSound.setPan() -> flashMovie.setPan(soundID:String, pan:Integer);

SMSound.setVolume() / flashMovie.setVolume(soundID:String, volume:Integer);

Maintaining State, Efficiently: Events and Polling

While sound objects in Flash are active and firing interesting events (eg., loading, playing or property changes), Flash will periodically call methods on the relevant soundManager.SMSound instance to update changed properties or fire events, accordingly. In order to maintain state, SMSound objects receive updates for the play, stop, onload, onfinish, whileloading and whileplaying events at a minimum, and will fire events for any user-assigned subscribers. For example, a Flash sound object's onload() event may fire, which in turn results in a call to an SMSound._onload() method, which may then pass through to a user-assigned SMSound.onload().

Nearly all interesting events can be subscribed to on the Flash side - load, play, finish and so on - but whileloading and whileplaying are not native to the API, and needed to be added within Flash and JS. To receive regular updates, a single interval timer fires within Flash that loops through all active sound objects, checking for state changes (eg., increases to the sound's bytesLoaded or duration properties.) If changes are found, a call is made to JS for the given SMSound instance with the relevant properties and their updated values, and the Flash sound object is then marked as being up-to-date.

Given use cases, the JS-side API should always be updated with the latest sound state as the user could read SMSound attributes at any time. Events are used internally by the API also for updating duration, position, bytes and other properties; if a user has an event handler assigned, then their code is called at the same time as well. It should be obvious, but it is important to reduce and optimize the number of calls between Flash and JS in either direction given the potential expense and overhead associated with each call.

ActionScript Events to JS

A straightforward example of a sound object's onload() handler in Flash, and how it makes the ExternalInterface call to JavaScript-land:

// Flash 8 / AS2
mySound.onload = function (loadedOK) {
  // call soundManager.sounds[soundID] event handlers with updated values
  // ensure _whileloading() is called with latest values...
  ExternalInterface.call("soundManager.sounds['" + this.sID + "']._whileloading", this.getBytesLoaded(), this.getBytesTotal(), this.duration);
  ExternalInterface.call("soundManager.sounds['" + this.sID + "']._onload", loadedOK);
}

Polling-based updates to JS

The whileloading() and whileplaying() events are implemented similarly, though driven by a single timer:

// Flash 8 / AS2
var checkProgress = function() {
  var oSound = null;
  for (var i=0, j=sounds.length; i<j; i++) {
    oSound = soundObjects[sounds[i]];
   /* PSEUDOCODE...
    *
    * if (oSound.getBytesLoaded(), getBytesTotal(), duration or position changed..) {
    *   ExternalInterface.call("soundManager.sounds['" + oSound.sID + "']._whileloading", bL, bT, nD);
    *   if (position has changed).. {
    *     ExternalInterface.call("soundManager.sounds['" + oSound.sID + "']._whileplaying", nP);
    *   }
    * }
    *
    */
  }
}
var timer = setInterval(checkProgress, timerInterval);

This approach is about as simple as it gets, doesn't try to be too smart, and simply calls the relevant SMSound instance methods with basic numeric data. It works quite well, and is fast.

Error handling between JavaScript and ActionScript

Occasionally, a JavaScript call to an ActionScript function will fail with an exception that originated within the flash movie. Flash will provide an error as such: uncaught exception: Error calling method on NPObject! [plugin exception: Error in Actionscript. Use a try/catch block to find error.] The debug message, while not immediately helpful, is accurate. By wrapping your ActionScript function code in a try..catch block, you can troubleshoot errors and write them console.log() style back to JS-land if you wish.

A try..catch example:

// Flash 8 / AS2
function badFunction() {
  try {
    causeAnError(); // method does not exist, etc..
  } catch(e) {
    ExternalInterface.call('console.log', 'Flash error: '+e.toString());
  }
}

Initializing Flash and ExternalInterface

To ensure JS/Flash support is present and working, the Flash movie reference must be created, loaded and a start-up test run. Ignoring the fun of object/embed differences for the moment, the process goes something like this:

  • Write Flash movie (object/embed) to DOM
  • Obtain DOM reference to Flash movie, depending on browser, via window['myMovie'] for IE, else document.getElementById('myMovie') or document['myMovie'] if the former is null for Safari. Perhaps this is related to the way ExternalInterface is executed between the different browsers - but frankly, it's a mystery.
  • Attempt to call a method on the Flash movie from JS; eg., swf.externalInterfaceTest(); using a try..catch block. If an exception (null/undefined) is caught, Flash plugin may not be present, missing SWF, flash blocker in effect, flash security blocking access to movie due to offline or cross-domain rules, or other error (corrupt install, or Flash version < 8 perhaps.)
  • If flash externalInterfaceTest() call runs (called from JS), it attempts to make a return call to a JS externalInterfaceOK() method, also within a try..catch block. If this succeeds, then bi-directional ExternalInterface communication is presumed to be OK and SoundManager 2 fires its onready/onload() handlers.

Object/embed quirks and Other Annoyances

When document.getElementById() doesn't always work (!?)

As noted above, you should assign your name/ID carefully and reference the movie in the DOM depending on what returns as being "truthy." SoundManager uses return isIE?window[movieID]:(isSafari?_id(movieID) || document[movieID]:document.getElementById(movieID)); for its getMovie() function, which has been well-tested over the years.

allowScriptAccess object/embed Property

This property is a bit odd, but should be set to sameDomain or more-freely, always (vs. "never") to allow ExternalInterface to do its work. In any event, Flash by default will not let the hosting page on domain A script a SWF on domain B, even if allowScriptAccess="always".

wmode (window vs. opaque/transparent) Weirdness

In a nutshell (augh, help, I'm in a nutshell!): If you set wmode to a non-default value such as opaque or transparent, a different rendering mode kicks in and weird things happen. This may also relate to using position:fixed and/or position:absolute.

Yes, you can potentially have the flash movie render with transparency, be "invisible" and use z-index to layer things on top of it, but, Firefox - possibly only on Windows - seems to then behave oddly and will not load the .SWF until it has scrolled into view. (This may actually be a good thing for most normal, non-SM2 cases.) This behaviour is bad for SM2, which cannot start until the movie has loaded and will fail with a timeout if the page loads and no initial Flash<JS "callback" is made within a reasonable amount of time. SM2 does have a built-in check for this edge case, and will reset wmode to its default when you have specified a non-null wmode value without specifying an infinite timeout. The movie will then load normally.

ExternalInterface: Implementation Notes

XML serialization JavaScript methods, injected(?) into the browser

ExternalInterface almost seems like one of Flash's dirty little secrets. It's an interesting implementation and usually "it just works" etc,. but just under the hood is some interesting stuff. If you inspect the DOM tab in Firefox, you'll see Flash adds several methods to the window.parent object; calling alert(window.parent.__flash__argumentsToXML), for example, will return the actual function code.

  • __flash__argumentsToXML(obj, index)
  • __flash__arrayToXML(obj)
  • __flash__escapeXML(s)
  • __flash__objectToXML(obj)
  • __flash__request(name)
  • __flash__toXML(value)

As you can see, these helpers do XML serialization of JS->AS method arguments before they're passed to Flash. __flash__argumentsToXML() gives an idea of the implementation:

function __flash__argumentsToXML(obj, index) {
  var s = "<arguments>";
  for (var i = index; i < obj.length; i++) {
    s += __flash__toXML(obj[i]);
  }
  return s + "</arguments>";
}

ExternalInterface Performance Considerations

Given the weight of XML serialization and potential impact on performance it's most logical to use flat objects with no nesting (no object literal/JSON structure stuff). Keep it dumb, and simple. Skipping arrays for a single string that you can perform split()/join() on both JS/AS sides, presumably, should be much more efficient than XML serialization of N array items and concatenation of an increasingly-large string. In the case of SoundManager 2, sometimes a 2x256-item array of numbers from 0-255 is passed over the ExternalInterface bridge for fancy Flash 9-based spectrum/waveform data, so efficiency is a concern.

Consider the following example, where a JS array is passed to ExternalInterface and serialized to XML vs. passing a comma-delimited string - flashMovie.myExternalInterfaceMethod([1,2,3,4,5,6,7,8]); serializes to:

<invoke name="myExternalInterfaceMethod" returntype="javascript">
  <arguments>
    <array>
      <property id="0"><number>1</number></property>
      <property id="1"><number>2</number></property>
      <property id="2"><number>3</number></property>
      <property id="3"><number>4</number></property>
      <property id="4"><number>5</number></property>
      <property id="5"><number>6</number></property>
      <property id="6"><number>7</number></property>
      <property id="7"><number>8</number></property>
    </array>
  </arguments>
</invoke>

Compare with flashMovie.myExternalInterfaceMethod('1,2,3,4,5,6,7,8'); which can then have a split(',') performed in AS:

<invoke name="myExternalInterfaceMethod" returntype="javascript">
  <arguments>
    <string>1,2,3,4,5,6,7,8</string>
  </arguments>
</invoke>

You can see where this is going.

For fun, here's an object literal example, myExternalInterfaceMethod({foo:'bar', baz:'biff'});..

<invoke name="myExternalInterfaceMethod" returntype="javascript">
  <arguments>
    <object>
      <property id="foo"><string>bar</string></property>
      <property id="baz"><string>biff</string></property>
    </object>
  </arguments>
</invoke>

The same logic, I believe, applies to the Flash -> JS direction as well - so again, it's best to use simple strings or numbers wherever possible.

Flash's Security Model

In short, Flash will not allow the following scenarios by default:

  • HTML on domain A scripting methods in a SWF on domain B
  • HTML served via file:// or c:/ or otherwise-offline (not over HTTP) viewing attempting to script local or remote SWF, since Flash can access network/internet resources.

Same-domain over HTTP is the most common use-case, and works without issue - provided Flash is present, the movie is able to load and is not blocked by FlashBlock/ClickToFlash et al.

Crossdomain.xml and other quirks

Flash will not read "metadata" including ID3 tags, waveform and spectrum data from resources that are on a remote domain unless it is granted permission to do so, typically done via crossdomain.xml on the root of the remote site. Flash can also sometimes be picky about the bitrate of MP3s, disliking bitrates < 32 kbps and > than 256 (or 320) kbps. 48 Khz sample-rate MP3s also may not play properly unless pre-loaded first. Flash 8 cannot really "cancel" a loading MP3 request without loading a new URL, or destroying the current sound object.

Was it good for you, too?

This has been an overview of some of the more interesting findings from working on and maintaining SoundManager 2 over the years. It's a bit bizarre - but if you can learn to live with and work around the quirks, you can make some pretty fun stuff.

As they say on that website, [that is all].

Related links