[audioworklet] AudioWorklets integration with WebAssembly

[juj]

Reading the AudioWorklet reference at /~https://github.com/WebAudio/web-audio-api/wiki/AudioWorklet-Examples and /~https://github.com/WebAudio/web-audio-api/wiki/AudioWorklet-IDL-Proposal, it looks like code that would utilize WebAssembly to render audio in an AudioWorklet would look something like the following:

registerAudioWorkletProcessor('WasmInAudioWorklet', class extends AudioWorkletProcessor {
  constructor (options) {
    super(options);

    // Compile and Instantiate Wasm Module here, communicate with SharedArrayBuffer with the remaining program
    this.wasmModule = ...;
    this.wasmInstance = ...;
    this.wasmHeap = new Float32Array(...);
  }

  process(inputs, outputs, parameters) {
    for(var i = 0; i < inputs.length; ++i) { // *
      var input = inputs[i]; // *
      for(var j = 0; j < input.length; ++j) this.wasmHeap[... + j] = input[j]; // *
    } // *
    this.wasmInstance.processAudio();

    for(var i = 0; i < outputs.length; ++i) {  // *
      var output = outputs[i]; // *
      for(var j = 0; j < output.length; ++j) { // *
        output[j] = this.wasmHeap[... + j]; // *
      } // *
    } // *
  }
});

Talking with WebAssembly developers and @lukewagner, he is suggesting that it would be made possible to register the WebAssembly heap at AudioWorklet construction time, so that one would be able to optimize away the copies on the lines marked with stars // *.

Would something like this be feasible?

[hoch]

to register the WebAssembly heap at AudioWorklet construction time

Do you mean AudioWorkletNode's object construction? Or the construction of the entire AudioWorkletGlobalScope? AudioWorklet per se is not an instantiable object. It is a singleton owned by window.

Would something like this be feasible?

Does this mean that we have to introduce some AudioWorklet-specific WASM behaviors? It sounds like WorkletGlobalScope or AudioWorkletGlobalScope should embrace such change.

PS: The exmaples on Wiki are not up-to-date, so please consider referring the current spec. I'll update the wiki pages so it won't get looked at any more.

[lukewagner]

I'm unfamiliar with the basic spec concepts here, so forgive me for imprecise terminology, but I would think that we would register the memory (to use as input/output) once per independent "audio stream" of which there could be multiple per global.

Also, I was imagining the registration would pass in the memory as typed arrays which means the API would work for both JS (and similarly be an optimization for JS) and wasm, not semantically caring which was used.

One potential problem and a question: judging from the code sample, inputs and outputs aren't simply arrays, but arrays of arrays. If the number of input/output arrays is static, then this seems easy enough to fix, you just register arrays of typed arrays up front. But if the number of arrays is dynamic, something fancier would be required.

[rtoy]

In general, the arrays are dynamic. But if you set up channelCount and channelCountMode appropriately, you can make it static. (I think channelCountMode = "explicit", channelCount = however many channels you want. Then the number of arrays is fixed.)

[lukewagner]

Ah, good to hear. Given the complications of a good solution for a dynamic count, perhaps this feature could be stipulated to require or set an explicit channel count.

[rtoy]

Which feature are we talking about here?

I think it's critically important that an AudioWorkletNode at least support dynamic channels for a single-input/single-output case. Then it is conceptually possible to define every current node with an AudioWorkletNode. And that's a great indication that the AudioWorklet has minimum set of features to be generally useful in constructing new nodes.

[lukewagner]

The feature of (in addition to what currently exists) being able to say up front (before process() starts being called) "use these typed arrays for input and output". In theory (I'd be interested to hear from implementers), this could allow less copying.

[hoch]

@lukewagner

But if the number of arrays is dynamic, something fancier would be required.

Yes, this exactly is AudioWorklet's case. I believe that the 1-in-1-out node will be the majority of use cases.

The feature of (in addition to what currently exists) being able to say up front (before process() starts being called)

I think what you're asking might be relevant to this issue. Basically this storage will be a static one, so you can cache it on the global scope then it is accessed by all other objects in the scope. Not a pretty picture.

What kind of API are you envisioning? How can we do "I need this memory upfront, but expose it only inside of process() method"?

[lukewagner]

One more-specific API idea would be to pass the array-of-typed-arrays for input and output to the AudioWorkletProcessor constructor as arguments. IIUC, that has the right associativity.

Question: does the number of input/output channels only change in direct response to the user calling some JS method/setter or does it ever change implicitly in response to something else? If the former, then perhaps said method for changing the number of channels could also take a new array-of-typed-arrays (matching the new number of channels). Such a method would also allow being called with the same number of channels as before, which would give the user a way to change what what memory was being used at runtime.

[rtoy]

The number of input channels can change by itself, sort of. Say you have an oscillator connected to your worklet. It's mono. Now connect() an AudioBufferSource with 5.1 channels to it. The number of input channels changes to 6. This is under your control. But when the AudioBufferSource is done playing, it can disconnect itself from the worklet at some point, and that changes the input count back to 1.

If this really matters to you (and it seems it does), you can arrange things so that the number of input channels is fixed (using channelCount = desired count, channelCountMode = "explicit"). It's then up to you to figure out if that's really appropriate for you application.

[lukewagner]

Ah, I see, thanks. So maybe if you specify your buffers up-front, then that automatically sets channelCount = number-of-given-channels and sets chanelCountMode = "explicit"?

[hoch]

One more-specific API idea would be to pass the array-of-typed-arrays for input and output to the AudioWorkletProcessor constructor as arguments. IIUC, that has the right associativity.

Currently AudioWorkletProcessor is not user-instantiable object. It is implicitly created when user creates AudioWorkletNode in the main global scope. This is a cross-thread operation, so any transaction between two instances will be serialized.

[lukewagner]

Ah, ok, I was reading Jukka's code in the first comment and thinking these arrays would be passed as arguments to the super() call.

[padenot]

@padenot pinging myself so I answer here tomorrow or something.

[joeberkovitz]

From the WG discussion today:

• We feel that Wasm is usable with the current AudioWorklet API, noting that more copying does take place than is strictly necessary -- but while the copying is annoying/inelegant, it's unlikely to be a big performance issue.
• We don't want to further complicate the API for the non-Wasm use case, i.e. we'd like to layer any further options on top of the current proposal rather than fundamentally changing it. (Just as an example, we could allow internal methods to be called to configure special buffer management inside the AudioWorkletProcessor constructor.)
• We recently enhanced AudioWorklet to dynamically adjust the number of channels based on the input to an AudioWorkletNode, and also allow static input/output channel configuration on a per-node basis.

Our conclusion is that we'd like to defer the issue of optimizing copying to the next version of Web Audio, so we come up with the best of the multiple possible approaches to this problem. In the mean time we'll learn a lot about where the meaningful gaps exist in AudioWorklet in its current form.

[joeberkovitz]

Just noting that we talked directly with the WebAssembly folks about this today at TPAC and resolved to keep working on it, taking a good look in the near future at a "bespoke tailored" connection between AudioWorklet and WASM.

[karlt]

An extension point to consider when designing new APIs is an alternative registerProcessorV2() method on AudioWorkletGlobalScope. From a wasm perspective, registering a function would be more natural than a class.

[rtoy]

Teleconf: Profiling the cost of copying to and from the WASM heap would be quite interesting.

[padenot]

An extension point to consider when designing new APIs is an alternative registerProcessorV2() method on AudioWorkletGlobalScope. From a wasm perspective, registering a function would be more natural than a class.

It can be made to work without changing the name of the function, and instead dispatching based on the shape of the second argument. If it's an ES6 class, it has to conform to the required prototype.

If it's an object, it needs to have the right keys, and those keys need to be functions. They can have the same names, and be implemented in js or directly in WASM. This clealry ties in to WebAudio/web-audio-api-v2#4. When WebAudio/web-audio-api-v2#4 is solved, then I believe the solution for WebAudio/web-audio-api-v2#5 will come naturally.

[carlsmith]

The original example above contains this block:

// Compile and Instantiate Wasm Module here, communicate with SharedArrayBuffer with the remaining program
    this.wasmModule = ...;
    this.wasmInstance = ...;
    this.wasmHeap = new Float32Array(...);

In practice, you cannot instantiate a Wasm module in a constructor like that, as instantiation is asynchronous, so the exports must be assigned to instance variables in a callback (after the constructor has returned, and possibly after the process method attempts to call an exported function). The issue linked to immediately above addresses that problem specifically.

The Wasm memory can (and often must) be directly created within the constructor, as you cannot (in practice) pass a buffer more than a few MB as an argument (from the main thread) without the audio thread dropping out. See #2269 .

[mdjp]

This would be a great issue to discuss across groups as TPAC 2021

[hoch]

At this point, I feel like this issue almost looks like #2442. I suggest marking this as duplicate and focus on the other thread since it has a proposal by @padenot.

[hoch]

@juj Since you opened this issue years ago - the WG believes that prioritizing registerBuffer() proposal over this issue is more sensible at this time. What do you think?

[juj]

Yes, registerBuffer() would directly address this need.

[hoch]

Thank you. Then I will close this issue and the WG will focus on #2442.