Architecting an Audio Plugin Host as Web Extension
The following text is submitted to the 9th Panhellenic Conference «Acoustics 2018», that will be held in October 8th-9th, 2018 in the Conference and Cultural Center of the University of Patras, Greece.
Listening to sounds produced by the browser is a highly common phenomenon in modern era. Nowadays, audiovisual content is available from multiple online sources ranging from websites, free video / tv / radio providers and social networks to paid content services for movies, music and training i.a.
In principle, listeners are unable to adjust the browser’s sound in any way, besides the primitive volume control, as none of the major browsers offers any kind of such functionality. Modern browsers though, can add new functionality to sites and the browser itself through script-based extensions (or add-ons). At first glance, the solution of utilizing browser web extensions processing audio, seems to be sufficient. True, for simple operations and demands. If one though, wishes to utilize more than one web extensions processing audio simultaneously, most likely is going to hit a wall as only one extension at a time is allowed to process audio in major browsers. Moreover, most of the existing extensions in the market are limited to particularly basic audio operations.
The work presented here is focused on overcoming the above problems, by architecting one web extension encapsulating a Javascript audio plugin host, based on HTML5, Web Components and Web Audio API (WAA). The host UI allows the listeners to dynamically add an arbitrary number of the available audio plugins and connect them in any sequence. Multiple instances of the same audio processor are allowed (e.g. two equalizers — one for the left channel and one for the right one). The audio plugin host is currently focused on audio effects. Advanced plugins are implemented such as HRTF-based 3D audio via headphones and virtual surround over two speakers based on crosstalk cancellation. The challenges behind the approach are numerous related to both performance and the very nature of web extensions.
An extension’s architecture depends on its functionality, but most extensions will include multiple of the available components: (a) manifest, which must always be included (b) background script, (c) UI elements (popup page, icons, etc.), (d) content script injected into the web content and (e) options page. The actual audio processing can take place in either the background script of the content script. The UI of each audio processor can be in a popup window or in the options page. In any case, the background and the foreground (UI) processes live in their own spaces and they can only communicate through asynchronous messaging. Moreover, the separate processes should share the same state so that they will always be in sync.
The proposed solution is based on background scripts for audio processing and popup pages for UI. For browsers not supporting audio stream capture at background, a fallback method is used based on processing in content scripts. It also utilizes reactive JavaScript, local storage and Redux, which is a predictable state container for JavaScript, for the state sharing between the background and the foreground processes.
The performance challenge is critical, especially for custom audio operations not directly supported by WAA. In the proposed solution, these operations are implemented in web assembly and/or AudioWorklets, which were recently introduced in Google Chrome. For browsers not supporting AudioWorklets yet, fallback processing is used with ScriptProcessorNodes.
In summary, a novel audio plugin host as web extension has been developed solving the problems described above. Ongoing work is focused on performance enhancements, audio synthesis plugins and loading audio plugins from remote hosts.
