On the fly WebP decoding using WASM and a Service Worker
For the people who know me, I have been super excited by the recent developments on the web platform, such as modules, service workers and now even Web Assembly!
It is amazing seeing the momentum and the progress being made, with the W3C working group, which I am a member of, just having release its first working draft of the standard:
A little while back, back in November, I did a presentation of Web Assembly at the yearly W3C TPAC meeting — this year in the bay area:
One of the quick demos that I showed, was decoding "webp" images in both Chrome and Firefox. libwebp already had some experimental WASM support based on SDL, which Emscripten has support for via the canvas element.
That worked fine, but wouldn't it be nicer if people could just import a library and then things would just work out of the box? I thought so, because as you see above, using webp required some invasive code changes like manually calling JavaScript functions and thus doesn't work everywhere like in CSS.
Service Workers allows a site to intercept fetches of resources, and as it happens to be, you can use WebAssembly in a Service Worker too! I was super excited about spending some more time on this, but being busy with lots of things, I only had time to look at this now.
What prompted me to actually spend some time on this was Surma's great tweets and later his article about how to write a WebP decoder using WebAssembly — I highly recommend you to check that out.
Status
I am happy to say that you can indeed make a Service Worker decode WebP images on-the-fly already with some quirks that doesn't make it deployable yet :-) — or at least I haven't found a way, yet.
[March 10: Update: It can be deployed now without any experimental features, check the update section at the end of this article]
First of all, in order to get a blob that the browser can render as an image (ie. image/png blob) you need to use a canvas. Canvas is not available in workers, and thus service workers, unless you use the Offscreen Canvas, which is only implemented behind flags in Chrome and Firefox — and only works reliable in the former. The API between the two also differs.
Another issue I ran into, is that you cannot use the same WebAssembly library in both the Service Worker and the main context, at least at this point. In the screenshot above, I decoded 3 images in the Service Worker and the 4th in the main context and as you can see from the DevTools output, the latter fails.
If I turn off the service worker (Bypass for network), the first 3 images are rendered directly by Chrome, which has native support, and the 4th by the WebAssembly library:
This also works fine in Firefox, which will only show one image, as it has no native support for WebP images:
Creating the library
The libwebp library is a really simple C API:
This makes it super easy to get started. All I need to do is expose two functions, decode() and getInfo(). The latter is needed in order to get the size of the image so that I can allocate the right sizes buffers etc. Unlike Surma, I am not creating my own allocate and deallocate methods because Emscripten provides those for me in the form of Module._malloc() and Module._free() . The libwebp docs will tell me to free via WebPFree() but that method just calls the regular C version of free so using Module._free() is fine!
Setting up an automated build system using CMake was a bit more of a challenge :-) but I managed with a bit of help from good friends:
In order to make it easy to compile, I added the build commands inside my package.json so that all I need to do is call npm run build:
Approach and complications
Now we have the library we need to wrap it into something nicer, so that the user doesn't need to know about buffers, allocations, de-allocations, and the special inner working of Emscripten.
I created a mini JS library called webp-decoder.js that should work from the main context and from a worker. It exposes an async function called fetchWebPDecoder returning a WebPDecoder object with a few methods.
Emscripten will create a global Module object, or use yours if you supply it. As we want to know when everything has been loaded, we need to supply our own. Though something like var Module = {} works fine directly in the main context or directly in a service worker, it won't work in libraries imported from these places, so the below fixes that — and is very similar to what Emscripten does itself.
We now create a Promise that will resolve when everything is properly loaded, which is nice because we can just block on it ("await" it) before we do further processing. In order for the onRuntimeInitialized to ever be called, and your promise resolve, we need to load the JS boilerplate code generated by Emscripten. In a worker that means calling the sync importScripts as Modules don't work in workers yet, even though you can configure a worker as being an ES module using { type: "module" } eg:
The final code looks like this:
I implemented a simple importClassicScript for the main context, and that works fine.
What unfortunately doesn't work fine, is calling importScripts from within a JavaScript file loaded via importScripts so that will have to be done manually by the user. ES module should solve all of this, so I am really looking forward to that working in workers!
To get access to the C functions we exposed, we use Emscripten's cwrap:
But as these are quite low level, I wrap it all up in a WebPDecoder class with a few methods. Let's look at these!
The constructor takes an ArrayBuffer, for which we create a typed array, an Uint8Array as libwebp excepts an const uint8_t* pointer. We need to copy the data to the WebAssembly heap, so first we allocate a buffer and then we copy the data. For allocation, we allocate byteLength bytes using _malloc and then we copy the data using Module.HEAPU8.set() — all convenience APIs exposed by Emscripten.
Exposing version() is easy as it just returns a number, so we just call the function we got back from cwrap. The info() method is a bit more complicated. In C, we return an integer array (pointer) of the size 3, where the first value is whether the method succeeded or not, and the next two, the width and height of the image, respectively.
On the JavaScript side, we use getInfo from Emscripten to get the values at their respective offset and then we free the memory afterwards, as getInfo allocated internally.
The C decode method also allocated internally, and it returns a buffer of uint8_t values, which we need to copy to the JavaScript side and expose as an Uint8ClampedArray. We need the "clamped" array because the ImageData objects can only work with that, and the data is already clamped :-)
Let's look at the resulting code:
We decode, then we create a "view" on top of the decoded data. This is then copied as we create the Uint8ClampedArray and we can therefore free the decoded data.
An Uint8ClampedArray is not that useful for actually displaying the image on the screen, as what we really want is a Blob of format image/png. As this is a bit of work, we create a convenience method, which takes a canvas (offscreen or not, so it works on main context and in a worker):
We basically need to create an ImageData object of the raw data and then paint that to a canvas. After that we need to check whether it is an OffscreenCanvas or not and here you notice that the experimental API differs between Firefox and Chrome.
If it is not an offscreen canvas, we need to just call toBlob() but as that API is designed before the times of Promises, we need to wrap it.
Yay! We now have a nice library for decoding WebP images!
How to use
Let's start with the main context. We just manually fetch a file, get it's buffer and then decode it. The result it then added to a new img element.
As the WebAssembly code and thus my WebPDecoder is loaded async, async/await really saves the day here:
Interestingly enough, the code looks quite similar in the Service Worker:
Here we intercept all GET calls to resources ending in ".webp", fetch the resources, get the array buffer, and the WebPDecoder, decode using an offscreen canvas, and then return a Response based on the blob data. Quite nice.
As I mentioned earlier, we cannot call importScripts from inside a JS file loaded via importScripts so I create a fetchWebPDecoderWithWorkarounds() function, that does that manually as well as blocks until the WASM module is fully loaded:
In order to be able to actually block, I need to create the promise on the global stack.
The offscreen canvas has to be created on the main context and sent to the service worker:
On in the Service Worker, we have to block on it being available as well, with a similar trick:
We all that set in place, you can now decoded webp images in both Firefox and Chrome, and in Chrome using a Service Worker! I am pretty excited about the future implications of this, especially when we got ES module support in workers!
Creating canvasses and sending them around is a bit complicated so I wish there were some easier APIs for that, as the actual canvas might not be required for this use-cases. I saw that there has been a few proposals for that before, like:
That is it for now, folks! Check out the code here:
A reminder
Also, don't forget to turn on the required offline canvas flags in Chrome and Firefox. Depending on your Chrome version, you might have to turn on "Experimental Web Platform Features" as well:
An update
Domenic Denicola pointed out to me on Twitter, that the web also supports BMP images, which is great because BMP images do not compress their graphical data. Wikipedia also has a great explanation on how the format works, so it was quite straight forward building an encoder:
The resulting encoder is around a mere 60 lines. Very nice!
With that the Service Worker code gets simplified to:
The cool thing is that this now all works in Firefox, as can be seen below!
