WebAssembly Is Fast: A Real-World Benchmark of WebAssembly vs. ES6
Disclaimer: All opinions thoughts / work were made personally by me and do not represent any of my employer’s thoughts or work.
Edit (12/21/18): Added Firefox performance bug / issue
Other WebAssembly Benchmarks
Colin Eberhardt, who runs WebAssemblyWeekly on Twitter, has a great response / TL;DR to one of the micro-benchmark stack overflow questions on the problems with micro benchmarking, and how Wasm should give about a 30% increase over asm.js in a real world case. Here is a link to the paper they are referring to for the Wasm performance increase claimed in the Stack Overflow response. Also, Colin has an A M A Z I N G talk on WebAssembly. The talk has a section that does a ton of comparisons of Wasm vs. Native vs. JS performance, and the talk illustrates this in much more detail than that response linked above.
WasmBoy Benchmarking Explained
The WasmBoy benchmarking tool works by loading each of the available WasmBoy core configurations, and then runs a specified number of frames of an input ROM / Game. The time it took to run each frame of the ROM is recorded in microseconds, using the npm package microseconds. This does not use the popular benchmark.js, since benchmark.js focuses more on running the same exact code multiple times. When benchmarking frame by frame, one frame we could be doing a ton of sound processing, and the next frame could just be moving around memory. Once we have all the times that it took to run each individual frame, we can process the data into other statistical values, and visualize on charts.
WasmBoy Benchmarking Setup
I then ran it on a variety of devices, and took screenshots (and merged them together into one large full page screenshot). The devices I tested on were:
- 2015 MacBook Pro, with the 2.9GHz Intel i5 configuration
- iPhone 6s (Geekbench)
- iPhone X (Geekbench)
- Google Pixel 2 (Geekbench)
- Moto G5 Plus (Geekbench)
- Toshiba CB35-B3340 Chromebook
I tested the benchmark on all major browsers, on the major browsers each device supported. The browsers we Chrome 70, Firefox 63.0.2, and Safari 12.1. I didn’t test on Edge because Microsoft recently announced Edge will be replaced with a Chromium based browser. Feel free to use the link to the tool mentioned at the beginning of this section to test on your own devices and their respective browsers.
To keep the article shorter, we will only highlight and embed some of the results in this article. However, the images and results for all other configurations can be found in the WasmBoy repo. To interpret our results we will be using the “Sum” row in the tables to represent the performance of each core. The “Sum” represents that total time it took to run each frame added together. Also, we will be interpreting our results using a clear “X times as fast” format, as explained by this article on explaining performance improvements.
For desktop, let’s take a look at the results of Back To Color of the 2015 MBP on Chrome, FireFox, and Safari. This is because, Back To Color is the more demanding of the two ROMs tested, the 2015 MBP is what I use to develop the emulator, and has support for all three major browsers.
MPB 2015, Chrome
MPB 2015, Firefox / Edit: (Bug / Issue)
MPB 2015, Safari
For mobile let’s take a look at the Chrome and Firefox on the Moto G5 Plus and Safari iPhone 6s results for the Back To Color ROM. As stated in the introduction, I wanted to get this running on more “budget friendly” devices, as these devices will have a harder time keeping up with the emulation.
Moto G5 Plus, Chrome
Moto G5 Plus, Firefox / Edit: (Bug / Issue)
iPhone 6s, Safari
Personally, my biggest question with WebAssembly was its mobile performance. And taking a deeper look into this, you can tell WebAssembly is D E F I N I T E L Y worth investing time into. A ~60% increase of mobile web performance on Android opens up a whole new realm of possibilities in terms of what we can run on mobile browsers for PWAs, hybrid applications built with Cordova, or frameworks built on top of Cordova like Ionic. I was very pleasantly surprised to see these results and definitely made my day!
WasmBoy Benchmarking Gotchas
I wanted to make sure we highlighted some Gotchas before closing out the article. By “Gotchas”, I mean some things about the benchmarking tool or WasmBoy that could possibly give skewed results.
Another Gotcha that may exist here that would actually improve our Wasm results, is the overhead that lies in jumping between Wasm and JS. Even though the over head in jumping is small, and continually being improved, it is still there and counted by this tool. This is important to bring up, because maybe you are building an application that does not need to jump back and forth 60 times per second to update the screen. Maybe you can have a long running task that doesn’t require as many calls per second, and this will actually improve your Wasm performance relatively.
Thanks for reading this ridiculously long deep dive into Wasm performance! I hope you got as much out of this as I did. Let’s do a quick recap of the original questions we had in the introduction:
- “Was I able to bring playable Gameboy emulation to budget mobile devices and Chromebooks using WebAssembly?”
Yes! Looking at the results of the Moto G5 Plus, 44 FPS average doesn’t seem like it would be playable, and you are right, it isn’t. But what this benchmark doesn’t show is that I put a bit of elbow grease into some configurable options to increase performance, sometimes at the expense of accuracy. For example, I have an option for “Batch Processing” (or Lazy Evaluation), where we won’t update anything about the sound until we actually need to do something with it, allowing us to just skip it most of the time for silent sections, or just sustained notes. Also, the npm install-able WasmBoy Lib which has a cleaner / easier to use API than the core was eventually rewritten to modularize its intensive code into Web Workers. This greatly improves performance, along with other options, and gets even demanding ROMs like Back To Color running at 60 FPS on the Moto G5 Plus! Which then let me build a Vaporwave inspired PWA GB / GBC Emulator built with Preact and Preact-CLI, called VaporBoy (Source Code).
- “Will WebAssembly allow web developers to write almost fast as native code for web browsers, and work alongside the ES6 code that we write today?”
Honestly, I will say that this depends on a lot of factors. We can tell from these results that it depends on your device, browser, language, and use case for WebAssembly to answer this question. If you plan to use WebAssembly on desktop Chrome, written in AssemblyScript, for Wasm’s Intended use case (Computational heavy-lifting tasks), then yes, Wasm is about 30% faster. But on mobile it can be much faster at around 60%, and on firefox it can be much much faster around 90%. The only time we really observed it being slower than 30% was on safari, but more so because it’s JS engine handled WasmBoy very well, not because it’s Wasm engine wasn’t that much faster.