A Brief History of Polarr Photo Editor, and the Last Legs of WebKit?
When we launched the very first version of Polarr Photo Editor, it was completely written with web technologies. To my surprise, our second version of the app, which was released in early 2015, is still working as expected, in very good shape compared to our version 6 which is currently available for iOS Testflight, and due to make its debut to the public in a couple of weeks.
We were blessed to have ridden the wave of fledging web stack five years ago. As you observed, our editor is quite snappy and works completely out of a web browser. This was contradictory to the conventional photo editing software we grown to love as kids. I remember installing Adobe Photoshop 5.0 as a child via a CD on my P5 running Windows 98. Setting up the software and booting it up took hours. But even in 2014, in order to use Photoshop CC, it required downloading gigabytes of binaries and installations that took close to an hour. Furthermore, users needed to have a powerful PCs/Macs with decent memory and processing speed to editing photos with a smooth experience. Polarr Photo Editor did not require high-spec hardware, actually, it did not even need to run any installation script. Users were able to edit photos within seconds from their favorite browser directly on a lowly $300 USD Acer Chromebook when it was launched in 2015. Our UI and underlying logic were written completely with HTML5, CSS, and Javascript. And as for rendering, popular to contrary beliefs where people thought the processing was done through a remote server, all of the image rendering was done locally via a technology called WebGL. WebGL allowed us to access local GPU directly without knowing the underlying operating system or graphics card make. The same code could be served on a iMac with a NVIDIA GTX through Safari or a Windows 10 machine that had an integrated Intel GPU.
Our real big break came in 2014 when iOS 8.0 officially supported WebGL which meant our app was able to run on the iPhones and iPads not just through the Safari browser, but released as an app served through WebKit. Contrary to what most existing apps written using web technologies and embedded through a WebView on the Apple App Store (Safari, Native Mail App, and yes, the App Store itself), Polarr utilized every single ounce of GPU resources made available by WebGL to compute simple shaders of the likes such as color temperature and exposure, all the way up to the most complex flows such as denoise and dehaze. The cross-platform compatibility gave the team huge boost in productivity and reach. We didn’t need to employ a large iOS or Android development team to ship our apps on these platforms. In fact, even as of today, our Polarr Photo Editor team only has 3 frontend developers with a total reach of 5 million monthly active users. Pushing updates to Windows, Chrome, iOS, MacOS, Android, Linux, web were all a breeze. The same Javascript, CSS, WebGL code was running on all these platforms without modifications. A funny anecdote — when we submitted our editor to the Universal Windows Platform App Store on Windows, the reviewer initially rejected our app as it had a localhost running on one of the ports when the app started and thought it was doing something malicious. We had to get explicit approvals from our internal Microsoft contact who knew what web technologies we used. The app received a lot of attention from the press, Business Insider called us a more advanced Instagram that had a quarter million downloads on iOS 48 hours upon launch, CNET did a cover story on the web technology behind Polarr. At that time, we were a team of 8 and already throwing punches above our weight class.
Over the years, we have reaped the rewards of web technologies but we have also seen an influx of native development and platform fragmentations that made developing Polarr through a universal web-base more and more difficult. For instance, a large part of our technology stack is on doing computer vision and deep learning directly on the device without uploading any content to the servers. Over the last few years, hardware phone manufacturers each have beefed up their deep learning frameworks to support image recognition use cases. We for example leverage deep learning to run segmentation on photos, to allow users easily select masks for editing purposes. To run these very complex neural networks and compute large matrix computations, companies like Qualcomm, Apple, Samsung, Huawei, Xiaomi, etc all developed their internal proprietary inference engines to harness the local compute to ensure image recognition and processing can happen in a realtime basis (>30FPS). Unfortunately, this is one area that has been lagging behind on the web front. While doing computations of neural networks are fast, which simply involve matrix multiplications, additions, clamping, and comparisons, moving data around is exponentially more expensive and time consuming. There have been a number of frameworks that began to support inferencing directly in the browsers such as tensorflow.js, ONNX.js, and WebDNN, comparing their performance against native implementation such as CoreML on iOS, SNPE on Qualcomm Snapdragon SoCs, or SNAP by Samsung is still abysmal. And for us, in order to leverage some of the latest layer support and new abstractions, Polarr has to have a deep learning inference layer that runs completely in the native context, and the rest in the web context. By having multiple contexts, interprocess communication becomes the bottleneck and make the application prohibitive when it comes to high performance processing or realtime video inference. Furthermore, over the last couple years, iOS as well as Android have begun massive investment in ARKit. In order for our current application to exploit these newly minted frameworks, it means that we may potentially consider going full native.
Stability and performance consistency have also been something on the back of my mind in the last couple years as we added more features to Polarr Photo Editor. Our tool now supports complex layering effect and fine local adjustments. Just by running WebView by itself requires around 300MB of memory, imagine manipulating a full 12 megapixel image and trying to do brushing on top of the image at the same time supporting unlimited undo and redo operations, it means that we would need to save recent brush strokes in memory and then buffer old history in storage. This by itself already incurs performance penalties. Furthermore, it is very difficult to precisely control and allocate memory utilizations in the javascript context, and the underly OS could shutdown the WebView without warning when it sees spikes in resource utilizations. Hence, for some of the lower model Android phones, it proves to be difficult to edit large photos with advanced tools reliably after long usage.
As more and more headwind come our way in the web technology stack, it will be interesting to see how big the gap it will be in performance, stability, and novel experiences between native stack and web stack. While I remain undeterred that web technologies will always have their place in solving customer problems and play a big part of our interactions with each other and the computer, it’s up to debate if future advanced photo and video editing tools can still reign supreme by using the web stack.
// Derek Z. H. Yan is the Co-founder & CTO of Polarr, Inc., a Series-A stage tech startup in the photography and artificial intelligence space based in Downtown San Jose. Please visit polarr.co/careers for job postings or email directly to stay in touch.
