Handling images in a truly dynamic web environment
Like the growth of the web, the number of devices connected to it has grown. Gone are the days of websites which were only accessible with Internet Explorer using a 800x600px resolution. The web has become dynamic!
Luckily for us, users. But also slightly more annoying for us, developers. Whereas the web used to be for PCs (and Macs, and all flavours of Linux), it is now increasingly accessed from mobile devices as phones and tablets as well. Add some mobile apps to the mix the sheer number of resolutions becomes baffling.
Design-wise the Bootstrap grid has seen quite an adoption. Many websites are powered using it (or its derivatives, or successors). And indeed with CSS media queries responsive design has become much better. However there was one area where this was lacking. Every single image on the Internet used to be quite bad.
The introduction of retina devices
This was the first step on PCs (okay, actually Macs in this case) where Apple fitted a very high quality display on a Macbook Pro. As the proud owner of one of the first 2012 Macbook shipped with the display it suddenly struck me. Imaging on the web really sucked.
- It took Facebook months after I had my laptop to update its logo in the top left corner so it would not be pixelated.
- Many sites used headers embedded in images. On the retina blown up resolution, you could actually picture yourself walking up and down the stairs since those images were pixelated really badly.
The reaction of the web
Quickly web developers learned (by now more and more high quality screens are added to laptops, and many mobile devices have absolutely stunning resolutions to be honest), and started supporting the weird
Browsers later added new features as
srcset, which made life again a bit easier.
What we did
For both companies I work for, at some point we realized we had a big problem. It started out with a designer which wanted to make avatars 10px wider and higher. Using our existing approach, that would mean we would have to download hunderds of thousands of original images, perform offline work to resize to the new size (and the retina size), and re-upload them. That was definitely not fun.
Additionally with new mobile apps around the corner, for which we wanted to limit download sizes to the proper resolution, we came up with a different approach. We built a live image resizer.
“What is an live image resizer?”
In our case, its like a self-hosted version of imgix. As both companies are startups, we do not have the money to pay somebody to do this. However we do have a number of good engineers to build it!
First we wanted to see how we could improve. Step 1 was ditching Java for images. Trust me, Java is great, but images are complicated. Using some Java libraries at the time resulted in regular artifacts on customer uploads (badly parsed alpha channels in PNG, green tint on JPG etc). Instead we picked an existing and simple solution: imagemagick.
Step two was making imagemagick accessible: API servers definitely should not spawn imagemagick processes themselves, but defer this to a specialized service. In the end we opted for Node.js.
Step three was the dynamic request of an image: if an original existed but a client needed a special size or format, that image should be generated. As a result, we also want to properly cache that image, since it does not change between runs and that would definitely save some cycles on our CPUs.
Step four was making the cache available using a CDN for quick usage by all clients, preferably in a location close to them. As we already leverage AWS for some other stuff, we opted for AWS S3 to store our caches. (Yes, I know about Cloudfront, I’ll get there in a jiffy. *)
The final service
After going through a number of rewrites (the prototype was intended for throw-away, but as it goes it performed well enough to go to production where it stayed for a few months, after some time I got fed up and rewrote it properly in a single weekend), we arrived at iaas: Imaging As A Service.
An API server can request a token from the service, which is passes to the frontend. The frontend will talk directly to iaas (using the token) while uploading the file. This way we circumvent any awkward middleware ;).
On upload of the image, the image is first correctly oriented. This sounds simple, but it is surprising how many system get it wrong. (At inventid we encountered this first hand when we started accepting avatar uploads directly from phone, and everybody with an iPhone in portrait setting was the wrong way up.) Next step is writing the original to disk and we determine its original height and width (after the orientation is fixed) which we send back to the frontend. The frontend can then simply use the id it got to save the image reference to the backend. **
Once an image is requested, we use
window.devicePixelRatio (or something equivalent) to determine what type of screen we are dealing with, and use the regular intended width and height of an image. We then combine that with the different ways to fit images in there, and send a request to iaas. The filename we request then looks like
/image_100_100_2x.jpg?fit=crop. Additionally you can also set blur parameters, and nowadays for jpg the intended quality as well. The point remains the same.
The request then hits iaas, which parses the incoming request. First it sends a request to Postgresql to see whether that image was generated before. If it was, we return a redirect using a
307 header to the cached location. If not, we need to generate the correct image directly, and return it to the requester. We reuse the same image to upload to AWS, and save the parameters and the image location to Postgresql, so we do not need to re-render it next time.
For convenience, we wrapped everything up in a Docker container.
Our service runs stable and handles thousands of request every day. We are quite happy with it :)
All in all, it allows our customers to enjoy our platforms as fast as possible, while we do not have to deal with any problems regarding design changes.
Other things it does
Usually this service does additional things, either because of an earlier problem or because we needed new features
- It exports metrics to influx if you want to.
- It allows serving of the original image.
- It can limit the maximum upload in resolution (yes, we had a client upload a 253MP PNG which takes really long to render to any resolution).
- It can limit the time required in imagemagick to prevent run away processes.
- It can serve
robots.txtfile which determines whether images should be allowed to be indexed by search engines.
*) You can also specify a Cloudfront distribution, which performs better than an S3 bucket.