How Computational Photography Works
Optics, using a lens and sensor, is the fastest and easiest way to create an image. We have a device for that, the camera. Images can also be created using non-optical means using computational algorithms. Welcome to the field of computational photography. It is widely used in smartphone cameras and it’s features are finding its way to commercial applications. Generally it is applicable to digital photography, not film photography. Examples of this include Apple’s iPhone portrait modes and Adobe’s Sensei for the Creative Cloud suite.
Let’s take a look at how computational photography works under the hood. What this does to an image is enhance it beyond the skill level of an average photographer, making it look much more like it was shot by a professional photographer. Include retouching, but this is all done in a matter of seconds thanks to advanced software and hardware. Some smartphone camera software will enhance the image by softening the skin to give it a less harsh look, sharpen the eyes and then apply a background blur aka bokeh effect. It happens instantaneously, which is a part of the feature.
Although you can create images computationally without using a camera, you will need first and foremost an image to work with. It can be acquired from various optical sources besides a camera, like a scanner or sensor. What we need is an image to get things started. For this example we will take a RAW image shot from a DSLR camera and convert it into a raster format (JPEG). It is a lossy type compression, but the image will be set to less compression for more detail at maximum resolution (in this case it is 4928 x 3264 pixels or 16 MP). Once we have our image ready, we can begin.
RAW Image
Let’s begin with the RAW image, which is the file created when you captured your image from the camera or device. This is an optical exposure that is created when the lens focuses the image onto the sensor and stored in digital media. If you shoot with a Canon you will most likely have a .CR2 file extension while Nikon shooters will have a .NEF extension. Each camera maker has their own format usually, but make sure that it is a RAW file format. Usually before you shoot you should check your camera settings to see if you are storing images as RAW uncompressed or as compressed JPEG files (refer to your camera documentation for more details).
We start with the RAW image because this is where the image creation process begins. Photographers prefer to shoot in RAW because it preserves the highest quality in terms of details and can be edited using a non-destructive editing software before being exported to a more manageable format. RAW files are not compressed, so they consume the most space and are thus in itself not ideal for editing. Instead it must be converted to a format like JPEG, which is compressed, so it shrinks the file size but at the expense of detail. Set your compression ratio to more detail but larger file size or more compression with less detail. For uploads to the web and social media, compressed formats are ok and do not usually have to be high resolution. For professional work, higher resolution is not just a requirement but the norm.
In computational photography, the RAW image capture is automatically converted to JPEG for the user. It happens quickly behind the scenes, the user will not even be aware of it. All they see is their captured image, which has already undergone post processing from the image processing from the ISP (Image Signal Processing) and smartphone camera software. The image captured also includes metadata called the EXIF (Exchangeable Image File Format).
When you apply certain computational photography effects (i.e. stitching, beauty mode, panoramic, etc.) using your smartphone, you are transforming that image already into something new. The smartphone software creates a new object for it and it is going to become a totally different image from the original.
Bitmaps
A bitmap is an array of pixels which can store more than 1 color value per pixel. This is created from the original image. The software begins the transformation by creating a bitmap of the original image which is then stored temporarily in the memory sub-system under a random type of name e.g. tmpzg3tjpyo.PNG (it can also use a BMP extension).
After the RAW -> JPEG conversion, the file must now be opened first and then loaded into memory.
In terms of coding, here is an example of how that would work using Python.
>>> img = Image.open("TestImage.jpg")>>> img.show()
The software in the smartphone camera (this is not the exact line of code, just an example) will first load the image as a bitmap into memory. This creates the working space to begin transforming the image. As a result of this, the EXIF is stripped from the original source image. This is also true when compositing multiple images, stitching and in HDR effects.
Image Enhancements
Now that we have a copy of the source image loaded into memory, we can run some routines to alter it. So let’s apply a filter to sharpen the image, then adjust the brightness and contrast.
>>> img = img.filter(ImageFilter.SHARPEN)>>> enhance1 = ImageEnhance.Brightness(img)>>> enhance_img = enhance1.enhance(1.8)>>> enhance2 = ImageEnhance.Contrast(img)>>> enhance_img = enhance2.enhance(8.4)>>> img2 = img.point(lambda p: p * 0.9)
After applying the sharpen, brightness and contrast setting, the image pixels are multiplied with 0.9 using a one-time lambda function to make it darker.
Now we can further make modifications by adjusting the image’s resolution (size).
>>> width, height = img2.size>>> resizedImage = img2.resize((int(width/2), int(height/2)))
The image was resized by 1/2 of its original size. This was done by taking img2 and creating a new object called resizedImage. The new object, which has been reduced in size, now has a resolution of 2464 x 1632 pixels. That is down from 4928 x 3264 pixels.
This is the final adjustment before the image is saved.
>>> resizedImage.save("FinalImage.jpg")Instant Retouching And High Quality Results
We just went over what happens during the time you release the shutter and view the image on the display. Everything happens so fast, yet it produces amazing results. Sometimes the images are just too stunning. This is because the smartphone camera uses AI (Artificial Intelligence) software to determine the best exposure. The camera hardware uses frame buffers to take various exposures of the image. It will select the sharpest image and then apply the filters for enhancement. The code used are just examples of what goes on during this whole process. This saves time by not requiring any post processing on the part of the user. The retouching has been done by the camera’s software application.
Computational photography is about transforming the image using algorithms, which are a set of routines that computers perform. It is an automated process which performs a sequence of steps to transform an image. Think of it like packaging a product on a conveyor belt. The product must first go through a line that performs steps to lead to the final packaging before it is shipped as the final product. With computers, these steps are performed so quickly because of fast microprocessors running the software application. The moment the image is captured by the device’s camera (e.g. smartphone) it goes through this “conveyor belt” process leading up to the final image.
The next time you shoot with your smartphone camera or other computational-aware device, think of the process. You don’t even need to use a camera to create an image, undergoing the same process discussed. All that is needed is an image and the rest is up to software and hardware to figure out the best way to enhance the image.
