Computational Photography is about conquering the limitations of digital photography with computation: in optics, sensors, and geometry. Image manipulation uses image processing to improve the way we process, manipulate, and interact with visual media. While we normally tend to think that this applies only to post-processing the end result (similar to photo editing), the possibilities are much richer since computation can be enabled at every step of the photographic process — starting with the scene illumination, continuing with the lens, and eventually even at the display of the captured image. Computational photography is common in digital cameras and especially smartphones, automating many settings to make for better point-and-shoot abilities.
Before diving deeper in different technical aspects of computational photography I think is necessary to observe an incomplete timeline of innovations in digital photography.
From tape to supermassive black hole
In 1975 Steven Sasson, an engineer at Eastman Kodak, invented the first ever digital camera . The 8 pound camera recorded 0.01 megapixel black and white photos to a cassette tape. The first photograph took 23 seconds to create. To play back images, data was read from the tape and then displayed on a television set.
In 1991 Kodak upped the game and introduced the Kodak DCS which sported a “massive” 1.3 megapixel image sensor and sold for a cool $13,000 USD.
By 1991, Kodak had produced the first digital camera that was advanced enough to be used successfully by professionals.
1999 — The Kyocera VP-210 introduced a concept that we still use frequently today– phone photography! It could store 20 stills and transmit live “video” at a rate of 2 fps. Sharp soon followed with its J-SH04, developed with inventor Philippe Kahn, whose 1997 prototype phone was the first to transmit a photo — of his baby daughter.
1988 — Thomas and John began presenting Photoshop to several Silicon Valley companies. At this time, pixel-based image editing software was just coming to the market. Some of the features in the earliest version of Photoshop were: displaying and saving several file formats, soft edge selections, levels for tone adjustments, color balance, hue, and saturation color adjustments, painting capabilities.
2010 — Instagram CEO and co-founder Kevin Systrom uploaded the first photo (X-PRO2 filter) to the app, which was called Codename at the time. The app allows users to upload photos and videos to the service, which can be edited with various filters, and organized with tags and location information.
2012 — Lytro camera hits the market. This is a light-field, or plenoptic, camera. An array of micro-lenses sits over the camera’s sensor, capturing all the light in the scene being photographed. The Lytro then saves your image in a proprietary file format to deliver a “living picture” that you can manipulate on your computer, much like a raw file. By manipulating key attributes, you can effectively change the focus of the image. Finaly, Google bought Lytro to use their technologies in Pixel.
2014 — HTC One (M8) is the first smartphone with two cameras.
2017 — Light L16 camera. This compact camera captures the details of the scene at multiple focal lengths (16 lenses), then uses sophisticated algorithms to combine 10+ images into a single, high-resolution photo.
2019 — Huawei P30 Pro isn’t just enhancing the image information the user captures but actually placing pre-existing imagery of the moon into the photo.
2019 — The first image of a Black Hole. No comments.Just read this.
Computational photography and programming languages
What programming language are popular in image analysis and processing for real life application? Experienced software engineers from this emerging field mostly recommend C/C++, Python, Java and MATLAB. The choice depends upon the tool and programming level you have.
Among many programming languages Python’s learning simplicity and some really useful libraries can be the best choice for beginners. NumPy and OpenCV (also can be used with Java) are the most popular libraries that can be used with just several month of coding experience. If you are brave enough, you can try Halide (new in the industry), which designed to make it easier to write high-performance image and array processing code. Halide was designed specially for image processing. Actually, this is not a standalone programming language — Halide is embedded in C++.
Also don’t forget about Ruby. Chunky_png, mini_magick, cairo gems can be super helpful if you need get a quick result without tons of complicated conditions. This perfectly works if you don’t need very high performance image processing and you don’t want to dive deeper in the problem.
Swarm of data or different techniques in computational photography
Simulate Depth of Field (DOF) — a hallmark of a professional-looking image, since it visually separates the subject from the background — is a good example. It enables you to take photos of people and other subjects and apply a background blur for a convincing bokeh effect. Google’s A.I. is incredibly adept at applying the blur to the correct areas, but it doesn’t always get it right. Sometimes it struggles with hairlines or complex mixed areas where depth and distance are tough to discern.
Generate images with a high dynamic range (HDR) — HDR is the ability to take multiple shots simultaneously or in rapid succession, and then merging them together to overcome the limitations of the sensor’s natural capability. In effect, HDR, particularly on mobile devices, can expand the tonal range beyond what the image sensor can capture naturally, allowing you to capture more details in the lightest highlights and darkest shadows. By combining the best of an overexposed, underexposed, and balanced shot, the HDR result can often be a professional quality photo.
Reduce motion blur — Motion blur can be caused by a moving subject, a moving camera or both. Reduce blurring resulting from several types of camera motion; including linear motion, arc-shaped motion, rotational motion, and zigzag motion.
Remove objects and other imperfections — algorithms are proposed for removing large objects from digital images. The challenge is to fill in the hole that is left behind in a visually plausible way.
Low-light shots — It can be tough to capture good photos in dark environments, and the flash tends to wash out your subject, lose the background, and create glare or reflections, among other things. Google’s solution is called Night Sight. It extends the same approach that HDR+ employs by taking several shots and stitching them together to dramatically boost brightness and color.
It can be reasonably expected that the future of photography is computational, not optical. This is a new paradigm in the era of unrealistic and surrealistic view on what surrounds us.
- Computational Photography. From Selfies to Black Holes.
- Как победит вычислительная фотография. Будущее фотографии — это кот
- Intro to Python Image Processing in Computational Photography
- Computational Photography Will Revolutionize Digital Imaging
- The Future of Photography is Code
- The Future Of Photography Is About Computation