CameraX: a glimpse into the past, present, and future
CameraX is an unbundled Android Jetpack library that helps you easily add camera capabilities within your Android apps. Traditionally building applications with camera capabilities has been difficult on Android due to the large variety of Android devices and a complex programming model. Thousands of developers are now using CameraX and many have realized APK size reduction, improvements in their business metrics, and also streamline their resources, enabling their engineers to focus on building cool new features.
This post talks about how CameraX flips the existing camera framework programming model, how it helps you scale your apps across Android, and introduces new capabilities since Beta, like an easy way to integrate smart capabilities like ML Kit into your application.
Reversing the Complexity of the Past
We started by scouring online for the various issues experienced by you and experimenting with a number of existing devices in the field. We were surprised to learn that many of you resorted to large if-else code blocks across manufacturers, chipsets, and even specific devices just to make things work. One of the reasons for this behavior is because with the Camera1 or Camera2 framework APIs, you start with a set of device- and OS-specific inputs to achieve a desired output, such as to take a picture. By design, this requires that you know the device configuration ahead of time to help you achieve your goal — making it difficult to scale your code when new devices are launched.
We had to fundamentally rethink the programming model. A lot of your feedback inspired us to flip the funnel and start with a use case based approach (represented by the diagram below).
With this new approach the team created an unbundled library, CameraX, which was agnostic of all the major Android releases and can work on over 94% of devices starting with API 21, without needing device-specific if-else blocks. In CameraX you first specify your goal by creating a use case, for example ‘Take a picture’ with ImageCapture use case. Then CameraX determines the OS and device specific configuration that will help you achieve your goal. Today, use cases underpin the CameraX experience, starting with Preview, ImageAnalysis, and ImageCapture.
Helping Developers Scale Testing
In addition to existing framework API implementation and usage complexity, many developers experience difficulties with testing their apps at scale across a wide number of Android devices and diverse camera configurations. What we learned is that even for the most basic camera apps, a lot of your resources were spent on testing a large device pool, resulting in device-specific workarounds as mentioned earlier. Given testing is a crucial part of your overall development, this inspired us to build a dedicated CameraX test lab that would provide test coverage across multiple Android devices.
Today this lab is filled with 90+ Android devices. Two phones are mounted inside a custom test box with controlled lighting and a motor to simulate rotation and orientation of the camera app. These 90+ devices represent ~450M active Android devices spanning all the OS levels (since Marshmallow), price tiers (our lowest-end phone has a retail price under $100), chipsets (covering MediaTek, Qualcomm, and Unisoc), and manufacturers (complete device list).
Closing the Gap
Over the past few years, Android manufacturers have made tremendous progress bringing new capabilities like High Dynamic Range (HDR), blurred bokeh, and Night photography to mobile devices. However, until now this capability has only been accessible via the phone’s native (or system) camera application. CameraX Vendor Extensions (a.k.a. Extensions) closes the gap and opens up five manufacturer-specific capabilities to all developers. This includes:
- HDR: uses the manufacturer’s multi-frame processing capabilities to merge multiple frames and provide an image that retains both the highlights and lowlights of a scene.
- Night: uses manufacturer’s low light multi-frame processing capabilities to provide more appealing pictures in low-light conditions.
- Auto: automatically determines and provides either an HDR or a Night image based on current ambient lighting conditions.
- Bokeh: provides an image with a blurred background, great for people photography.
- Beauty: enables face beauty effects within the image.
The best part: when enabling Extensions you do not need device-specific if-else blocks — code once and it works on all supported devices. You can check if Extensions are enabled on a particular device by using ExtensionsManager.ExtensionsAvailability. If an Extension is available, you can show necessary UI elements to the user (if applicable) and configure both the Preview and ImageCapture use cases to use Extensions instead of normal image capture. With just a few more lines of code, you can dramatically improve image quality and overall user experience.
These Extensions are available across Samsung (all flagships from S10+ onwards), Motorola (majority of their devices upgraded or launched with Android 10), Oppo (Reno Pro 2), and several entry-level phones that use the Unisoc mobile processors. We are excited about closing the gap with Extensions and thankful to all the manufacturers for continually expanding the number of devices with these capabilities.
Enabling New Capabilities
Fast-forward through CameraX’s Alpha launch at Google I/O 2019 , Beta in February 2020, to today. With the latest Beta release, we are thrilled to share about CameraX’s improved compatibility, introduction of PreviewView to get a seamless preview, and new samples and utilities to make usage of CameraX even easier.
Starting with improved device compatibility, we’ve been able to address several device-specific issues like wrong device aspect ratio (on LG G3), failure to switch camera under certain instances (on Pixel 2), and fixing tap to focus (on Samsung J7) by adding fixes and workarounds within CameraX. In addition, based on your input from the public bug reports (thank you!), we’ve added additional tests within the CameraX test lab, such as UI widget integration and aspect ratio tests. If you see any device-specific issues, please let us know and we would be happy to address the issues.
While CameraX does provide the Preview use case, many of you have been asking for an even simpler solution to getting a preview on screen. PreviewView is the simplest way to add a view to your application which interacts with the camera. It requires no in-depth knowledge about camera APIs or the underlying hardware, and since it’s powered by CameraX it is power efficient and presents very low latency for users. It achieves this by using SurfaceView by default and, if the device is not compatible with SurfaceView, it will fallback to TextureView automatically.
PreviewView also helps improve the user experience. The new getPreviewStreamState() API enables your apps to be notified when the preview is streaming or idle. You can use this new API to show a placeholder image on top of the PreviewView when the preview is idle and hide the placeholder when the preview is streaming. This avoids an unpleasant black screen while the preview is starting up. You can learn more on how to use PreviewView in this blog post.
Finally, ImageAnalysis starts by providing per-frame buffer access. Once we combine this use case with ML Kit’s turn-key APIs, it opens up a lot of new possibilities like text recognition, image labeling, and even detecting and recognizing objects using your own trained Tensorflow Lite model. Alternatively, you can use CameraX directly withthe TensorFlow Lite run-time (GitHub). For an in-depth view of the latest CameraX updates, check out this video:
There are thousands of apps on the Play Store currently deployed with CameraX across automotive, banking, dating, chat, e-commerce, search, social, and many more categories. Some of the developers that have found CameraX helpful include the following:
- Monzo Bank used CameraX to a) eliminate 9,000 lines of code, including UI code, and b) reduce the dropout rate from identification image capture and selfie video recordings from 25% to around 5% (case study).
- YouCam Perfect was able achieve 70% fewer lines of code with CameraX (over camera2) and it has greatly improved the efficiency of new product developments.
- Ufoto used CameraX to streamline their resources, enabling their engineers to focus on building cool new features. They also observed a measurable improvement when migrating from Camera1 to CameraX. For example, in some devices image capture is 10% faster and in some devices, the preview frame rate is 30% higher.
What’s Next
We will continuously be improving device compatibility and reliability across Android while making CameraX even easier to use for you (list of fixed CameraX issues reported by developers). For example, we will be adding additional devices into our CameraX test lab over the coming months. We strongly believe that developers should not have to worry about the underlying camera stack or device-specific quirks so that they can focus on building awesome apps that users love. With over 2,000 developers using CameraX in production, we recommend you try adopting and deploying the latest version of CameraX Beta.
If you want to get your hands dirty with CameraX, check out our tutorial and our official sample. After you try CameraX, we would love to hear about the types of improvements you see. And if you observe any issues with the latest release please file a bug with bugreports where possible (we triage weekly). You can also email the Android CameraX Discussion Group with questions or post on StackOverflow with the tag ‘android-camerax’.
Our most heartfelt thanks to you, our amazing developers, partners, manufacturers, and CameraX team members for helping shape and build the CameraX Jetpack library.
