Using Jetpack CameraX

The picture is provided by Unsplash

As a matter of fact, CameraX is a Android Jetpack support library, built to help you make camera app development much more easier. This means it supports a consistent and easy-to-use API surface, which works across most Android devices with having backward-compatibility to Android 5.0 or API level 21. This essay aims to discuss some main issues in using Jetpack CameraX and the new features that have been released recently by Google for Android developers.

Introduction and Overview

Basically, CameraX is a Jetpack support library to make camera app development easier. Currently, it is in Beta version. The CameraX API is built on top of Camera2 API, and provides its capabilities. However, it uses a simpler use-cases-based approach that is lifecycle aware. Above all, this concise API results in an average of 70% fewer lines of code than the Camera2 API. As developers, we know writing a camera app could not be an easy task. First, you have to deal with Camera2 API, which is powerful and flexible; however, it is a challenging process to do. Second, given the large number of Android devices in the market, it is time consuming to achieve a consistent experience across all the devices types. In other words, often this needs a long running if/else section within your code. Eventually, if you are publishing your app on the Google Play, testing on a real device is a must, and testing across a large number of devices can be both costly and time-consuming as your code may require device-specific solutions to allow for a consistent experience.

Now, Google has tried to tackle and solve these issues by introducing CameraX as an efficient method to diminish some major obstacles to use camera in Android apps as follows:

1. Easier to implement and understand by reducing the total amount of codes you must write in your app.

2. Consistent behavior across 94% of Android devices starting from API 21.

3. Google conducts 24/7 on-device testing across a wide range of devices and API levels.

Use cases

Fundamentally, CameraX introduces use cases, which allow you to focus on the task you require to accomplish instead of spending time handling device-specific nuances. There are main use cases as follows:

1. Preview: get an image on the display.
2. Image analysis: access a buffer seamlessly for use in your algorithms.
3. Image capture: save high-quality images.

However, Video Capture has not officially provided yet. In addition, these use cases work across all devices running Android 5.0 (API level 21) or higher.

Preview

In fact, when you want to add a preview to your app, you must use PreviewView, which is a view that can be cropped, scaled, and rotated for proper display. The image preview streams to a surface inside the PreviewView when the camera becomes active. PreviewView is a very useful and easy UI widget for Android application to show a preview. It needs no transform. Also, PreviewView makes your app much more power efficient and low latency. It does this by using SurfaceView under the hood. It also supports scaleType. Thus, you can choose how to scale a preview instead a PreviewView. Finally, it makes it very easy to implement a tap-to-focus.

So, to implement Preview by using PreviewView, you should follow these steps in your code:

1. Optionally configure a CameraXConfig.Provider.
2. Add a PreviewView to your layout.
3. Request a CameraProvider.
4. On View creation, check for the CameraProvider.
5. Select a camera and bind the lifecycle and use cases.

For instance, the following code shows a sample code for the last step as follows:

fun bindPreview(cameraProvider : ProcessCameraProvider) {
    var preview : Preview = Preview.Builder()
            .build()

    var cameraSelector : CameraSelector = CameraSelector.Builder()
          .requireLensFacing(CameraSelector.LENS_FACING_BACK)
          .build()

    preview.setSurfaceProvider(previewView.createSurfaceProvider())

    var camera = cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, preview)
}

Image analysis

The main goal of image analysis is to make it super easy to integrate libraries that are great at analyzing things, such as TensorFlow Lite, ML Kit, or your own image library.

The image analysis use case provides your app with a CPU-accessible image to perform image processing, computer vision, or machine learning inference on. The application implements an analyze method that is run on each frame.

Initially, images are processed by passing an executor in which the image analysis is run and an ImageAnalysis.Analyzer parameter to the setAnalyzer() method. Image analysis can work in two ways: blocking and non-blocking. First, Blocking mode is enabled by calling setBackpressureStrategy() with STRATEGY_BLOCK_PRODUCER. In this mode, the executor receives frames from the camera in sequential order. Second, Non-blocking mode is enabled by calling setBackpressureStrategy() with STRATEGY_KEEP_ONLY_LATEST. In this mode, the executor receives the last available frame from the camera at the time that the analyze() method is called.

For instance:

val imageAnalysis = ImageAnalysis.Builder()
    .setTargetResolution(Size(1280, 720))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .build()

imageAnalysis.setAnalyzer(executor, ImageAnalysis.Analyzer { image ->
    val rotationDegrees = image.imageInfo.rotationDegrees
    // insert your code here.
})

cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, imageAnalysis, preview)

The power of image analysis is not so obvious, because just provide frame buffer access. However, when we want to combine it with state-of-the-art Machine Learning libraries, it suddenly becomes a super powerful tool. It allows the application to do something very interesting, things like text recognition, image labeling, or detect your own object using your own trained model.

Image capture

The image capture use case is designed for capturing high-resolution, high-quality pictures and supports auto-white-balance, auto-exposure, and auto-focus functionality. In fact, the caller is responsible for deciding how to use the captured picture, including the following choices:

1. takePicture(Executor, OnImageCapturedCallback): This method provides an in-memory buffer of the captured image.

2. takePicture(OutputFileOptions, Executor, OnImageSavedCallback): This method saves the captured image to the provided file location.

Basic controls for taking pictures are supported in this use case. After configuring your camera, the following code could be used for taking a picture based on events:

fun onClick() {
    val outputFileOptions = ImageCapture.OutputFileOptions.Builder(File(...)).build()
    imageCapture.takePicture(outputFileOptions, cameraExecutor,
        object : ImageCapture.OnImageSavedCallback {
            override fun onError(error: ImageCaptureException)
            {
                // Your code....
            }
            override fun onImageSaved(outputFileResults: ImageCapture.OutputFileResults) {
                // Your code....
            }
        })
}

The new features in the first half of 2020

1. Enhanced device compatibility with Beta

Recently, Google has made a lot of efforts into fixing device issues, and they has enhanced test lab infrastructure in terms of scale and the test coverage. Ever since CameraX reached beta, Google has been focusing on the stability and the device compatibility. Therefore, they have fixed a number of device specific issues by feedback from developers such as wrong preview aspect ratio, fail to switch camera, or tap-to-focus not working. For example:

1. Wrong preview aspect ratio LG G3

2. Failed to switch camera Pixel 2

3. Tap-to-focus not working Samsung S7

Also, they have improved their rotation, UI widget integration, and aspect ratio test.

2) Preview: PreviewView new APIs, OpenGL sample

As a result, for the Preview use case, they added more API to the PreviewView in order to satisfy developers’ needs. Furthermore, for the needs of advanced developers, they added OpenGL sample. In short, this is a guidance for developers who want to use OpenGL on top of CameraX.

Today, Google has added two important APIs to the PreviewView. First, getPreviewStreamState API. This API allows application to be notified when a preview is streaming or idle. This is useful for application who want to avoid this very temporary rest screen when a preview is starting up. For instance, they can show a placeholder image on top of PreviewView when the preview is idle, and hide the placeholder when the preview is streaming. Second, getBitmap API. This API allows application to get a snapshot of current preview displayed in the PreviewView. As you know, obtaining a bitmap is not an easy task as it appears, because, firstly, the underlying view could be a surface view or tetra view, which has various ways to get a bitmap. Secondly, the bitmap you get from the view, it is not appropriately transformed or cropped like the PreviewView carries out. Thus, the new getBitmap API allows application to get a bitmap 100% identical to what you observe in the PreviewView.

Besides, you probably want to draw some advanced effects on top of camera output. Or perhaps you would like to integrate the camera preview into your existing OpenGL bucket or game engines that are in the app. So, for accomplishing these tasks, you should follow some steps as follows: The first step is you create a SurfaceTexture in your render insert. Then, you create a surface using the SurfaceTexture. After that, you provide the Surface to the Preview SurfaceProvider. This SurfaceProvider will also provide a signal to your Android app when it is safe to release the Surface. Eventually, your app can now easily receive the camera frame in the SurfaceTexture, and your render insert can do any task you want on the SurfaceTexture using OpenGL.

3) Image analysis: YUV to RGB conversion utility, ML Kit sample, TensorFlow Lite sample

For the image analysis use case, they has introduced samples for both convert YUV to RGB and use ML Kit or TensorFlow Lite to analyze the image retrieved from image analysis process. Google has provided you a list of sample calls that can make integration easier. First, YUV to RGB converter. The image you retrieve from image analysis is in YUV format. This is a native format in the Android camera system, but as you know lots of the libraries need the image to be RGB format. As a result, these utilities provide an easy and fast way to convert from YUV to RGB. It uses Renderscript under the hood. So, it is efficient and highly accelerated. In addition, it is able to achieve 30 FPS with 640*480 resolutions across devices like Pixel 3 XL, Samsung S10, and Huawei P20.

In conclusion

In fact, CameraX is a Jetpack support library to make camera app development much more easier. Currently, it is in Beta version. The CameraX API is built on top of Camera2 API, and provides its capabilities. This article considered some main issues in using Jetpack CameraX and the new features that have been released recently by Google for Android development.