Reactive selfies with Camera2 API on Android — Part 2

Published in

Bumble Tech

7 min readApr 9, 2018

This is part two of an article, where I show how you can use RxJava2 to build logic on top of asynchronous API. I chose Android Camera2 API as an example (and didn’t regret it!). This API is not only asynchronous, but also has hidden distinctive implementation features which are not described anywhere else. So, there is a potential double benefit for readers of this article.

For whom is this post intended for? I’m assuming that the reader is an experienced and inquisitive Android developer. Basic knowledge of reactive programming is highly desirable, as is an understanding of Marble diagrams. The post will be useful for those who want to gain insight into a reactive approach, and also for those who are planning to use Camera2 API in their projects.

You can access the project source code on GitHub. Make sure you read part one first!

Setting the task

At the end of part one I promised to address the issue of waiting for autofocus/auto exposure to activate.

Let me remind you what the operators chain looked like:

Observable.combineLatest(previewObservable, mOnShutterClick, (captureSessionData, o) -> captureSessionData)
                .firstElement().toObservable()
                .flatMap(this::waitForAf)
                .flatMap(this::waitForAe)
                .flatMap(captureSessionData -> captureStillPicture(captureSessionData.session))
                .subscribe(__ -> {
                }, this::onError)

Right, what do we want to obtain from the methods waitForAe and waitForAf? We want the autofocus/auto exposure processes to be launched and, when completed, to receive a message that the camera is ready to take a picture.

For this to happen, we need both methods to return Observable, which emits the event when the camera notifies us that the converging process has occurred (to avoid repeating the words ‘autofocus’ and ‘auto exposure’ from now on I am going to use the term ‘converge’). But how do you run and control this process?

The non-obvious distinctive features of the Camera2 API pipeline

At first, I thought that it would be sufficient to call capture with the necessary flags and to wait for the onCaptureCompleted method to be called in the CaptureCallback.

It seemed to make sense: you activate the query, you wait for it to be completed — so the request has been completed. And the code in question was even sent off to production.

However, we then noticed that on some devices in dark conditions, even if the flash did go off, the photographs were not in focus and came out dark. At the same time, the system camera was working excellently, although, truth be told, it did take much longer to get ready to take the picture. I started to suspect that in my case the autofocus feature hadn’t managed to focus in time for onCaptureCompleted.

To test out my theory, I added a delay of one second — and the pictures started to come out fine! Of course, I couldn’t help but be happy about this decision and I started to look around for how to know that the autofocus has been triggered and that things are ready to proceed. I didn’t manage to find any documentation on this subject and I had to go to the source code for the system camera. Fortunately, the code is accessible as part of Android Open Source Project. It turned out that, unusually, the code was quite hard to understand, and I had to log the camera and analyse the logs when the camera took pictures in the dark. I found out that after calling capture, accompanied by the necessary flags, the system camera calls the setRepeatingRequest to extend the preview and then waits for the onCaptureCompleted call-back, accompanied by a specific set of flags, at TotalCaptureResult. This response might only be forthcoming several onCaptureCompleted later!

Once I realised this quirk, Camera2 API pipeline’s behaviour started to make sense. But it took so much effort to obtain this information! Anyway, now we can move on to describe the solution.

Right, here’s the plan of action:

● send the capture command, accompanied by the flags which trigger the converge process;

● send the setRepeatingRequest command to extend the preview;

● obtain notification from both methods;

● wait for special flags in the onCaptureCompleted notification, confirming that the converge process has been completed.

Right, let’s go!

Flags

We create a ConvergeWaiter class with the following fields:

private final CaptureRequest.Key<Integer> mRequestTriggerKey;
private final int mRequestTriggerStartValue;

This is the key and value for the flag which will trigger the necessary converge process when a capture command is sent.

For autofocus this will be CaptureRequest.CONTROL_AF_TRIGGER and CameraMetadata.CONTROL_AF_TRIGGER_START respectively. For auto exposure it will be CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER and CameraMetadata.CONTROL_AE_PRECAPTURE_TRIGGER_START respectively.

private final CaptureResult.Key<Integer> mResultStateKey;
private final List<Integer> mResultReadyStates;

And here is the key and set of expected values for the flag obtained from the onCaptureCompleted result. Once we see one of the expected key values, we can deem the converge process to have been completed.

Here is the list of values for the autofocus key value CaptureResult.CONTROL_AF_STATE:

CaptureResult.CONTROL_AF_STATE_INACTIVE,
CaptureResult.CONTROL_AF_STATE_PASSIVE_FOCUSED,
CaptureResult.CONTROL_AF_STATE_FOCUSED_LOCKED,
CaptureResult.CONTROL_AF_STATE_NOT_FOCUSED_LOCKED;

Here is the list of values for the auto exposure key value CaptureResult.CONTROL_AE_STATE:

CaptureResult.CONTROL_AE_STATE_INACTIVE,
CaptureResult.CONTROL_AE_STATE_FLASH_REQUIRED,
CaptureResult.CONTROL_AE_STATE_CONVERGED,
CaptureResult.CONTROL_AE_STATE_LOCKED.

Don’t ask me how I got all this!

Now we can create ConvergeWaiter instances for autofocus and exposure, for which we create a factory class:

static class Factory {
    private static final List<Integer> afReadyStates = Collections.unmodifiableList(
        Arrays.asList(
            CaptureResult.CONTROL_AF_STATE_INACTIVE,
            CaptureResult.CONTROL_AF_STATE_PASSIVE_FOCUSED,
            CaptureResult.CONTROL_AF_STATE_FOCUSED_LOCKED,
            CaptureResult.CONTROL_AF_STATE_NOT_FOCUSED_LOCKED
     )
 );
 
    private static final List<Integer> aeReadyStates = Collections.unmodifiableList(
        Arrays.asList(
            CaptureResult.CONTROL_AE_STATE_INACTIVE,
            CaptureResult.CONTROL_AE_STATE_FLASH_REQUIRED,
            CaptureResult.CONTROL_AE_STATE_CONVERGED,
            CaptureResult.CONTROL_AE_STATE_LOCKED
     )
 );
 
    static ConvergeWaiter createAutoFocusConvergeWaiter() {
        return new ConvergeWaiter(
            CaptureRequest.CONTROL_AF_TRIGGER,
            CameraMetadata.CONTROL_AF_TRIGGER_START,
            CaptureResult.CONTROL_AF_STATE,
            afReadyStates
        );
 }
 
    static ConvergeWaiter createAutoExposureConvergeWaiter() {
        return new ConvergeWaiter(
            CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER,
            CameraMetadata.CONTROL_AE_PRECAPTURE_TRIGGER_START,
         CaptureResult.CONTROL_AE_STATE,
            aeReadyStates
        );
 }
}

capture/setRepeatingRequest

To call a capture/setRepeatingRequest we need the following:

● a CameraCaptureSession which has been opened earlier and is accessible in CaptureSessionData;

● a CaptureRequest, which we will create using CaptureRequest.Builder.

We create a method

Single<CaptureSessionData> waitForConverge(@NonNull CaptureSessionData captureResultParams, @NonNull CaptureRequest.Builder builder)

As the second parameter we will send a builder, set up to ‘preview’. So CaptureRequest for preview can be created straightaway using the command

CaptureRequest previewRequest = builder.build();

To generate a CaptureRequest to run the converge procedure, we add a flag to builder which will trigger the necessary converge process:

builder.set(mRequestTriggerKey, mRequestTriggerStartValue);
CaptureRequest triggerRequest = builder.build();

And we use the methods introduced earlier to obtain Observable from the methods capture/setRepeatingRequest:

Observable<CaptureSessionData> triggerObservable = CameraRxWrapper.fromCapture(captureResultParams.session, triggerRequest);Observable<CaptureSessionData> previewObservable = CameraRxWrapper.fromSetRepeatingRequest(captureResultParams.session, previewRequest);

Generating an operator chain

Now we can generate a reactive stream which will contain the events from both Observables with the help of the merge operator:

Observable<CaptureSessionData> convergeObservable = Observable
         .merge(previewObservable, triggerObservable)

The convergeObservable obtained will emit events with the results from the callback onCaptureCompleted.

We need to wait for the moment when CaptureResult, sent to this method, contains the expected flag value. To this end, we create a function which receives CaptureResult and, if it contains the expected flag value, sends back the response true:

private boolean isStateReady(@NonNull CaptureResult result) {
    Integer aeState = result.get(mResultStateKey);
    return aeState == null || mResultReadyStates.contains(aeState);
}

The check for null is required in the case of poor implementations of Camera2 API, so that it doesn’t get stuck in waiting mode forever.

We can now use the filter operator to wait for the event for which isStateReady has been performed:

.filter(resultParams -> isStateReady(resultParams.result))

We are only interested in the first event, so we add:

.first(captureResultParams);

In full, the reactive stream looks like this:

Single<CaptureSessionData> convergeSingle = Observable
            .merge(previewObservable, triggerObservable)
            .filter(resultParams -> isStateReady(resultParams.result))
            .first(captureResultParams);

Should the converge process drag out too long or if something goes wrong, we add a time-out:

private static final int TIMEOUT_SECONDS = 3;
 
Single<CaptureSessionData> timeOutSingle = Single
      .just(captureResultParams)
      .delay(TIMEOUT_SECONDS, TimeUnit.SECONDS,
       AndroidSchedulers.mainThread());

The delay operator retriggers events with a set delay. By default, it will do so in the thread belonging to the computation scheduler, so we move it over to mainThread using the final parameter.

We now combine convergeSingle and timeOutSingle, and the first one to emit the event is the winner:

return Single
 .merge(convergeSingle, timeOutSingle)
 .firstElement()
 .toSingle();

Full function code:

@NonNull
Single<CaptureSessionData> waitForConverge(@NonNull CaptureSessionData captureResultParams, @NonNull CaptureRequest.Builder builder) {
         CaptureRequest previewRequest = builder.build();
 
         builder.set(mRequestTriggerKey, mRequestTriggerStartValue);
         CaptureRequest triggerRequest = builder.build();
 
         Observable<CaptureSessionData> triggerObservable = CameraRxWrapper.fromCapture(captureResultParams.session, triggerRequest);
         Observable<CaptureSessionData> previewObservable = CameraRxWrapper.fromSetRepeatingRequest(captureResultParams.session, previewRequest);
         Single<CaptureSessionData> convergeSingle = Observable
      .merge(previewObservable, triggerObservable)
      .filter(resultParams -> isStateReady(resultParams.result))
      .first(captureResultParams);
 
         Single<CaptureSessionData> timeOutSingle = Single
      .just(captureResultParams)
      .delay(TIMEOUT_SECONDS, TimeUnit.SECONDS)
      .observeOn(AndroidSchedulers.mainThread());
 
         return Single
      .merge(convergeSingle, timeOutSingle)
      .firstElement()
      .toSingle();
}

waitForAf/ waitForAe

The main bulk of the work has been done; now all we have left to do is to create instances:

private final ConvergeWaiter mAutoFocusConvergeWaiter = ConvergeWaiter.Factory.createAutoFocusConvergeWaiter();private final ConvergeWaiter mAutoExposureConvergeWaiter = ConvergeWaiter.Factory.createAutoExposureConvergeWaiter();

and use them:

private Observable<CaptureSessionData> waitForAf(@NonNull CaptureSessionData captureResultParams) {
    return Observable
        .fromCallable(() ->
         createPreviewBuilder(captureResultParams.session,
         mSurface))
        .flatMap(
            previewBuilder -> mAutoFocusConvergeWaiter
                .waitForConverge(captureResultParams,
                 previewBuilder)
                .toObservable()
        );
}
 
@NonNull
private Observable<CaptureSessionData> waitForAe(@NonNull CaptureSessionData captureResultParams) {
    return Observable
        .fromCallable(() ->
         createPreviewBuilder(captureResultParams.session,
         mSurface))
        .flatMap(
            previewBuilder -> mAutoExposureConvergeWaiter
                .waitForConverge(captureResultParams,
                 previewBuilder)
                .toObservable()
        );
}

The main thing to notice here is that we are using a fromCallable operator. We might be tempted to use a just operator. For example, as follows: just(createPreviewBuilder(captureResultParams.session, mSurface)). However, in this case the createPreviewBuilder function will be called at the same time waitForAf is called, but we only want it to be called once the subscription to our Observable appears.

Conclusion

As you know, the most valuable part of any article is the comments! So, I invite you to share your thoughts, feedback and links to better implementations in the comments.

You can access the project source code on GitHub. Pull requests are welcome!