Solving image rotation on Android using Camera2 API

I wrote this post back in April 2017, and just discovered it yesterday sitting around, unpublished. I probably meant to proofread it and refine it, yada yada before hitting the publish button, but almost a year later, I’m just going to put it out there because why not?

I don’t blog very often, but when I do, I’m usually motivated by irritation. This is no exception.

If you’ve ever done anything with camera hardware on Android, whether that be with the original Camera API or with Camera2 API, you’re already probably familiar with the usual pain points that come packaged with that. You might have even written some hacks that look along the lines of

if (Build.MANUFACTURER.equalsIgnoreCase("samsung")) {
    ... // This code will make you die a little inside.
}

I’m telling you now, if you’re at all tempted to do this, don’t. It’s a slippery slope.

My latest task has been to update the Seesaw Classroom app (Update: I don’t work here anymore) from using the original Camera API to use Camera2 API. The intention of this was to take advantage of the features and control that Camera2 API provided to address many of the outstanding issues being reported which involve the camera hardware. One of the top issues being that images were incorrectly rotated.

When I first started the refactor, I was excited to play with the Camera2 API and thought that it would not only solve these rotation bugs, but that it would be relatively straight forward and fun! That was a bit overly optimistic. My prior experience with Camera API was pretty limited, but I had done enough to know it was simple to get off the ground, and then not much else after that would be something I might use the world “simple” to describe. Camera2 on the other hand wasn’t exactly simple to get started with, although not overly daunting, it definitely took a bit of effort and doc reading. I kept thinking, it’s ok though, it’s a reasonable trade-off because I’ll be able to query the hardware for all the info and have all the control.

In fact, more control than I even needed for an app mostly used by primary school aged children. This turned out to be mostly true. You see, things worked fabulously! When they worked.

Let’s discuss the part that let me down the most about Camera2 API. Image rotation. If you haven’t heard of Seesaw (unless you’re in the education sector or have a kid whose teacher uses it at school, you probably haven’t) it’s an app that teachers can use in their class to allow the students to create and publish pieces of work. Sometimes this might involve using the tablet’s or phone’s camera hardware to take photos or create video.

To take a photo you need to create a CaptureRequest using CaptureRequest.Builder like so:

final CaptureRequest.Builder captureBuilder =
        cameraDevice.createCaptureRequest(TEMPLATE_STILL_CAPTURE);

You’ll also need to set a target which you’ll set according to the intended use of the CaptureRequest such as showing a preview or taking a still photo. Here we’re taking a still photo so the target needs to be our ImageReader surface.

captureBuilder.addTarget(imageReader.getSurface());

Next we can set the behavior and properties we want. One of which will be the degree to rotate the captured image in order to produce a JPEG image that is shown to the user in an upright fashion.

captureBuilder.set(
        CaptureRequest.JPEG_ORIENTATION, 
        getJpegOrientation(deviceRotation));

The getJpegOrientation(int deviceRotation) method is a method you need to create to calculate the angle at which to rotate the resulting image so that it’s always upright to the user based on the sensor data provided. The sensor data you’ll need to use are:

Device rotation — the physical rotation/orientation of the device.

activity.getWindowManager().getDefaultDisplay().getRotation();

and sensor orientation which is the physical rotation of the device’s camera sensor.

android.hardware.camera2.CameraManager manager = activity.getSystemService(Context.CAMERA_SERVICE);
CameraCharacteristics characteristics = manager.getCameraCharacteristics(cameraId);
sensorOrientation =  characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION);

With this you can fetch surface rotation which you’ll need to use to calculate jpeg orientation rotation angle like so. The code shown below to get surface rotation is provided in googlesamples/android-Camera2Basic sample app.

private static final SparseIntArray ORIENTATIONS = new SparseIntArray(4);

static {
    ORIENTATIONS.append(Surface.ROTATION_0, 90);
    ORIENTATIONS.append(Surface.ROTATION_90, 0);
    ORIENTATIONS.append(Surface.ROTATION_180, 270);
    ORIENTATIONS.append(Surface.ROTATION_270, 180);
}

Now you can use the value of deviceRotation to fetch surfaceRotation.

surfaceRotation = ORIENTATIONS.get(deviceRotation);

Finally, we can calculate the angle we need to rotate the image to produce an upright result.

int jpegOrientation = 
(surfaceRotation + sensorOrientation + 270) % 360

You might have noticed that this logic isn’t quite the same as what Google provided in the actual Android documentation for CaptureRequest.JPEG_ORIENTATION. That looks like this.

private int getJpegOrientation(CameraCharacteristics c, int deviceOrientation) {
     if (deviceOrientation == android.view.OrientationEventListener.ORIENTATION_UNKNOWN) return 0;
     int sensorOrientation = c.get(CameraCharacteristics.SENSOR_ORIENTATION);

     // Round device orientation to a multiple of 90
     deviceOrientation = (deviceOrientation + 45) / 90 * 90;

     // Reverse device orientation for front-facing cameras
     boolean facingFront = c.get(CameraCharacteristics.LENS_FACING) == CameraCharacteristics.LENS_FACING_FRONT;
     if (facingFront) deviceOrientation = -deviceOrientation;

     // Calculate desired JPEG orientation relative to camera orientation to make
     // the image upright relative to the device orientation
     int jpegOrientation = (sensorOrientation + deviceOrientation + 360) % 360;

     return jpegOrientation;
 }

While the logic provided in the documentation and in the sample project works on many devices, it didn’t work on all. The Pixel C is an example of a device that this logic did not work on for all rotation angles.

TL;DR: OK, For the majority of you who stumbled upon this post in desperate need of a solution or code to copy/paste (sorry, that you didn’t find a copy pasta solution here), but here’s what you need to know. Don’t rely on setting a fixed JPEG_ORIENTATION in the CaptureRequest to work on all devices. Don’t rely on ExifInterface.TAG_ORIENTATION to exist and to provide reliable data on all devices. The only solution that I’ve found to be reliable across devices is to use the sensor data to calculate the correct rotation angle (see blog post above for more on this) so that you can rotate the resulting bitmap yourself.

Hope this helped!