How we doubled the photo resolution from Flutter camera on IOS

Hi, my name is Sergei and I am a mobile software engineer at Brickit.

Some time ago, we at Brickit finally migrated both iOS and Android apps to one Flutter codebase. We are pretty stoked about how it turned out, but the transition certainly wasn’t smooth. One part of our grand journey was setting up and using a camera plugin, which turned out to be a bit tricky on both iOS and Android. In this article, I will describe the problem with high resolution photos on iOS, explain in part how the original plugin works and provide our solution with code samples on how to make it better. Link to full code at the bottom.

What was wrong?

In short, the highest photo resolution from the camera plugin wasn’t good enough for our use-case, while we also knew that it is possible to go higher with the native camera.

In the Brickit app, the camera is a crucial part of the core functionality. Users take photos of their creations, set profile pictures, and, most importantly, capture images of their bricks for the app to scan. Since there can be thousands of bricks in some cases, we need to obtain the highest possible resolution that the hardware can provide.

If you have used the flutter camera plugin, you may have noticed that it includes some presets to control the resolution. The issue is that setting the ResolutionPreset.max or ResolutionPreset.ultraHigh resulted in a final picture resolution of 3840x2160, which is not only in a 16:9 aspect ratio instead of the usual 4:3 for photos, but also is not the highest resolution for most iPhones currently available.

So, after spending days looking at StackOverflow and issues on GitHub, and searching for an alternative plugin without any luck, we have decided to fork Flutter Camera and fix this ourselves.

How it works?

Setting a resolution for the camera in original flutter plugin happens in three steps:

• resolution preset is set in flutter code when initialising CameraController like this:

final CameraController cameraController = CameraController(
      cameraDescription,
      ResolutionPreset.max, //or any other preset
      enableAudio: enableAudio,
      imageFormatGroup: ImageFormatGroup.jpeg,
    );

• then, under the hood the ResolutionPreset enum value is mapped to its native IOS analog

typedef NS_ENUM(NSInteger, FLTResolutionPreset) {
  FLTResolutionPresetVeryLow,
  FLTResolutionPresetLow,
  FLTResolutionPresetMedium,
  FLTResolutionPresetHigh,
  FLTResolutionPresetVeryHigh,
  FLTResolutionPresetUltraHigh,
  FLTResolutionPresetMax,
};

• finally, camera resolution is set in native code using the converted preset value

- (void)setCaptureSessionPreset:(FLTResolutionPreset)resolutionPreset {
  switch (resolutionPreset) {
    case FLTResolutionPresetMax:
    case FLTResolutionPresetUltraHigh:
      if ([_videoCaptureSession canSetSessionPreset:AVCaptureSessionPreset3840x2160]) {
        _videoCaptureSession.sessionPreset = AVCaptureSessionPreset3840x2160;
        _previewSize = CGSizeMake(3840, 2160);
        break;
      }
      if ([_videoCaptureSession canSetSessionPreset:AVCaptureSessionPresetHigh]) {
        _videoCaptureSession.sessionPreset = AVCaptureSessionPresetHigh;
        _previewSize =
            CGSizeMake(_captureDevice.activeFormat.highResolutionStillImageDimensions.width,
                       _captureDevice.activeFormat.highResolutionStillImageDimensions.height);
        break;
      }
    case FLTResolutionPresetVeryHigh:
      if ([_videoCaptureSession canSetSessionPreset:AVCaptureSessionPreset1920x1080]) {
        _videoCaptureSession.sessionPreset = AVCaptureSessionPreset1920x1080;
        _previewSize = CGSizeMake(1920, 1080);
        break;
      }
    case FLTResolutionPresetHigh:
      if ([_videoCaptureSession canSetSessionPreset:AVCaptureSessionPreset1280x720]) {
        _videoCaptureSession.sessionPreset = AVCaptureSessionPreset1280x720;
        _previewSize = CGSizeMake(1280, 720);
        break;
      }
//and so on

As you can see from the code, setting the maximum preset results in going through all cases of the switch until the highest available quality can be used.

When the ultraHigh preset is set, the AVCaptureSessionPreset3840x2160 is obtained if it is available. This is the correct behavior for using a video output, as 4k is currently the highest video resolution for iOS devices. However, if you are using the camera to take photos (as in the case of Brickit), there is another preset you can use - AVCaptureSessionPresetPhoto.

What did we change?

After looking through the plugin code and researching Apple’s documentation, here is what we came up with:

- (void)setCaptureSessionPreset:(FLTResolutionPreset)resolutionPreset {
  switch (resolutionPreset) {
    case FLTResolutionPresetMax:
   if ([_videoCaptureSession canSetSessionPreset:AVCaptureSessionPresetPhoto]) {
        _videoCaptureSession.sessionPreset = AVCaptureSessionPresetPhoto;
        _previewSize = CGSizeMake(4032, 3024);
        break;
      }
      if ([_videoCaptureSession canSetSessionPreset:AVCaptureSessionPreset3840x2160]) {
        _videoCaptureSession.sessionPreset = AVCaptureSessionPreset3840x2160;
        _previewSize = CGSizeMake(3840, 2160);
        break;
      }
    case FLTResolutionPresetUltraHigh:
// and so on

In case of max preset, we try to set the 4032x3024 resolution for the camera.

Method canSetSessionPreset returns a boolean that tells you if the requested session preset can be set for your device. If that turns out to be true, we then actually pass the preset to session and set a preview size to the correct 4032x3024. All done!

A little bit of a safety net — since camera can be used for both video and photo and because the app can be used on wide variety of devices, we can’t assume that 4032x3024 will always be available. So, in case canSetSessionPreset returns false, we try to use good old 4k just like in original plugin.

Since we use 4:3 aspect ratio for the photo of the bricks in our app, with the default plugin we would have needed to crop 3840x2160 (16:9) photo to 2880x2160 (4:3). And that would be a 6 megapixel photo (6 220 800 pixels to be precise). With our little fix, we don’t need to waste time and resources cropping the image, and at the same time get 4032x3024, 12 megapixels (12 192 768 pixels to be precise) photo. As you can see, it is actually twice bigger than with the original plugin!

You can find full code along with an example here.

GitHub - brickit-app/brickit-flutter-camera at feat-ios-camera-best-resolution

We have successfully used this code in production for about 6 months without encountering any issues. There is a plan to propose these changes as a pull request to the original plugin, but that’s a topic for a whole other article.

Thank you for reading and don’t hesitate to reach out with questions or feedback!