Image processing using MetalKit

While developing, we may think about how image filtering works and how we can achieve it. Apple provided one in-built framework “Core Image” which has plenty of in-built filters, and we can create our custom filters too.

● Left: Normal ● Right: Metal view filtering ● Photo by Unslash

What is the issue with old image processing?

1. For high resolution photo, CPU will need more time to process thus slow result
2. Load on the CPU around 90% of usage
3. User needs to wait until filter process done, loaders 🥱
4. No real time results

How can we resolve this issue?

Hey Olivia

We will tell GPU to do our image processing work and for that we have in-build library MetalKit before MetalKit, OpenGL (Open Graphics Library) was widely in-use.

OpenGL was used to cover all graphic functionalities, and for that we had GLKit (Graphic Library Kit) API, although after introducing iOS12+, Apple deprecated OpenGL and completely migrated to MetalKit.

What is MetalKit? (By Apple)

The Metal framework supports GPU-accelerated advanced 3D graphics rendering and data-parallel computation workloads. Metal provides a modern and streamlined API for fine-grained, low-level control of the organization, processing, and submission of graphics and computation commands, as well as the management of the associated data and resources for these commands.

A primary goal of Metal is to minimize the CPU overhead incurred by executing GPU workloads.

Before we start, let’s get familiar with some terms:

1. Metal Device: MTLDevice is a protocol that defines an interface to the GPU which draws graphics. In order to run other metal objects like command queues, buffers, textures you will need instance of metal device, you can create it using MTLCreateSystemDefaultDevice().
2. Command Queues: It handles the list of tasks that are created for the GPU.
3. Textures: MTLTexure is a resource which holds formatted image data.
4. Metal View: MTKView is a subclass of the UIView which can render buffer, pixel formatted elements and textures of graphics on screen, thus you can draw with Metal elements as it's directly connected to GPU.

We’re creating toggle between Metal View and Image View filtering to witness the differences. We will start with normal and old way to apply filter to begin.

Giphy

Normal way to apply CIFilter directly on Image View

We’re using the default filter provided by in-build API, Apple categorized too well, checkout this documentation.

1. Saturation filter falls under ‘CIColorControls’, so we will initialize filter by its category name as well we have input image from image URL.
2. Customize the CIFilter by setting input image key and saturation key.
3. Set the final output image on the image view.

CPU usage of implemented filter on an Image View

Usage load with Image View + CIFilter

As we can see, above result for applying (CI)filters on a imageView, CPU’s percentage of usage is very high, also user need to wait until this process ends.

New and fast way to apply the filter

For real-time result, we don’t need to wait for an output result to get processed image instance for UIImageView when user changes the value of the slider.

We need one time setup for rendering, and we don’t need to worry about changing value of the slider for that we will initialize the metal view and render our image buffers on the view and metal view do its work.

We also need to confirm the delegates to our view controller, these delegates will be going to give us control what we want to render and how we want to render on the metal view.

By confirming the delegate, we will have two required methods, i.e.mtkView(_:drawableSizeWillChange:) and draw(in:)

Let’s see what Apple says about these required the delegate.

1) The view calls the mtkView(_:drawableSizeWillChange:) method whenever the size of the contents changes.

This happens when the window containing the view is resized, or when the device orientation changes (on iOS). This allows your app to adapt the resolution at which it renders to the size of the view.

2) The view calls the draw(in:) method whenever it’s time to update the view’s contents.

In this method, you create a command buffer, encode commands that tell the GPU what to draw and when to display it onscreen, and enqueue that command buffer to be executed by the GPU. This is sometimes referred to as drawing a frame.

You can think of a frame as all the work that goes into producing a single image that gets displayed on the screen. In an interactive app, like a game, you might draw many frames per second.

We will be using draw method to display image and filtered update on the Metal View.

1. First check if currentDrawble is nil or not, this will be your current frame which is being rendered on the metal view.
2. We have sourceTexture created from our image URL, we need CIImage as inputImage for the CIFilter.
3. Add required to be input elements which are inputImage and saturation value from slider for both of key like we did for image view.

Finally to present our final output, we will tell commandBuffer to notify the GPU to present textures and commit our changes on the GPU.

Final Result:

Now, we are good to go now whenever you change value of the slider you can see the effect is changing in no time.

Smooth right!

Conclusion

Great! You have successfully created image filtering using Metal, with faster and real-time editing of a photo.

What you can do more?

1. Customized and multi chained filters without loading
2. Real time filters as well for video editing
3. You can also build an app with custom camera controls

Thank you for reading! 🙇‍♀️

Checkout the attached demo 👇

mobile-simformsolutions/SSMetalImageFilter