Fast image processing in Android with RenderScript
RenderScript turns out to be one of the best APIs for running computationally-intensive code on the CPU or GPU (that too, without having to make use of the NDK or GPU-specific APIs). We can use some existing intrinsics or create our new kernels that describe the computation and the framework takes care of scheduling & execution. In this code I have explained how to use ScriptIntrinsicYuvToRGB
intrinsic that is available in Android APIs to convert an android.media.Image
in YUV_420_888
format to Bitmap
.
Note to readers
- I have a habit of explaining concepts before showing the code, if you want to see the money, no code — go to Java code section.
- Despite lacking a good sense of humour, I tend to try to write funny.
- For this article, I expect readers to be familiar with Android, Java, JNI, Images in YUV and Bitmap formats.
- This article also serves as an example of how to use
RenderScript
in Android, and it's performance benefits - even if this is not the exact use case you are targeting.
Originally published at https://blog.minhazav.dev on July 11, 2021.
RenderScript
You can read much more about RenderScript in Android’s documentation. In short:
- RenderScript is a framework for running computationally intensive tasks at high performance on Android.
- Primarily oriented for data-parallel computation (very well suited for image processing).
- Runtime parallelizes the work across available processors such as multi-core CPUs & GPUs.
- Developers can focus on the algorithm rather than it’s scheduling.
- Language derived from C99
YUV_420_888 to Bitmap in Android using RenderScript
If you are here, I assume you are fairly aware of YUV_420_888
format & Bitmap
.
If you are not, and you want to learn more or find other ways to convert YUV_420_888
to Bitmap
in Android, you can check out another article:
ImageFormat#YUV_420_888 is one of the most common image format supported by Android Cameras. It’s a multi-plane YUV (YCbCr) format represented by three separate planes in android.media.Image. This format can be used for processing the input frames before saving to disk or some other action. A very common question around YUV is how to consume it in Android. In this article, I’d describe different ways it can be used. The most common question is how to convert YUV to Bitmap or jpeg format in Android?
ScriptIntrinsicYuvToRGB
ScriptIntrinsicYuvToRGB is an intrinsic for converting Android YUV buffer to RGB. The input allocation is expected to be in NV21
format and the output will be RGBA with alpha channel set to 255.
In NV21 format, we have full Y channels and subsampled U & V channels. That is each U & V pixel caters to 4 pixels in the Y channel. Also, the NV21 format is single plane with Y channel data first followed by interleaved V & U pixels. For a 4X4 image it would look like YYYYYYYYYYYYYYYYVUVUVUVU.
BTW if you are wondering what an intrinsic
means:
intrinsic: Normally, “intrinsics” refers to functions that are built-in — i.e. most standard library functions that the compiler can/will generate inline instead of calling an actual function in the library.
(source: What are intrinsics? — StackOverflow)
How to use this intrinsic
So basically, let’s say we want to use the Java APIs available. We:
- Initialise a
RenderScript
Context. - Create input and output
Allocations
. - Copy input and output to the allocations.
- Execute the kernel.
Some definitions:
- An Allocation is a
RenderScript
object that provides storage for a fixed amount of data. This needs to be provided by the caller. - Allocations are of type Element. In this case we would want to create one allocation for input (YUV) and other for output (
Bitmap
). - For
Bitmap
we can directly use theElement
of type RGBA_8888. - For
Image
we don't have a direct element type, and we need to convert the data into a flatNV21
byte[]
.
Java Code
Since the allocations are expensive, I’d design the allocations to be reusable.
Note: In this example I am not taking care of the thread-safety or making the code asynchronous or whether we should do resource allocation in the constructor. Leaving all of those tasks to you :)
So we can start with a skeleton of our YuvConvertor
class
Constructor
Note: There is a certain overhead associated with construction of this object (driven by initialization cost). Be mindful of when to initialize this. On a mid-tier Android device I observed it to take between
0ms - 14ms
. The cost could be higher on a low-end device.
toBitmap(Image image) method.
Since RenderScript
inherently doesn't support YUV_420_888
Image
- we need to convert the Image
to a byte[]
. This is going to be slow down every thing.
Unless of course we can write a fast highly parallelized way to do so (which should be doable). So I have found two ways to do it — one is fast other is not. Also, important to note that neither of these approaches leverage the parallelizability of this copy.
Let’s start with our Java approach which is slower.
toNv21(Image image) Java Approach
I have the split the performance numbers into two part:
- Full
toBitmap(..)
method. toBitmap(..)
withouttoNv21(..)
to show the cost difference of the intrinsic and the buffer copy.
Do you see just how sad that Java copy was? It took 83%
of net computation time. BTW, this total time is still faster than pure Java approach but slower than full native approach mentioned in How to use YUV (YUV_420_888) Image in Android.
Let’s do better with a native toNv21(..)
method.
toNv21(Image image) Java Native (JNI) Approach
In this case we basically port the java copy code to native and plug it in to the Android code with JNI. Here I am assuming you know how to set up JNI.
Java code:
JNI code:
I have the split the performance numbers into two part:
- Full
toBitmap(..)
method. toBitmap(..)
withouttoNv21(..)
to show the cost difference.
I hope we can make do with these numbers! This is faster than the Java or Native implementation we had in How to use YUV (YUV_420_888) Image in Android.
Even faster?
I am fairly sure this can be made much faster by:
- Either, a custom RenderScript kernel than can read the three planes in Y, U & V directly.
- Faster NEON based implementation of native code.
Ok fine, take what you came for!
YUV_420_888 Image to Bitmap using ScriptIntrinsicYuvToRGB — GitHub Gist
Performance verdict
Appraoch with Native YUV 420 -> NV21 byte[] is faster that java or direct native implementations. See How to use YUV (YUV_420_888) Image in Android for more numbers.
But the GPU implementation using Vulcan or NEON extensions could be faster. I plan to do this full investigation soon. (I’ll update this article).
Just FYI, RenderScript is being deprecated in Android S (Android 12)
What why? I just learned about it
:(
And why the hell, did you just explain the whole stuff?
Relax, read more below first!
With Android S (Android 12), the team seems to have announced deprecation of RenderScript for following reasons in the blog post:
- RenderScript was introduced to allow developers to run computationally intensive code on CPU/GPU without making use of NDK or GPU specific APIs. It was abstracted by RenderScript.
- With Android’s evolution, NDK and GPU libraries using OpenGL have significantly improved. They give low level access to GPU hardware buffers and RenderScript no longer seems the most optimal way to accomplish the most performance critical tasks.
- Your current RenderScript will continue to work on existing devices, it would still compile for Android but on future Android devices, the internal implementation may be CPU only.
- Android team seems to have written a toolkit for migrating core intrinsics in RenderScript to highly optimized C++ implementations — android/renderscript-intrinsics-replacement-toolki
I’ll try out the intrinsic example by Android team and write about it!
If you know of a faster way to get this done with RenderScript or with other APIs in Android please let me know in the comment section.
References
- RenderScript — Android documentation
- C99 language — Wikipedia
- How to use YUV (YUV_420_888) Image in Android
- What are intrinsics? — StackOverflow
- RenderScript depreciation — Google Blog
- RenderScript migration — Android documentation
Attributions
Originally published at https://blog.minhazav.dev on July 11, 2021.