Android : Simple and fast image processing with RenderScript

Want to make image editing fast with a few lines of code? Want to use the computing power of your phone’s GPU without the complexity of OpenCL? Well, renderscript is meant for you.

Example of blur on the right of the Image

Still not convinced of the utility of this powerful tool? Let’s speak numbers:

I compared the renderscript blur to the java based fastblur that you can find here. The image of the moutain has a resolution of 4806x3604 pixels. When processing blur on it on my Nexus 6P, it took 738ms for the renderscript. The fastblur didn’t even work (out of memory)! So I tried on a smaller image (1944x1944), and the fastblur worked in 1,354ms, so I tried again with renderscript and it took 160ms, it’s more than 8 times faster.

You can find below a comparison of the java’s an renderscript’s performances on the Gaussian Blur:

Blur performances (taken from Jonathan’s article on degree53)

I won’t talk here about NDK as I don’t have enough knowledge, but you can find a renderscript vs NDK comparison here. I didn’t go further in this direction as it was hard to set up and would not work for every phone unlike renderscript.

Renderscript is based on C99 (Ed. C language), so you need to be familiar with that language. It shouldn’t be hard to know the basics if you already know java.

First of all, you need to add those two bold lines in your build.gradle file :

android {
    compileSdkVersion 23
    buildToolsVersion "23.0.3"

    defaultConfig {
        minSdkVersion 8
        targetSdkVersion 19

        renderscriptTargetApi 18
        renderscriptSupportModeEnabled true

    }
}

If your app is minSDK 16 and lower, you should use support mode as a lot of methods were added since API 17.

The renderscriptTargetApi goes up to 23, but you should set it to the lowest API level able to provide all the functionality you are using in the scripts. If you want to target API 21+ with support mode you have to use gradle-plugin 2.1.0 and buildToolsVersion “23.0.3” or above.

Renderscript will use scripts written in C which will parallelize calculations for each pixel of your image. A script is a file with ‘.rs’ extension that must be placed in app/src/main/rs. Android Studio won’t generate this folder or any script file for you.

To illustrate this article, I’ve made a sample app that computes histogram equalization on the Y channel of the YUV colorspace (see picture below), and that blurs the image. It’s available on github.

Histogram Equalization on Y channel (before and after)

First example : Blur image

We will start with an easy task : blur an image. For this example, no renderscript code is required because a class is provided by the API : ScriptIntrinsicBlur.

public static Bitmap blurBitmap(Bitmap bitmap, float radius, Context context) {
    //Create renderscript
    RenderScript rs = RenderScript.create(context);

    //Create allocation from Bitmap
    Allocation allocation = Allocation.createFromBitmap(rs, bitmap);
    
    Type t = allocation.getType();

    //Create allocation with the same type
    Allocation blurredAllocation = Allocation.createTyped(rs, t);

    //Create script
    ScriptIntrinsicBlur blurScript = ScriptIntrinsicBlur.create(rs, Element.U8_4(rs));
    //Set blur radius (maximum 25.0)
    blurScript.setRadius(radius);
    //Set input for script
    blurScript.setInput(allocation);
    //Call script for output allocation
    blurScript.forEach(blurredAllocation);

    //Copy script result into bitmap
    blurredAllocation.copyTo(bitmap);

    //Destroy everything to free memory
    allocation.destroy();
    blurredAllocation.destroy();
    blurScript.destroy();
    t.destroy();
    rs.destroy();
    return bitmap;
}

As you already understood, this method returns a blurred bitmap. Let me introduce you to 3 important objects used in the above lines:

1. Allocation: memory allocations are done on the java side so that you should not malloc in a function that is called on each pixel (OOM is a pain in the ass). The first allocation I created is filled with the datas contained in the bitmap. The second one is not initialized, it contains a 2D array of the same size and the same type as the first allocation.
2. Type: “A Type describes the Element and dimensions used for an Allocation or a parallel operation.” (taken from developer.android.com)
3. Element: “An Element represents one item within an Allocation. An Element is roughly equivalent to a C type in a RenderScript kernel. Elements may be basic or complex.” (taken from developer.android.com)

Second example : Histogram equalization

Now that you understand the basics, we can start programming our own scripts.

The algorithm for Y histogram equalization is simple :

1. Convert RGB to YUV colorspace.
2. Compute the histogram of the Y channel.
3. Remap the Y channel according to the histogram.
4. Convert back from YUV to RGB colorspace.

Note : I edited the code thanks to Stephen Akridge (see comments). It‘s now 20% faster. Big thanks to him!

We are now ready to create our rs file : histEq.rs, located in the rs folder

#pragma version(1)
#pragma rs_fp_relaxed
#pragma rs java_package_name(com.example.q.renderscriptexample)

#include "rs_debug.rsh"

int32_t histo[256];
float remapArray[256];
int size;

//Method to keep the result between 0 and 1
static float bound (float val) {
    float m = fmax(0.0f, val);
    return fmin(1.0f, m);
}

uchar4 __attribute__((kernel)) root(uchar4 in, uint32_t x, uint32_t y) {
    //Convert input uchar4 to float4
    float4 f4 = rsUnpackColor8888(in);

    //Get YUV channels values
    float Y = 0.299f * f4.r + 0.587f * f4.g + 0.114f * f4.b;
    float U = ((0.492f * (f4.b - Y))+1)/2;
    float V = ((0.877f * (f4.r - Y))+1)/2;

    //Get Y value between 0 and 255 (included)
    int32_t val = Y * 255;
    //Increment histogram for that value
    rsAtomicInc(&histo[val]);

    //Put the values in the output uchar4, note that we keep the alpha value
    return rsPackColorTo8888(Y, U, V, f4.a);
}

uchar4 __attribute__((kernel)) remaptoRGB(uchar4 in, uint32_t x, uint32_t y) {
    //Convert input uchar4 to float4
    float4 f4 = rsUnpackColor8888(in);

    //Get Y value
    float Y = f4.r;
    //Get Y value between 0 and 255 (included)
    int32_t val = Y * 255;
    //Get Y new value in the map array
    Y = remapArray[val];

    //Get value for U and V channel (back to their original values)
    float U = (2*f4.g)-1;
    float V = (2*f4.b)-1;

    //Compute values for red, green and blue channels
    float red = bound(Y + 1.14f * V);
    float green = bound(Y - 0.395f * U - 0.581f * V);
    float blue = bound(Y + 2.033f * U);

    //Put the values in the output uchar4
    return rsPackColorTo8888(red, green, blue, f4.a);
}

void init() {
    //init the array with zeros
    for (int i = 0; i < 256; i++) {
        histo[i] = 0;
        remapArray[i] = 0.0f;
    }
}

void createRemapArray() {
    //create map for y
    float sum = 0;
    for (int i = 0; i < 256; i++) {
        sum += histo[i];
        remapArray[i] = sum / (size);
    }
}

We have different methods here :

• bound(float val): this method is used to keep the result between 0 and 1.
• root(): this method is called for each pixel of the input Allocation (it’s called a kernel). It converts the pixel from RGBA to YUVA, it puts the result in the output allocation. It also increments the value for the Y histogram.
• remaptoRGB(): this method is also a kernel. It remaps the Y value and then converts back from YUVA to RGBA.
• init(): this method is automatically called when creating the script in java. It initializes the arrays with zeros.
• createRemapArray(): it creates the remap array for the Y channel.

Note that you can create methods like you are used to in C. But here, if you need to return something as I do in bound(), the method must be static.

Call scripts from Java code

Now that your script is ready, you have to call it from your Java code.

Java classes will be generated for the scripts when building the project (so remember to build before using your scripts in java). If you have a script called foo.rs, a class named ScriptC_foo will be generated. You can instantiate it by passing the RenderScript object in the constructor.

You can call kernel methods by calling the forEach_root() method with input and output allocations as parameters, and it will compute the root method for each pixel.

When your rs script uses a global variable, setter and getter are generated in the java code, so if you use a global variable called value, you can set it with script.set_value(yourValue) and get it with script.get_value().

If you want to use an array as a global variable, you can either declare an Allocation type, and use the set or get method, or declare an array of a type and bind it with script.bind_variableName(yourAllocation). You will then access it in your script with rsGetElementAt_type(variableName, x, y) and set values with rsSetElementAt_type(variableName, element, x, y).

Here is my sample java code for Y histogram equalization :

public static Bitmap histogramEqualization(Bitmap image, Context context) {
    //Get image size
    int width = image.getWidth();
    int height = image.getHeight();

    //Create new bitmap
    Bitmap res = image.copy(image.getConfig(), true);

    //Create renderscript
    RenderScript rs = RenderScript.create(context);

    //Create allocation from Bitmap
    Allocation allocationA = Allocation.createFromBitmap(rs, res);

    //Create allocation with same type
    Allocation allocationB = Allocation.createTyped(rs, allocationA.getType());

    //Create script from rs file.
    ScriptC_histEq histEqScript = new ScriptC_histEq(rs);
    
    //Set size in script
    histEqScript.set_size(width*height);

    //Call the first kernel.
    histEqScript.forEach_root(allocationA, allocationB);

    //Call the rs method to compute the remap array
    histEqScript.invoke_createRemapArray();

    //Call the second kernel
    histEqScript.forEach_remaptoRGB(allocationB, allocationA);

    //Copy script result into bitmap
    allocationA.copyTo(res);

    //Destroy everything to free memory
    allocationA.destroy();
    allocationB.destroy();
    histEqScript.destroy();
    rs.destroy();

    return res;
}

Debugging RenderScript

You cannot use the debugger to analyze your renderScript yet (c.f Stephen Hines comment on his reply here), but you can use logs.

To use logs in a rs file you need to include “rs_debug.sh”. You can then use the rsDebug method, taking the log message and one or more variables as parameters.

#pragma version(1)
#pragma rs java_package_name(com.example.q.renderscriptexample)
#include "rs_debug.rsh"

void root(const uchar4 *v_in, uchar4 *v_out, const void *usrData, uint32_t x, uint32_t y) {
    float4 f4 = rsUnpackColor8888(*v_in);

    rsDebug("Red", f4.r);
    *v_out = rsPackColorTo8888(f4.r,f4.g,f4.b,f4.a);
}

I’m Android Image processing engineer at Pictarine, and we compute a complex auto enhancement script on 100,000+ pictures everyday. It uses a lot of scripts, like the histogram equalization (twice) and it takes generally less than one second to compute. We’re more than happy with this powerful tool and if you plan to do image processing I strongly recommend you to use it.

Example of auto enhancement (on the right)

Thanks to Baptiste (my Lead Developer) who encouraged me to learn renderscript and to write an article about it.