Node.js get-pixels: Getting Pixels at Specific Sectors of an Image using ndarray

npm has a handy package for extracting pixel data from a given image, but it doesn’t give us any (obvious) way to extract pixels from a given sector of the image with a defined width and height. Doing this on the client side is easy, since we have tools like HTML5's canvas element, but server side, we’re kind of stuck.

Let’s take a look at how we can implement the math to do this on a node.js server, given an image with a certain width and height and assuming we don’t have something like canvas to abstract this away for us.

First, let’s set up our module to require get-pixels:

var getPixels = require(‘get-pixels’);

Next, we’ll build out a function that can invoke getPixels on a given image. For the sake of illustration, let’s imagine that the property req.body.imagePath contained the path to our image:

var pixelGetter = function(req, res){
  getPixels(req.body.imagePath, function(err, pixels){
    if (err){
      console.log(err);
      return;
    } else {
      console.log(pixels.data);
    }
  });
}

pixels.data returns a buffer object with a length property, and we can iterate over it similar to how we’d iterate over the indices of an array. The first four indices will be RGBA values, respectively, for the first pixel; the second four indices will be RGBA values for the second pixel, and so on. Here’s what this might look like:

[255, 255, 235, 0, 253, 143, 32, 0…]

Imagine we had a .png image with a width of 5 pixels and height of 5 pixels. When getPixels is invoked on this image, it would return a buffer object with 100 elements: 5 pixels by 5 pixels is 25 pixels total, each with an R component, a G component, a B component, and an A component — 25 pixels with 4 properties each gives us 100 total elements.

Our problem is that when accessed on pixels.data, all of these elements exist in a totally flat array! Fortunately, iterating through the array stored on the .data property isn’t the only way for us to access pixels in the image, since get-pixels actually returns a ndarray. ‘n,’ in this case, refers to the number of dimensions the array has — ours has 3, and might look something like this:

Think of the red, blue, green and gray squares as representing the RGBA values for a given pixel, and that data for a given pixel is four elements “deep.”

Now, assume we had an ndarray representing image that was 5 pixels by 5 pixels (like the visual representation of the ndarray to the left) and simply wanted pixels for the the top-left 3px by 3px segment of the image. This isn’t too hard to accomplish using a couple of nested for loops:

var flatArray = [];
for (var y = 0; y < 3; i++){
  for (var x = 0; x < 3; j++){
    for (var z = 0; z < 4; k++){
      flatArray.push(pixels.get(x, y, z));
    }
  }
}

Let’s have a look at everything we’ve done in the chunk of code above, starting with the code contained in the innermost for loop.

pixels.get takes as many arguments as there are dimensions, and these appear in the order we’d expect them to — in our case, we have x for width, y for height, and z for “depth.” (With animated gifs, there’s even a fourth dimension, representing pixels at each frame. That’s right — time!) Dimension z can be understood metaphorically as “depth” — and it makes sense if you think of each pixel as 4 elements deep due to it having an r, g, b and a value, respectivley. Here’s our image again:

Our iteration starts at the red square in the top-left of the cube — we traverse across the red face of our cube, left-to-right, top-to-bottom, and at each stop on a red square, we dive inward and grab the red, green, blue, and alpha values for the pixel. When constructing a flat array, we want to push each of these r, g, b and a values (whatever they may be) to the to the array before moving on to the pixel to its right (this happens as we iterate through the x loop). And of course, once we’ve reached the 5th pixel in a horizontal row of pixels (x=4), we increment y once to move to the row immediately below it, reset x to zero, and repeat.

If we wanted to get a 3x13 chunk of pixels in the top-right quadrant of our image, our code would look really similar:

var flatArray = [];
for (var y = 2; i < 5; i++){
  for (var x = 2; j < 5; j++){
    for (var z = 0; k < 4; k++){
      flatArray.push(pixels.get(x, y, z));
    }
  }
}

In either case, our variable flatArray will end up looking something like this:

[235, 216, 130, 0, 253, 143, 32, 0…]

Turning this back into an ndarray is trivial. First, install ndarray using npm:

npm install ndarray

Now require it in your node module so we can get access to all of its nifty methods:

var ndarray = require(‘ndarray’);

Creating an ndarray with our desired dimensions is as easy as invoking the following on our variable flatArray:

var pixelArray = ndarray(new Float64Array(flatArray), [10,10,4])

Now, we have an ndarray that lets us access the rgba values of pixels using the handy getter and setter methods that come bundled with the module.

Here’s a twist, though — the above is nice for knowing what’s going on under the hood, but it turns out that ndarrays have methods that allow for us to abstract a lot of this math away.

We can actually slice this array in three dimensions using .hi and .lo. Think of .lo as clicking and dragging the upper-left corner of our image towards the bottom right of the image to grab some pixels. Using the 5x5 pixel model illustrated above, we could grab the R values for the top-left 3x3 corner of our image by invoking this (remember, arrays in JavaScript are zero-based):

pixelArray.lo(2, 2, 0);

The resulting selection would be an ndarray that looked like the orange segment of our model below:

We can chain .lo with .hi to select only the portion produced by the overlap of each method. If we wanted only the R value of of the pixel located at (2, 2), we could write code that looked like this:

pixelArray.lo(2, 2, 0).hi(2, 2, 0);

Here’s what the selection looks like on our model, with the overlapping portion colored in a darker shade of orange and with the non-overlapping portions of the chained selections stripped away, respectively:

While it serves for a nice illustration of how these methods function, using chained .lo and .hi methods wasn’t really necessary to grab this pixel, since all we’d really have to is write the following code:

pixelArray.get(2, 2, 0);

There are plenty of cases where we’d want to use this method chaining, however.

Let’s say we wanted to select the bottom-left 2x2 sector of the image. Our code would look like this:

pixelArray.lo(2, 4, 0).hi(4, 2, 0);

Here’s what this would look like on our model:

We haven’t explored grabbing depth yet. If we wanted to grab both the red and the green portions of this 2x2 chunk of pixels, all we’d have to do is change our code to look like this:

pixelArray.lo(2, 4, 1).hi(4, 2, 1);

Again, here’s what that would look like on our model:

That’s pretty much it! ndarrays are extremely handy for grabbing elements in a multi-dimensional array, and is especially handy for when we want to grab and perform analysis on certain pixels in an image.