Let’s Build: Digital Sand Paintings with P5js
Sand paintings mesmerize me. I love the way they slowly build by sorting the different size grains into layers and how the same, slow process builds a range of sandscapes.
I wanted to create a similar effect using pixels from an image instead of sand grains. The goal was to use pixel sorting algorithms to create a slowly evolving image, gradually converting a photograph into a sandscape.
Pixel sorting is the process of applying list sorting algorithms to the data contained in an image. A single pixel has many properties that can be quantified and sorted; the red, green, or blue color channels can be used or the hue, saturation, or brightness. Since pixel data are arranged in a two-dimensional array, pixels can be sorted horizontally, vertically, or both.
For this project, we’ll be working with a bubble sorting algorithm. This algorithm is ideal for an animated process because data are swapped with their nearest neighbors. Individual pixels don’t move far from their original positions each iteration, and one “edge” of the sorting stabilizes quickly.
No sorting project would be complete without addressing the speed of the sort algorithm. In this case, however, we are less concerned with the overall time to sort our pixels, but with the time to complete one sort iteration for each pixel in the image. A 200x100 pixel image contains 20000 pixels, which means that one iteration of the bubble sort involves 20000 comparisons and up to 20000 swaps of the pixel data. Additional, artistically-motivated operations will also need to be repeated this many times.
This is where a challenge arises; the P5js pixel set() and get() methods were not designed for speed, but for simplicity (see discussion here). Since these will be used so frequently in pixel sorting (both for comparison and for swapping), we will look at creating something more efficient for this project. These fastSet() and fastGet() functions can be used to directly manipulate the pixels[] array multiple times before updating it for display.
From here on, we’ll actually start building using P5js. You should be familiar with the use of variables, functions, loops, and arrays. Several platforms work, but I recommend (and will provide links for content in) the P5js editor and OpenProcessing.
Sorting Pixels (Quickly) in P5js
Open the Slow Pixel Manipulation sketch (P5js Editor, OpenProcessing) and take a look through the three main functions. preload() loads the image panel into memory, setup() creates a canvas with enough room for a frame around the panel, and draw() loops through every pixel in the image, sorting by brightness, before displaying the updated panel (line 38).
The implementation of the bubble sort algorithm for our pixel sorting is to look at each pixel in the image (the loops in lines 26 and 27), compare this pixel’s brightness to its neighbor (line 28), and swap the pixels if the pixel on the left is brighter. The swap requires storing the color of the right pixel (line 29), moving the color from the left pixel into the right pixel (line 30), and then putting the stored color into the left pixel (line 31). Finally, the pixels are all updated (line 32). The result, after many iterations, is each row sorted from by brightness from dark to light.
Running this code doesn’t inspire confidence. It takes a long time to sort the pixels in a small image. We can do better!
Now take a look at the Fast Pixel Sorting sketch (P5js Editor, OpenProcessing). Most of the code remains recognizable. There is an addition to setup() to ensure that the monitor pixelDensity is set to 1. More importantly, calls to the panel.loadPixels() and panel.updatePixels() have been added and moved, respectively. In the previous code, loadPixels() was called as part of each get() or set() use. Additionally, updatePixels() needed to be called after each swap to prepare for the next get() method use. These are responsible for the slowdown.
The major change is the use of the functions fGetPanelPixel() and fSetPanelPixel() in the comparison and swap. These functions manipulate the pixel[] array directly, extracting the [r,g,b] values for a pixel and writing these values back. Using these functions requires explicitly calling panel.loadPixels() first, but only requires one call to panel.updatePixels() at the end of the updates. These functions are not as safe as get() and set(), but they are much faster. For more details about how these functions were created, visit the P5js reference page for pixels.
With these updates, we can now complete the sorting task on an image with 9 times as many pixels in 1/5 of the original time (on my machine), a nearly 50x improvement.
Now, armed with some faster pixel manipulation tools, we can start to work on creating the sand painting effects by modifying our sorting algorithm.
But first…
There are many different properties that we can use to sort our pixels. In place of the brightness() function, try using hue(), saturation(), red(), green(), or blue() to sort on these properties.
If you’re feeling really adventurous, try comparing one property to a different one (say, red() > saturation()). You can get some ongoing dynamics instead of a static result.
If you want to make sure you understand the code, try sorting from top to bottom instead of side-to-side. Be careful not to read outside of the pixel[] array.
The Sand Swap Algorithm(s)
The bubble sorting algorithm proceeds too uniformly and directly, and it only operates along one axis. Fortunately, we have some options!
Open the Sand Swap sketch (P5js Editor, OpenProcessing). Two additional functions have been added to allow easy switching between different styles of sorting. The sandSwap(style, i, j) function has 12 different styles, corresponding to sorting by hue, brightness, and saturation either up, down, left, or right. You could easily add more for sorts on red(), green(), or blue(). The first integer parameter chooses the style of the sort for the pixel at (i,j).
This new function streamlines the draw() loop substantially and makes it easy to move between sorts on different properties. Choosing randomly starts to build some of sand-like structures, but the transition between a horizontal and vertical sort is jarring. Let’s fix that.
But first…
You likely noticed that the pixels always seem to “move” from right-to-left and bottom-to-top. This is because of the way the two for loops have been set up (incrementing upward). Can you update the draw loop so that they move in a different direction? Be careful not to read outside of the pixel[] array.
A Pixel Sandscape
The last step toward creating the Pixel Sandscape is to smoothly combine a horizontal and vertical sort. Open the sketch in either the P5js Editor or OpenProcessing; the changes are all in the draw() loop.
To smoothly sort pixels both horizontally and vertically, we do both at the same time. In lines 46 to 52, a choice is made for each pixel to use either a selected horizontal sort (hSortStyle, line 18)or a vertical sort (vSortStyle, line 19) for that pixel. This also introduces some interesting dynamics as the sketch evolves; some pixels quickly become fixed in place while others make their way to their final positions slowly and circuitously. This is just what I’m looking for!
To return a little more order to the dynamics, a biasValue is generated using the Perlin noise function. The expression for biasValue (which can be adjusted using the biasScale and biasFreq constants) produces a smoothly varying value between 5% and 95% that is used in line 48 to determine what fraction of the pixels sort horizontally. This creates a sort of current or wind that changes direction over time.
There are now many parameters that we can use to adjust the final result of the sandscape and the dynamics of how it reaches the end. Before moving on to the final sate, try adjusting these to your liking.
Selecting a New Panel
After the sorting algorithms have run for a while, the panel will settle into a fixed, or near fixed state (just like a physical sand painting). The original image can be reloaded and a slightly different experience will be created, but we can do better than that. As a final step, let’s look at a way of restarting the Sandscape with a different panel selected from the same larger image.
The approach that we will take is to preload the full image (shown here) and select a random portion of it to use as the panel. This will be done with a new function called getNewPanel(). Both the setup() and mousePressed() functions will call this function. The resulting program can be found here for P5js Editor and OpenProcessing.
Now, it is time to personalize your work.
Try adding your own image for the panel. Pop open the file manager with the arrow at the top of the line number column. You can add an image of your own, and update the preload() function to Sandscap-ify a different image.
Along the way, you’ve tried sorting based on several properties. Update the sandSwap() function to produce something more calming or more chaotic!
And don’t forget that there are parameters scattered throughout the code that will adjust sizes, rates, and so forth.
Finally, if you are inspired to go further with pixel sorting, you might consider downloading the desktop IDE for Processing. You can install a P5js mode to run locally (faster) or port this code to Java (much faster). You can take advantage of this additional speed increase the detail in your work.
Just like a beach, images are built up from tiny grains. Pixel sorting algorithms are like the wind, gently pushing the grains against each other so that they group, reorder, and build structures. If there is a deeper meaning that I find in pixel sorting, it is the reminder that the world around us is built from countless smaller parts shaped by simple forces to create all manner of beautiful things.
