The Anatomy of a Jellyfish
Creative / Technical Overview of ‘Maru’
pppolymer.net || Isaac Cohen ( @cabbibo ) || https://patreon.com/cabbibo
Hey there and thanks for stopping by to check out this write-up on Maru, an interactive audiovisual website made in collaboration with Plaid for their new album ‘Polymer’! It’s partly about the creative decisions I made, and partly about the technical ones. I’ve usually tried to focus on one or the other in write-ups, but here, I’m going to see if I can blend the two!
I’ll try and keep stuff pretty simple, but if anything doesn’t make sense, just skip it! I’m doing my best at learning how to explain things better, but this is all new to me! If you have more questions or there is more you want to chat about, you can also reach out to me on Twitter @cabbibo or shoot me an email at hello@cabbi.bo! I’m also just starting a Patreon, so if you like this article, and want to see more, please consider supporting: https://patreon.com/cabbibo
If you haven’t checked out the experience itself definitely do that first!
It’s non-mobile only, but you can find it at pppolymer.net! If you can’t see it first hand, here is a link to a video, but I’ve got to tell you, compression is my kryptonite…
Also, the fine folks over at Warp have let Ed, Andy & I Open Source this project for all y’all to dissect and learn! So if you are a coder ( or just wanna go hunt for some cool stems! ) go check out the repo: https://github.com/cabbibo/PlaidPolymer
So it begins…
In the process of chatting with Ed, Andy & James about what we were going to do for this piece, we thought about going in a bunch of different directions. Infinite microscopic landscapes pulsing to music, Oil slick simulations, collaborative unlocking of stems, etc.
In the end we decided that it would make the most sense to try and create a piece that illuminated the juxtaposition between the organic and the polymer. This is quite a difficult task when creating in real-time in WebGL, but after a bit of research / experimentation, we figured that the best idea would be to make a jellyfish surrounded by polymer nodes, which the participant could turn organic with a simple click.
The word Juxtaposition feels like it’s a bit overused, but to me this building and releasing of tension, the balance of opposing forces, is the core of all art ( and even life! ). By making these floating crystalline polymers an environment our flowing jelly could explore, we let the crystals be more crystal, and the jelly be more jelly!
What Software do you use?
Deciding on what libraries etc. to use was probably the easiest part of this process! We knew we wanted to publish on web, so that as many people could see it without downloading anything, and if you are publishing on web, unless you are planning on writing raw WebGL, your best bet is the super friendly, super accessible, super open THREE.JS, an API for making 3D graphics on the web!
Pretty much every single 3D website you’ve seen uses three.js in some way shape or form! I learned to code by just downloading examples, altering them until they broke, and then re-downloading and starting over!
Knowing I was going to be working in an API that felt like home made me super excited, as I hadn’t gotten to do much of it since I’ve been doing mostly XR stuff in unity.
There are obvious limitations when working in the web, but the distribution abilities alone are WAYYY worth it!
With this fundamental part of the project decided on, it was onto actually fabricating the reality!
POLYMER
I started off trying to make the polymer. If you listen to Plaid, you know that their music is powerful, mesmerizing and precise. If you don’t, you are making a big mistake. In our previous project ‘The Digging Remedy’ ( http://thediggingremedy.com/ ) as well as ‘Maru’, we knew we wanted to make something that was as interesting visually as aurally. Making art that is as good as music is a pretty hard task for me, but as with any project that has audio, I knew there were ways to syphon the life-force of the music, and channel it into making the piece feel alive.
Plaid knew they wanted the polymer in the experience to mimic the look and feel of the polymer in the pack shot, but we also wanted to make sure viewers knew that they were clickable, becoming some strange geometric sequencer for another Plaid Banger…
By making the polymer low poly, as well as disforming a pulsing to the rhythm of the stem that it would play, we were both able to signal the action to the user, while still making the object seem polygonal and non-natural. After a while, I could tell which shapes played which loops, just by the movement of the spheres!
Just this was cool enough, but so much of the theme of the album was about the natural vs. synthetic, and without a corresponding natural shape that balanced this synthetic one, there wasn’t much of a story.
ORGANIC
To then make these objects feel ‘organic’ so that the participant was helping move the narrative from man-made to natural, I tried subdividing the icosahedron until it was smooth and then used this same audio to help shade and deform the new smoother object
Although I wish the objects could look even *smoother* , there are very real limitations to how many triangles when you create real-time objects, it seemed to work for the size of the polymers on the screen.
FFT
But just how did we get these polymer to pulse to the audio? Enter the magic of FFT.
FFT stands for Fast Fourier Transform, and is a way of decomposing a piece of audio into its the different ‘frequencies’ that make it up. You’ve probably seen FFT’s in action with out even realizing it, usually looking something like this:
This function is *luckily* built into the web audio API and makes it easy enough for even the newest javascript developer to play with the analyser functions ( check out https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Visualizations_with_Web_Audio_API for a tutorial )
In this case, every polymer had its own loop, which could be turned on and off. To make the polymers pulse to the sound of their own stem, it mean that I had to do an FFT analysis for every polymer on the screen.
In the above gif, the ‘strength’ of the signal at each frequency corresponds to the ‘height’ of the graph. In our case, it will correspond to ‘brightness’, with black being a weak singal and white being a strong signal.
Each vertex in the polymer would then see the strength at a certain value in the fft, and extrude along its normal as the sound got louder! To put it in other terms: This means that if there was no sound, the polymer would be perfectly spherical, but when audio came in, the polymer would bulge!
This same ‘FFT value deformation’ shows up again and again, and we’ll see it later on when we look at the jellyfish! Keep it in your noggin because we’ll be seeing that gif again!
COLOR
The FFT texture above is greyscale, moving from black to white. However, the gif of the ‘organic’ polymer itself has a slight ‘chromatic aberration’ to it. Chromatic aberration is a *widely* overused term, that usually doesn’t mean what the user thinks it does. That’s 100% true in this case as well, but it helps describe the effect so I’ll stick with it (incorrectly) anyway! In reality though it’s quite a fascinating phenomena that happens because different frequencies of light refract differently through a lens.
This bending of light means that the edges of images blur out across the spectrum giving it a hint of color, and a bit more life! To create the color out of the black and white FFT, I used a trick that I’ve used in pretty much every single audio visual experience I’ve ever created. Instead of just feeding in 1 value for each frequency band of the FFT, I send in 3 ( 1 for red, 1 for green, 1 for blue ). Each of these values samples the 3 closest frequencies, which will always be similar but not *exact* .
These slight variations in frequencies turn into slight variations of color, suddenly using the audio to breath life into each object that uses this texture
This single texture was all that was needed to shade the organic polymer, and all that was needed to deform the synthetic ones. But the polymer were not the only piece of the puzzle, and time constraints were *very* real, so I had to move on to the real star: Our Jellyfish!
JELLYFISH
I’m a firm believer that its only with real-time reactivity that you can make a creature really feel ‘alive’, like something that is moving and breathing and in an environment ( like ours ) that is changing! If you haven’t watched it, Bret Victors’ ‘Stop Drawing Dead (jelly)Fish’ is a good place to really grok why I love creating for computers.
This means that although we could have made a pre-scripted jellyfish animation it wouldn’t feel organic. And to make it feel organic, it needed to behave according to laws of nature, in this case *physics* . I figured that if I could create a procedural jelly out of springs, masses, and other physical constructs, it would behave fluidly, be responsive, and almost feel alive.
With that in mind set out to work our little sparkly friend.
PHYSICS ( GPU )
We knew the organic element needed to feel as luscious as possible, with many intricate moving parts. Because of this, I knew that I wouldn’t be able to rely on typical calculations on the CPU and would have to spent time converting all of my physics so they could run in parallel on the GPU! Its a very technical challenge, but luckily people have found ways of tricking the computer to do this before!
A GPU is *REALLY* good at making pixels certain colors, but thats pretty much all it is allowed to do in WebGL. However, if we tell the computer we are making a texture with certain colors, and then reusing those same Red, Green Blue values to correspond to a position in 3D space, I could create ‘textures’ that would store the positions and velocities of every single point on the jellyfish head, tail and plume. Though not nearly as interesting as the jellyfish itself, you can see what the main tails ‘position textures’ look like below :
In this case red is X position, green is Y position, and blue is Z position. For Instance, this means a white pixel would correspond to a position that is in the positive X, positive Y and positive Z Octant. ( In this case the pixel values ‘max out’ at 1, but in reality, the positions can be *far* greater than 1 ( it just makes it harder to show… ))
Although its a bit hard to tell, this texture is telling us that all of the points in the jellyfish are in the positive z (blue) direction, and that the jellyfish itself is moving in the positive X(red) and positive Y(green) direction. Its a bit nerdy, but if this is something that you want to know more about, you can check out the library I build for doing physics just like this over here: http://cabbi.bo/PhysicsRenderer/
MATH
The actual Math behind the Jellyfish is quite simple. Each particle simply applies a force attracting it to the one above it:
That is, there is a force that moves the green particle towards the red particle that is dependent on the distance between the two particles! ( “m” in this case is just some multiplier to make the force stronger or weaker ). Chain a bunch of these particles in a row, and we get particles being attracted to the ones above them, forming a trail
Where each particle is ‘following’ the one above it by applying a force towards it. In action we see:
Here the first ‘particle’ is following our mouse position ( the white dot ), and every subsequent particle is following the one before it, creating the smooth spine.
Sub Section
Now this movement is nice, but we are making a Jellyfish, not a snake! The shape of the jellyfish needs to have a bit more ‘width’ to it, so it looks like a fern more than just a line! Again, our goal was to make something that felt organic & lush, while still making it run in real time. I felt like the best way to do this, was to create a plethora of simple objects, all connected together to form a larger, complex / luscious one.
( As an aside, interestingly enough, the Man of War which is not an actual jellyfish, mimics this thinking, as it is a siphonophore, a colony of organisms, not an animal! )
In order create this complexity through simplicity, I put 4 particles surrounding the each particle in the ‘spine’ of the jellyfish. These particles all attract to the particle in the spine but REPEL from the other 3 particles that are attached to that spine. Below is a simplified diagram, with spine particle P1, and Sub particles P2 & P3
This means that P3 and P2 are actually pushing against each other, while at the same time, attracting to P1! This makes them evenly distribute around the spine, giving it a bit of width. Just like in our initial diagram, “m” is just a value we multiply by to make the force stronger or weaker.
By making that same “m” value grow along the spines tail, we will make it so the further along the tail, the more the ‘sibling’ particles repel from each other, giving the spine more girth near the base.
We can see this in action below:
Combining this Sub Section Layer and the Spine Layer we get
Sub Sub Section
This is much better, but I still wanted more decadence! To do this, I repeated this process one more time! That is to say, for every Sub Particle I created 4 ‘Sub Sub’ particles, which all repel from each other while simultaneously attracting to the their ‘parent’ Sub Particle !
Doing this means that for every spine point we will have 4 sub particles that each have 4 sub sub particles, giving us a total of 21 particles per spine section! Multiply that by the length of the spine and you start pushing on the border of complexity.
Now thats looking luscious!
HEAD
A jellyfish, however, would be nothing without its head, so let’s look at that next!
The spine seems so beautiful and smooth, I wanted to make sure that the head of the jellyfish followed its movement, meaning that that we needed to place the different vertices of the jellyfish head in relationship to the jellyfish spine!
This essentially meant a tube, placed in a smooth curve along the spine, like so:
Of course I needed the radius to be variable to give it more of a ‘jellyfish’ look:
This was looking good, but the ‘resolution’ of the head ( AKA its smoothness ) was of upmost importance! The Spine had 32 particles distributed along the whole tail, while the head had 1024 distributed in only a fraction of that area.
This meant I couldn’t place the points linearly along the spine, but had to turn the points of the spine into a curve, making it smoother. In the gif below, look at how the purple line curves smoothly through each red point of the spine ( I’ve have shown only the first 7 points of the spine for visualizations sake )
In order to create this smoothness, I used a ‘cubic bezier curve’, which is a mathematical formula which lets us create a smooth curve connecting points. Its a bit too much maths to go into here, but you can read more about the curves here : https://en.wikipedia.org/wiki/B%C3%A9zier_curve
Essentially though, stretch the head along as much of the tail as I wanted, making for some strange variations
Combining one of these variations with the body of rest of the particles, we start to see a glimpse of our jelly to be:
Now thats starting to look a little jelly!
But what are those yellow dots doing? They certainly are a nice touch, but they aren’t the spine, sub sections, or sub sub sections. So what are they and where did they come from?
PLUME
Every medium has its constraints. With sculpting its gravity, with paint its the bristles of the brush, the texture of the canvas, and with computers, it’s a bunch of stranger, more arbitrary rules. In the case of the physics renderer, the number of ‘particles’ we render needs to come in the form of (2ⁿ)*(2ⁿ) particles.
This means we could simulate 1 particle [ (2⁰) * (2⁰) ] , 4 particles [ (2¹) * (2¹) ], 16 particles [ ( 2²) * (2²) ] , 64 particles [ ( 2³) * (2³) ] , 256 particles [ ( 2⁴) * (2⁴) ] , 1024 particles [ ( 2⁵) * (2⁵) ] and so on!
In this case, I decided 32 x 32 was the best dimension set ( 1024 particles ) because I wanted the tail to be relatively long, but a spine with a length of 64 felt pretty ridiculous. This meant that the texture layer looked something like this:
Where the Y dimension is the position along the spine, and the X defines what section we are looking at.
Column 1 ( in yellow ) corresponds to the particles in the spine
Column 2–5 ( dark to light orange ) correspond to the particles in the ‘sub sections’ ( 4 particles per spine )
Column 6–21 ( dark to light red ) correspond to the particles in the ‘sub — sub sections’ ( 4 particles per particle sub section. AKA 16 particles per spine particle )
If this was all we were doing, then we would have a texture with dimension 21 x 32 , however because of our constraints this means we have an extra 11 columns to play with! Not one to waste memory or computational power, I decided to make those final 11 columns a ‘plume’. This made the particle break down of a single row look something like
These plumes, would behave like the spine!
The same as one of our first diagrams, with the difference being that the very first particle in the chain would follow the bottom edge of the jellyfish *head* instead of the the very tip of the jellyfish.
Singling out just the plume looks a little bit like this:
They say constraint breeds creativity, and I couldn’t agree more! If it wasn’t for those 11 extra columns, I would have had no reason to say ‘well what do I do with these’ and our jellyfriend would have been slightly more banal.
more! More! MORE!
Now even though our jelly was looking pretty majestic physics wise ( we will get to shading soon! ) I wanted to take it even further! To do this I created 6 more of the same 32x32 textured tails and repelled them from each other!
Additionally, I also placed these textures plumes evenly around the jellyfish head! By also making the forces that pulled this particles stronger, these tails become shorter and tighter filling out the top section of the jelly
This felt especially important, as with real jellies, most of the ‘stuff’ is packed up close to the top:
Combining these trails with the main trails gave us:
The shape of our Jelly was finally looking good!
That being said, the colors still look disgusting, and there our little friend wasn’t dancing to the music at all! So I went back to the techniques I used for the polymers, and applied them to the jelly to give it a bit more life!
SONIFYING / SHADING THE JELLY
The first thing I did with the sound was to make it affect the way that the jellyfish moved. Although there are a few ways to do this, one of the ones I found the most exciting was to take that initial function:
Which depended on the constant ‘m’ and to use different parts of the FFT to change the value of ‘m’. That is, in the gif below, darker would mean smaller ‘m’ and brighter would mean lighter ‘m’
I continued to use this same texture to affect different parts of the jellyfish!
The size of the particles was dependent on the fft, the color of the particles used the chromatic aberration fft, even the width of the lines connecting the particles was defined, all using the fft
Close up, this looked pretty majestic, and far back:
Even Better! So much of the feeling of ‘complexity’ or ‘lusciousness’ feels like it comes from an object having detail at many different ‘scales’ . In this case, the FFT texture felt like it was the perfect way to add this sort of detail at both the minute scale, with beautiful chromatic aberration happening within each particle, as well as the larger scales that define the shape, size and movement of the full entity!
There are a few different stems that reallllly help illustrate this, so next time you go and check out pppolymer.net , try testing out just listening to a single loop at a time to see what it does to the jelly!
See the way that the polymer lights up at the same time as the jelly? Its the magic of audio Reactivity!
FINISHING TOUCHES
With both the polymer and the jelly looking tip top, it was time for the finishing touches!
We tried out different background colors, different arrangements of polymer, different loading screens etc.
We ended up trying to parallel the pack shot, but with a more ‘deep sea / immersive’ feel to it. Combine that with me floundering around in HTML & CSS to make the main, and we finally made something that felt complete!
EPILOGUE
Creating art is really hard for me, and it turns out that explaining it is even harder! I’m so happy with the way the jellyfish turned out, and I hope that you learned at least one new thing by reading this post! I’m sorry if it was too Artsy or too Nerdy for you, but it turns out, that’s right where I seem to find the most beauty :p
Again, the repo is up for you to play with if that’s your kind of thing: https://github.com/cabbibo/PlaidPolymer
Give the experience one more go:
http://www.pppolymer.net ( not on a cell phone )
And for the love of god go pre-order/pre-save the album! Let me tell you first hand, its majestic:
https://plaid.ffm.to/polymer-pre
Huge thanks to Andy & Ed, James over at Warp and everyone else for helping make this a reality!
❤,
Isaac!
Twitter : @cabbibo
Email : hello@cabbi.bo
Help me make more of these ( and other weird stuff ): https://patreon.com/cabbibo
