Coding Stories: Simulating Sight

This Spring during the COVID-19 lockdown I was spending a lot of time in my home. Much of that time was dedicated to my school job and spending time with my family. But given that I stopped commuting, was living with teenage children who have their own lives, and was prevented from doing many of the activities that usually consume my time, I was on the lookout for a new project.

So I took up drawing and painting. I have always loved photography. Taking walks, looking closely at the world around me, and attempting to capture it in the camera frame is a good match for my personality. Good photography is about becoming aware and in tune with your surroundings. So I wondered if drawing and painting might increase my awareness and build upon what I already love about photography. Also through painting from my imagination I could explore the natural landscapes that I love despite being confined most of the time to a sofa.

Now for some reason I have always been drawn to digital tools. In middle school the computer keyboard saved my school career after years of failing the subject of handwriting. In college I used digital tools to stay organized, avoid losing my work and improve productivity. In my current life I turn to the computer to create interesting animations and bring the arts to the STEM disciplines. So it was only natural that when adding drawing and painting to my life I immediately picked up an Apple Pencil, started using my iPad again, and bought the Procreate App. I knew that learning how to draw and paint was not a small task and technology does not lessen the need for practice. So I started watching James Julier’s YouTube channel and followed along with my iPad. They call him the Bob Ross of Procreate. I grew up on PBS programming so I had seen my share of Bob Ross tutorials featuring “happy trees”. I went through four or five of Julier’s videos making fun imaginary landscapes. As I worked through the tutorials I realized how much I love creating art on the iPad. Using layers means that you can correct specific parts of a painting without messing up the rest of it. I think one of the things that always held me back in drawing and painting is that I struggle to follow a careful procedure. For example moving from the back to the front of a scene in laying down color. Now I could move from layer to layer in any order that I wanted and slowly make it better over time. If I forgot something I could easily layer it in later. Here is one piece that I created based on one of Julier’s videos that I used as my Zoom background while teaching this past spring.

After spending a good chunk of time getting familiar with the brushes, color selector and other important features of my new art tool, I realized there were a few core ideas that were going to be critical in my progress as a painter. In this piece I want to focus on one of these core ideas, tone. The tonality of a scene is key to creating a sense of depth as tone is intimately connected to the illumination of the scene. The directionality of light creates the illusion of three dimensions in a two dimensional painting. It also plays a powerful role in defining the composition of a piece as well as conveying emotion to the viewer.

Interestingly tone is just as important in photography and despite years of taking photographs I was suddenly thinking about tone differently as I chose shades of color to use in my paintings. I was starting to notice tone in all parts of my life including the varying shades of the off white wall in my living room. Lighting also impacts saturation and hue as well but for the purposes of simplicity I will focus exclusively on tone for this piece.

I decided to start playing around with these ideas by getting down to basics. One of the most basic shapes you can draw is a sphere. A sphere has the exciting property that no matter what vantage point you look at it from, it has the shape of a circle in your field of view. So you can just draw a circle on the page and it is a sphere. As luck would have it for a beginner with limited fine motor skills such as myself there is a feature in Procreate to create a perfect circle on the canvas. However, a circle does not automatically look very three dimensional. To get that three dimensionality you need to think about the lighting of a sphere. So I started shading in one part of the sphere and then putting highlights on the other side. By adding black and white with a low opacity I could adjust the tonality of the circle making it appear three dimensional.

After fiddling with the shading for a while the one on the left looked pretty realistic. When you are painting there is some formal thinking to be done such as where is the light coming from and where will the shadow fall. But my process consisted more of seeing if it looks right and then making small adjustments so that it looks a little better over time. The iterative process of painting like this is very soothing and is a big part of what drew me to painting in the first place. Now I am quite sure that the lighting on the sphere I created above has some technical flaws, but it is close enough to convince someone it is a sphere.

But then I started to think about the sphere as a scientist. I wondered how the rules of Physics could help me better understand this sphere. I wondered if looking at the science of tonality could be just as rewarding as my experience with it artistically. I suppose that is what my blog is all about. I want to figure out where the science and the art intersect. Immediately I thought of bringing my sphere onto the computer screen and seeing what p5.js could offer to this exploration.

But now we need to get even more fundamental in our thinking. What does it mean to see something? I remember once watching an educational series called Minds of Our Own created by the Annenberg Foundation. In the video they asked students what they could see if they were in a room with absolutely no light. None of the students guessed the correct answer. Without light entering your eye, all you see is darkness as it is the light that allows you to see. So in order to properly shade our sphere the light from the sphere will have to somehow enter my “eye”. And the amount of light that enters from a particular direction will determine the tonality of that part of my vision.

One way to shade my sphere or really the whole field of view in front of me is to draw a series of lines from my eye outward. Those lines represent pathways traversed in the opposite direction by light coming from the scene that can be interpreted into tonal information by rods in the retina of my eye. The next step in this process is to draw a rectangle between this imaginary scene and the point that we are calling the eye or camera. You can think of this rectangle as the computer screen that will display the scene in two dimensions. Computer screens are made up of a rectangle of pixels. Each pixel is assigned a particular color value. Together these pixels are what you see on the screen giving you the illusion of three dimensions.

In the diagram above I have created an imaginary setup that I will use to demonstrate how to shade a sphere computationally. First I created a three dimensional coordinate system. At the origin (0,0,0) I placed a point which I am calling camO for the camera origin point. The camera plays the role of my eye and it is from this point that I will capture the light that will be used to visualize the scene. Next at the point (0,0,7) I placed a square in the xy plane. This square is the canvas that is instantiated by createCanvas(400,400). However, to adapt to some conventions in 3D graphics I will make the center of my square canvas at 0,0 instead of 200,200. Also I will call the corners (-1,1), (1,1), (-1,-1), (1,-1) instead of (0,0), (400,0), (400,400), (0, 400). Next I will include a sphere of radius 2 in the scene placing its center at (0,0,17). Finally I will illuminate the scene with parallel rays of light moving in the direction given by the vector (1,1,1). So I have a camera, a canvas, an object to be rendered, and light to illuminate the scene. That should be enough. Let’s see how this works.

To assign color values to each pixel on the screen I will create a for loop.

for (let i=0;i<width*height;i++) {
  const x = 2*(i%width)/width-1;
  const y = 2*int(i/width)/height-1;
  const camO = createVector(0,0,0);
  const camL = createVector(x,y,7).normalize();
  trace(i,camO,camL);
}

Each index i represents one of the 1600 pixels that make up my canvas. Later I will render these pixels using the loadPixel() and updatePixels() functions built into p5. The first step in rendering this scene is to define a line that begins at camO and passes through the given ith pixel. As explained above, this line which is technically speaking a ray will be used to determine the color value of the ith pixel based on any light from the scene that will traverse the ray back to camO. A line can be defined by a point and a vector. All the lines pass through camO so the point part is taken care of. The vector of each line which I call camL is simply an arrow that starts at camO and points at the given ith pixel. I normalize this vector for mathematical simplicity.

The trace() function finds the color value for each of the pixels by using the arguments i, camO and camL. The first step in the trace() function is to find where the line intersects the sphere. Below is a function that returns the distance to the point of intersection from camO. It returns null if the line does not intersect.

function findIntersection(o,l) {
  const t1 = o.copy().sub(c);
  const t1l = t1.mag();
  const t2 = l.dot(t1);
  const s = t2*t2-t1l*t1l+r*r;
  if (s>=0) {
    const sd = min(-t2+sqrt(s),-t2-ss);
    if (sd >=0) {
      return sd;
    } else {
      return null;
    }
  }
}

If there is no intersection then I can make the pixel black as no light will reach the eye along that line. If it does intersect the sphere then I can mathematically determine the point on the sphere where it intersects. I can do that by just multiplying camL by d and adding that vector to the position of the camera. Here is a line of code that I can include in the trace() function to accomplish that task.

const iPoint = camO.copy().add(camL.copy().mult(d));

I can also determine the normal vector to the sphere at the intersection point as it just points away from the center of the sphere.

const iNormal = iPoint.copy().sub(c).normalize(); 

Now we just need to calculate the amount of light that will be returned to the camera from each intersection point.

One type of lighting that will impact the pixel value is called diffuse lighting. This type of light assumes that the sphere scatters the light in many directions. The tone of the light is given by the dot product of iNormal and -lightDir. It makes the sphere look like this.

Another type of light that we will compute is the specular reflection of light. This assumes that the sphere is a good reflector. This is what creates the typical highlight point on a sphere. Below is an image that only contains the specular component of the light reaching the camera. It is subtle but quite noticeable when combined with the other lighting.

To render our sphere we just put the components together. Now it looks like a realistic planetary body.

It is not such a convincing rendition of a sphere in normal room lighting however. This is because typical room light not only originates from the parallel rays of the sun but also bounces off of other surfaces and creates new sources of illumination. This bouncing can be quite complicated to compute so a simple way to deal with it is to add an ambient light component to our scene. The ambient light is equal across the circle. The ambient light looks like this.

Okay so putting all three together:

Now we have a convincing sphere that looks 3D. It could perhaps be a red ball tossed into the air. If you placed it on a surface such as a table we would need to do more work calculating the shadows and reflecting light beams off of the tables surface. I will save that work for another time. Here is a link to view the above project in the p5 web editor

You might notice that the project loads slowly in p5. This is because there are many calculations happening and they take time. Keep in mind that this sketch is not even animated. For this reason computer graphics need to be handled quite differently. A language GLSL was created for faster rendering of graphics through the utilization of the GPU instead of the CPU. On top of this have been build more complex full featured tools such as WebGL. The javascript library three.js was created on top of WebGL. Using these tools you can define the camera, the screen, the light source and the objects in the scene and then all of the calculations are done for you. This can be quite convenient once you figure out how to use them. But I recommend spending the time to figure out how the light interacts with your scene and gets to the camera. This can be helpful to understand what is going on especially when you run into an unexpected problem that you can’t explain. But perhaps more importantly understanding how light behaves can bring you enjoyment perhaps similar or complementary to the joy of painting a scene. In fact I think I am already enjoying the original green sphere that I created a little bit more now. After writing this article I completed another Jullier tutorial of a beach scene. I think I captured a little bit more of the sunlights’ affect on the water and the sand. Painting this made me think of Cape Cod in the summer.