Circadian Landscape: Part 2. Development

I recently completed a project that builds a photo slideshow based on the current time of day and the colors that appear in a set of photos. I’ve named this digital artwork Circadian Landscape. It was built using PHP, MySQL, HTML/CSS, and JavaScript.

This post is Part 2 in a series of articles that explain why and how I built it (Part 1, where I discuss goals, and how I planned and prepared for this project). In this post, I’ll describe how I thought through the project at the time I was doing it. I’ll do a tear down of my decisions and performance in Part 3, but this is meant to be an overview of the decisions, emotions and challenges in tackling a new project.

In this section I’ll share:

• The step-by-step process of how I got from color analysis to slideshow
• The obstacles I faced and the methods I used to identify alternate solutions
• A breakdown of the actual results of each stage of this process, including some explanation of the code and what it is doing

Code repo: https://bitbucket.org/alightergreen/circadian-app

Rainbow Cosmos by Alexis Coram

Analyze the Photos

I wanted to stay focused on completing an end-to-end prototype of the system and not get distracted trying to optimize these settings.

As laid out in my Product Requirements Doc, I initially planned to move through the steps in a linear manner: start at analyzing the photos and moving through each step until I could display a dynamic slideshow.

My first step was processing the photos using the PHP Color Analysis library I had found. I started by taking a quick tour through the code to understand its capabilities. The library came with an example file (index.php) that returned and displayed the analysis results to the screen, much like you see on the CoolPHPTools website. I began my experimentation by making a copy of this example file and introducing small changes to the code so it would work with the photos I wanted to use.

Coding

The Directory is the folder where my photos were stored

The most glaring limitation of the code that I needed to address was the fact that you could only select one photo file at a time to analyze. I had an entire folder of photos that I wanted to analyze in one pass.

To compensate for this limitation, I added code to populate an array of file names based on the contents of a selected folder. The code then looped through that array and plugged in each name into the original “one-at-a-time” processing logic.

At this stage, I avoided playing around with the various settings to see what kind of different results I would get (See the above photo for a sense of the additional settings. I’ll talk about these later). I wanted to stay focused on completing an end-to-end prototype of the photo selection system and was determined to not be distracted trying to optimize settings, when I couldn’t yet judge their impact on the final output. There were bigger risks to focus on. Including the fact that I didn’t know how I was going to group photos, or use the color and percentage data that I was getting back from this processor.

The second issues was that the initial file just displayed the results to the screen. I could have just taken extended the photo-by-photo logic to take those results and do more processing on them. However, I wanted there to be more seams between different activities so I also added a snippet to toss the results in a database. That way if I changed the way an activity was working, as long as it was pushing the final data into the same data fields, I knew my later activities would keep working. This also allowed me to easily stop in between activities, take a look at the results and ask, “Is this doing what I want it to do? Is this returning the kind of data I need?” Storing the results in a database was the quickest and easiest solution to implement that would give me a permanent record of the results without needing to rerun the file every time.

Output

The result of this work is the “Select directory for extract” file (seldir4extract.php). This file brings up a form that allows you to set the variables the photo analysis library uses:

• Number of Colors for how many color results you want back for each photo
• Delta for how finely it will separate colors
• Reduce Brightness and Reduce Gradients to normalize photos for these two things
• Extensions for if you want to filter files selected by a certain extension
• I also added a radio button to let you select the directory where the photos you want to process are. This isn’t part of the library, but was convenient for my purposes while building.

Example of results displayed for a photo

The results are color hex codes and percentage of the photo that color makes up. The results are returned sorted by percentage (highest to lowest) so you will always get the most common colors regardless of what number of results you choose.

When you submit the form, it processes the photos and displays the results along with some log data on the screen. After each photo is processed the results are added to the main database table before the next photo is processed.

Depending on the size of your photos and how many you are processing at one time, there’s a good chance this will time out on you. (Or it may just be my computer. I used my ancient personal Mac when building this and it tuckers out easily.) If you do experience this, no worries. Before it processes a photo, I added a check to see if the file’s name is already in the database. If it times out, you just have to refresh the page and hit run again (make sure to reset your settings if you changed them from the defaults). It will skip all the photos that have already been processed and have records in the database, and start on the next new image. There’s pros and cons to this from an iteration perspective, but it was the fastest way to get around the timeout issue with what I knew how to do well.

Color Assignment, Round 1

I had assumed if I could analyze photos for colors that the results would be immediately useful.

Since I had a color analysis of all my photos, now I needed to be able to select photos by those colors. Which turns out to be a really difficult process!

The biggest issue is that there aren’t nice, “user-friendly” names to be able to select by for most of the colors that exist in the RGB spectrum. Since I didn’t want to have to become an expert on color theory or RGB encoding in order to finish, I was willing to introduce some manual work into the process to finish quickly. If I could get the colors for each photo down to 1 or 2 results, I could name the colors “blue” or “pink” by hand within a reasonable amount of time. Then it would be easy to code the slideshow to “Show me pink photos at dawn.”

The first step in doing this was finding a way to automatically get 1 or 2 colors that really represented the photo from the analysis results. Here’s where the roadblocks started. I had assumed if I could analyze photos for colors that the results would be immediately useful. I did not find that to be the case.

Coding

I tried out several ideas about how I might get this to work.

Here’s a good example of what I’m talking about

1. Raw Results. The first was just taking the top color from the results and using that as the photo’s “color.” But this was not particularly accurate. For example, say a photo has a lot of black in it, maybe 25%, and the photo also has lots of different colors of blue that comprise 75% of the photo in total. If no individual blue is greater than 25%, then the photo would be listed as a “black” photo even though when you look at it you would say, “This photo is ‘blue’.”

Anyone that’s ever done anything with computer vision is probably laughing right now…

In the 60s, Marvin Minsky assigned a couple of undergrads to spend the summer programming a computer to use a camera to identify objects in a scene. He figured they’d have the problem solved by the end of the summer. Half a century later, we’re still working on it.

2. Color Groups. To resolve this issue, I then decided to squish all the like colors I was getting together into groups of colors. Then all the blues from a photo would get added together as blue. In the example above, the results would be that the photo is 75% blue and 25% black, and the photo would get listed as “blue”. To do this the colors would need to be labeled as “blue” or “black” in order to be grouped.

This brought me back to my original trouble — there aren’t standardized color groupings for hex colors as far as I could find. The closest is this resource which groups standard web colors into color groups. Getting those color groups applied to my color results was the obvious next step. It looked like it would be effective based on the small set of photos I was using to test with…but it definitely headed into territory I was hoping to avoid in this project, as mentioned above. I didn’t see how I could get around it, though, so I decided to move forward figuring out how to match my result colors up with the Wikipedia color table.

Again I started with several assumptions that progressively got tested, discarded, improved and tested again.

1. Please just match! I had hoped that the results I got from the analysis would be pretty close to or exactly the colors that are listed in the sheet from Wikipedia so I could just do a simple matching process. This was a hypothesis driven by desperation, and me really not wanting to have to figure out how to do math on hexcodes. But…There are millions of hexcode results possible in the RGB spectrum, and only a little over 100 results in the table I was matching too. Only 6 out of over 200 color results matched in my first run and they were all perfect black (#000000) or perfect white (#FFFFFF). Not very helpful.
2. FINE! I’ll do the math…It seemed like there was no way around it. I was going to have to write the logic to figure out which of the color group table colors a result was closest too. This requires doing hexcode math — Figuring the total distance of two colors from each other by looking at the distances between their three distinct Red, Green and Blue elements — which is something I had really wanted to avoid and why I had gone to look for a library that would do the color analysis for me in the first place.

Output

At this point, I was pretty frustrated. Everything above looks simple written down, but a ton of coding went into trying different ways to make each attempt work.

Since I had been hoping to avoid having to deal with the hexcodes at all, I hadn’t done any research on the logic behind them or how to work with them in PHP. Even after all that, I still had to go all the way back to the beginning to be able to move forward. I wasn’t familiar with programming with hexcodes outside of their basic front-end applications (Make links blue, Make Header2 orange, etc.).

In retrospect, I recognize that having to rewind to the research phase is what made me so frustrated. I felt like I had already completed the research and planning phase, and now I was supposed to be doing things. Having to hit the books again to solve this problem felt like moving backward. It felt like failing. (Stay tuned for Part 3 for more discussion on this.)

I completed my research and got all my resources together that I thought I would need to solve this (PHP documentation, stackoverflow answers, some general definition stuff on hexcodes and hexidecimals numbers in general). I could tell I had a reference to solve every piece of the problem, but they weren’t clicking together in my head into a codable solution. I felt like my brain just wouldn’t engage and I was spinning my wheels.

Also, I had estimated originally that I could get a prototype for this project together in about a week. My week was up, and a big portion of it had been committed to this color grouping problem without getting me any closer to a finished system. I decided it was time to take a step back from this piece.

I wrote down what I had tried, what I had working and what I knew I was still missing, and decided to move on to the next steps — getting the photos to be chosen by the time of day and displayed in a slideshow. I also gave myself a hard deadline of one more week before I had to shelve the project. I had a job hunt I needed to get back to full-time!

Finding a Photo’s Top Colors

Of course, this wasn’t going to scale well. It was a work-around solution. It had done its job and now I was able to finish building the process.

I got around the fact that I couldn’t immediately finish the hexcode math problem by going back to my original solution — do it by hand. However, instead of trying to just look at the top 1 or 2 colors I kept all of them. I pulled the results for my 12 test photos into excel and also threw them up in a quick HTML file so I could see the hex color. Then I set the color group for all of their 200+ color results by hand and imported that column back into the database.

Now, at least, I had something to figure off of. This was a great work-around in this initial phase! Considering I had over 100 photos I had already bought for the slideshow though this was NOT going to scale well…

I cut those thoughts off immediately. Of course, this wasn’t going to scale well. It was a work-around solution. It had done its job and now I was able to finish building the process.

Coding

Since I now had some color groups to work with, I was able to get back to smooshing each image’s results down from 19 kinds of blue and 1 black, to the Total% of blue and the Total% of black.

I started with a copy of my “Select Directory” file because I also needed the file list array to complete this step, and the Select Directory file already had working code to get it. Once I had the file list, I was able to pull each file’s results from the database, add together any that had the same color group, and save the “condensed” results back to the database in a separate table.

The matching and mathematical logic here is simple, but required nested for loops and if statements. These gave me a bit of trouble up front, especially thinking through what needed to happen in each of the “then” or “error” branches. Unlike the hexcode situation, it wasn’t anything that bull-headed debugging couldn’t uncover.

I also added logic to retrieve the condensed results, sort them from greatest to least, and write over them. It would have been easy to do this on the fly at the beginning of the next step. I did it here and immortalized it in the database to keep the next step focused. Again this was about creating seams, and trying to keep the logic for different activities together in their own file. This way all the logic around ranking colors was in this file, and the next file just had to match them to the time of day.

Output

This was completed in “Top Colors” file (topcolors.php).

Just like in the original Select Directory file you can select the folder you want to work with. In this case, we’re really just using the folder to build the list of files we want to check for in the database and condense our colors around. We’re not actually doing anything with the files themselves.

Matching Photos by Color to Time of Day

The next step was associating color groups like black, blue, red, and pink with times of day. I had already learned it is so much easier to improve something that you know already works than to try to figure out something really complicated and write it out from scratch. I was getting down to the wire on my time, so I gave up on building any sort of scientific data set around this.

I made a simple table that listed the hours of the day using 0 to 23 in one column, and a color group names from my Wikipedia table in the other. What hours got what colors was just based on my gut instinct, and what I thought would look pretty.

Coding

Everything else needed to finish off the process is in the “Slideshow” file (slideshow.php). This file does more things than any of the other files:

• Figures the current time of day
• Finds the time of day in the database table
• Gets the color group(s) assigned to that time of day
• Selects the photos whose first or second top color matches that color group(s)
• Randomizes this list of photos
• Selects some photos to display
• Uses this list to initialize the slideshow

Even though it does a lot, each step is very straightforward, so this was a breeze to program. The biggest change was in the slideshow.

Similar to the example file in the color analysis library, the slideshow tutorial I was using was made to be “hard-coded” — it meant for you to create a slideshow file and type in the names of the files you wanted the slideshow to use. Since I wanted it to be able to select photos dynamically, I had to break down the structure used for each slide and replace hard-coded strings with variables I could pass in. Then I put this in a forloop so it could write up a slide for each file in the list the PHP code returned.

Output

I had accomplished the initial goal! I had gotten from a folder full of files to a slideshow of the photos based on their main colors and the time of day! YES!

Example of the *wide* variety of photo sizes I needed to deal with. LOL! So Punny…Grizzly Panorama by Alexis Coram

However, I discovered two big issues the slideshow code I had used to create my slideshow:

The other extreme. Jasper-Maligne Canyon by Alexis Coram

• Although it didn’t jump out to me at the time, a quick review revealed all the photos used in the tutorial had been resized to be the exact same size and orientation. I have both horizontal photos and vertical photos of varying dimensions, and the slideshow isn’t set-up to handle this attractively.
• The second difficulty was making the slideshow recheck the time of day occasionally, and refresh the page the page if needed. Otherwise, if you start it at night you are just going to see night photos forever until you manually refresh.

I determined that both of these needed to be solved with JavaScript, which is the language in all of this I am least familiar with. Now that I’d had a few days to stew on the hexcode problem, I’d figured out how I needed to solve it. Instead of opening up a whole new can of worms, I decided to go back and finish automating this process while my head was still in the PHP code.

Automated Color Grouping, For Reals

At this point, I was pretty stoked. I had a functional pipeline outlined to get from a folder full of photos to a slideshow. It still required manual help, but all of the pieces that I wanted it to have were there in at least their most basic form. At least I could run a slideshow using my 12 test photos.

There was no way I was going to set color groups by hand for the other 100 photos I had purchased. 100 photos x 20 results each = 2000 colors to review! Before my time limit was up, I wanted to see if I could finish automating the process so I could import the rest of my photos.

Coding

I needed to find the grouped color whose absolute difference from the ungrouped color was lowest. I could do this by splitting the color codes into their three component pieces: red, green and blue.

Then I subtracted each component of an ungrouped color from the matching component of a grouped color. This gave me 3 numbers showing how far my ungrouped color’s red was from the grouped color’s red, ungrouped blue from grouped blue, etc. To get 1 number to compare off of, all of these numbers were turned into their absolute values and added together. Then that result was set in a variable for further comparison along with the compared color’s group.

Example:
05A6FF — 00AAFF
Split into components:
Red | Green | Blue
05 — | A6 — | FF -
00 | AA | FF
Change to Base 10:
Red | Green | Blue
5 — | 166 — | 255 -
0 | 170 | 255
Subtract Each Column (Color)
Red = | Green | Blue
5 | -4 | 0
Take the Absolute Values
Red = | Green | Blue
5 | 4 | 0
Add them all together to find the total distance
05A6FF is 9 units away from 00AAFF
05A6FF would be labeled with 00AAFF’s color group if 
9 is less than the result of the previous color 05A6FF 
was compared with.

Then I’d do this again for the same ungrouped color and the next grouped color. If the result was lower, the saved variable tracking difference and color group was updated. If not, the comparison was ignored. After running through all the grouped colors, the ungrouped color was set to match the group of the color with the lowest difference.

It took a ton of debugging and iteration to get this to work consistently. Primarily again because of the setup of forloops and if statements necessary to build the logic.

Output

Figuring out a way to solve this and actually executing it is still the thing that I am most proud of in this app!

The result of all this work is the “Color Group Match” file (colorgroupmatch.php). This file is used between the Select Directory and Top Color files. It replaces the manual work I did to label all the color results the the photo analysis returns with a user-friendly color group name like pink or blue.

Isabella White Flower by Alexis Coram

Refactoring

Over the course of the app, I learned a lot about how to do some of the more basic and repetitive tasks. Consequently, my later files had much cleaner code and were easier to debug or even just avoid errors in the first place.

I needed to go through and retest the whole process from beginning to end because I had broken some of the connections in getting the Color Group Match file to work. While I patched things up, I decided to make a few things more consistent across all the files.

Coding

I cleaned up how I was making database calls, how my forloops and errors were organized, and how I was incrementing my forloops.

For example, I had originally learned forloops in Octave and just jumped into this project using that syntax.

Then my husband, who actually codes in PHP, looked at what I wrote and was like, “WHAT ARE YOU DOING?! #phpterror”.

But hey, what I did worked! The standard PHP method is MUCH cleaner, however, which makes it nicer for you to read and nicer for me 6-months from-now-and-I don’t-really-remember-any-of-this to figure out what I did.

Output

A lot of the clean-up involved moving repetitive tasks into a separate file and making them callable functions. This meant that when you opened any of the “executable” .php files for this project, you would just see the logic for what that file was trying to accomplish. Repetitive stuff around opening and closing database connections, and accounting for errors was moved into photoapp.inc.php.

Lessons Learned

Before my week was up, I even had time to add all 100 photos to the folder, process them all through the entire algorithm, and see them in slideshows! This meant that the entire process from folder of photos to slideshow could be completed just by clicking run on a few files whenever I wanted to add more photos. I was also perfectly set-up to condense the process all into one file, or jump back in to fix up the slideshow issues whenever I wanted.

V.0.1 is officially complete!

The key lessons I learned in the implementation phase of the project were:

• A lot about how to code PHP well
• Keeping code clean
• Giving yourself distance from a problem can help you solve it
• It really is easier to refactor than to build
• There is always more “fog of war”

In part 3 I’ll break down what I hoped to accomplish, the goals I laid out in my PRD, and how the actual experience stacked up against it.