Last week I finished my first website for Project 52, my 1-project-a-week project! I should have posted this a week ago with the project, except I’ve been a bit busy (and also putting it off) ANYWAY…

What’d I make?

The idea of this 1 page site was to show what TV shows I am currently watching but not finished with. I watch a lot of shows, though I rarely see the episodes on the same day as they air. I’m usually behind, hence the name: “Don’t Spoil It For Me”.

This site doesn’t have many practical uses. Nonetheless, it was fun to make.

How’d I make it?

Overview

• Data: TV Calendar website
• Scraping and moving data: PhantomJS & CasperJS, Node.Js
• Database: Firebase
• Design frameworks: Bootstrap with AngularJS
• Visualization: Angular directive for nvd3

Finding the data

For a while now, I’ve been using this TV Calendar website made by PoGDesign to keep track of my shows. Once you register, you add your TV shows, and it’ll display a calendar with the days and times of new episodes. As you watch episodes, you can check them off your list. I’m always forgetting where I left off, so this saves me the hassle of going to a show’s Wiki entry and reading through the descriptions of all the episodes until I find the one I haven’t seen.

Getting the data

There was one page of my TV Calendar profile that proved to be very helpful. Most of the data I needed (TV show title, image, and # of episodes) could be seen on this page:

However, TV Calendar has no API of its own, so I had to find my own way of getting the data off their site. I couldn’t do a simple scrape because I needed to log in to the website in order to visit to my page — none of this info is public.

I turned to PhantomJS, a headless browser, which automates any actions a user performs on a webpage by using simple functions written in Javascript. I had heard of PhantomJS before, but had never worked with it. With this, I could write a script to automatically press the log in button for me and navigate to the page that displayed my unseen episodes. After playing around/struggling for a while, I discovered an add-on to PhantomJS called CasperJS (Don’t you just love these names?) to make the process of navigation on sites easier. This let me grab the divs I needed and I stored the contents in a json file.

I also had a Node.js script that took the json file created by my previous script and put it into my Firebase database. I didn’t particularly need the real-time capability Firebase provides, but I could structure my data the way I wanted to (JSON objects, not tables) and the JavaScript library made it really easy to insert and update objects. They also have a library for AngularJS that made fetching the data on the front-end as easy as (literally) 3 lines of code, plus no issues with the data format. And the great part is that it’s all stored online.

Since I wanted my information to be up to date, I ran the Casper and Node scripts on my server using cron so that I could get new data every 12 hours.

Displaying the Data

Instead of displaying static HTML pages, AngularJS dynamically displays your data through a model, view, and controller (MVC) pattern. The ngRepeat directive is the best when you have a list of objects. No more using for loops in JavaScript/JQuery to append to your HTML.

And finally, for the graph: a simple Angular directive for nvd3.

What went wrong?

At the beginning of the project, I foolishly thought that I should save the data from each day in separate json files and then use the most recent file on the front-end side. When I decided to add a d3 chart to the site around day 5, I realized that it would be stupid to iterate through each file to get data from each day; It would be so much easier to store the info in a database, consequently I switched to Firebase.

I also spent a lot of time trying to insert directly into Firebase from within my CasperJS script. I ended up creating a separate Nodejs file because my “require(‘firebase’)” line wouldn’t work even after reading up on the difference between Casper and Node ‘requires’.

After seeing data flow in to my site for about a week, I noticed that there’s a small bug once the number of unseen episodes for a show goes from 1 to 0. You can see this in the graph picture above. (The Mindy Project started at 1 on 1/6/15 and stopped before 1/11/15, but never went down to 0.) Only shows with 1+ unseen episodes are recorded in the database and as a result, when I catch up with a show, it isn’t shown on the graph. d3/nvd3 doesn’t have any capabilities of filling null values with zero, so I’d have to write my own function to insert the zeros (which I don’t feel like doing at the moment).