Going Digital to Analog: Turning Webcams into Thing Counters
-
At Timbr.io we’ve been plugging away at building a platform to share reusable algorithms and data sources. There are so many fantastic algorithms emerging in academic papers and as open source libraries we think it is a good time to help organize the work we are collectively creating. Instead of having to decipher and implement a new algorithm from scratch, we want to enable the community to build on each other’s work. That way we can avoid reinventing the wheel and can focus on creating new ideas.
The concept can get a bit esoteric, so we figured an example would help tell the story. We recently did an internal hack day to build a demo for a start up pitch event. The team kicked around several cool ideas, and the winner was a pipeline that scrapes images from a webcam and an algorithm that counts transient things in those images. We really liked the idea of turning a digital image stream into an analog counter. Below is a video showing that work being repurposed by our product manager, Bill:
The fun bit with the video is Bill doesn’t code but was able to repurpose the team’s work to create his analytic.
How it Works
One of our integration points for the platform is Github. Specifically, the ability to share your algorithms as gists and post your pipeline data snapshots to Github. Since we are in the middle of our beta I’ll share this example workflow through those publicly exposed points. We’ll start things off with the code for the webcam scraper
Using this source we can build a data snapshot. For the purposes of portability to a Jupyter notebook I used Timbr.io’s native HDF5 format to share the data here. With a nicely structured data snapshot we can dive into a Jupyter notebook for some analysis. With a new data source I like to check to make sure I’ve configured my polling interval correctly first. Pramukta wrote a nice unique timeline visualization (inspired by) that is super handy to check this.
The output ends up looking like so:
Each unique color represents a unique image — till you run out of colors and it repeats. Good news is we have all unique images for our snapshot, so we are all set to do a bit analysis with it.
Detecting Objects in Images
During a previous hack day Winnie had tested out the facial recognition library in OpenCV to see if it would detect cat faces in an Imugr stream. Pramukta took that example and started evolving it into a people counter. He tuned it specifically for the Steamboat resort webcam we’d collected data for and ended up with this:
He added an image slider in the snippet so you can scroll through the image collection and get people counts for each one — example below:
The trick with getting most algorithms to work well is tuning them to the data. Sometimes we can do this through parameterization or often with training data. This is what makes generalizing algorithms to broad use cases so difficult, and why with Timbr.io we tackle algorithm sharing with a search, iterate and publish approach.
Repurposing an Algorithm
A good example of this was when one of our other data scientists, Jamie, wanted to create a car counter for a border crossing webcam he found. Instead of reinventing the wheel he did a search for object detection algorithms and found Pramukta’s work. Using the shared code he tuned the detection parameters and added a couple of new methods for his car snapshot.
In the original ski area counter Pramukta took an image subtraction approach then detected contours. Since the background in ski area images was pretty consistent this worked well. For the border crossing webcam the movement of the sun and shadows over time made this trickier. So, Jamie added a sliding window average to the method as well as opening the morphologies in the image to reduce noise then closing them, so cars did not get joined into one object. Resulting in the following:
Once Jamie had it well tuned he published it to Timbr.io and now the next person has a car counter and person counter to choose from. We hope this approach will create a virtuous cycle of contributions. Each new algorithm or data source makes the community that much smarter, while creating new opportunities for novel analysis. Our job will be recommending the best algorithms and giving you the best environment to iterate your custom solution.
While Github is an awesome place to post up snippets and data sets we think there is a biggish missing piece of fusing data+algorithms together seamlessly. An environment that lets you easily collect and structure data then mesh algorithms with it in one spot. If you’d like to check it out — hit up Timbr.io to sign up for early access. We’ll be posting more examples as we keep building the platform.
