Shazam the High Line

The High Line has more than 4 million visitors each year. This spring the High Line has installed their newest art exhibition, Mutations. We thought this would be the perfect project to try a new concept, Shazam for Objects. HBO’s Silicon Valley’ always timely satire covered this with their recent Shazam for Food. So, let’s build a Shazam for Art with the High Line’s Mutations as our pilot project.

The concept is simple, take a picture of the art and see more information about the artist and the artwork.

Building the Tool

First, we need to train a model to recognize each piece of art.

An example of the images we used to train the model

We’ve built custom models with Clarifai’s Custom Model training for client projects and got some inspiring results, so we’ll use that to create the classifier. Next step is to gather training data to train the High Line model. The classifier needs tagged images of each artwork.

Clarifai’s Custom Model Visual Interface

We gathered the training data by walking the High Line and taking extensive pictures of each artwork. We trained the model and after some brief debugging we were able to get 90%+ confident classification for most of the artwork.

Prediction confidence is shown in the top right of this image

Now that our classifier is correctly identifying the art we need to build a user interface for people to use the classifier.

Making it a Product

The best platform for this would be a stand alone Augmented Reality application like Shazam, but I don’t want to download an app just for the High Line. Let’s use an existing platform where people are going to share their pictures. We are going to prove the concept using Twitter. If a person Tweets with the hashtag #HighlineNYC we will run the custom model to see if they are looking at Mutations and if so which piece it is. Instagram would be a great platform, but their API is too restrictive for this.

What should we return when we correctly identify the artwork? For now we’ll respond with the name of the artwork, the artist, and where you can find more information. The possibilities here are endless and exciting. Its a case where you have to really focus or you’ll get lost in the huge pool of possibilities.

The returned info format (for us brief text and a URL) will change based on what platform the product is on. For example it is our experience that links work well in Twitter because they can connect users to substantially more information than in 140 characters, or with Facebook messenger its better to return structured cards with buttons for users. A dedicated app would have the benefit of allowing you to customize all of this.

Trying it out

Let’s try out our new model by tweeting about checking out this new art at the #Highline or #HighlineNYC:

Looks like it worked! It recognized Dan Bader’s installation Chess. Now what happens if it doesn’t recognize the artwork?We inform the user that they can snap a pic of the art to learn more about:

Awesome! So what’s next? What else can we do with this model?

Expanding the Product

The product works well. The art is visually distinctive enough that the classifier could identify it with little training resources. This could be recreated easily for other installations, galleries and museums.

There is a ton of tagged public artwork available. We aren’t going to list them all, but here is an example of what’s out there:

So, Shazam for Art is easy enough to build and can reach users in a fairly frictionless manner. What else can we build?

What’s Next?

Shazam for art is a fun start, but there’s so much more you can do once you can recognize an object in the real world. What if we want gamify and personalize the High Line experience?

An artist can extend their artwork beyond the physical space and into a personalized interactive experience. What if you could play an interactive game of asynchronous human chess. You could take a picture of the chessboard and it would tell you what move to play next. You take a picture of you making that move and at the end you could stitch together all the images into a summer long game of chess.

Up next…we build interactive games using our custom computer vision and the High Lines new Mutation installation.

Part 2 coming later this week.