Computer vision so good.

Many of you are following the stories about the revamp of our collection online and the use of computer vision to create needed keywords, but also to identify objects that share visual similarities. The use of computer vision to extract subjects in works of art remains a tricky bit and we will be reporting more about that soon. The use of computer vision to show visual similarity, however, is so good that it’s almost too good.

For the visual similarity bit, we’ve been working with Ahmed Elgammal from Rutgers and his startup Artrendex. Ahmed’s work is strictly focused on the visual relationships of images and we were extremely excited about his results — we thought this part was going to be our toughest challenge of the project, but he had so much experience it turned out to be one of the most seamless. That said, we noticed something very quickly. The similarity results were so good they were too good — at least for the Barnes.

Visually similar results for a Renoir painting are, well, visually similar, but not exactly how Barnes was thinking about things. You can imagine if we just used these results, the viewer would simply go down a rabbit hole of the same with no way to get out — the exact opposite of how our ensembles work in practice.

Albert Barnes installed his ensembles based on the formal elements — light, line, color, space — and in highlighting visual relationships, he often made connections between objects in ways that were surprising. He displayed visually related works that also had substantial differences in terms of medium, time period, and historical origin; this part of the ensemble is that magical place that keeps viewers guessing. For example, Barnes owned 9 Cézanne bather paintings, but you never see these together in the same room — he’d mix up a Cézanne bather with something unexpected, like an old master painting or a metal hinge, so you could think about how those objects are similar.

What makes an ensemble an ensemble and how can we apply those ideas to the visually similar results? Much discussion as curator Martha Lucy tours Ahmed and Cassie through the galleries.

In looking at Ahmed’s results, we realized very quickly they were incredibly visually similar, but not necessarily Barnes-ian. And, this collection online project is at the Barnes, so even though we don’t want to replicate the actual ensembles on the web, we do want to honor what makes the Barnes unique and extend that idea into our collection online. As a result, we spent some time mathematically analyzing an ensemble to create a recipe. Very generally, we found an ensemble is made up of differing sizes of works, media, artists, and cultures.

Analyzing the ensembles at the Barnes Foundation. This is the work of Steve Brady, our Director of IT.

The resulting algorithm will be applied across the site whenever you are using the tools to “visually browse.” Also, this same method will be applied when looking at visually similar results of a specific work — here AREA 17 has designed a slider bar with “more similar” at one end and “more surprising” at the other. More “similar” will display Ahmed’s results more natively while more “surprising” will apply the Barnes algorithm on top of those results — the default will be somewhere in the middle.

The similar/surprising slider bar will weigh strictly visually similar results on one side with Barnes logic on the other. Hat tip to Kemp Attwood of AREA 17 for this idea.

So, this is what we are working on in theory and by the time we get to the end it may change a little in practice. The idea is to get viewers on the web deeper into the collection just like we would in the galleries, rather than deliver a lot of the same which would present the user with a dead end.

I’m happy to say, too, that we’ve shifted from our completed backend development by Girlfriends Labs to our frontend development, which is being led by Micah Walter and his team Rachel Nackman and Eric Chaves. Micah has been consulting as a lead engineer at the Barnes for quite a while now and we consider ourselves extremely lucky to have someone with his depth of experience in thinking about and building these types of projects — we’re not the only ones. No doubt, you’ll be hearing from him on the Barnes Publication in the coming weeks.

The Barnes Foundation collection online project is funded by the Knight Foundation and our code is open source. Follow the Barnes Foundation on Medium.