The complexity of visual elements in digital connections.
I’m not going to sugar coat this one — this has been a technically complex project. This complexity is in our use of computer vision to determine visual similarity of collection objects and, also, the UX challenges associated with doing something really new with a collection online. The more we move through this project, I’ve realized how fortunate we are to have Micah Walter and his team — Rachel Nackman, Eric Chaves, and Jonathan Dahan — working on it.
A few weeks ago, we did a stealth “soft launch” figuring we’d be ready to “launch” pretty quickly on the heels of it, but the idea of a “ta da” moment on a project this novel puts a lot of pressure on the product. The adjustments needed to get us to a launch-ready state were proving complex enough that we felt like we could benefit from an extended soft launch.
But, we’ve also decided to take the somewhat unusual step of publicly releasing the soft launch because it means we can discuss some of the challenges before or as we tackle solutions. As I point you to the soft launch, stay patient and stay with me because we’ll be writing a lot over the next few weeks.
As a refresher, we’re using computer vision to power visual similarity and to determine when works of art have similar formal elements. By doing so, we hope to provide a way to “see” these elements online in the same way Dr. Barnes taught people to engage with works of art in his galleries. What you’ll find when you look at the online collection are ways to search objects by various colors, by deep or shallow space, by diffuse or concentrated light, and by various types of lines.
The hope is that when you search by objects in this way, that you can begin to see visual connections among them; the reality is a bit tricky. Take, for example, search by color. When you search for a single color like blue (far left screenshot) or red (center) or a combination of blue + red (right) — the results have a lot of clarity.
This is because there is a lot of density in the overall colors that can be easily seen and the search parameters are narrow enough to produce results you can sort of “get” pretty easily.
Now, take a look at what happens when you start adding more colors to the search results. This time we’re searching blue, green, red, and gray.
While the results are “true” they start getting visually muddy and it’s harder to understand what’s happening. Some colors, too, don’t help us in this regard because there are too few matches or the matches are murky. (Do a search on any of the three violet colors and you’ll see what I mean.)
We think the fix may involve only letting you select a single color at a time or perhaps just two colors in combination. We may also need to think about eliminating selections from the color bar that yield few or overwhelmingly strange results (here’s looking at you…that violet-next-to-the red).
The idea of additive searching is implemented across the site, so for example, you can search for a bunch of colors and then click over to “lines” and add some line types into that mix. That’s a kind of cool feature, but the more elements you add, the less clear the results become and it’s more difficult to see the formal elements at work. This also points to a solve that may involve a single element search because the limitation helps clarity and you can start to make connections.
For example, let’s look at this cookie cutter. You will find this object pop up when you search by both “broken” and “unbroken” line. Computers have interpreted the outline as unbroken, but the patina as broken. In fact, if you keep playing around with this you keep seeing more objects that fall into multiple formal element categories. Some objects will have strong vertical and strong horizontal elements, etc. And it takes you a minute to parse all of this — why is that broken, unbroken, vertical, horizontal, deep, shallow, etc? The more parameters, the more to parse; the more to parse makes it harder to see. This wouldn’t be such a problem if, like in the galleries, we have staff who help show you how to look, but online things are a bit more unforgiving and you need a certain clarity from the get go.
Now that we are on the subject, let’s think a bit more about the online space versus the physical space. An ensemble at the Barnes is a fixed element that doesn’t change. In effect, Dr. Barnes was the curator of which elements in which paintings would formally relate to each other because he arranged things to draw out those connections. In digital space computers are doing the curation, but we are finding that the same constraints of physical space may help online as well. Just like an ensemble is fixed because two paintings can’t be in the same room at once, online you’ve got more flexibility — that painting can dynamically appear anywhere — but you still need some constraints in order to make the connections more powerful.
So, what you will likely seeing us doing in the next few weeks is tweaking this considerably. We may eliminate or significantly reduce searching by multiple elements and we may also think about adjusting the sorting based on very strong characteristics versus some that may be weaker connections. We think the constraints may bring the greater clarity to help you “see.”
The Barnes Foundation collection online project is funded by the Knight Foundation and our code is open source. Follow the Barnes Foundation on Medium.
