SMK’s collection search levels up
Or Why They Put Art in Artificial Intelligence…
You can put your collection online all you want, but if it can’t be searched it won’t be found. Or rather: To realise its potential, an online collection should be generously sprinkled with connections. Connections between artworks and connections between the art and the user.
Into Danish golden age master Eckersberg, today? Great, we’ll get you everything we have. Got a cat project? Allow us to get you every collection object with with a feline presence.
One way of achieving this would be to employ a sizeable army of art historians for many years and have them assign “correct” categories to each artwork. There’s much to be said for this approach, but it is costly and would require an intensive discussion about what to include and what to leave out.
What’s in this painting by C.W. Eckersberg? A lady? A church? A staircase? Sculptures? Lions? What about “street”? “City life”? “Blue skies?”. And what if, some years after this great manual classification, you decide that something else was more important? Perhaps color scheme or composition?
At the time of writing we’re designing a new online interface to search SMK’s collection. It will (at least that’s the plan and of course I tremble to write this) be snappy, responsive and inviting. But it will also be something else and less visible: It will be super-connected and thus eminently searchable. We’ll achieve this by getting help from the robots.
In other words: We’ll submit all the artwork photos to analysis by machine learning tools and thereby assign whole new spectra of classification. So even though no human ever told any system that a painting contained a “horse” our robot friends will supply this piece of inforation and thus create a plethora of new search options.
More specifically, we (and our friends at Strømlin) plan to do the following.
Is it a bird? Automatic object recognition
Every artwork is submitted to Microsoft Vision Services which returns its best estimate of the objects seen. Thus we’ll get a large number of tags such as “horse”, “cow”, “cloud” or “portrait”. The service will return each of these tags with a “level of confidence” and we’ll set our threshold somewhere relatively safe (i.e. we’ll get false positives but not too many).
Microsoft’s service was trained on photos and it struggles somewhat with paintings (of course, even we humans start to struggle when abstraction sets in). We’re considering how to improve the model to turbo-charge recognition of drawn or painted motifs but this will probably need its own future project.
Looking for likeness
Everything goes through a neural network which spits out a “fingerprint” for each image, based on composition, material, color scheme and motif. Each fingerprint is based on several thousands of properties of each image.
In this way, we can compare fingerprints to find similarities between artworks — sometimes surprising ones!
For the technically inclined, we’re talking about a Convolutional Neural Network using Keras and Tensorflow.
A rose is a flower is flora
Our collections database contains a lot of hints about what’s in an artwork. This can be hidden in prose or it can reside indirectly in the title. This is no problem if you’re looking directly at an artwork, but let’s say you dropped by to search for “sailboat” and you didn’t find the perfect artwork because the title mentions a schooner. Now, a schooner is a sailboat but your average search engine has no clue.
Similarly, if you search for an object hidden inside a compound noun (like the “vogn” (carriage) in “hestevogn” (horse carriage) a standard search engine might be in trouble.
To get around this, we’ve trained a Danish language model using machine learning. Among other things, this model identifies nouns inside titles and finds their basic form (or “lemma”).
We combine this with automatically generated hypernyms or “umbrella terms”. So, if you search for “kitten”, we could give you “cat” and “animal”. But more importantly, if you search for “animal”, we can get you those cats and kittens because the search engine understands the relationship.
Navigation by colors?
We’re keen to offer alternative, explorative ways into the collection. To this end we’ve performed a range of color analyses on our images to determine color dominance and color scheme. How exactly to implement color search is still to be determined but the analysis lets us offer color-based search, artworks with related color schemes and automatically generated background color schemes for specific interface elements.
Oh, and we’re doing this through pixel analysis and a KMEans algorithm.
All together now
In the grander scheme, we are trying very hard to perform all the hard work behind the scenes — to let the user interact as simply and intuitively as humanly (and robotically) possible. If you know all about Eckersberg, Hammerhøi, and Jerichau-Baumann then that’s fantastic. And if all you know is “pink” or “unicorn” then we’d like that to be an entirely valid starting point as well.
Oh, I know what you’re thinking, and sorry we actually have zero pink unicorns at the moment, but the collection is ever-growing and we’ll pass your request on to our director.
- Full developer cred goes to Nikolaj Erichsen and Anton Stonor. I really am just the messenger :-)
