AI and Museum Collections.
Faced with a Museum collection so large it may never be completely digitally catalogued, can AI offer a solution?
At Auckland Museum, we have a truly amazing collection; estimated to be around 7 million objects. The collections spans art to archives, cultural collections to natural science specimens, an impressive war collection, and a research library. Our largest ever digitisation and cataloguing project is well underway. We are uploading 2000 new records and images every month. But even so we estimate that it will take decades to digitally catalogue all the collection. This backlog of work may never be completed, and I am always conscious that it is there… a collection waiting, unseen and full of potential.
So can technology, new working styles and the gig economy help to tackle this backlog head-on?
Using Artificial Intelligence or Machine Learning to auto-tag collection images isn't a new idea for museums. The Powerhouse Museum in Sydney started doing this a decade ago when they used OpenCalais to add social tags to images from the collection. Most recently Google Arts has led the way with the the Google Arts experiments — a series of projects that allow for serendipitous exploration of collections indexed by the Arts and Culture project by harnessing the power provided by AI.
With these examples , we see the potential of AI technology to enrich the user experience, enabling users to explore the collections in new ways. What I am interested in however, is how we can start enhancing collection data at its root source in our collection management systems? Surely we need to get the foundation strong before we build the experience. So, how could we use these technologies, right from the start of the cataloguing process, to create basic records and open up more of our collections?
I wanted to see at what stage we could bring in A.I. technology to help with our backlog — I wondered, could we use the computer vision systems to catalogue previously unseen images and get a basic record online that could be enhanced by a cataloguer at a later date.
Is it better to have a bad AI generated record online than no record at all?
IBM vs Microsoft vs Google vs Clarifai
When looking at computer vision systems there are four big contenders, each offering a similar service. I was instantly drawn to Clarfiai and Microsoft as the free tier would allow for more images to be processed each month.
We have used all four in some format, using the strength of each system to enhance our metadata, for this project the Microsoft service won me over. The ability to add captions alongside the tags just seemed to enrich the experience and provide a better level of basic context. Alongside the tags and caption, the service also provides a confidence score that indicates how accurate the AI thinks the caption and tags are. This function would give us a programmatic way to select which images could be published with the AI generated content, and which should go back for review.
I ran 2,000 collection images through the Microsoft system to see how this would perform in practice. I discovered that when the confidence score was high, the caption appeared to be pretty accurate, but when it was low, well, it gets it really wrong.
As such the confidence score is a good measure and an important feature. It would allow us to automatically reject any record that is AI cataloged with a lower 60% confidence score, ensuring that we don’t publish any records that could be misleading or embarrassing.
To get the system to work and to bulk load thousands of images into the system I used a python script that was written by someone on Fivver. Fivver is a freelancer platform that is part of the Gig economy. Fivver allowed us to quickly and cheaply get the script set up and tested by someone remote from the institution, with the technical ability to create the working script in a very short time frame. This trial of the GIG Economy was a test of how we might remove the technical barrier of us struggling to overcome this hurdle with our limited in-house knowledge of the Microsoft API (I talk more about this here).
These 2,000 images were processed with an average confidence score of around 60%. We were able to take all those with a high score (i.e. >60%) and import them directly into our source system. I added a classification identifying these as auto-created records for administrative transparency.
The next step is to work out how we might show these records online. For instance, ethically, do we need to flag these records as ‘auto generated’ so the public are aware they are looking at a record which was not created or verified by a museum staff member?
We are also looking at how we process those records with a ‘medium’ confidence score, around 50–60%. Some of these records may well still be accurate but we need a quick way to review them before they go online and so prevent them becoming their own backlog. I wonder, could we use crowdsourcing to help process these ‘medium’ confidence records?
As we start collecting more born digital collections, the rate of our acquisitions are growing exponentially. Last year we acquired a single collection that contained over one million images. If we ever want to get ahead of the backlog, to make these images at least searchable and findable, we need to look at new solutions to open the collections. From my first initial tests, computer vision could provide a simple solution to providing basic records, allowing users (internal and external) a new way into the collection. But we need to continue to invest time and imagination to develop the practical employment of AI in this way and work through the inevitable limitations to fully embrace a new era of collection cataloguing.
To hear more, check out my presentation at the 2017 National Digital Forum.
