v1s10n (Preliminary 3 questions)
--
1 — Overview
Within recent years, museums and online art collectives around the world have started to digitize their art collections. A proportion of these high-resolution images have been made available in the public domain, while others remain private. The influx of images to the online arena has provided an opportunity for scientists and researchers alike to compare pieces of artwork through careful image analysis studies.
In the past, neural networks have been used for pattern recognition and to produce pattern transfer (i.e. Google DeepDream). In addition, several studies have been conducted regarding painting image classification (Google and Stanford). We would like to build upon these ideas and create an interface of art and computer vision where we can classify paintings based on various attributes. (i.e. painter, style, date, etc). We would also like to explore the data through an interactive visualization. This could be helpful for museums, art historians, and art enthusiasts seeking to better understand artistic correlations between paintings.
For this project we will conduct research under the guidance and mentorship of Dr. Serre from Cognitive, Linguistic, and Psychological Sciences (CLPS) at Brown University. He has a lab on campus (Serre Lab) where he conducts research in the area of visual perception and cognitive science. Dr. Serre has given us access to a high performance computing environment consisting of Common Internet File System (CIFS) storage and a set of good workstations at his lab.
We are excited about this project and look forward to seeing what observations we can gather from the study.
2 — ETL (Extract, Transform, and Load) + Database
Overview:
At this stage in the project we have created a target list of online painting image archives. Many of these websites only allow single-image downloads of high-resolution artworks. In addition, we have reached out to several of the larger museums for access to their (Application Programming Interface) API, so that we may download a batch of images at once. In some cases, this functionality is not built into the API or may require granting additional access privileges. We will download with an API if possible, but we plan on building Python-based web scraping tools (e.g. Scrapy, Beautiful Soup) to quickly extract images from the online repositories and museum digital archives. For each online repository, our goal is to download the high-resolution image, and output a CSV file containing the corresponding image attribute data (e.g. style, artist).
We are hoping to work with at least a few thousand paintings but the size of the data is TBD. Ideally, we would like as many images as possible (on the order of hundreds of thousands to millions). We will strive to collect paintings from different time-periods, artists, and painting styles.
ETL:
Scraping the Web Gallery of Art was a relatively simple experience, as the website is over 20 years old and is built using rudimentary HTML with little in the way of security protocols or other anti-scraping measures. A simple index was available that listed all of the artists with some metadata, and from there each artist had a link to a separate page that contained a table of their paintings with additional metadata for each. The procedure was to simply extract the artist links, then extract the image links for each artist. The more complicated part was scraping the metadata into a csv file. There were a few typos and misplaced commas that I had to code exceptions for. At present, all of the information is present, though some of it is grouped into a “misc” column for reasons of practicality and uncertainty of how frequently such information would be available down the line. Another issue was that some artists appeared on the artist list multiple times under different aliases. In this case, the urls were repeated but there was always text of the form “(see …)” to indicate the duplicate. With that, removing that redundancy was no problem at all.
Please see below for a snippet of the code used to extract images from Web Gallery of Art, as well as collect attribute data into a CSV file.
Database:
Below is the schema for our SQL database, which will contain image attribute data. The primary keys were created from specific columns in the CSV file. Since we are extracting images from a wide range of online image databases, we are aware that certain image attributes (e.g. artist_school, release_year) won’t be collected. We will aim to collect as many attributes as we can for each image off a particular online database. A star schema was selected because efficiency is important, and we don’t need a highly normalized table form. Each record has a unique image_filename which serves as the path to where the image is saved in storage.
3 — Exploratory Statistics
Within PyCharm, the CSV file containing image data was imported into a Pandas DataFrame structure. Then, some basic statistics were calculated from the data (e.g. # of paintings by style, artist, and western nations).
Image processing in Keras (TensorFlow)
In order to get a better feel for how Keras can be used for image classification, we did some very basic proof of concept testing with the Keras software using a total of 800 painting images. 400 images corresponded to the style of cubism and an additional 400 corresponding to the style of Baroque. We used an additional 80 samples from each class to be used as a validation set. In classification, Kares expects a very specific directory structure (which we will not get into here). An epoch is essentially referring to one full forward and backward pass of the training data, whereas batch size corresponds to the number of training images used in a given pass.
This image is an example of setting up the training configuration. Images have RGB values ranging from 0–255 but these values are too high to model for most preferred learning rates so we rescale the RGB values to be between 0 and 1 by scaling with 1/255. The train_generator and test_generator feed in images from the respective train and validation directories into the kares model.fit_generator function allowing training to begin and accuracy to be determined. Finally, we can save the model to be used later to test on additional images by simply calling the kares model.save_weights function.
