Content-based artwork recommendation: integrating painting metadata with neural and manually-engineered visual features
Pablo Messina, Vicente Dominguez, Denis Parra, Christoph Trattner, Alvaro Soto
This paper discusses different ways to perform content-based recommendation, in the context of artwork pieces in an online website. These paintings are one-of-a-kind, so collaborative filtering it’s not an option. Different methods are explored to perform the recommendation, such as Most Popular Curated Attribute Value, Favorite Artist, and visual features: extracted from a CNN, and more traditional methods like LBP. The methods are compared with different metrics and are also combined to create hybrid systems.
Would have been possible to perform fine-tuning in the CNN used to extract features? They could have used the pre-trained network, and train it with the artwork dataset, trying to predict the attributes from the metadata. Maybe this way the net would be better suited to extract relevant features from the paintings.
I find it interesting the use of LBP to extract features. The traditional LBP method needs a grayscale image, and I think that any art buyer would tell you that the color is one of the most important attributes of an artwork. Interestingly, LBP was a helpful feature in the hybrid methods, with a 14.5% ponderation, so this indicates that there are other strong aspects of a painting besides the color.
I wonder why did they use expert curators to validate the recommendations of the system, when the final users are not experts, but normal people looking for art. Moreover, these experts were all part of UGallery, so at certain level, they root in favor of the success of the system. This produces a little bias, and the recommendations could be a bit better than what they are to a neutral person. The use of only 8 experts didn’t help either, as more samples from different people could have helped in reducing this little bias.
I find it great that in the paper, accuracy was not the only metric used, and diversity and coverage were also considered. Nevertheless, in the use of coverage they computed user coverage and session coverage, when probably for a store, the most important it’s item coverage. I think so, because a store doesn’t want some paintings to never be recommended, and you should try to recommend from every artist.
