To sell your car faster, take a side picture first!
Traditional e-commerce sites usually display stock photos of the products for sale. These professionally-taken photos are high-quality: they display the product in the foreground and with good lighting, making the object of interest clearly recognizable.
In a marketplace like letgo, however, sellers take their own pictures of the items or services they want to sell, which gives rise to more variety in the quality of the images. Sellers’ cameras vary a lot, lighting conditions are not always ideal and what looks like a sharp picture on a small screen can be blurry on larger devices. In addition, sometimes pictures are taken in the item’s context, resulting in cluttered photos in which it may be hard to recognize the product being sold.
Showing products with better, clearer images is good for our users and for the marketplace ecosystem: the products will be more attractive to buyers so they’ll garner more interest (this article provides more data on the impact of image quality on peer to peer marketplaces).
Since images are important for a marketplace’s reputation, in the data science team we decided to spend some time studying the relationship between image quality and a listing’s performance.
Assessing image quality via proxy measures
When we started to work on this project, we thought, naively, that it would be easy to use our users’ feedback to assign a “good” or “bad” quality label to our product images.
For example, we initially considered indirect measures of a product’s image quality based on its Click-Through-Rate (CTR, the percentage of impressions that convert to a visit to the product). Our hypothesis was that a high CTR may be indicative of better image quality, and vice versa.
Surprisingly, the labels generated with such a procedure didn’t quite match our expectations: there wasn’t much of an aesthetic difference between images with high and low-quality scores. After some investigation, we realized that the feedback provided by users doesn’t correlate well with what we think is a good image for a product. In fact, many users visit a product even if the image is bad just to find out more about it in the description, or to contact the seller and ask for more information.
As a result, in our first modeling attempts, product features such as price, location, or the item’s category turned out to be more important than image features (although they still had some importance) in predicting the CTR or the labels we generated; in addition, this model’s predictive performances were not as high as we initially hoped.
At about the same time, a paper by Alexandre Maros et al. presented at SIGIR 2019 in Paris arrived at similar conclusions:
This indicates that the image alone is not completely responsible for the user’s decision to click on a product. Other factors such as the price range and the title of the product may influence customers actions. But, as we can see from the results, a product’s image quality does have some influence and could possibly be used to improve search results for an EC platform.
— Alexande Maros et al., SIGIR 2019 —
Seeing an independent study corroborating our work was actually quite comforting but we still had a question: can we say something on the relationship between image quality and selling success?
Simplify and conquer: focus on car images
Following these findings, we decided to focus our modeling efforts to the car vertical to reduce our scope and simplify the problem.
Given the caveats discovered during our attempts to do automatic labeling, we decided to instead rely on manual annotations. However, before investing in the labeling of a large number of images using a service like AWS Mechanical Turk, we decided to validate our ideas by manually annotating a small dataset of 2500 car images using labelme, an open-source tool.
Even though manual labeling doesn’t sound too exciting to most fellow Data Scientists, it is a process that we can’t recommend enough, especially when you are embarking on a new project, the scope of your project is still not well defined or you are not too familiar with the data domain. During the process you will:
- get a feel about the data you are working on,
- identify potential issues and quirks that might be present,
- come up with new hypotheses that could be valuable to your business.
For example, at the beginning of our manual labeling process, we planned to simply label car images as “Good” or “Bad”. Upon inspection of a few images, we came up with the new hypothesis that the orientation of the vehicle had a strong impact on our perception of a car listing quality. To validate this hypothesis, we decided to expand beyond the Good/Bad classes and adopt a version of the set of labels described here.
We assigned an “overall quality” label (Good or Bad) to each image based on (rather subjective) criteria like ‘the vehicle is centered in the image’, ‘the image has good enough lighting’, ‘the image is relevant to sell a car’, etc.
We also assigned an “image type” label to each image: “Good” images have been further labeled depending on the orientation of the vehicle (Front, Front-Side, Side, Rear-Side, and Rear); “Bad” images have been labeled with one of the classes “Car-Interior”, “Car-Parts”, “Other” or “Bad-Picture”.
Using two sets of labels allowed us to independently assess the relationship between Good/Bad images and CTR, as well as the dependency between vehicle orientation and CTR.
Below we provide an example for each class — in UPPERCASE we show the “image type” label, in (parenthesis) the overall quality label — :
A car’s image quality impacts its selling success
Our results showed that cars with Good images have 11% higher CTR, generate 10% more visits immediately after publication and 5% more conversations than cars with Bad images.
A quick analysis of our data revealed some differences between the distribution of CTR for products labeled with “Bad” first images and those labeled with “Good” first images:
On average, cars with Good images have a CTR that’s 20% higher than cars with Bad images (similar differences were observed also for other metrics like how often products are visited immediately after being listed or the number of conversations it generated).
In order to see if this difference generalized to products outside of our annotated set, we build a quick Good/Bad classification model in Keras using transfer learning on a MobileNet V2 pre-trained on Imagenet.
Our classifier achieved an AUC of 85% on the test set and, after some manual inspection, we decided that the model was indeed good enough to classify new images into our Good/Bad classes. We created a new dataset of about 20K car products (never seen in training), evaluated them through our categorization model and compared several metrics between the two predicted categories. Our results showed that cars with Good images have 11% higher CTR, generate 10% more visits immediately after publication and 5% more conversations than cars with Bad images.
A car’s image vehicle orientation impacts its selling success
Next, we assessed the hypothesis that the orientation of the vehicle in the images plays a role in making the car more appealing. Seeing important differences in the metrics as a function of the vehicle-orientation label in our labeled dataset, we decided to train a vehicle-orientation categorization model using transfer learning on a MobileNet V2 network.
Our Vehicle Orientation model showed good enough performance, despite the fact that some of the classes had very few samples. Below you can see the confusion matrix on the test set.
Next, we ran our model on the 20K unlabeled cars dataset and unveiled very interesting results! For example, below you can see the expected CTR for each of the predicted Vehicle Orientation classes (excluding REAR and REAR-SIDE which had a small number of samples.)
Cars with a SIDE orientation have a CTR 4% higher than cars with FRONT views and 6% higher than FRONT-SIDE views, and about 14% higher than cars with a Bad Picture. Also, it is quite interesting to observe that cars with Bad pictures, while having a CTR lower than cars with Good images, do perform better than cars whose first image displays the car interior. It’s even more interesting that irrelevant products like Car Parts or Other products have an even lower CTR than proper cars with Bad Images, which stresses the importance of showing relevant products to our users (stay tuned, we’ll soon be writing a post on search and relevance at letgo…).
Such differences are not limited to CTR — we observe similar differences across other important metrics for letgo like the number of visits a product receives and the number of times a seller has been contacted.
Conclusions and next steps
In this post, we have introduced you to the challenges of assessing image quality in marketplaces and its impact on metrics related to the likelihood of selling success. In particular, we have shown that a simple computer vision model can predict a label that correlates strongly with a car listing’s CTR, the interest it generates shortly after publication and the number of contacts it receives.
We are currently working on improving and bringing this model to production to make letgo even more enjoyable for our users! If you like what we have done and would like to help us enable letgo’s buyers and sellers succeed, take a look at our open positions.
