Modern RecSys

The future of visual RecSys is an exciting one. Let us explore some of the most cutting edge techniques and ideas that we can incorporate into our recommenders.

Style2Vec (2017) — Combining Multiple Convolutional Neural Networks

Lee, H., Seol, J., & Lee, S. (2017). Style2Vec: Representation Learning for Fashion Items from Style Sets. Open Access From https://arxiv.org/abs/1708.04014

The paper is short, but the idea is fascinating. The use of two CNN models, where one generates embeddings for the target image while the other generates embeddings for similar context items, is a novel application of the techniques derived from Word2Vec to images…


Modern RecSys

Building a Recommender That Evolves with Seasons

One of the biggest challenges in the design of any recommender system (RecSys) is in handling temporal shifts. Since the world of fashion evolves with time, recommenders we design must adjust to the changing tides as well.

In this article, we will consider how we can factor in the seasonal outfit subtleties into our Convolutional Neural Network (CNN) model using temporal weights.

The Business Problem

Our imaginary eCommerce company, HappyPanda Co., is expanding globally and will like to launch Seasonal Collections. The product requirements are:

  • Given collections of outfits, how can we adjust them to suit the various seasons across different countries? …


Outfits selected by our RecSys. Left: “Anything with Shorts”. Right top: “Urban Lifestyle”. Bottom Right: “Flowery Dreams”. Outfits from DeepFashion, open-source by Liu Z. et al.

Modern RecSys

We will make use of transfer learning, approximate nearest neighbors, and embeddings centroid detection in PyTorch to build our recommender.

I have worked in the data industry for over seven years and had the privilege of designing, building, and deploying two recommender systems (RecSys) that went on to serve millions of customers. In this article, I will introduce modern approaches to visual recommender by walking through a case study with code and sharing some of my experience designing RecSys.

The Business Problem

Imagine you are hired by an eCommerce, HappyPanda Co., to work on their Fashion Collection feature. The product team has outlined the requirements:

  • The model should be able to scan across all 280,000+ product images and automatically generate a group of…


Modern RecSys

We will cluster COVID-19 X-ray images based on severity with our CNN RecSys flow using transfer learning, Spotify’s Annoy, and PyTorch

This work is meant as a proof-of-concept of how we can apply the same framework we set up in the previous CNN chapter onto a completely different domain.

We will swap out the training data and employ a more powerful pre-trained model (Resnet152); the rest of the code remains identical to the one we used for DeepFashion images. We aim to identify clusters of X-ray images with similar severity in infection using Approximate Nearest Neighbors.

This work is not intended as medical research nor representative of how we can use CNN to detect COVID-19.

This is part of my Modern…


Target image on left, recommendations generated by our model on the right. Outfits from DeepFashion, open-source by Liu Z. et al.

Modern RecSys

We will build a recommender with transfer learning, Spotify’s Annoy, PyTorch, and return visually similar products across 240K images in 2ms

This is part of my Modern Visual RecSys series; feel free to check out the rest of the series at the end of the article.

The Data

We will be using a subset of DeepFashion data open-sourced by Liu Z. et al., The Chinese University of Hong Kong. Our data consists of 280K fashion images across 46 categories. You can download the data from their website.

Furthermore, the team has released an updated version with additional data. You will need to fill up a Google form to gain access to the data.

What is Convolution?

Convolution is not a new technique. In essence, we are…


Modern RecSys

In this chapter, we will explore the “hello world” data for visual models, the FashionMNIST dataset from Zalando with PyTorch, Tensorboard and Colab.

This is part of my Modern Visual RecSys series; feel free to check out the rest of the series at the end of the article.

FashionMNIST and the visual challenge

Source: FashionMNIST by by Kashif Rasul & Han Xiao

The data consists of:

  • Training set of 60,000 images and a test set of 10,000 images.
  • Each image is 28x28 grayscale, across 10 classes: T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot.

What are embeddings and why we need them?


Modern RecSys

For this chapter, I will introduce the RecSys Design Framework with a case study of Amazon.

This is part of my Modern Visual RecSys series; feel free to check out the rest of the series at the end of the article.

RecSys Framework — Amazon case study

Recommendations on my Amazon homepage

An eCommerce website like Amazon is heavily reliant on having a good RecSys. After all, users cannot be expected to browse through millions of products that are on the platform, while sellers will like exposure for their products. …


Modern RecSys

In this series of articles, I will introduce modern approaches to visual recommender systems. We begin with a case study of Spotify.

I have worked in the data industry for over seven years and had the privilege of designing, building, and deploying two recommender systems (RecSys) that went on to serve millions of customers. For each chapter, I will walk through case studies and share my experience designing RecSys.

This is part of my Modern Visual RecSys series; feel free to check out the rest of the series at the end of the article.

RecSys Basics — Spotify Case Study

We begin with a case study of Spotify to understand how RecSys works and introduce several key concepts, including a modern approach called convolutional neural networks (CNN), applied…


I have the privilege of hosting Avi (VP — Data Sciences, AI & ML, InMobi), Hong Ting (CEO and Cofounder, Botbot.AI) and Kenny (Head of R&D Singapore, DataRobot) during the recent Tech in Asia Singapore 2018 conference. Here are the key takeaways.

Q: What is AI and does the definition matter?

  • From computer science angle, what we deem as “AI” in recent news is just Narrow AI — machines that can do a single task well like self-driving cars, AlphaGo, Google Home and so on. We are still very far away from General AI — the likes of J.A.R.V.I.S. from…

Kai Xin Thia

Snr Data Scientist at Refinitiv Labs, M.S. CS Georgia Tech. 9+ years in data, found ❤️ in RecSys, NLP, Computer Vision, Applied R&D. linkedin.com/in/thiakx

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store