Capturing Beauty — Hybrid Incremental Learning for AI-powered Tinder profile swiping : An Implementation in Keras.
Introduction
“Beauty is in the eye of the beholder”- a well quoted line about the standards of human beauty. But what is beauty? Can we define or quantify the superficial attractiveness of an individual? More importantly, should we abstract something so sensitive and subjective to a black box? While some universal attributes such as symmetry have been identified to be cross culturally shared as being beautiful, the fact remains that judging the attractiveness of an individual remains a subjective affair, highly dependent on the environment the person exists in.
Tinder is the world’s most popular dating application, with over 4.1 million paying users alone, and over 20 billion matches made daily in 2018. It’s popularity can be attributed to its simplicity : a user is presented if profiles, consisting of a series of photos and some textual information. Based on this information, a user swipes right or left depending on whether they find the profile attractive. Should two users swipe right on each other, a match is made and the two users can then chat with each other. Swiping is heavily influenced by superficial factors, eschewing more deeper connections until further down the line.
Automatic swiping scripts have existed for dating Apps such as Tinder for as long as the app has existed, although the application has heavily cracked down on the use of its APIs. These scripts rely on the quantity over quality approach, assuming that given enough right swipes, a match is statistically more likely. Indeed, research has shown that men are significantly less selective when it comes to right swiping than women. However, such approaches are hampered by Tinder’s own internal safeguard — the ELO score.
The ELO rating system, named after its inventor Arpad Elo, is a calculation method for judging the relative skill of players within a zero sum game. While originating in Chess, use of the system, has moved to other competitive sports, being particularly useful at the professional level for league rankings. Within the context of Tinder, it allows for users to be ranked against each other through a combination of certain weighted statistics, such as match and right swipe rates. These rankings then affect how , how often a profile is seen by other users. While the complete formula is not available to the public, observations among users have shown that a swiping right consistently causes the user to be punished, and their profile pushed to the bottom of the stack, making it less likely that other users would encounter their profile.
To overcome this, existing autoswiping applications have utilized static or variable probabilities to introduce left swipes at regular or randomized intervals. However, this is still an inadequate solution , as the ELO score of the target profile that one swipes right on has also been shown to have an impact on the score of the swiper — for example, swiping right on a lower-ranked individual would have a detrimental effect on your own ELO score.
The optimal solution to this problem would hence involve namely tailoring the likes of an swiping algorithm to the preferences of a user. By gathering information on what profiles a user swipes left and right on, we can build a classifier capable of only swiping right on profiles that the user judge aesthetically pleasing. One problem to this approach is the lack of data at the beginning of the training process –as information on a user’s preferences is only created when they perform a swipe. To address this, we can utilize incremental learning — a training approach where the model is continuous trained with a individual datapoints collected over time. Such an approach would not only allow for a continuously improving classifier to be available for immediate use, but also accommodate for possible changes in preference over time.
Note while a Tinder API exists, implementing an autoswiper or scraping data off from Tinder would run afoul with its Terms and Regulations. Instead, we will be using a simulated incremental learning environment instead: we will use a reduced total dataset size of 500 image for each of our two classes representing faces with desirable and undesirable characteristics.
We’ll be examining two hypothetical scenarios:
- A user who is exclusively partial to Asian female faces.
- A user who is exclusively partial to Asian female faces above a certain threshold of attractiveness.
To address this, we will utilize the SCUT-FBP dataset collected by the Human Computer Intelligent Interaction Lab at the South China University of Technology, which consists of 500 female Asian faces ranked on a scale of 1–5 for “attractiveness” by three researchers. Against this class, we’ll be using a dataset of 477 images collected from NVIDIA’s Flicker Faces dataset, scrubbed of any Asian female examples.
Implementation
In a real incremental learning situation, we’d fit our model on individual datapoints, save our progress, and then continue training once a new datapoint is collected. However, we’ll settle with a simulation by restricting our batch size to one or two images. Essentially, this means that the network will be trained using only a few images at a time (not including just-in-time Keras data augmentation effects).
Our code was implemented in Google Colaboratory using Keras and Tensorflow. As usual, the full code can be found on the GradientCrescent Repository.
To start off, we import our datasets into our notebook.
!pip install vis
!pip install -I scipy==1.2.*!mkdir datafrom numpy.random import seed
#seed(6) #1
from tensorflow import set_random_seed
#set_random_seed(5) #2#import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)# Input data files are available in the “../input/” directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directoryimport os#TRAINING SETS#########################################Download the FFHQ dataset minus asian femals!gdown https://drive.google.com/uc?id=1neeeqFXWcdgTognHrPlndnjqPUxBBL1i
#Downlaod SCUT-FBP Asian Dataset!gdown https://drive.google.com/uc?id=1IUqrFv9Fal5S3bujWfEx0hIlquecDVFn
#Download Ratings CSV — this will be used for beauty ranking (3)!gdown https://drive.google.com/uc?id=1LBP5MnfB-IY2hSjkt_qsaoqEJv3OS6kD#Unzip all folders!unzip asian_tinder.zip -d data
!unzip real_tinder.zip -d data
You’ll note that we also download a CSV spreadsheet along with our datasets. This is the attractiveness ranking done by the SCUT-FBP researchers, which we’ll hold on to for a latter part of the study. You’’ll also notice the next cell of code — this sorts through the SCUT-FBP to only include examples ranked above an attractiveness rating of 3. This will be only enabled when considering the second scenario, so leave it as it is for now.
print(os.listdir(“/content/”))#Following code is for modifying the dataset in accordance to the rankings of the Asian Female Face dataset. Disabled by default, enable it if you wish to replicate the beauty-ranking part of tutorial
“””base_dir_A =’/content/data/asian/’
base_dir_B =’/content/data/real/’
base_dir_Beauty = ‘/content/data/beauty/’os.mkdir(base_dir_Beauty)import pandas as pd
import shutilranking = pd.read_csv(“tinder_labels.csv”)
#print(ranking)#Iterate over all of the labels within the df
for i, j in ranking.iterrows():
#print(int(j[“Image”]))
filenumber = int(j[“Image”])
filename = “SCUT-FBP-”+str(filenumber)+”.jpg” #Iterrate over all images and move them to beauty. Rest move to real. #Then set new A class as beauty folder
#Define source and destination, copy it over then
src =base_dir_A + filename
dst =base_dir_Beauty + filename
shutil.copy(src, dst)
#delete the file you just copied from source directory
os.remove(src)
#Once we do this for all files within the dataframe, move the leftovers to directory Bfor filename in os.listdir(base_dir_A):
leftsrc =base_dir_A + filename
leftdst =base_dir_B + filename
# rename() function will
# rename all the files
shutil.copy(leftsrc, leftdst)
os.remove(leftsrc)
#Sanity check
print(os.listdir(base_dir_A))#After this, remember to set the classs directories appropriatelyl
# /content/data/asian/SCUT-FBP-11.jpgos.rmdir(base_dir_A)
“””
Next, let’s define our preprocessing functions and ImageDataGenerators. All of the preprocessing done is native to Keras itself. You’ll notice that we’ve created two sets of them here, with one training and validation pair covering a batch size of 1, and another covering a batch size of 5. We’ll be using these two sets to implement a hybrid training methodology.
from tensorflow.keras import backend as K
from tensorflow.keras.models import Model ,load_model
from tensorflow.keras.layers import Flatten, Dense, Dropout
from tensorflow.keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input
from tensorflow.keras.optimizers import Adam, RMSprop
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.callbacks import ModelCheckpoint
import numpy as np
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras.applications.inception_v3 import InceptionV3
import tensorflow as tfDATASET_PATH = ‘/content/data’IMAGE_SIZE = (299, 299)
NUM_CLASSES = 2
BATCH_SIZE = 1 # Batch size one to simulate online learningNUM_EPOCHS = 150
LEARNING_RATE = 5e-4 #Slow learn rate as we are transfer training5e-5. 2E-4 tried,
DROP_OUT = .5#Train datagen here is a preprocessor
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input,
rotation_range=50,
featurewise_center = True,
featurewise_std_normalization = True,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.25,
zoom_range=0.1,
zca_whitening = True,
channel_shift_range = 20,
horizontal_flip = True ,
vertical_flip = True ,
validation_split = 0.3,
fill_mode=’constant’)# test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input,
# fill_mode=’constant’)train_batches_5 = train_datagen.flow_from_directory(DATASET_PATH,
target_size=IMAGE_SIZE,
shuffle=True,
batch_size=5,
subset = “training”,
class_mode=’binary’
)valid_batches_5 = train_datagen.flow_from_directory(DATASET_PATH,
target_size=IMAGE_SIZE,
shuffle=True,
batch_size=3,
subset = “validation”,
class_mode=’binary’
)train_batches_1 = train_datagen.flow_from_directory(DATASET_PATH,
target_size=IMAGE_SIZE,
shuffle=True,
batch_size=1,
subset = “training”,
class_mode=’binary’
)valid_batches_1 = train_datagen.flow_from_directory(DATASET_PATH,
target_size=IMAGE_SIZE,
shuffle=True,
batch_size=1,
subset = “validation”,
class_mode=’binary’
)
Next, we define our InceptionV3 model. We’ve used this same model before — a simple vanilla model loaded with ImageNet weights, with a custom top-end for binary classification. Using a pre-trained benchmark model cuts down on training time, allowing for rapid implementation and idea validation.
To begin with, let’s inspect how our InceptionV3 model performs using a purely incremental learning-based approach, with a batch size of one for 150 epochs and a learning rate of 2E-4. We can do this with the .fit() command:
result=model.fit_generator(train_batches_1,
steps_per_epoch = 100,
validation_data = valid_batches_1,
validation_steps =50,
epochs = 150,
)Training should take less than an hour . Once finished, we can inspect our results with the Matplotlib library:
def plot_acc_loss(result, epochs):
acc = result.history[‘acc’]
loss = result.history[‘loss’]
val_acc = result.history[‘val_acc’]
val_loss = result.history[‘val_loss’]
plt.figure(figsize=(15, 5))
plt.subplot(121)
plt.plot(range(epochs), acc[ :150], label=’Train_acc’)
plt.plot(range(epochs), val_acc[ :150], label=’Test_acc’)
plt.title(‘Accuracy over ‘ + str(epochs) + ‘ Epochs’, size=15)
plt.legend()
plt.grid(True)
plt.subplot(122)
plt.plot(range(epochs), loss[ :150], label=’Train_loss’)
plt.plot(range(epochs), val_loss[ :150], label=’Test_loss’)
plt.title(‘Loss over ‘ + str(epochs) + ‘ Epochs’, size=15)
plt.legend()
plt.grid(True)
plt.show()
plot_acc_loss(result, 50)
That’s pretty terrible. Our validation accuracy is roughly 50%. Trying to fine-tune this model with a lower learning rate of 2E-5 doesn’t help either.
Doesn’t bode well for pure incremental learning. With the small size of each batch, we simply don’t have enough data to allow for gradient descent to converge toward the optimal parameters, resulting in the persistent poor performance. Note how the accuracy and loss values oscillate widely, a known weakness of incremental learning — as the number of images per batch in an epoch is very limited, the relative contribution of each individual image is significantly magnified, resulting in the wide swings observed.
So how do we overcome this problem?
Let’s assume that we don’t start training immediately after receiving data, but wait until a certain minimum dataset size is reached. We could then train the model traditionally for a few epochs using a larger batch size, and only then allow for incremental learning to take place. If all data is of a shared domain, this approach would allow gradient descent to converge toward the correct parameters more rapidly, with incremental learning playing a more minor role similar to fine tuning.
So let’s simulate this. Let’s assume the user swipes over a few days, swiping across 50 profiles, achieving a roughly equal data distribution matching our criteria. We could then use a batch size of 5 images (train_batches_5) to train our model for 6 epochs at a higher learning rate of 5E-4, with 20 images left for validation at a batch size of 3 (valid_batches_5). We then switch to our incremental model, training with a batch size of 1 and a learning rate of 2E-5, essentially fine-tuning our model with fresh data. Let’s take a look at the results:
With a validation accuracy of roughly 90%, we’ve created a a competent classifier for our application — the intial higher learning rate took strides toward the correct optimum, allowing for the smaller rate to converge to it more rapidly. We do observe some over-fitting past 60 epochs however, and so early stopping may be warranted.
Now that we’ve covered our first scenario, what if our user was more selective, preferring to only swipe on users above an attractiveness threshold? Let’s enable our preprocessing code, which should result in 119 examples being transferred within the new “Beauty” directory, and the rest shifted to the auxilliary class. We then train our model with the same hybrid methodology as before, but keep the fine-tuning to 50 epochs to avoid overfitting:
You’ll notice that our validation accuracy converges relatively fast here, with the loss seemingly increasing with time. This suggests that the model is not truly learning, which could be attributed to an overlap of the data domain — namely, the small differences between what’s considered a “beautiful” a more average face example, are too small to be distinguished with this model. Perhaps this is for the best. We wouldn’t want to AI’s to remove the magic of connecting with the people we find beautiful, now would we?
Beauty truly is in the eye of the beholder.
We hope you enjoyed this article on GradientCrescent. To stay up to date with our articles, please consider subscribing to the publication. Next-up, we finally move way from convolutional neural networks, and discuss reinforcement learning.
