Thompson Sampling with Django
I just finished studying up on Reinforcement Learning using Thomspon sampling algorithm but i felt the urge to implement this in a real world scenario so i came up with this problem.
Problem statement:
A company has 10 versions of an image they intend using to run an advert to the users of their website and the projection of the cost of doing A | B testing is quite high so they need to find a way to explore these ads while at the same time exploiting the one with the most user click (i.e they need a program that displays the ads to users and at the same time is calculating which one is having the most impact thereby converging the particular ad being shown to the ones with highest user clicks).
In this first part, i would be showing you how to setup your environment and the core of the Reinforcement Learning algorithm itself. It’s basically meant to wet your appetite and let you get a feel of implementing your Machine Learning Algorithm on a web application.
Prerequisites
To make this tutorial as direct as possible, i would assume you have python and django installed, you’re familiar with python, django, and setting up virtual environments(I used anaconda for my environments but whatever way you choose should work too.
Let’s get started:
We would activate our environment(i named mine django) and then create a new django project
activate django
django-admin.py startproject optimizer
cd optimizer
Then create the django app:
python manage.py startapp ads
At this point, the directory should look like this:
optimizer/
ads/
migrations/
__init__.py
__init__.py
admin.py
apps.py
models.py
tests.py
views.py
optimizer/
__pycache__/
__init__.py
settings.py
urls.py
wsgi.py
manage.py
Let’s test our project to see if it works.
python manage.py runserver
You would get some migration soft warnings, don’t worry about that for now. Just open your browser and navigate to localhost:8000
Voila!
With the project and app created, we need to sync our database for the first time(This would fix those warnings we saw earlier) and create an initial user and set a password for that user when prompted
Please take note of these credentials
python manage.py migrate
python manage.py createsuperuser
Now, open the optimizer/settings.py
file and add the ads
app to the list of INSTALLED_APPS
INSTALLED_APPS = [
'ads.apps.AdsConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]
Furthermore, open the optimizer/urls.py
file and add include
to the imports from the urls module, then add urls for the ads
app in the urlpatterns list:
...from django.contrib import admin
from django.urls import path, includeurlpatterns = [
path('admin/', admin.site.urls),
path('', include('ads.urls'))
]
Adding links to pages
We need to create a urls.py
file in our ads
folder. Then add the following code to it.
from django.urls import path
from . import viewsurlpatterns = [
path('', views.home, name ='ads-home'),
]
Let’s add the home
function to the ads/views
folder. It doesn’t need to do anything fancy at the moment but this is actually where our algorithm would go
from django.shortcuts import renderdef home(request):
pass
For this tutorial, I am using the default sqlite
that comes with django
as my Relational DataBase Management System. You can use postgres
or any other RDBMS you prefer.
With that you now have your project setup and ready to code the ads optimizer.
Creating the models for the 10 ads and then user interactions
Open your ads/models.py and add the following code
class Ad(models.Model):
image = models.ImageField(default = 'default.jpg', upload_to = 'advert_images')
def __str__(self):
return f"Ad_id - {self.id} "#Determines how these images are named when queried or in the backendclass Interaction(models.Model):
ad_1 = models.PositiveSmallIntegerField(default = 0)
ad_2 = models.PositiveSmallIntegerField(default = 0)
ad_3 = models.PositiveSmallIntegerField(default = 0)
ad_4 = models.PositiveSmallIntegerField(default = 0)
ad_5 = models.PositiveSmallIntegerField(default = 0)
ad_6 = models.PositiveSmallIntegerField(default = 0)
ad_7 = models.PositiveSmallIntegerField(default = 0)
ad_8 = models.PositiveSmallIntegerField(default = 0)
ad_9 = models.PositiveSmallIntegerField(default = 0)
ad_10 = models.PositiveSmallIntegerField(default = 0)
Here we are specifying the column of each of the tables that our program would use.
Ads table would have only one field to store the image name. We specified the folder where images would be uploaded to, and also a default image.
Interactions contains a columns representing each ad and they can be 0 or 1 depending on whether a particular user clicked on the ad or not. A record here is a particular user’s interaction with all the ads
Since we are using django
, we can also add our model to the admin. We shall see a little bit later how we can use the admin section to add images of each advert. Go ahead and add the following lines of code to the ads/admin.py
file.
from .models import Ad, Interactionadmin.site.register(Ad)
admin.site.register(Interaction)
Thereafter, make your migrations and then migrate
python manage.py makemigrations
python manage.py migrate
Adding images of our ads:
Navigate to http://127.0.0.1:8000/admin
in your browser and login with the super user credentials that you added during the setup phase of the project. Once logged in, you will have an interface before you that looks like this.
We need to tell Django where to look for and store our media files, so add the following code to the bottom of your optimizer/settings.py
file
...MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
Then add the following code to the bottom of your optimizer/urls.py
file
from django.conf import settings
from django.conf.urls.static import staticif settings.DEBUG: #If we are in development mode, access images this way
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Now we are ready so click on the Add button to ad images of your choice. For the purpose of this tutorial, we would be adding only 10 images.
The algorithm
We need a way to persist each user’s interaction on the app, so i used a session named user_interaction_id
for this
Open your ads/views.py
file and add the following code
from .models import Ad, Interaction
from django.shortcuts import render, redirect
from django.urls import reverse_lazy
import math
import pandas as pd
import randomdef home(request):
if not request.session.get('user_interaction_id', None):
new = Interaction()
new.save()
request.session['user_interaction_id'] = new.id
Let’s get the interaction of different users for our ads as seen in our database
df = pd.DataFrame.from_records(Interaction.objects.all().values(), columns=['ad_1', 'ad_2', 'ad_3', 'ad_4', 'ad_5', 'ad_6', 'ad_7', 'ad_8', 'ad_9', 'ad_10'])
N = len(df)#Number of users
d = len(df.columns) #Number of ads
ads_selected = []
ads_rewards = [0] * d
numbers_of_rewards_1 = [0] * d
numbers_of_rewards_0 = [0] * d
ads_selected == The list of ads shown to each user in the system
ads_rewards == This list would contain records of clicked ads as seen in the db
numbers_of_rewards_1 == The times each ad was clicked
numbers_of_rewards_0 == The times each ad was not clicked
Next
for n in range(0, N):#For each user
ad = 0
max_random = 0
for i in range(0, d):#for each add
random_beta = random.betavariate(numbers_of_rewards_1[i] + 1, numbers_of_rewards_0[i] + 1)
if random_beta > max_random:
max_random = random_beta
ad = i
ads_rewards[i] += df.values[n, i]#Accumulate this ad reward ads_selected.append(ad)
reward = df.values[n, ad]
if reward == 1:
numbers_of_rewards_1[ad] += 1
else:
numbers_of_rewards_0[ad] += 1
The intuition behind this:
For each ad of a particular user’s interaction, use the number of times the ad has been clicked and the number of times it hasnt been clicked to generate a random value between 0 and 1 using random.betavariate
, now take the ad that got the highest of the randomly generated values and display it to the user
Second step:
Check if in the real world(i.e the db’s records), this selected ad was actually clicked by this user. If it was, increase the number_of_rewards_1
variable, else increase the number_of_rewards_0
variable of this particular ad.
This determines the magnitude of the value generated by the random.betavariate
for that particular ad and this ultimately determines whether or not this ad would end up being selected
This loop uses reinforcement learning to keep adjusting itself and as user interaction of our web pages containing this app increases, the selected ad would start to converge on the one with the highest likes
Take aways
So far, we have seen how to setup a Django project that uses images, how Thomas sampling can be used to optimize ad selection, we also saw how i used sessions to create records for each user accessing our system. Finally, this algorithm uses reinforcement learning to keep adjusting itself and as user interaction of our web pages containing this app increases, the selected ad would start to converge on the one with the highest likes
In the second part, i will be discussing how the ads gets displayed on a page, adding buttons to enable users ‘Like’ or ‘Dislike’ images thereby helping the program get better.
Wow that was a long one, so i’m glad you made it here. I hope this was helpful to you and I want to thank you for reading this post. Hit clap in case it was helpful so that others may find it too. You can also give suggestions in the comment section.