Image Credit : Instapage

Thompson Sampling with Django

I just finished studying up on Reinforcement Learning using Thomspon sampling algorithm but i felt the urge to implement this in a real world scenario so i came up with this problem.

Problem statement:

A company has 10 versions of an image they intend using to run an advert to the users of their website and the projection of the cost of doing A | B testing is quite high so they need to find a way to explore these ads while at the same time exploiting the one with the most user click (i.e they need a program that displays the ads to users and at the same time is calculating which one is having the most impact thereby converging the particular ad being shown to the ones with highest user clicks).

In this first part, i would be showing you how to setup your environment and the core of the Reinforcement Learning algorithm itself. It’s basically meant to wet your appetite and let you get a feel of implementing your Machine Learning Algorithm on a web application.

Prerequisites

To make this tutorial as direct as possible, i would assume you have python and django installed, you’re familiar with python, django, and setting up virtual environments(I used anaconda for my environments but whatever way you choose should work too.

Let’s get started:

We would activate our environment(i named mine django) and then create a new django project

activate django
django-admin.py startproject optimizer
cd optimizer

Then create the django app:

python manage.py startapp ads

At this point, the directory should look like this:

optimizer/
   ads/
        migrations/
            __init__.py
        __init__.py
        admin.py
        apps.py
        models.py
        tests.py
        views.py
   optimizer/
        __pycache__/
        __init__.py
        settings.py
        urls.py
        wsgi.py
    manage.py

Let’s test our project to see if it works.

python manage.py runserver

You would get some migration soft warnings, don’t worry about that for now. Just open your browser and navigate to localhost:8000

Voila!

If you get this page then we’re on track.

With the project and app created, we need to sync our database for the first time(This would fix those warnings we saw earlier) and create an initial user and set a password for that user when prompted

Please take note of these credentials

python manage.py migrate
python manage.py createsuperuser

Now, open the optimizer/settings.py file and add the adsapp to the list of INSTALLED_APPS

INSTALLED_APPS = [
    'ads.apps.AdsConfig',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Furthermore, open the optimizer/urls.py file and add include to the imports from the urls module, then add urls for the ads app in the urlpatterns list:

...
from django.contrib import admin
from django.urls import path, include
urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('ads.urls'))
]

Adding links to pages

We need to create a urls.py file in our ads folder. Then add the following code to it.

from django.urls import path
from . import views
urlpatterns = [
    path('', views.home, name ='ads-home'),
]

Let’s add the home function to the ads/views folder. It doesn’t need to do anything fancy at the moment but this is actually where our algorithm would go

from django.shortcuts import render
def home(request):
    pass

For this tutorial, I am using the default sqlite that comes with django as my Relational DataBase Management System. You can use postgres or any other RDBMS you prefer.

With that you now have your project setup and ready to code the ads optimizer.

Creating the models for the 10 ads and then user interactions

Open your ads/models.py and add the following code

class Ad(models.Model):
 image = models.ImageField(default = 'default.jpg', upload_to = 'advert_images')
 def __str__(self):
  return f"Ad_id - {self.id} "#Determines how these images are named when queried or in the backend
class Interaction(models.Model):
 ad_1 = models.PositiveSmallIntegerField(default = 0)
 ad_2 = models.PositiveSmallIntegerField(default = 0)
 ad_3 = models.PositiveSmallIntegerField(default = 0)
 ad_4 = models.PositiveSmallIntegerField(default = 0)
 ad_5 = models.PositiveSmallIntegerField(default = 0)
 ad_6 = models.PositiveSmallIntegerField(default = 0)
 ad_7 = models.PositiveSmallIntegerField(default = 0)
 ad_8 = models.PositiveSmallIntegerField(default = 0)
 ad_9 = models.PositiveSmallIntegerField(default = 0)
 ad_10 = models.PositiveSmallIntegerField(default = 0)

Here we are specifying the column of each of the tables that our program would use.

Ads table would have only one field to store the image name. We specified the folder where images would be uploaded to, and also a default image.

Interactions contains a columns representing each ad and they can be 0 or 1 depending on whether a particular user clicked on the ad or not. A record here is a particular user’s interaction with all the ads

Since we are using django , we can also add our model to the admin. We shall see a little bit later how we can use the admin section to add images of each advert. Go ahead and add the following lines of code to the ads/admin.py file.

from .models import Ad, Interaction
admin.site.register(Ad)
admin.site.register(Interaction)

Thereafter, make your migrations and then migrate

python manage.py makemigrations
python manage.py migrate

Adding images of our ads:

Navigate to http://127.0.0.1:8000/admin in your browser and login with the super user credentials that you added during the setup phase of the project. Once logged in, you will have an interface before you that looks like this.

We need to tell Django where to look for and store our media files, so add the following code to the bottom of your optimizer/settings.py file

...
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

Then add the following code to the bottom of your optimizer/urls.py file

from django.conf import settings
from django.conf.urls.static import static
if settings.DEBUG: #If we are in development mode, access images this way
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Now we are ready so click on the Add button to ad images of your choice. For the purpose of this tutorial, we would be adding only 10 images.

The algorithm

We need a way to persist each user’s interaction on the app, so i used a session named user_interaction_id for this

Open your ads/views.py file and add the following code

from .models import Ad, Interaction
from django.shortcuts import render, redirect
from django.urls import reverse_lazy
import math
import pandas as pd
import random
def home(request):
 
 if not request.session.get('user_interaction_id', None):
  new = Interaction()
  new.save()
  request.session['user_interaction_id'] = new.id

Let’s get the interaction of different users for our ads as seen in our database

df = pd.DataFrame.from_records(Interaction.objects.all().values(), columns=['ad_1', 'ad_2', 'ad_3', 'ad_4', 'ad_5', 'ad_6', 'ad_7', 'ad_8', 'ad_9', 'ad_10']) 
 N = len(df)#Number of users
 d = len(df.columns) #Number of ads
 ads_selected = []
 ads_rewards = [0] * d 

 numbers_of_rewards_1 = [0] * d
 numbers_of_rewards_0 = [0] * d

ads_selected == The list of ads shown to each user in the system
ads_rewards == This list would contain records of clicked ads as seen in the db
numbers_of_rewards_1 == The times each ad was clicked
numbers_of_rewards_0 == The times each ad was not clicked

Next

for n in range(0, N):#For each user
   ad = 0
   max_random = 0
   for i in range(0, d):#for each add
      random_beta = random.betavariate(numbers_of_rewards_1[i] + 1, numbers_of_rewards_0[i] + 1)
   
      if random_beta > max_random:
         max_random = random_beta 
         ad = i
      ads_rewards[i] += df.values[n, i]#Accumulate this ad reward
   ads_selected.append(ad)
   reward = df.values[n, ad]
   if reward == 1:
      numbers_of_rewards_1[ad] += 1
   else:
      numbers_of_rewards_0[ad] += 1

The intuition behind this:

For each ad of a particular user’s interaction, use the number of times the ad has been clicked and the number of times it hasnt been clicked to generate a random value between 0 and 1 using random.betavariate, now take the ad that got the highest of the randomly generated values and display it to the user

Second step:

Check if in the real world(i.e the db’s records), this selected ad was actually clicked by this user. If it was, increase the number_of_rewards_1 variable, else increase the number_of_rewards_0 variable of this particular ad.

This determines the magnitude of the value generated by the random.betavariate for that particular ad and this ultimately determines whether or not this ad would end up being selected

This loop uses reinforcement learning to keep adjusting itself and as user interaction of our web pages containing this app increases, the selected ad would start to converge on the one with the highest likes

Take aways

So far, we have seen how to setup a Django project that uses images, how Thomas sampling can be used to optimize ad selection, we also saw how i used sessions to create records for each user accessing our system. Finally, this algorithm uses reinforcement learning to keep adjusting itself and as user interaction of our web pages containing this app increases, the selected ad would start to converge on the one with the highest likes

In the second part, i will be discussing how the ads gets displayed on a page, adding buttons to enable users ‘Like’ or ‘Dislike’ images thereby helping the program get better.

Wow that was a long one, so i’m glad you made it here. I hope this was helpful to you and I want to thank you for reading this post. Hit clap in case it was helpful so that others may find it too. You can also give suggestions in the comment section.