My first full stack data science project (Part 5)…

Link to previous article: Here

After verifying that our frontend we made in 4th article and model we made in 3rd article works perfectly and not malfunctioning we can start connecting everything we did in 2nd, 3d and 4th articles together by python backend framework Django.

Django follows MVT(model, view, template) architecture. We already have model and template ready. After setting up Django project, urls.py, database settings and static-files settings in settings.py file we come to views.py file of our app which will act as a bridge between our model written in python in 3rd article and template. views.py file will be receiving AJAX requests from frontend Javascript and send request to backend.py file accordingly. backend.py file will return data from get_on_display() function to views.py file and it will return data back to template so that it can display to user. Task of displaying to user is done by 'get_moves_list.js' file.

Functions fourthUserShow() and fourthRatingShow() written in ‘get_moves_list.js’ file takes response which is a JSON object returned by get_on_display() to views to template. Here 'on_click_ok.html' is doing task of sending requests to views.

sendRequestUserOk() is called when user clicks 'ok' button to search for specific username's data. Data from template(on_click_ok.html) to views(views.py) is sent in format of : username, game class and moves list. Task of sending this request to views.py is done by urls.py file. After views.py receives request in userOk() function it will then call backend functions to initialize user data and also initialize rating data.

Why are we initializing ratings data again? Because we don't want user to restart again clicking all the moves again to reach same position. This is also the answer to why are we sending moves list in the request data. After initializing both user and rating data we click in the backend all the moves that are there in the moves list so we make the data reach current position and not stay at starting position.

sendRequestUserOk() is doing one more task. If the data of user is not found in the database then either the username doesn't exist or the data is not with us. In this case we want to store the data in one table so that we can come back to it and add that user's data in the games table. Finally display my data on display because keeping the username empty is not compatible with backend. Also one more thing is that while the data is loading we don't want user to move click or send 'ok' requests again so we store the status requests of whether some requests are still running or not in 'doneRequests'.

sendRequestRatingOk() is doing similar thing to above function. Sending rating range and moves list data to backend, views will receive this request in ratingOk() function, it will initialize user and rating data and play the moves in backend to reach current position. Why initialize user data again? Answer is same as we did in above function.

Sending requests for move clicks, going back a move and changing color have very similar syntax so not showing that here but you can see that in ok_click_ok.html file in source code.

Now we are done with all the coding regarding project and it works just fine and the project works just fine on the local server. But the only 2 thing left are to upload our database on cloud and to host our application on cloud.

First part is to migrate our data stored in MySQL server localhost to Azure SQL database. I created Microsoft SQL server on Azure portal and then a database named ChesscomData in that server. Created three tables named chesscom_games, user_ratings and new_users in the database.

For the migration part I was first of all able to export all the data in tables to CSV and also to SQL insert statements by pycharm IDE, then I tried different methods like through SSMS(SQL Server Management Studio), Azure Data Studio, SSMA(SQL Server Migration Assistant for MySQL), MySQL Workbench migration wizard, tried to insert data online on Azure database query editor on portal itself but none of them worked. Not worked in the sense that it was taking time in minutes and you can't even track progress to know how many hours it would take. SSMA was not able to configure the ODBC driver and working on it for hours to configure.

Finally the two methods that worked are: to use pyodbc module in python to connect with Azure SQL database and execute all the insert statements which we exported to one file already in a loop and another method was to use 'import data' on a table because I was able to connect Azure SQL database to my PyCharm IDE. Both methods were giving almost same speed which is 17 rows/sec. I know this is very low because we want to migrate a million rows. I know there are of course better methods but I just couldn't make those work. Good thing is it’s an improvement over how we migrated CSV files to MySQL database. Code for migration is given in 'mysql_to_azuresql.py' file in source code.

Final thing left is to host the Django app on Azure. The code had to be uploaded on github through which it was connected to Azure. Uploading was not an issue but the configuration of settings really took huge amount of time as there were lot of things involved like urls.py had to be changed, static-files settings and paths, debug settings, database settings, driver version settings, storage settings, template paths etc. As of now there are already 70 commits in my private github repository which is connected to Azure.

Finally I made a nice looking home page from which you can direct to source code, this project report and contact me through email.

Everything works just fine and project is over…

Except. Still one last error left to be resolved. On one device the website is working ok but if I try to run it parallelly on two or more devices still the home page opens but the data of moves it’s showing on website is malfunctioning. It was huge surprise for me to learn that I must not declare global variables in Django. First thing came to my mind was can I transfer this running on server ‘backend.py’ to front end where global variables are stored on local website storage but the problem is limit with storage which is 5MB only. Now instead of storing data-frames as global variables, I have to store them in database only. Means to find a way that we only interact with database and not global variables(data-frames in this case) in ‘backend.py’. But with the speed of 17 rows/sec it’s impossible to deal with these many requests.

I found another solution that ‘CSRF’ is a unique token for client(visitors on website) which can be sent as data through AJAX requests. So I did a trick to keep a dictionary of keys as CSRF tokens as a global variable in ‘backend.py’ file. This way I separated out data of different CSRF token holders and the problem is solved.

Finally everything is sorted and website works perfectly fine on parallel devices..

That’s it for this project. Your comments and suggestions on the project are appreciated. Thank you for reading.