Deploy your AI model the hard (and robust) way

Build a Machine Learning web service with ROI

Previously, in the garage …

As a young and ambitious data scientist, you discovered how you could quickly deploy a model in production, through a REST API.

The approach however was simple and naïve, and real life is full of terrible dangers that need to be taken care of in a more sophisticated way.

Key elements for a robust predictive application will require to:

• use a real http server instead of the testing server provided with flask
• put some testing in place to ensure your code is correctly working, even when you refactor and tweak it
• document your code, and your API
• track the inner workings through logging
• handle multiple tasks with a basic queuing system

Fasten you seat belts and get ready to become a full-stack data scientist, one that brings money to the table.

Description

Here it is: a complete example of exposing your AI model as a REST API accessible on github! To start it, first download/clone/fork the project, install docker and docker-compose go in the folder docker and launch the command

sudo docker-compose up

My AI is a machine learning model that needs to be trained on data (here the data are stored in csv files); after that, you would be able to predict given new input data.
In this project, we will provide AI services through two URL routes: /train and /predict. All the information will be send to (and received from) API will be formatted as JSON. The JSON file format is a text file structured as key-value pairs. My API comes with the documentation available via URL http://localhost:5000:

We can easily start training our model by sending JSON data with a POST request (run curl in your terminal):

curl -X POST -H "Content-Type: application/json" -d '{"target":"price","csv_file_name":"kc_house_data.csv"}' http://localhost:5000/train

where in the JSON,target is the column to predict and csv_file_name the name of the csv file to use.
We can then check if the model is available with the GET request to the same route and then make a prediction using the command:

curl -X POST -H "Content-Type: application/json" -d @to_predict_json.json http://localhost:5000/predict

where to_predict_json.json is a json file:

{"id":7210,"bedrooms":5.0,"bathrooms":2.75,"sqft_living":3595.0,"sqft_lot":5639.0,"floors":2.0,"waterfront":0.0,"view":0.0,"condition":3.0,"grade":9.0,"sqft_above":3595.0,"sqft_basement":0.0,"yr_built":2014.0,"yr_renovated":0.0,"zipcode":98053.0,"lat":47.6848,"long":-122.016,"sqft_living15":3625.0,"sqft_lot15":563}

The returned JSON contains the prediction:

{"status": "ok", "prediction": 1008773.4048572383}

The training is done asynchronously so you can continue predicting while training.

The components of the project

Here, I will give a small description of each component of the project and the complete explanation of how they work in the next chapters.

• docker:
  OS-level virtualization, easy to install/deploy system.
• supervisor:
  a client/server system that allows its users to monitor and control a number of processes. It starts nginx gunicorn and Redis and keeps monitoring their logs.
• nginx:
  high performance proxy server
• gunicorn:
  a python Web Server Gateway Interface HTTP server
• Flask:
  a popular python web server library
• Flask-RESTPlus:
  an extension of Flask for quickly building REST APIs
• Blueprint:
  documentation structure of the API REST from Flask
• sklearn:
  python machine learning library
• Redis:
  an open-source in-memory database. Here this database is used to host the queue system.
• RQ:
  python-rq is a module to manage job queues on Redis.
• logger:
  a python module to implement a flexible event logging system.
• pytest:
  python module framework that makes it easy to write tests.

Docker

Docker is a program for virtualization, it permits to manage "containers", that pack together virtual OS, software and its dependencies. Containers are easy to set up and deploy. You can find the documentation here (it is not difficult to handle by reading the quick start then you can find a configuration on github that fits your needs). There are two steps in the “dockerization”, the installation and the setup. These two steps can be separated in two files: the Dockerfile and the docker-compose.yml

The list of the different commands in this Dockerfile

• FROM: the container image from which we start the installation
• RUN: a command to run inside the container
• ADD: add a file inside the container
• ENTRYPOINT: a command to start when we run the container

• version is just the reading format version of the file.
• services starts the section with all the different services (mainly containers, but could be other entities), here there is only one container.
• flask is the service name.
• build the path to the Dockerfile to build.
• container_name is the container name.
• volumes all the links between folder in your server that will be shared with the container.
• ports all the ports that the docker will listen to from the outside.

A list of the different commands to interact with the container is available in the folder docker/docker_utils

• sudo docker-compose build will build docker
• sudo docker exec -it house_prediction_REST_API bash launches a bash terminal inside the container.

Supervisor

Supervisor allows to control processes on UNIX-like system, it permits to restart them when there is a crash and to handle the logging of each of them in a centralized way. By default, it starts all the file in /etc/supervisor/conf.d/*.conf. In this project, we link this volume inside the container with docker/supervisor-conf.d . The structure of a file that launches a process is

[program:process_name]
command=command_to_launch
directory=/directory/to/launch

Through supervisor, we will launch 4 processes:

• gunicorn
• nginx
• redis-server
• redisQworker

You can easily restart all of them with the command: supervisorctl restart all

gunicorn & nginx

To interface your API REST with the world, you need gunicorn and nginx.
Why?
Because Flask is well designed for it.
Why?
Read the link below

Why do I need Nginx and something like Gunicorn?

docker, gunicorn, nginx and the swagger documentation have a weird interaction in this project, I will attempt to describe it, and it will work fine.
gunicorn launches the python flask server entrypoint ai/app.py with the command gunicorn app:app -b 0.0.0.0:5000 .
gunicorn listen on the port 5000.

nginx configuration is in the file docker/src/nginx_flask_proxy.conf

server {
 listen 5001;
location / {
  proxy_pass http://0.0.0.0:5000;
  proxy_set_header Host $host:5000;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
 }

nginx is listening on the port 5001 and proxy to 5000, to gunicorn server.

docker listens the port 5000 from outside the docker and sends it to 5001 and everything work fine:

world --5000--docker--5001--nginx--5000--gunicorn

Why didn’t I make nginx listen on the port 5000 in a way that it will work the same inside and outside a docker? It was my first attempt, but gunicorn was redirecting everything to his proper port and it was not possible to use the nice swagger documentation.

Flask RestPlus

Flask-RESTPlus is an extension of Flask that adds support for quickly building REST APIs.

A regular Flask object is instantiated with app = Flask(__name__). This app is given as an argument to Api object: api = Api(app, version=’1.0', title=’house pricing prediction REST API’, description=’RESTfull API house pricing prediction’)

Blueprint permits to setup all the swagger documentation.

We chose to separate everything about the routes in an other file name ai/main_namespace.py and we add it in the api api.add.namespace(main_namespace)

Above is a small part of the code to show how to define a route. You first have to define the Namespace, then the model of the inputs and outputs, then you need to create a class Train that inherits from Resource this class is decorated with route method from your new namespace and the different get post delete put class methods corresponding to the route method. Finally, you can add comments using namespace doc and response methods.

Redis & python RQ

In this project, we want to be able to retrain the model without stopping the service, there are many ways to do it, the key is to use asynchronous programming. Here we chose to use python RQ based on Redis server. It permits to queue jobs and to store the results — the trained model.

On one side, we have a regular Redis server that was just started by supervisor. We have also a python script that simply listens to a Redis queue and executes the jobs.

We will launch this python script via supervisor.

When the server and the worker are up, we can use the queue bellow and all the useful functions that I used in this project:

scikit-learn pipelines

I did nothing complicated for machine learning models, but a clear structure with different pipelines. A pipeline is a combination of some transformers and a model. We select the best model via a cross validation method. From the csv, we only select numeric columns with the command line data = data.select_dtypes(include=[np.number])

the predict function is just a wrapper around the pipeline model. It converts a dictionary of values into a vector and returns the prediction.

Putting it in production, making changes and updating

If you are writing your code in one shot and never read it again, never make it evolve or never reuse it, you can skip this part. But if you plan to make it evolve or if your script will be a part of a bigger project you should read bellow.

Logging and testing are very important in big projects, because everything big is a mess to debug, you want to be able to grab as much information as you can. If you plan to rewrite some parts, but keep all the functionalities you should have all these functionalities written as test.

Logging

In reality, logging is important. When you transfer money, there are transfer records. When an airplane is flying, black box (flight data recorder) is recording everything. If something goes wrong, people can read the log and have a chance to figure out what happened. Likewise, logging is important for system developing, debugging and running. When a program crashes, if there is no logging record, you have little chance to understand what happened. For example, when you are writing a server, logging is necessary.

my logging system is quite simple, in each script part that I want to log I added these 3 lines:

import logging
logging.config.fileConfig('logging.conf')
log = logging.getLogger(__name__)

Where logging.conf is a configuration file (this second line should be in your main script, but it is optional in all your imported modules).
Then, I just have to log strings with 5 logging levels:

log.critical('text')
log.error('text')
log.warning('text')
log.info('text')
log.debug('text')

Then everything is in the config file logging.conf

In this file, there are 3 objects: loggers, handlers, formatters, all defined on the top.
The loggers collect logging ouputs, here there are collected through there name with qualname this name is the one given by logging.getLogger(__name__) using __name__ is convention.
We also defined in the logger the logging level debug is the lower, it means that it logs everything, the level info will not log log.debug.
The handlers specify the display channel (here it is either a file or the stdout. It specifies also witch formatter to use.
The formatters specify the output format.

pytest

Testing applications has become a standard skill set required for any competent developer today. The Python community embraces testing, and even the Python standard library has good inbuilt tools to support testing. In the larger Python ecosystem, there are a lot of testing tools. Pytest stands out among them due to its ease of use and its ability to handle increasingly complex testing needs.

All functionalities of a main function should be tested in all the possible configurations and the python module pytest has been made for it. I created a folder named ai/tests where I put all my testing scripts. Bellow is the script that tests the functions train and predict from the file model.

This script is a very simplified example of testing and in real life each function should be tested separately. For the simplicity, here, I am testing train and predict in the same pytest function.

The 3rd argument target, df and input_columns will take two different values defined by the decorator @pytest.mark.parametrize for a total of 8 combinations. To launch specifically this test, we just have to be in the folder ai and to send this command: pytest tests/test_model.py. To start all the tests just execute pytestand it will start everything in the python script that starts with “test”.

pytest-flask

pytest-flask is a python module to test flask application via pytest. To do so, we just have to add a file conftest.py that initializes the app decorated with @pytest.fixture:

Then we can test the app as above:

Conclusion

That's it for today and congrats! You have learned everything you need know to expose your model to world as a web API. It is a really valuable knowledge because a model hidden from world in your closet is useless. Now, you can easily adapt this full example (see github) to work with your own peculiar model. You can even make it evolve (into AI?) to another big and complex project (and take over the world?).

I would be happy to answer all your questions in comments below and please don't forget to follow our blog dataswati-garage and clap/share with you loved ones. In the future posts of this series, I will show you how to handle other kinds of data (like image, sound or video) through the API REST and expose other kind of models (the next episode might be about Deep Learning with pytorch). Stay tuned and see you then!