Virtualizing Cuda Environment Using Nvidia-Docker and Deploying APIs

In this quickly evolving domain of Artificial Intelligence and Deep-Learning, advancements are so quick that nobody is able to quickly scale to the latest ones. Many algorithms are there which run on specific versions of Cuda and switching from one version to another is a complete hassle. An easy workaround is to use nvidia-docker to launch compatible container’s which will not only save time, but will also let you use different configurations simultaneously and deploy different micro-services on different environments concurrently.

Prerequisite of using nvidia-docker2 is

• docker-ce
• Remove if already installed :nvidia-docker (Version 1)

For Installing Docker-ce version :

1. Make sure you don’t have any earlier versions. If you have those uninstall using the following command :
   sudo apt-get remove docker docker-engine docker.io
2. Install the Docker Community Edition, using the following command.
   sudo apt-get update
sudo apt-get install docker-ce

Subsequently, we will install nvidia-docker2

1. First, we make sure that we do not have nvidia-docker (Version 1) installed by running these commands.
   docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 \
docker ps -q -a -f volume={} | xargs -r docker rm -f
sudo apt-get purge nvidia-docker
2. Secondly, we add the package repositeries
   curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
3. Next, we install nvidia-docker2 and would reload the docker daemon configuration
   sudo apt-get install nvidia-docker2
sudo pkill -SIGHUP dockerd
4. Try the nvidia-Docker to check if it is running fine
   docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi

nvidia-smi Command’s Output from docker container

Carrying forward, Let us learn how to deploy a hello-world API using apache on this docker container and how to bind it using the IP of host system.

We open a docker container binding port 82 of host system to port 80 of the docker container.

sudo docker run -v <code_dir>:/code/ -p 82:80 \
--name=cuda8_cnn7_image_docker -it --runtime=nvidia [ --rm] \
<cuda_env> /bin/sh

Lets visit the options of the docker run command for our reference.

• --rm : By default a conatainer’s file system persist even after the container exits. To automatically clean up the container and remove the file system when the container exits, we use the — rm flag.
• -v or --volume : We can add -v or — volume flag to bind mount a volume in the following format.
  -v or --volume=[host-src:]container-dest[:<options>]
  Where options can be comma-separated settings for binding a volume
• --name : Provide the name of the container, so that you can attach to the same container easily later.
  BTW, to detach from a container and letting it run in the background, Press and hold <Ctrl>Key and then press P and Q keys and to attach to the same container again. Type in the below command
  sudo docker attach <container’s name>
• -p : Publish a container᾿s port or a range of ports to the host
  format: ip:hostPort:containerPort | ip::containerPort | hostPort:containerPort | containerPort
  Both hostPort and containerPort can be specified as a range of ports.
• <cuda_env> : A complete list of available cuda environment can be found on the below mentioned link :
  https://gitlab.com/nvidia/cuda
• sh : It is the command that we want to run on the Docker’s container when we load it. We would like to open the shell when we open the container. So that we can install the modules we want to.
• -it : Here we have combined two flags, which is used with docker when we want to use interactive shell on docker container.
  -i : Keeps standard input open
  -t : Allocates a pseudo-tty.

You can use the below mentioned command as an example for yourself.

sudo docker run -v “/path/to/code/”:/code/ -p 82:80 --name=cuda8_cnn7_image_docker -it --runtime=nvidia --rm nvidia/cuda:8.0-cudnn7-runtime-ubuntu16.04 /bin/sh

After completion of this downloading and loading of container.
Install the necessary things on the container, but first update the existing packages.
apt-get update

1. Vim : apt-get install -y vim
2. Python : apt-get install -y python3-pip
3. Apache 2: apt-get install -y apache2
4. WSGI : apt-get install -y libapache2-mod-wsgi-py3
   Enable WSGI Module
   a2enmod wsgi
5. Flask : pip3 install flask

Add the following files in the /path/to/code/ directory

1. app.py : This will create a Flask application having endpoints and returning different replies to different endpoints.

from flask import Flask
app = Flask(__name__)
@app.route(“/”)
def hello():
    return “Hello World!”
if __name__ == “__main__”:
    app.run()

2. app.wsgi : The Web Server Gateway Interface (WSGI) is a simple calling convention for web servers to forward requests to web applications or frameworks written in the Python programming language. Here we just import the app variable which we created in the app.py file.

import sys 
sys.path.append(“/code/”) 
‘’’We either need to add this directory in path here or while editing 000-default.conf. ‘’’ 
from app import app as application

Now edit the 000-default.conf configuration file of Apache2 in the container. By default it is in /etc/apache2/sites-enabled/000-default.conf path and add the following lines in the VirtualHost *:80 which actually binds the port 80 of the docker container to the flask application that we are running in the above file using WSGI as the middleware.

WSGIDaemonProcess docker_test python-path=/usr/local/lib/python3.5/dist-packages/
WSGIScriptAlias /nvidia-docker/test /code/app.wsgi
WSGIScriptReloading On
ServerAdmin abc@xyz.com
DocumentRoot /code/
<Directory /code/>
 WSGIProcessGroup docker_test
 WSGIApplicationGroup %{GLOBAL}
 Require all granted
</Directory>

Restart the server now using the following command:
service apache2 restart

Try hitting on the URL using the Host system : Remove if already installed
http://localhost:82/nvidia-docker/test/
OR
http://<private-ip>:82/nvidia-docker/test/

I hope you find this article informative and easy to learn, if you have any queries feel free to reach me at info.ankitp@gmail.com