Understanding the Workflow of TorchServe using Densenet

TorchServe for deploying PyTorch Models — Part I

Photo by Josh Olalde on Unsplash

This will be a series of articles to understand, use and customize TorchServe for deploying PyTorch models. This first post is about getting familiar with the workflow. For that, we will use a pre-trained model and only consider what happens when our model is ready for deployment. The other articles of this series will cover the following:

• 👉 Part II — Understanding the Base Handler
• 👉 Part III — MNIST Example with a customized Handler Script

What is Torchserve?

TorchServe is an open-source model serving framework for PyTorch that makes it easy to deploy trained PyTorch models performantly at scale without having to write custom code. [1]

To get started with TorchServe, we will clone their GitHub repository [2].

git clone https://github.com/pytorch/serve.git

We will use the densenet model as an example. You can find much more examples in the repository. I highly recommend reading through it.

Safe Densenet Model

We will use a pre-trained densenet to understand the workflow of deployment. If you want to have the same folder structure as in this post create the folders as indicated below and save the model there. We can download it using wget.

mkdir densenet_example
cd densenet_example
mkdir models
cd models
wget https://download.pytorch.org/models/densenet161-8d451a50.pth

Other options for saving the model are via an executable script module or a traced script as described in the TorchServe GitHub repository.

Save the Model as an archived File

In order to deploy the model, we need to create a “.mar” file of it. This is the file we use for deployment and it contains all the information needed in order to do this, e.g. the model script file and the state-dict of the model. We will go through the needed files in a bit, but first, we need to install the needed dependencies in order to create this file. The easiest way to do that is to move to the cloned git repository and get all the needed dependencies using the following commands:

cd serve
pip install .

cd model-archiver
pip install .

You can also install the needed dependencies directly, e.g.

pip install torchserve torch-model-archiver torch-workflow-archiver

Now we can create the “.mar” file using the following command.

torch-model-archiver --model-name <your_model_name> \
                     --version 1.0 \
                     --model-file <your_model_file>.py \ 
                     --serialized-file <your_model_name>.pth \
                     --handler <handler-script> 
                     --extra-files ./index_to_name.json

Explanations:

• — model-name: Model name
• — version: Define version number (optional)
• — model-file: Python model script (.py-file)
• — serialized-file: Location of the model file, which contains the state-dict of the model (.pth-file)
• — extra-files: Other documents needed for the handler script
• — handler: Script, that defines preprocessing, inference, and postprocessing steps of the model (.py-file)

When our model is ready for deployment, we have a model script (model-file) and also a saved state-dict (serialized-file). For this post, we downloaded the saved state-dict. The model script, we will copy from the git repo we cloned. The handler is an important concept, it handles how to preprocess data, apply the model and how to postprocess the output of the model. TorchServe provides a BaseHandler class, which can be customized for individual needs. In the next post of this series, we will go through this class in detail. TorchServe additionally offers default handlers for the most important tasks such as image classification, object detection, text classification, or image segmentation. A complete list of default handlers can be found in the TorchServe documentation. In this example, we are considering an image classification task. The default image classification handler is suitable for models trained on ImageNet dataset. The output of this handler is the top 5 predictions and their respective probability of the image [3].

For the image classification handler we need an additional document called “index_to_name.json”. This is a dictionary that maps the predicted index to the class. Before copying the model script and the dictionary we have to change the rights.

cd serve/examples/image_classifier/densenet_161/
chmod +rwx model.py
cd serve/examples/image_classifier/
chmod +rwx index_to_name.json

Then move back to our created folder and copy the files.

cd densenet_example
mkdir models
cd models
cp ../../serve/examples/image_classifier/densenet_161/model.py .
cd ..
cp ../serve/examples/image_classifier/index_to_name.json .

For our example the command to create the “.mar”- file looks like this.

torch-model-archiver --model-name densenet161 \
                     --version 1.0 \
	             --model-file models/model.py \  
		     --serialized-file densenet161-8d451a50.pth \   
                     --extra-files index_to_name.json \
                     --handler image_classifier

Next, create a folder “model-store” and move the densenet161.mar file into it.

mkdir model-store
mv densenet161.mar model-store/

The final folder structure looks like this.

Folder structure

Deploy the Model using Docker

We will use docker to deploy the model. For that, we download the latest image for TorchServe.

docker pull pytorch/torchserve:latest

For more details — also on specific images refer to the docker section in the TorchServe git repository. (There is also described how to create the .mar-file directly in docker.) With our created files we can then run this image.

docker run --rm -it -p 8080:8080 -p 8081:8081 \
           --name mar \ 
           -v $(pwd)/model-store:/home/model-server/model-store \
           pytorch/torchserve:latest \
           torchserve --start \ 
           --model-store model-store \
           --models densenet161=densenet161.mar

We map to ports 8080 and 8081 for predictions and models, which are the default ones. We further include our model-store folder. You can check which models are available using this command.

curl http://localhost:8081/models

This should output

{
  "models": [
    {
      "modelName": "densenet161",
      "modelUrl": "densenet161.mar"
    }
  ]
}

Apply the Model

Now we can use the model to make predictions. The densenet model is trained on ImageNet, which contains 1000 different classes. A list of the classes can be found on WekaDeeplearning4j. We will apply it to see, if it can classify different snakes, but you can choose any other classes from the list. Let’s have a look at some example images.

Photo by David Clode on Unsplash

We can then make predictions via:

curl http://localhost:8080/predictions/densenet161 -T david-clode-Ws6Tb1cI0co-unsplash.jpg

This gives the result:

{
"rock_python": 0.4521763324737549,
"night_snake": 0.35173436999320984,
"king_snake": 0.10417615622282028,
"boa_constrictor": 0.04921364784240723,
"sidewinder": 0.02006538398563862
}

Nice! The model’s highest predictions are all for snakes and the highest is “rock_python”. I’m no expert on snakes, but the image was described as “carpet python”. Although the model is not very certain for this image as the highest probability is only 0.45. Let’s consider another example:

Photo by Alfonso Castro on Unsplash

The prediction is analogue to the previous example:

curl http://localhost:8080/predictions/densenet161 -T alfonso-castro-HaGwCk2AD84-unsplash.jpg

This time we get the following result:

{
"green_snake": 0.7169939279556274,
"green_mamba": 0.1878765970468521,
"vine_snake": 0.02244602143764496,
"night_snake": 0.01946556195616722,
"green_lizard": 0.007101472932845354
}

Fantastic! The prediction is “green_snake”, this is also how the image was described. Next one:

Photo by Mohan Moolepetlu on Unsplash

Here are the results:

{
"Indian_cobra": 0.9904808402061462
"water_snake": 0.005116844084113836,
"thunder_snake": 0.002371615031734109,
"night_snake": 0.000596951344050467,
"sea_snake": 0.0005211854004301131
}

This image was described as “cobra”, so this prediction is almost perfect. Let’s have a look at one last example:

Photo by Timothy Dykes on Unsplash

This pretty yellow snake is classified as follows:

{
"boa_constrictor": 0.4590628147125244,
"rock_python": 0.34132421016693115,
"horned_viper": 0.12637333571910858,
"sidewinder": 0.025241760537028313,
"king_snake": 0.024632452055811882
}

In this case the model is not very certain as in the first example. The highest prediction probability is only 0.45. As far as I understand it this yellow snake is also called “Banana Boa Constrictor”. This is no class the model has been trained on. That is the model even recognized it as a Boa Constrictor although the color is unusual.

That’s it! 💥 That is the workflow of model deployment using TorchServe.

👉 Have a look at the next episodes of this series. There we will have a closer look at the handler script, which we need to customize when we want to use our own models.🚀

🔥Understanding TorchServe’s BaseHandler

🔥Use TorchServe with a customized handler script

Further Recommendations for getting started with TorchServe

• Shashank Prasanna, Introduction to TorchServe, an open source model serving library for PyTorch, https://youtu.be/AIrrI8WOIuk
• https://cceyda.github.io/blog/torchserve/streamlit/dashboard/2020/10/15/torchserve.html

References

• [1] AWS, Introducing TorchServe: a PyTorch model serving framework (2020), https://aws.amazon.com/about-aws/whats-new/2020/04/introducing-torchserve/
• [2] PyTorch, TorchServe Examples (2022), https://github.com/pytorch/serve/tree/master/examples
• [3] PyTorch, TorchServe default infrence handlers (2020), https://pytorch.org/serve/default_handlers.html

Find more Data Science and Machine Learning posts here:

More

Get an email whenever Pumaline publishes.

Mlearning.ai Submission Suggestions