Understanding the Workflow of TorchServe using Densenet
This will be a series of articles to understand, use and customize TorchServe for deploying PyTorch models. This first post is about getting familiar with the workflow. For that, we will use a pre-trained model and only consider what happens when our model is ready for deployment. The other articles of this series will cover the following:
- 👉 Part II — Understanding the Base Handler
- 👉 Part III — MNIST Example with a customized Handler Script
What is Torchserve?
To get started with TorchServe, we will clone their GitHub repository [2].
git clone https://github.com/pytorch/serve.git
We will use the densenet model as an example. You can find much more examples in the repository. I highly recommend reading through it.
Safe Densenet Model
We will use a pre-trained densenet to understand the workflow of deployment. If you want to have the same folder structure as in this post create the folders as indicated below and save the model there. We can download it using wget.
mkdir densenet_example
cd densenet_example
mkdir models
cd models
wget https://download.pytorch.org/models/densenet161-8d451a50.pth
Other options for saving the model are via an executable script module or a traced script as described in the TorchServe GitHub repository.
Save the Model as an archived File
In order to deploy the model, we need to create a “.mar” file of it. This is the file we use for deployment and it contains all the information needed in order to do this, e.g. the model script file and the state-dict of the model. We will go through the needed files in a bit, but first, we need to install the needed dependencies in order to create this file. The easiest way to do that is to move to the cloned git repository and get all the needed dependencies using the following commands:
cd serve
pip install .
cd model-archiver
pip install .
You can also install the needed dependencies directly, e.g.
pip install torchserve torch-model-archiver torch-workflow-archiver
Now we can create the “.mar” file using the following command.
torch-model-archiver --model-name <your_model_name> \
--version 1.0 \
--model-file <your_model_file>.py \
--serialized-file <your_model_name>.pth \
--handler <handler-script>
--extra-files ./index_to_name.json
Explanations:
- — model-name: Model name
- — version: Define version number (optional)
- — model-file: Python model script (.py-file)
- — serialized-file: Location of the model file, which contains the state-dict of the model (.pth-file)
- — extra-files: Other documents needed for the handler script
- — handler: Script, that defines preprocessing, inference, and postprocessing steps of the model (.py-file)
When our model is ready for deployment, we have a model script (model-file) and also a saved state-dict (serialized-file). For this post, we downloaded the saved state-dict. The model script, we will copy from the git repo we cloned. The handler is an important concept, it handles how to preprocess data, apply the model and how to postprocess the output of the model. TorchServe provides a BaseHandler class, which can be customized for individual needs. In the next post of this series, we will go through this class in detail. TorchServe additionally offers default handlers for the most important tasks such as image classification, object detection, text classification, or image segmentation. A complete list of default handlers can be found in the TorchServe documentation. In this example, we are considering an image classification task. The default image classification handler is suitable for models trained on ImageNet dataset. The output of this handler is the top 5 predictions and their respective probability of the image [3].
For the image classification handler we need an additional document called “index_to_name.json”. This is a dictionary that maps the predicted index to the class. Before copying the model script and the dictionary we have to change the rights.
cd serve/examples/image_classifier/densenet_161/
chmod +rwx model.py
cd serve/examples/image_classifier/
chmod +rwx index_to_name.json
Then move back to our created folder and copy the files.
cd densenet_example
mkdir models
cd models
cp ../../serve/examples/image_classifier/densenet_161/model.py .
cd ..
cp ../serve/examples/image_classifier/index_to_name.json .
For our example the command to create the “.mar”- file looks like this.
torch-model-archiver --model-name densenet161 \
--version 1.0 \
--model-file models/model.py \
--serialized-file densenet161-8d451a50.pth \
--extra-files index_to_name.json \
--handler image_classifier
Next, create a folder “model-store” and move the densenet161.mar file into it.
mkdir model-store
mv densenet161.mar model-store/
The final folder structure looks like this.
Deploy the Model using Docker
We will use docker to deploy the model. For that, we download the latest image for TorchServe.
docker pull pytorch/torchserve:latest
For more details — also on specific images refer to the docker section in the TorchServe git repository. (There is also described how to create the .mar-file directly in docker.) With our created files we can then run this image.
docker run --rm -it -p 8080:8080 -p 8081:8081 \
--name mar \
-v $(pwd)/model-store:/home/model-server/model-store \
pytorch/torchserve:latest \
torchserve --start \
--model-store model-store \
--models densenet161=densenet161.mar
We map to ports 8080 and 8081 for predictions and models, which are the default ones. We further include our model-store folder. You can check which models are available using this command.
curl http://localhost:8081/models
This should output
{
"models": [
{
"modelName": "densenet161",
"modelUrl": "densenet161.mar"
}
]
}
Apply the Model
Now we can use the model to make predictions. The densenet model is trained on ImageNet, which contains 1000 different classes. A list of the classes can be found on WekaDeeplearning4j. We will apply it to see, if it can classify different snakes, but you can choose any other classes from the list. Let’s have a look at some example images.
We can then make predictions via:
curl http://localhost:8080/predictions/densenet161 -T david-clode-Ws6Tb1cI0co-unsplash.jpg
This gives the result:
{
"rock_python": 0.4521763324737549,
"night_snake": 0.35173436999320984,
"king_snake": 0.10417615622282028,
"boa_constrictor": 0.04921364784240723,
"sidewinder": 0.02006538398563862
}
Nice! The model’s highest predictions are all for snakes and the highest is “rock_python”. I’m no expert on snakes, but the image was described as “carpet python”. Although the model is not very certain for this image as the highest probability is only 0.45. Let’s consider another example:
The prediction is analogue to the previous example:
curl http://localhost:8080/predictions/densenet161 -T alfonso-castro-HaGwCk2AD84-unsplash.jpg
This time we get the following result:
{
"green_snake": 0.7169939279556274,
"green_mamba": 0.1878765970468521,
"vine_snake": 0.02244602143764496,
"night_snake": 0.01946556195616722,
"green_lizard": 0.007101472932845354
}
Fantastic! The prediction is “green_snake”, this is also how the image was described. Next one:
Here are the results:
{
"Indian_cobra": 0.9904808402061462
"water_snake": 0.005116844084113836,
"thunder_snake": 0.002371615031734109,
"night_snake": 0.000596951344050467,
"sea_snake": 0.0005211854004301131
}
This image was described as “cobra”, so this prediction is almost perfect. Let’s have a look at one last example:
This pretty yellow snake is classified as follows:
{
"boa_constrictor": 0.4590628147125244,
"rock_python": 0.34132421016693115,
"horned_viper": 0.12637333571910858,
"sidewinder": 0.025241760537028313,
"king_snake": 0.024632452055811882
}
In this case the model is not very certain as in the first example. The highest prediction probability is only 0.45. As far as I understand it this yellow snake is also called “Banana Boa Constrictor”. This is no class the model has been trained on. That is the model even recognized it as a Boa Constrictor although the color is unusual.
That’s it! 💥 That is the workflow of model deployment using TorchServe.
👉 Have a look at the next episodes of this series. There we will have a closer look at the handler script, which we need to customize when we want to use our own models.🚀
Further Recommendations for getting started with TorchServe
- Shashank Prasanna, Introduction to TorchServe, an open source model serving library for PyTorch, https://youtu.be/AIrrI8WOIuk
- https://cceyda.github.io/blog/torchserve/streamlit/dashboard/2020/10/15/torchserve.html
References
- [1] AWS, Introducing TorchServe: a PyTorch model serving framework (2020), https://aws.amazon.com/about-aws/whats-new/2020/04/introducing-torchserve/
- [2] PyTorch, TorchServe Examples (2022), https://github.com/pytorch/serve/tree/master/examples
- [3] PyTorch, TorchServe default infrence handlers (2020), https://pytorch.org/serve/default_handlers.html
Find more Data Science and Machine Learning posts here: