Advanced Pytorch Practices&Functions #1

Introduction

Deep learning achieved incredible success in artificial intelligence literature. We owe a huge thanks to deep learning frameworks and the mindset of open source coding, of course alongside the increase in GPU power and data. Pytorch is one of the most used deep learning frameworks where it allows the creation of deep models so easily. Basics of Pytorch can be learned quickly, you can search Pytorch tutorial, and many examples/tutorials/practices will come to us. However further steps may be difficult. In this series, I will try to explain some advanced functions or usage of Pytorch. The concept is describing some Pytorch functions (2–5 per blog) which are rarely used but are effective.

If you are good at Pytorch but don’t know how to go further, this series is the right one to follow.

Pytorch Profiler

Since we don’t have an unlimited computational source, it is very critical to trace the memory of both GPU and CPU. Pytorch Profiler traces the memory during training and inference. Although it is possible to use some other profilers, Pytorch one provides good functionalities.

In the above example, a Resnet18 model is defined and a Pytorch profiler is created to record model activity. Output:

As seen in the above figure, a very detailed tracing is provided by Pytorch Profiler. GPU version is also available.

register_buffer()

register_buffer() is a function that defines a value in the model, but this value is not a parameter, it is like a constant in the model. It is more efficient to define variables with this function if the defined variable is not learnable.

There are 2 good explanations (1, 2). In summary, if a variable is not a model parameter, it is more efficient to use this function.

Conclusion

In this blog, I mentioned 2 different Pytorch usages. In the following series of this blog, I will explain and mention other rarely used things in Pytorch.