Parallel Processing using the Virtru Python SDK
“If you ain’t first, you’re last.” — Ricky Bobby
I have been working with the Virtru Data Protection Platform to solve various real-world problems related to securing data by using the Virtru SDK. In my effort to secure all the data, I have inadvertently neglected application scaling and performance. Encryption and decryption take time, because…MATH!!! This is my first post related to parallel/concurrent processing related to the Virtru SDK. I will be focusing on the Virtru Python SDK in this article.
Virtru Python SDK Background
The Virtru Python SDK shares the same base SDK as the Virtru C++ SDK. The base SDK is written in C++, which helps consolidate the codebase. This is fantastic as the C++ and the Python SDK both have the same calls and approaches to encryption, decryption, and policy management. This also introduces the dreaded GIL (Global Interpreter Lock).
There are 2 approaches that I will explore:
To not repeat how the threading and processes work, I am going to reference this article.
Multithreading, in a python context, means a single process of python runs multiple concurrent threads. In certain areas of Python (IO-bound calls) a thread can wait for IO on one thread and another thread can perform CPU bound tasks. This means that a program will probably not scale linearly for every thread added as not all parts of Python are multithread capable. Let’s look at my example for multithreading the base project.
What I did find is that even if the number of threads was increased, the performance did not follow. Test output for 40 files comparing run time to the number of threads:
Watching the output and computation time, the threads looked like they are executing serially, similar to this image:
Multiprocessing is when multiple processes of Python run which uses more resources but is the only way to accomplish parallel processing using the Virtru Python SDK. After some reading, I figured I would want some control over the concurrency of the application. If for example, I had to encrypt 1 million files, I would probably not be able to create a million python processes on a single host to concurrently encrypt all files. To ensure I didn’t bring my computer to its knees I decided to use a Multiprocessing Pool. By using a pool, I can declare the maximum number of processes I think my computer can support. Other than having to put the proper controls in to ensure there is no contention on files (no file should be processed more than 1 time) I was ready to go.
Bingo! As soon as I started to step up the processes, the performance increased as expected. Test output for 40 files comparing run time to the number of threads:
Watching the output and computation time, the processes looked like they executed similar to this image:
Global Interpreter Lock
GIL (Global Interpreter Lock) is never invited to the party but tends to show up uninvited. As I was chasing my tail trying to encrypt more files faster using Threading and Async/Await…I began to doubt my python chops, but after talking to one of the Virtru Engineers, we came to the conclusion that it was indeed the GIL.
Little did I know when I started how hard it is to figure these issues out. I thought I was just wrong all over the place and I finally started to use my GoogleFu to my advantage by asking the right question. I came to the same conclusion as this post, but “if you don’t know, now you know”. I started to get some additional help from the Virtru engineering team and after giving them my code to reproduce, what I thought was my complete lack of understanding of threading and multiprocessing and programming in general turn into a confirmation that my hunches were indeed correct, the GIL was to blame.