RAPIDS.ai — Pandas Accelerator Mode for cuDF
Credits: This article as inspired by Avi Chawla’s article on Daily Dose of Data Science, here.
The devs from RAPIDS.ai did a great job. Just with one line of code on your Jupyter Notebook, you may use back your old pandas code and it will automatically accelerate most of the commands. Although the website claims it can be 150x faster, this highly depends on the Nvidia GPU that you have on hand.
Before we get started on the code, lets just see how cudf.pandas work.
Basically, what happens is, when cudf.pandas is enabled, it kind of proxies or links your code to the relevant codes that executes on your GPU. If the proxies are not available, it will fall back on the traditional method of using your CPU to process instead. In short, when available, it will use the GPU, and if not available, it will use the CPU.
Just input two lines of code BEFORE import pandas as pd. One to import the module, the other to load the module extension.
import cudf
%load_ext cudf.pandas
import pandas as pdAvi Chawla has done a simple code to run the it directly on cudf.pandas. The link to his Google Colab can be found here.
I have went a step further to:
1) Time the difference between not using cudf.pandas and using cudf.pandas.
2) Run the code on my local machine. You may find the guide on how to install RAPIDS.ai on your local machine (if you meet the system requirements) in my article titled “RAPIDS.ai — Harnessing Nvidia GPU power via RAPIDS on a Local Windows 11 Installation Guide”.
The code and results are as per the Jupyter Notebook below.
I would like to highlight the importance of loading the module extension of cudf.pandas BEFORE importing pandas as pd. It is clearly seen that if we load the module extension AFTER, pd will not be redirected and there will be no effects / improvements.
The results can be summarized into the below table.
We can see a great improvement in terms of processing time (be it CPU Total Time or Wall Time), when using RAPIDS.
Generally speaking, the speed is 10–50x faster on my setup.
Just for info, my setup is:
1) 11th Gen Intel Core i7–11800H @ 2.3Ghz
2) 16GB RAM
3) Windows 11 Pro
4) Nvidia GeForce RTX 3060 Laptop GPU 6GB
You may also find more information about your Nvidia GPU by typing nvidia-smi on your Windows Command Prompt.
There are also line magic functions that can be used (which I did not include in the code above).
%%cudf.pandas.profile creates a table that shows the GPU / CPU calls for individual functions so that you know which function is the bottleneck.
%%cudf.pandas.line_profile creates a line by line explanation on which line uses GPU / CPU and the time spent for each line.
I would consider the performance (for cuDF.pandas) to be a great step up especially if the dataset is huge and requires huge amounts of computations. The line magic functions also help in better understanding and optimizing the code.
Some additional info:
1) In detail examples by sentdex on youtube. Link
