Run NVIDIA’s RAPIDS on a eGPU: How to Do Advanced ML/DS on a Laptop
Access cutting edge computing with a GPU on a budget
TL/DR: Go to Section 2 on for the How-To steps.
What are we trying to do and why?
One of the main barriers to entry to learning advanced machine learning and data science is the cost and availability of equipment necessary. Especially when you do not want to use cloud compute resources or are more comfortable with local resources, cost can be prohibitive. Many only have a laptop to use and buying/building a full blown workstation is out of budget.
Already have a relatively new laptop? You can use an eGPU and a NVIDIA GPU to get around these cost issues to do RAPIDS. This package is one of the most advanced ML & data science suites available and leverages graphics cards to significantly accelerate processes. For data science / ML practitioners that have the resources to build a workstation, you can buy a eGPU instead and reallocate the rest of the remaining funds to purchase a better GPU.
In my own work, I do onsite demonstrations and being mobile is key. Sometimes there’s poor or zero no wifi / ethernet. There are projects I’m working on where I’m legally prevented from using the cloud and have to maintain strict data security. So I need to be able to have high compute power, but be able to move around easily, and be able to do setup quickly and easily. The eGPU idea came out of these needs. Also I dumped all my resources into a great NVIDIA GPU card for research when one became available and didn’t have much money left for much else, haha.
1. What do we need?
Starting from the basic, the 3 main things you need are 1) a laptop or computer with USB-C/Thunderbolt connection, 2) an eGPU to contain GPU, and 3) a NVIDIA GPU that meets RAPIDS compute requirements.
If your computer/laptop is not currently running or dual booting Linux, you will need a free USB stick that can install a Linux distribution.
In terms PC hardware requirements, the main thing we need from your computer/laptop is a free USB-C/Thunderbolt connection. There are three different types of USB-C connections, the one we need is the latest one, USB-C 3.1, with the ‘Thunderbolt’ branding. While not as fast as a PCIe 3 connection to the motherboard, it has the speeds necessary for eGPU use. A quick check for USB-C/Thunderbolt is looking for a visual lightning bolt symbol next to the connection. If your USB-C connection is completely unmarked but you were able to connect a monitor to it, look up the motherboard specs of your machine to see if it has Thunderbolt. If the connection can not drive a monitor, the machine is not viable for this project.
The more system RAM you have, the better. This way RAPIDS can offload data to system RAM when CPU functions are called without too heavy a hit on performance. Try to have 2x the amount of RAM as your GPU has VRAM. If you do not (like in the setup shown), you have to pay particular attention to do as much work on the GPU as possible. Which is good policy in general.
RAPIDS runs natively on Linux. Windows 11 runs the suite but for now, we will focus on the standard installation. I will use 64-bit Ubuntu but you can also use the 64-bit CentOS Linux distro.
GPU requirements are that is and NVIDIA card and has enough VRAM for your purposes. In my experience a good rule of thumb is to have 3–4x as much VRAM compared to size of your typical work datasets. The more VRAM on the card, the less work for the user and the system overall.
Note: not every NVIDIA GPU is compatible with RAPIDS. Please check NVIDIA’s product page to make sure your GPU has “compute capability” of 6 or above:
Complete Pre-Reqs for RAPIDS summarized from RAPIDS’ “Getting Started” below but main thing is to focus on the Linux OS and GPU compatibility. Everything else we will handle in the how-to:
GPU: NVIDIA Pascal™ or better with compute capability 6.0+
OS: Ubuntu 18.04/20.04 or CentOS 7/8 with gcc/++ 9.0+
- See RDN 8 for recent changes to
gcc/++9.0 requirements - RHEL 7/8 support is provided through CentOS 7/8 builds/installs
Docker: Docker CE v19.03+ and nvidia-container-toolkit
- Legacy Support — Docker CE v17–18 and nvidia-docker2
CUDA & NVIDIA Drivers: One of the following supported versions:
External GPU (“eGPU”) is how we house the GPU and connect it to the computer/laptop using the USB-C/Thunderbolt connection. Main considerations are the size of eGPU and the power supply it contains. Make sure the eGPU housing’s size can fit the GPU of your choice and meets its electricity demands. Try to get an eGPU with more than the max electricity draw of your GPU. Spring extra for extras like USB connections and the ability to power the laptop. From personal experience, make sure that the eGPU wasn’t optimized for AMD cards and can handle NVIDIA cards. NVIDIA cards that can handle RAPIDS are usually built for high performance, so do not take eGPU compatibility as a given.
1.1 What I’m working with:
For this how-to, I used my 4 year old HP Spectre 360x. I brought it for writing , casual tablet & media use,and honestly nothing else. It has 16 GB of RAM, a 512 GB SSD and a 2 core Intel CPU. This whole project turns this thin, convertible ultrabook into a machine learning beast.
Next thing is the eGPU. External GPUs come in many sizes and forms, you can even build one yourself What I’m working with:
I went ahead and brought the Razer Core X Chroma refurbished off of eBay. Thought it was worth spending more for all the extras. Felt laptop power delivery, 700 Watt power supply, several USB connections and Ethernet cable was definitely worth it. It’s also got some dumb lights.
A spare USB stick was converted to an Ubuntu USB installer. Here are links to make one for Ubuntu and CentOS.
2. Step by Step Procedure
2.1 Preparing the PC/Laptop for Dual Booting Linux & Windows
Go to this great how-to first to familiarize yourself with the process.
I recommend creating a partition on your PC’s hard drive for Linux to live in. From Windows, look up ‘Create and format hard disk partitions’ or hit the Disk Management tool. Choose a disk drive that has ample space for Linux and your machine learning / data science needs and use the “shrink volume” function. I split my hard drive in half and set aside ~250 GB. Reboot your computer and on startup boot from the USB stick with the Linux distro.
If you need help on how to do this try going here. I used a stick with Ubuntu (20.04.2.0) and ran the demo on the laptop. Go ahead and try out the Linux distro. So far nothing will change for your computer. Check to see if everything is working, like mouse, screen(s), etc just to make sure there’s a high level of hardware compatibility. If you have audio jack issues, that unfortunately is common.
Once you’re ready to go, hit the install icon and start the process.
From there do the normal installation
The Linux installer will realize that you have a Windows setup and ask if you want to do a duel boot or erase everything and install Linux only. Go with the duel boot option and install the OS into the free partition created by the ‘shrink volume’ from before.
Ok, now you got a duel booting machine! Let’s poke around
2.2 Understanding Your Setup Before Attaching the eGPU
Let’s first see what the machine looks like before adding the eGPU. Go to ‘Terminal’ and use the following command:
lspci | grep VGA
Here we can see a single GPU, HD Graphics 620 that is on the laptop’s CPU. Let’s put together the eGPU.
2.3 Install the NVIDIA GPU into the eGPU
Follow the instructions included with your eGPU.
It show go something like this: 1) take the enclosure apart, 2) put in the the graphics card on the PCI connector, 3) attach the appropriate power chords from the power supply to the graphics card, 4) close up the enclosure, 5) attach USB-C/Thunderbolt cord from the eGPU to the laptop/pc, 6) attach cord to monitor (DON’T, will explain why later), and 7) attach power cord from outlet to the eGPU enclosure.
Here are some pictures of me in the process:
2.4 Making the proper connections between the eGPU and the Linux OS
Windows will most likely self install the drivers and ask for some permissions once you attach the eGPU to the computer and power it up. Linux will have some extra steps, for security reasons.
First it’ll let you know that something has to be “authorized.”
Go to Settings and search for “Thunderbolt” , that should bring you to the following screen:
Click on both and “authorize and connect” them. Now your eGPU has direct connections to the CPU through PCIe slots.
Close Settings and let’s check if our computer now sees the 2nd GPU using the terminal:
lspci | grep VGA
The NVIDIA graphics card has definitely been found and connected. Most likely it is using the “Nouveau” generic graphics card drivers. Let’s get started on that.
2.5 Setting the right graphics drivers for RAPIDS
First let’s update the operating system fully, using the following command. You can update using the “Software & Updates” utility in the GUI as well.
sudo apt upgrade && sudo apt update
This may require your system to reboot. Go ahead and log back on to Ubuntu.
With the base operating system up to date, we will install the NVIDIA drivers. This is best done using the GUI interface and “Software & Updates” utility.
When it opens, it default opens to “Updates,” as shown below. Click on the “Additional Drivers” tab.
The OS will scan for hardware and generate a list of drivers available for your NVIDIA GPU (below). As expected the system is currently using the default Nouveau display drivers, which are perfectly fine. But RAPIDS needs to use the 460 (proprietary) drivers. Notice this is NOT the latest NVIDIA drivers nor the server drivers. For reliable results do not use those and do not upgrade to the latest (470 as of writing) until its been tested by others to be compatible.
Click on the “Using NVIDIA driver metapackage from ‘nvidia-driver-460 (proprietary)” and then apply changes. Reboot afterwards if the OS asks.
2.6 Final Hardware Check
Log back on to Ubuntu check if everything is in working order. With NVIDIA graphics cards, the most commonly used diagnostic to see if the card is interfacing with the operating system correctly is the following command:
nvidia-smi
If the above screen pulls up on the terminal, everything is working. The computer is using the eGPU and the graphics card inside it!
If this install of linux is not fresh and you had a previous installation of CUDA installed, the CUDA version the system may be using could be different from 11.2. RAPIDS is compatible with CUDA versions 11.0 and 11.2 , you will have to take steps to remove CUDA altogether and install one of those.
2.7 Install Anaconda Python
RAPIDS is a combination of Python and C++ programing, with our eGPU working within Linux, we can now get the package going. First thing to do is install Python in the system. I recommend installing Anaconda or Miniconda as it is very robust and comes with Jupyter Notebook. Go to Anaconda’s homepage and grab the latest install package appropriate for your machine. In most cases it will be the Individual Edition, Python 3.8, 64-bit for x86 systems (scroll all the way down).
If hard drive space is limited, use Miniconda instead. The installer can be found here. Make sure to choose the appropriate Linux package and Python version 3.8.
Go to where the installer file was downloaded to and use the bash function to install the Anaconda file (usually Downloads):
bash [Anaconda3/Miniconda3]-[VERSION/STABLE DATE]-Linux-x86_64.sh
After installing Anaconda or Miniconda, do a quick update:
conda update conda && conda update anaconda
2.8 Install NVIDIA RAPIDS
Now we are ready to install the RAPIDS package. Go to the “Getting Started” page of the RAPIDS homepage and use the RAPIDS release selector applet to get the right code for your machine.
For most people the selector should look like this and yield the code to download it. Below I updated the code to add in Jupyter Lab and also install everything automatically:
conda create -n rapids-21.10 -y -c rapidsai -c nvidia -c conda-forge \
rapids-blazing=21.10 python=3.8 cudatoolkit=11.2 jupyterlab
Apply that code using the terminal and RAPIDS will install.
2.9 RAPIDS Check
Afterwards, go ahead and check to see if everything is working by first activating the RAPIDS environment under Python, then running Juptyer Notebook:
conda activate rapids-21.10
jupyter notebook
With the Jupyter Notebook open, load a dataframe using “cudf” and a csv data file of your choosing. Then use the nvidia-smi function to check that the dataframe is being hosted into the VRAM of your GPU. You can use the following example code. Here is documentation for cuDF, if you have questions on how to import datasets online or offline:
import cudf
gdf = cudf.read_csv(“https://raw.githubusercontent.com/rapidsai/cugraph/branch-21.10/datasets.karate.csv”, delimiter=’ ‘, dtype = [‘int32’, ‘int32’, ‘float32’], header=None)
!nvidia-smi
Here is the example code
Hopefully you see a process running on the GPU, indicating that RAPIDS is running on your system without any problems.
You are now capable of doing next level data science and machine learning using NVIDIA’s RAPIDS!
2.10 Troubleshooting
If there is a compatibility issue with your eGPU, the NVIDIA GPU, and the laptop/PC, try using this guide. I found using the open source “eGPU Switcher” to be helpful when the problem stems from the Linux OS. An NVIDIA Developer blog on using eGPUs was also helpful when the issue had to do with video drivers. Also from personal experience make sure to keep Secure Boot off, here’s how.
3. Notes on using and choosing a eGPU
One thing that I highly recommend users to do is NOT try to push on the GPU eGPU’s USB-C/Thunderbolt IO due to lower bandwidth than what we’re used to with PCI-E connection on the motherboard. RAPIDS excels because it does all your ETL on the GPU and you’ll need to leverage this as much as you can. Use the connections on your PC as much as possible and avoid using the eGPU’s connections. This even includes using the GPU’s video connection(s) too.
- The more you use the videocard directly, the more processes and VRAM is made unavailable for RAPIDS to use. Even with the 24 GB VRAM of my Titan RTX card, the VRAM can be used up very quickly.
- Regardless of how much VRAM your NVIDIA GPU has, if the size of the data you are working with is too large, RAPIDS will start using system RAM. This causes a significant hit on performance. If the Python processes you are running run out available on board RAM and VRAM, Linux will start using the swap file on the hard drive. This is catastrophic in terms of time, speed, and accuracy.
- Always try to minimize data transfers between the computer and GPU as much as possible. As you get serious with using GPUs to do machine learning and data science, managing memory will be a part of project design. You can use this humble eGPU setup to start learning how.
- The more non-GPU data traffic the USB-C/Thunderbolt connection between the laptop/workstation has, the slower everything will be. So don’t use the eGPU’s USB connections for external hard drives, keyboard, mouse or other peripherals. But this doesn’t mean you have to abandon the use of all of the eGPU’s extra connections and features, you can alternate how the connection is being used. A practical example of this that I do regularly is having portable hard drives and ethernet internet transfer data to and from the laptop’s SSD when the GPU is idle
Understand the enclosure you are housing the GPU. My experience with the Razer Core X Chroma really showed me the importance of having a tough, beefy enclosure. I’ve been able to take it everywhere, cars, trains, etc. and it was able to take a pounding while protecting the GPU. There were lighter, smaller options with less metal and at cheaper price points. But having a knock around capable housing really let me be mobile without worry. I was able to carry mine around on a small rolling suitcase without problems or attracting attention. It’s significantly better than carrying around a mid tower PC & projector.
- Maybe I could’ve gotten away with a custom small factor machine. For this project a off the rack eGPU was chosen so more people can replicate it.
- If you don’t plan on taking your eGPU out and about, then toughness is not worth paying extra.
- If you plan on using a NVIDIA GPU that doesn’t have a large electricity draw, you can also use a eGPU with a less powerful power power supply. Always make sure your eGPU’s power supply can push out more than the GPU’s max power draw. This will physically protect all the components and prevent throttling of the GPU.
- Enclosures with extra IO available are not necessary or recommended for data science / ML work but can definitely be a nice to have for other activities.
4. Conclusions
After two months of use I can honestly say that my thin ultrabook built casual has turned into a ML & data science beast. It’s now capable of doing so much, it’s mind blowing. It is viable to first generate a prototype with this setup then use cloud computing to completion using dask and multi-GPU packages. My original plan was to build a custom workstation for work this fall but with this eGPU setup, those plans are delayed by at least a year or two.
I honestly feel like this experiment with an alternative hardware setup has been a success. Much better than my expectations. By using an eGPU instead of building a new workstation, the money savings were redirected to upgrading the GPU used. Which then allowed me to use RAPIDS at a much higher level and do more advanced work, faster. The eGPU also allowed me to be mobile and do presentations in secure environments or ones without internet. I plan to keep using this setup for prototyping and presentation work. In 2–3 years I will build a full workstation and use this as a full time mobile setup, backup work setup.
Using the eGPU for RAPIDS is viable and very cost effective.
Future Projects
RAPIDS recently became compatible with Windows 11, if there is interest I can do a write on how to set that up.
