Install RAPIDS and Jupyter Notebook on Google Cloud Platform

Process data using GPU using Google Cloud platform

Photo by sergio souza on Unsplash

Go to your Google Cloud Platform Console https://console.cloud.google.com/

Find the Cloud Shell

In your upper right corner, you will find the button to activate the Cloud Shell

Create a VM Instances

Run the following command to create and initialize the instances:

export IMAGE_FAMILY="rapids-latest-gpu-experimental"
export ZONE="us-west1-b"
export INSTANCE_NAME="rapids"

gcloud compute instances create $INSTANCE_NAME \
  --zone=$ZONE \
  --image-family=$IMAGE_FAMILY \
  --image-project=deeplearning-platform-release \
  --maintenance-policy=TERMINATE \
  --accelerator="type=nvidia-tesla-t4,count=1" \
  --metadata="install-nvidia-driver=True"

Connect to the instances via SSH

Go to the https://console.cloud.google.com/compute/instances

Updating RAPIDS

The GCP RAPIDS Image for January 2020 is 0.7 a very outdated one.

We are going to create a Conda environment, and update RAPIDS

>> conda create --name rapids_blazing python=3.7
>> conda activate rapids_blazing
>> conda install -c rapidsai -c nvidia -c numba -c conda-forge \
    cudf=0.13 python=3.7 cudatoolkit=10.0
>> conda install --name cudf -c rapidsai -c nvidia -c conda-forge -c defaults dask-cudf=0.13
>> conda install -c blazingsql/label/cuda10.0 -c blazingsql -c rapidsai -c nvidia -c conda-forge -c defaults blazingsql python=3.7 cudatoolkit=10.0

You need to install Jupyter in this new Conda environment

>> conda install -c conda-forge jupyterlab
>> ipython kernel install --user --name=rapids_blazing

Now, through the instance terminal run this command

>> jupyter notebook --ip=0.0.0.0 --port=8888

Copy the URL generated in the terminal and replace the 127.0.01 with the external IP of your instance.
If you have problems entering the Jupyter Notebook, create the following rule in https://console.cloud.google.com/networking/firewalls/list

If you found this tutorial outdated, please let us know.

Opening port to see Dask status

Go to Setup firewall rules

Create a new firewall rule

On Targets select All instances in the network and in tcp select enter 8787

Troubleshooting

Libraries not found in Jupyter Notebook

In some cases, jupyter notebook can not detect the libraries just installed. If it happened to you, check that the python path is pointing to the correct conda environment.

From the notebook try:

import sys 
print(sys.executable)

If the path is correct you should see something like

/home/xxxxxx/.conda/envs/rapids_blazing/bin/python

If not, get your kernel path using

jupyter kernelspec list

find kernel.json and edit using nano . You should find something like

{
 "argv": [
  "/home/xxxxxx/.conda/envs/rapids_blazing/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "rapids_blazing",
 "language": "python"
}

Be sure that the path to Conda env is correct. you can find your Conda’s dev

conda info --envs

More info https://github.com/jupyter/notebook/issues/1524

Not enough GPUs

For use GPUs, you need to expand your quota sending an email to Google

Please navigate to https://console.cloud.google.com/iam-admin/quotas

Select your project:

Filter by Metric and be sure you select your GPU type

Select Compute Engine API and press EDIT QUOTAS . A sidebar will appear on the left. Fill in the blanks, submit your data. You will receive an email about the request status.

Add more GPUs

First, stop the instance

Then, click on the instance to get into the detail and press Edit

Press CPU platform and GPU and select the GPUs you want, save the changes.

Restart the instance.

External IP

If you want to use and static IP you can create one and assign it to your VM. Select your project and go to VPC network -> External IP addresses and click Reserve Static Adress

You are done.