RAPIDS Accelerates Kubeflow Pipeline with GPUs on Kubernetes
Vartika Singh (NVIDIA), Jeffrey Tseng (PM for RAPIDS, NVIDIA), Pete MacKinnon (Red Hat), Abhishek Gupta (Google)
Data science workflows are complex, non-trivial to manage, and compute intensive. NVIDIA and the Kubeflow team are trying to simplify the process and, at the same time, speed them up.
Yesterday, Kubeflow announced the availability of the RAPIDS GPU-accelerated data science libraries as an image on the Kubeflow Pipelines.
Inherently complicated, data science pipelines span the iterative phases of ingestion, validation, training, deployment, and more. They scale across clusters of servers running software from different parts of the workflow. And they are often compute and IO intensive. All this results in slow machine learning model development and deployment cycles.
The integration of RAPIDS with Kubeflow Pipelines streamlines the model development workflow and drastically decreases end-to-end model iterations times by automating the deployment of open, GPU-accelerated data science tools. Combining the simple orchestration of machine learning pipelines with RAPIDS, a collection of CUDA-accelerated libraries, data scientists can train and deploy machine learning pipelines significantly faster to solve business problems.
Kubeflow is a cloud-native platform for machine learning built on top of Kubernetes that reduces the time to production for machine learning models. It provides development tools for end-to-end machine learning workflows in an environment that is easily translatable to production and a collection of Google developed and other OSS frameworks that allow data scientists and ML practitioners to develop their end-to-end pipelines.
RAPIDS on Kubeflow
Kubeflow allows users to spawn Jupyter Notebooks using pre-built or custom Jupyter runtime environments and deploying them to production as seamlessly as possible. A RAPIDS image using NVIDIA GPUs and RAPIDS libraries, on Kubeflow pipelines, shortens the time to deployment from ingestion.
Users simply need to go to the Kubeflow JupyterHub Spawner interface and select the appropriate container image and specify the resource requirements on the container including the requirement of NVIDIA GPU. The spawner interface lists a few images by default in a dropdown list, but also allows the user to type in a path to an image. For the RAPIDS image, you can paste in gcr.io/kubeflow-dev/kubeflow-rapidsai-notebook:latest.
Once the notebook is ready, users can easily experiment and develop the data transformation and training with RAPIDS.
With the availability of RAPIDS-based Jupyter images, end users can build and execute an accelerated, end-to-end data analytics and machine learning pipeline on Kubeflow and NVIDIA GPUs.
Originally published at medium.com on December 12, 2018.
