Keith Kraus
May 21 · 3 min read

Like many projects, it has been a struggle for RAPIDS to integrate with the PyPI ecosystem without introducing all sorts of dependency problems for end users. In order to maintain the best user experience, we’ve come to the conclusion that we’ll no longer be able to support PyPI. And beginning with the RAPIDS v0.7 release, we will deprecate PIP installation packages. We did not make this decision lightly, and we strongly feel this is the best path forward for our end users and the Python ecosystem.

Currently, there isn’t a reliable way to integrate modern C++ or CUDA code into the PyPI and PIP ecosystem. The current PIP packaging ecosystem requires a build environment called “manylinux” which is based on CentOS 5 with GCC 4.8.2 which supports C++11 and older. In addition to CentOS 5 no longer being supported by CUDA, RAPIDS is a C++14 codebase which makes the manylinux build environment impossible for us to use to build compliant PIP packages. Other libraries making use of CUDA or modern C++ features, such as Tensorflow, do not use the manylinux build environment. However, they publish packages by incorrectly labeling their packages as manylinux compliant. This causes very erratic behavior and errors for end users. For example:

The Python community is eager to find a solution to these current limitations and issues. The RAPIDS team is also actively working with the community towards finding a viable and sustainable solution. There are numerous discussions on Twitter, Github, and the official Python development forums on this topic with developers voicing their support to help address the problem. For example, check out this thread with Core Python, Apache Arrow, and PyTorch maintainers:

Additionally, here’s another active discussion on GitHub related to PyPI rejecting non-compliant packages from being uploaded:

We will continue to re-evaluate PIP packages and look forward to finding a new specification defined and accepted by the Python community that we can conform to. In the meantime, we strongly encourage other developers to stop publishing non-compliant packages to PyPI and to join the ongoing discussions in driving towards a solution.

If you are looking to use RAPIDS today, we strongly encourage you to check out using Conda as it has been designed from the ground up as a package manager able to handle a combination of Python and modern C++ libraries. We’d love to get your feedback on this topic or others. We encourage you to download our latest RAPIDS Conda packages and and start contributing today!

RAPIDS AI

RAPIDS is a suite of software libraries for executing end-to-end data science & analytics pipelines entirely on GPUs.

Thanks to Mike Beaumont

Keith Kraus

Written by

RAPIDS AI

RAPIDS AI

RAPIDS is a suite of software libraries for executing end-to-end data science & analytics pipelines entirely on GPUs.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade