Like many projects, it has been a struggle for RAPIDS to integrate with the PyPI ecosystem without introducing all sorts of dependency problems for end users. In order to maintain the best user experience, we’ve come to the conclusion that we’ll no longer be able to support PyPI. And beginning with the RAPIDS v0.7 release, we will deprecate PIP installation packages. We did not make this decision lightly, and we strongly feel this is the best path forward for our end users and the Python ecosystem.
Currently, there isn’t a reliable way to integrate modern C++ or CUDA code into the PyPI and PIP ecosystem. The current PIP packaging ecosystem requires a build environment called “manylinux” which is based on CentOS 5 with GCC 4.8.2 which supports C++11 and older. In addition to CentOS 5 no longer being supported by CUDA, RAPIDS is a C++14 codebase which makes the manylinux build environment impossible for us to use to build compliant PIP packages. Other libraries making use of CUDA or modern C++ features, such as Tensorflow, do not use the manylinux build environment. However, they publish packages by incorrectly labeling their packages as manylinux compliant. This causes very erratic behavior and errors for end users. For example:
pyarrow: Segmentation fault (core dumped) · Issue #1450 · apache/arrow
Getting a segmentation fault, when loading this after Keras (see below) Keras version = 2.1.2 tensorflow version =…
The Python community is eager to find a solution to these current limitations and issues. The RAPIDS team is also actively working with the community towards finding a viable and sustainable solution. There are numerous discussions on Twitter, Github, and the official Python development forums on this topic with developers voicing their support to help address the problem. For example, check out this thread with Core Python, Apache Arrow, and PyTorch maintainers:
Additionally, here’s another active discussion on GitHub related to PyPI rejecting non-compliant packages from being uploaded:
Run auditwheel on new manylinux uploads, reject if it fails · Issue #5420 · pypa/warehouse
What's the problem this feature will solve? Projects are purposely uploading invalid manylinux1 wheels to PyPI which…
We will continue to re-evaluate PIP packages and look forward to finding a new specification defined and accepted by the Python community that we can conform to. In the meantime, we strongly encourage other developers to stop publishing non-compliant packages to PyPI and to join the ongoing discussions in driving towards a solution.
If you are looking to use RAPIDS today, we strongly encourage you to check out using Conda as it has been designed from the ground up as a package manager able to handle a combination of Python and modern C++ libraries. We’d love to get your feedback on this topic or others. We encourage you to download our latest RAPIDS Conda packages and and start contributing today!