Publish a python package to Conda

https://github.com/conda/conda

Conda is a python package manager similar to pip. If you are working on building a python library then it’s highly likely you will be publishing it to Conda as well. Otherwise, Conda users won’t benefit from your library which is a negative point for your users and you as well. Conda documentation has step by steps on how to publish a python package to Conda from scratch. But if you have no prior experience with Conda publishing then you will face a lot of troubles and the official documentation will not be much helpful for you. Here I am going to explain each and every step in detail which will help to clarify all of your problems.

What will you need?

You need to have these tools installed on your build environment.

1. Miniconda
2. conda-build
3. anaconda-client

Let’s start!

Let’s assume I’m going to publish the library named my-py-lib to Conda.

The most important task is preparing the meta.yamlfile for your python library. This is the only required file for Conda publishing except for setup.py file. Yes, the documentation says build.sh and build.bat are also required, but trust me it’s not! You can use the same setup.py file you used for pip, to publish to Conda as well. You can place the meta.yaml file anywhere you want. But it’s always a good idea to create a new directory called conda(or with any other name you prefer) in the root directory of your project and place meta.yaml in it. Here is my project structure.

.
|-conda
| |-meta.yaml
|-setup.py

Here is the bare minimum meta.yaml file required to publish a Python package to Conda.

meta.yaml

package:
  name: "my-py-lib"
  version: "0.1.0"
about:
  summary: "This is an awesome Python library"
source:
  path: ..
build:
  script: python setup.py install

Important points:

• source.path - should be the relative path from your meta.yaml to setup.py file.
• build.script - is the command to build your python distribution. This can be varied depending on your project.

Now you are ready to publish your package to conda. Execute this command to build the distribution from the project root directory.

conda build --output-folder ./conda-out/ ./conda/

Input argument for conda build should be the directory where your meta.yaml file is located.

If you have enabled anaconda upload in your build environment this will upload the package to anaconda after successfully building it. If you have disabled anaconda upload then you may have to run the following command to upload the distribution to anaconda (Update the path of your tar.bz2 bundle as necessary. If you haven’t login to anaconda from your CLI before then you have to set up it as described here first.

anaconda upload ./conda-out/osx-64/my-py-lib-0.1.0-0.tar.bz2

Congratulations! Your package should be now available in Anaconda. Let’s dive into some advance configuration options you might need when publishing your package to conda.

Build for multiple python versions

There might be special requirements with your library where you require the same python version at runtime which you used at the build time. Let’s assume you are using python 3.8 and you need to install a variant of your library which was built using the same python version which is 3.8. This can be achieved with version pinning. Theconda_build_config.yaml file will let you do this easily.

conda_build_config.yaml

Create a file named as conda_build_config.yaml in the same directory where your meta.yaml file is located. Creating a separate directory as I said before (in my case condadirectory) for conda specific files will be handy when you have more files besides meta.yaml. Here is my conda_build_config.yaml

python_version:
  - 3.7
  - 3.8
  - 3.9

I want my library to be built for python 3.7, 3.8 and 3.9 versions separately. Then you have to refer these python versions in your meta.yaml. Here is my updated meta.yaml.

package:
  name: "my-py-lib"
  version: "0.1.0"
about:
  summary: "This is an awesome Python library"
  
requirements:
  build:
    - python={{ python_version }}
  run:
    - python={{ python_version }}
source:
  path: ..
build:
  script: python setup.py install
  string: py{{ python_version | replace(".", "") }}

If you want to pin the versions of other dependencies as well, then you can include them also in the meta.yaml and conda_build_config.yaml. Now if you run the conda build command you will see multiple build variants in your output directory. If you want you can modify the build variants’ naming pattern by changing the build.string in meta.yaml.

Build for multiple platforms

You might have noticed that when you execute the conda build in the previous section, your build artefact should have been created in a sub-directory with the platform name of your build environment (i.e osx-64 , linux-64). That means after you pushing this build to the anaconda, this can be downloaded only by the same platform. Therefore if you want to support multiple platforms then you have to convert this build artefact for other platforms and upload to anaconda again. You can easily do that with the following command. I am converting the artefact built for osx-64 to linux-64here.

conda convert --platform linux-64 ./conda-out/osx-64/my-py-lib-0.1.0-0.tar.bz2  -o ./conda-out

Then you have to upload this again.

anaconda upload ./conda-out/linux-64/my-py-lib-0.1.0-0.tar.bz2

You can find all supported platforms here. But you have to do this if and only if your library is platform dependent. If it’s not you can build that as an architecture independent package by updating the buildsection of your meta.yaml as follows. Look into docs for more options.

build:
  noarch: python

Use variables in meta.yaml

There may be many situations where you need to define variables and read environment variables when defining your meta.yaml. Thanks to Jinja2 support in meta.yaml it’s possible.

Use environment variables

package:
  name: "my-py-lib"
  version: "{{ VERSION }}"

conda build will look for the VERSION environment variable in the build environment as the version value.

Read setup.py arguments.

{% set data = load_setup_py_data() %}

package:
  name: "my-py-lib"
  version: {{ data.get('version') }}

conda build will extract the value of version field of setup.pyas the version value.

Support complex build scripts

If you have a complex build script you can remove build.script section from meta.yamland create a separate build.sh file.

build.sh

Create a build.sh file in the same directory where your meta.yaml file is located. Write your build steps in this file. build.sh is only for Unix like build environments. If you are building on a Windows environment then you have to create a build.bat file complying with Windows shell standards. Look into the Conda documentation for more info.

#!/usr/bin/env bash
$PYTHON setup.py install

Extra Notes

• For full syntax reference of meta.yaml- Defining metadata
• More details on conda_build_config.yaml- Build variants
• If you are using Github actions there are many prebuilt actions.