Conda multi variant builds

We often need to build a package for multiple combinations of dependencies or platforms. For example its common to build python packages for 2.x and 3.x version separately.

Also some low level libraries could require separate packages and the build needs to be done with those specific libraries.

This is done in conda packaging by using something called as build variants, and more about that here.

Any build that needs to be done for multiple combinations is defined in a file called conda_build_config.yaml. The default for this is to be in the same directory as meta.yaml. The exact rules used for searching conda_build_config.yaml is mentioned here.

For example consider the content of conda_build_config.yaml below.

python:
    - 2.7
    - 3.5
    - 3.6
r_base:
    - 3.3.2
    - 3.4.0

This will give upto 6 possible combinations depending which outputs refer to the items in conda_build_config. If something only refers to python, that part will be expanded to 3 build definitions. Something that refers to r_base only will expand to two build definitions. Something that refers to both will expand to 2 times 3 = 6 build definitions.

The values in the conda_build_config.yaml file are available as jinja variables in the meta.yaml file. So for the above build config, you can have the python version in the file name by setting the build string like below.

build:
    string: _py{{ py_version | replace(".", "") }}

This will include a string like _py27, _py35 and _py36 in the file name.

You can also provide build variants at build time like this.

conda build recipe --variants "{'python': ['2.7', '3.5'], 'vc': ['9', '14']}"

For exact process of expanding build variants see here.

Finally two questions remain unanswered from the point of view of the person who will be using the packages.

1. How does one ensure unique file name for every variant?

The answer to this is to generate a hash of the variant values used for that particular build and include that in the build string. Conda does this automatically for you. If you don’t want the hash in the file name, ensure you take care of naming your files to take care of every variant and then use the --old-build-string parameter. More about this is documented here

_py{{ py_version | replace(".", "") }}

2. How does the end user know which variants the package is build with, and try to get the right file?

For this there is a command like below.

conda inspect hash-inputs <package path>

which produces an output like below.

{'python-3.6.4-h6538335_1': {'files': [],
                            'recipe': {'c_compiler': 'vs2015',
                                        'cxx_compiler': 'vs2015'}}}

There are lot many things to be covered but this is enough to get one started. For a full and exhaustive explanation read the official documentation here.