How to use Conan to manage OpenCV binaries in GitLab?

Introduction

Using a package manager makes dependency management much more simple. Package managers for different languages are plentiful: pip, conda, poetry, npm, maven, gradle etc. and using them is a common practice. But things change when you are dealing with C or C++ projects. Package management for C/C++ becomes much more complicated when you need to manage the source code of the package as well as the binaries. Using a package manager in a C/C++ project is not so common as in Python or JavaScript. However, the good news is that nowadays there exist several package managers for C/C++: buckaroo, hunter, dds, vcpkg, conan. Conan has advantages over other managers because it manages binaries and has integration in GitLab. In this article, we demonstrate how to use Conan for managing the OpenCV library.

What is Conan?

Conan is a free open source package manager for C and C++ projects with the permissive MIT license. One of the most powerful features of Conan is that it can manage pre-compiled binaries. Using pre-compiled binaries saves significant time for developers and CI servers, while also improving the reproducibility and traceability of artifacts. All projects managed by Conan are stored in packages. Every package has a package reference in the form package_name/version@user/channel:package_id_hash. The “package_name” is the name of your library. The “version” is the version of your library. The “user” is used to avoid collisions of libraries with the same names and versions. It could be associated with the name of the creator or your company. The “channel” is used to show the state of the package development. If it is equal to “testing” it means that the package is not stable yet. When you are sure that the package will not be changed further, you can change “channel” to “stable”. As the Conan documentation says:

It is strongly recommended that packages are considered immutable. Once a package has been created with a user/channel, it shouldn’t be changed. Instead, a new package with a new user/channel should be created.

The structure of the Conan package

The main file in the package is the receipt. It is a Python script with the name conanfile.py. It specifies how to build binaries from the sources, how to store the artifacts of the build, the dependencies, etc. The build configuration is ruled by the settings and options. The settings is a set of keys and values, like the operating system, compiler, build type and are declared in the settings.yml file. The options are similar to the settings except they could be defined by the recipe creator. A set of different settings, options, environment variables and tool requirements used when working with packages is called profile. A Conan package contains the recipe, the source code and the binary packages generated for different configurations of options and settings. Every binary package has a unique id generated as a hash of the settings, options and requirements. It is called the package id.

Conan is decentralized and based on the client-server model. The main purpose of the server part is to store the packages. It does not build them from the source. The client parts are used for downloading and uploading packages. If binaries are built from the source the compilation is done by the client application.

Conan client-server model

The client application is a command-line application. The Conan client application is written in Python and you should notice that starting from 1.22 version Python 2 is no longer supported. It could be installed via pip and it is the recommended way.

As for the server part you have different opportunities where to store and where to download the packages from:

1. The recommended Conan server is JFrog Artifactory Community Edition. You can use it for creating your private server. It is a free community edition of JFrog Artifactory for Conan packages, including a WebUI, multiple auth protocols, Virtual and Remote repositories to create advanced topologies, a Rest API, and generic repositories to host any artifact.
2. The conan_server is a small server distributed together with the Conan client. It is a simple open-source implementation and provides basic functionality, but no WebUI or other advanced features.
3. ConanCenter is a central public repository where the community contributes packages for popular open-source libraries. It contains around 1200 packages. All recipes are stored in the conan-center-index repository. It is actively developed and grows very quiсkly.
4. The most popular CI platforms such as Travis CI, Appveyour, Circle CI and GitLab have the Conan packages repository support. We suppose that it is a very good choice to store Conan packages in the same CI/CD platform along with the source code.

The ability to manage not only the source code but also the binaries and integration with the GitLab CI platform are the main reasons why we have chosen the Conan package manager. So now let’s find out how to use it. As an example, we take one of the most popular libraries in the field of computer vision — OpenCV. We are using GitLab 13.12.15 and Conan 1.43.0.

How to manage the OpenCV library package

OpenCV contains more than 2500 optimized algorithms. It is used by many well known companies like Google, Yahoo, Microsoft, Intel, IBM, Sony, Honda, Toyota.

We also use this library and sometimes use the DNN inference module. It could be compiled with CUDA backend and it is sometimes useful (when you are not sure about the device version) to use it instead of TensorRt that is provided by Nvidia.

Conan-center-index repository contains around 1200 recipes of the most popular libraries as well as a receipt for the OpenCV. However, unfortunately, it does not mean that ConanCenter contains all binary packages for all combinations of the settings and options. For example, it does not contain binaries with CUDA. We need to build it ourselves to be able to store and reuse them.

So let’s concentrate on fundamental aspects required for using Conan binary packages for OpenCV with DNN module stored in GitLab. As developers, we would like to achieve the following goals:

1. Build OpenCV with the DNN module w/ and w/o CUDA support.
2. Use the GitLab Package registry for storing the OpenCV Conan package.
3. Be able to load OpenCV binaries from GitLab.

Let’s go step by step to achieve those goals.

Generate an access token in GitHub

We want to use source code from GitHub and first we need to create an access token. We can do it by clicking on the profile image and choosing “Settings” -> “Developer settings” -> “Personal access tokens” -> “Generate new token”.

You need to set the “Note” field (what this token is for) and the expiration date. After pressing the “Generate token” button the token will appear. Save it somewhere because you will have no chance to restore it!

Fork the conan-center-index repository

We need to fork the conan-center-index to be able to import it into GitLab. Fork it by pressing the “Fork” button in GitHub web GUI. We fork the whole conan-center-index repository for simplicity but you can cut only the OpenCV receipt and upload it into GitLab as a separate project.

Import the conan-center-index repository into GitLab

Go to the GitLab and create a project for the conan-center-index repository. Select “New project” and choose “Import project”.

Fill in the access token and press the “Authenticate” button.

Select the conan-center-index repository and press the “Import” button to start the importing process. If it takes too long, refresh the page and press the “Go to the project button”.

Create Access Token for the conan-center-index project in Gitlab

We also need the Access Token for the conan-center-index project in GitLab. In the left sidebar go to the “Settings” -> “Access Tokens” and fill the “Name” field, the “Expires at” field, enable the “api” check box in the “Scopes” section and press the “Create project access token”.

Copy the token and go to the “Settings” -> “CI/CD”, expand the “Variables” section and add a new variable by pressing the “Add variable” button.

Fill the “Key” field with the name of the token “CI_JOB_TOKEN”. Paste your token into the “Value” and check the flags “Protected variable” and the “Mask variable”.

Customizing settings

Now we show how to add a new setting that controls the CUDA version. We need to modify the ~/.conan/settings.yaml file. By now we just show what we will add to this file. We will do it in the following “Write CI/CD pipeline” section.

We create a new setting “gpu” and create a new sub-setting “cuda” with a sub-setting “version”. For the purpose of this article we add only two variants “10.2” and “11.5.1”. But you could add as many values as you need.

gpu:
  cuda:
    version: ["10.2", "11.5.1"]
  None:

We should note that it is not absolutely necessary to modify the settings. We could get the same result by adding the CUDA version parameter as an option. But it is a good chance to see how customizing settings works. Also we suppose that CUDA version is much more similar to the system settings like “os”, “compiler” and “build_type” than to the package options. You can learn more about customizing settings in the Conan documentation.

Customizing the receipt to use the CUDA version and architecture

We need to add the use of the new setting into the receipt. Also we want to add a new option “cuda_arch”. By default OpenCV is built with all possible CUDA architectures. It is very time consuming and not always necessary. We provide the patch for the OpenCV receipt.

Let’s discuss each new line.

Add the use of a new setting “gpu”.

Add a new option “cuda_arch” with a list of values. The first value is “None” for building without CUDA. The second value is “all” which corresponds to the default OpenCV behavior. The rest values correspond to the existing CUDA architectures.

For the new option we add the default value “None” because the default value for the option “with_cuda” is “False” by default.

Delete the “cuda_arch” option if “with_cuda” is “False”.

The “cuda_arch” option is needed if the “with_cuda” option is “True”. Add a validation for this.

If the “cuda_arch” option is set to “all” use the default value of the “CUDA_ARCH_BIN” CMake option.

Write the CI/CD pipeline

We suppose that you or your GitLab administrator have configured at least one GitLab runner at some machine. If you don’t, you could follow the instructions provided by the GitLab. Go in “Settings” -> “CI/CD” -> “Runners” -> “Show Runner installation instructions” to create a runner for the specific project.

Go to the OpenCV Conan receipt project “CI/CD” -> “Editor” and push the “Create new CI/CD pipeline” button to create a .gitlab-ci.yml file.

Fill in the next code.

Let’s find out what’s going on.

Define some variables. The first variable is “CONAN_USER”. At it is said in the GitLab documentation you need to replace the “GITLAB_PROJECT_PATH” with the project path in your GitLab and replace every “/” with “+”. The “GITLAB_URL” should be replaced with your GitLab URL.

For example you project location is:

https://gitlab-example.com/foo/bar/project-name

Then your variables will be:

“GITLAB_PROJECT_PATH”: foo+bar+project-name

“GITLAB_URL”: https://gitlab-example.com

Add the “deploy” stage. It is the only stage for our example.

The “.common” part include the “stage”, “tags” and “before_script”. In the “stage” section we set the “deploy” stage. In the “tags” section you should specify the tags of your GitLab Runner.

The “before_script” section is the most interesting. In this section we add code for preparing the build system environment. We will use Nvidia docker for building because it has necessary CUDA libraries but it does not have some libraries and tools that are needed for the OpenCV build. We need to install libgtk2.0-dev, libva-dev, libvdpau-dev and cmake for the OpenCV build. We also need to install python3-pip for the Conan installation.

Initialize the Conan settings.yml file to be able to customize it. Add your GitLab to the list of the remote servers. Complete the settings.yml file with a new setting “gpu” as was described in the “Customizing settings” section.

Next we discuss the “with_cuda_build_jobs” section. It uses the “extends” GitLab directive to extend the job with the “.common” section.

For creating the build matrix we use the “parallel:matrix” GitLab directive. It will create several jobs: ubuntu_18.04 + cuda_10.2, ubuntu_18.04 + cuda_11.5.1, ubuntu_20.04 + cuda_11.5.1.

Set the desired docker in the “image” section. It uses the “CUDA_VERSION” and the “UBUNTU_VERSION” that was declared in the “parallel:matrix” section.

In the “script” section we call the two commands: “conan create” and “conan upload”.

For the “conan create” command we must specify the path to the directory where conanfile.py is located. Also we specify the set of settings and options. The “cuda_arch” option is set to 7.5 for all builds for simplicity but you can add it as a variable to the “parallel:matrix” section.

For the “conan upload” command we must specify the package reference in the form “package_name/version@user/channel”. Also we need to specify the “CONAN_LOGIN_USERNAME” that is equal to “ci_user” and the “CONAN_PASSWORD” that is equal to Access Token variable that was created in the “Create Access Token for the conan-center-index project in Gitlab” section. We provide the “--all” flag to upload both package receipt and packages and the “--remote=gitlab” to upload it to the GitLab Package Registry.

The “without_cuda_build_jobs” section is very similar to the “with_cuda_build_jobs” except it does not specify the CUDA related options.

To run the CI/CD pipeline you need to commit the CI/CD yaml file. You can see the progress of the pipeline in the “CI/CD” -> “Pipelines”. When the CI/CD pipeline is finished you can check the result of your build in the “Packages & Registries” -> “Package Registry”. You should see the OpenCV package. Click on it and you will see the information about that package in the Package Registry.

Check the OpenCV Conan package

To install the OpenCV binary package on your local system you need to do the following.

Install Conan on your local machine using pip.

pip3 install conan

Initialize the Conan settings.yml and customize it as in the “Customizing settings” section.

conan config init
echo "gpu:" >> ~/.conan/settings.yml
echo "  cuda:" >> ~/.conan/settings.yml
echo "    version: [\"10.2\", \"11.5.1\"]" >> ~/.conan/settings.yml
echo "  None:" >> ~/.conan/settings.yml

Add the “gpu” setting values into the profile. You can do it with the “conan profile” command.

conan profile update settings.gpu=cuda default
conan profile update settings.gpu.version=10.2 default

Add the GitLab remote. Replace the “GITLAB_URL” with your GitLab address.

conan remote add gitlab https://GITLAB_URL/api/v4/packages/conan

Add the credentials for the GitLab remote. You should provide your GitLab user name and Access Token. You could create the Personal Access Token in your GitLab profile “Preferences” -> “Access Tokens”. Replace “USER” with your GitLab username and “ACCESS_TOKEN” with your Personal Access Token.

conan user USER -r gitlab -p ACCESS_TOKEN

And now you can install the OpenCV package from the GitLab remote. You should notice that you must specify the same set of options to be able to download the binaries. You don’t need to specify the settings for “gpu” and “gpu.version” because we have add it into the default profile already. Replace the “GITLAB_PROJECT_PATH” with your project path.

conan install opencv/4.5.3@GITLAB_PROJECT_PATH/stable -s compiler.libcxx=libstdc++11 -s gpu=cuda -s gpu.version=11.5.1 -o opencv:with_cuda=True -o opencv:contrib=True -o opencv:with_cublas=True -o opencv:with_cudnn=True -o opencv:dnn_cuda=True -o opencv:cuda_arch=7.5

Conclusion

In this article, we show how to use Conan with GitLab to manage the OpenCV library. We customize the settings, receipt and create a CI/CD pipeline for building and uploading binaries into the GitLab Package Registry. Now you can use the OpenCV package in your projects. To see how to build the project using Conan and CMake please refer to the “Getting started” page.

We tried to make this guide as straightforward as possible. However, we acknowledge that some operations could be done differently.

It seems obvious not to hardcode the settings customization into the CI/CD pipeline but to make a separate docker image with all settings. You also can manage the settings.yml file with the “conan config” command and save it somewhere. There is the conan-package-tools repository that allows the creation of the Conan packages for different configurations. But it all needs separate sections for a description.

As a C/C++ developers we are excited to see the progress of the Conan project. We find it to be very promising and hope that it becomes even more popular with the release of the second version.

This project was conducted by Deelvin. Check out our Deelvin Machine Learning blog for more articles on ML.