Compiling OmniSci Core: Ubuntu 16.04 LTS (GPU)
In my role as a Developer Advocate at OmniSci, one of my many responsibilities is to monitor our various ‘Help’ areas like GitHub issues and our Community message board. While doing talks and contributing to open-source is super fun (and valuable in its own right), the task of answering people’s questions or tracking feature requests provides such a rich understanding of what our product currently does, what our users would like OmniSci to do, and where our documentation could be a little bit better.
In the spirit of ‘…where our documentation could be a little bit better’, this blog post was born. In this post, I’ll demonstrate how to compile OmniSci Core (open-source) on Ubuntu 16.04 LTS for NVIDIA GPUs.
This post is focused on those who want to contribute to OmniSci Core (and thus need an actual build environment); if you just want to try OmniSci Core, we provide Docker images, as well as tarball downloads and even OmniSci Cloud (paid, 14-day free trial).
After answering dozens of ‘Why doesn’t this compile?’ questions, without a doubt my most frequent suggestion is ‘Can you use Docker?’ While OmniSci Core isn’t a gigantic code base, there are enough dependencies where building inside of a Docker container makes a lot of sense. This is especially true when it comes to the NVIDIA CUDA dependencies.
In order to use Docker with GPU support, you need to install nvidia-docker. After installing nvidia-docker, we can use the following command to build the Docker container interactively:
With the above command, Docker will download an Ubuntu 16.04 LTS image with all of the NVIDIA development tools installed (CUDA, nvcc, headers, etc.), then run the image in interactive mode and provide a Bash shell.
Updating Image and Installing Standard Dependencies
Once we’ve got our Docker image up and running, we can update the image and install common dependencies as follows:
The second command installs
git so that we can clone the OmniSci Core repo, which contains a script to build all of the software dependencies. That script in turn uses
sudo, so we install that as well.
Installing the OmniSci Core Dependencies
The longest step in this build process is installing all of the dependencies needed to compile OmniSci Core. Because Ubuntu 16.04 LTS is a long-term support release, the base libraries such as the GCC compiler are fixed for a given LTS version. Of course, new releases of libraries continue to come out every day, and rather than constrain ourselves to older versions we can provide a build script that installs the versions of the updated libraries we need:
This step can easily take 20–30 minutes to run as compilers are installed, specific library versions are downloaded and compiled, etc. The final step in the process is to call the
mapd-deps.sh script which sets various environment variables.
Before we move on to compiling OmniSci Core, it’s a good sanity check to make sure we have the GPU support we’re expecting in the container:
nvidia-smi shows the state of your GPUs similar to the htop utility in Linux and
nvcc is the CUDA compiler. With both in place, we should have everything we need for successfully compiling OmniSci with GPU support.
Compiling OmniSci Core With GPU support
With all of our dependencies installed, it’s time to actually compile OmniSci Core! OmniSci Core uses
cmake to manage the build process:
The cmake file has numerous flags that can be set to control what gets built. To see the various flags possible, see the ‘Build’ section of the OmniSci Core repo. The code snippet above should be sufficient for most people wanting to do development on a GPU-enabled build.
Compiling OmniSci from scratch takes 5–10 minutes, depending on the value of
-j provided to make (
-j sets the number of jobs that can run concurrently). Depending on how many processors your computer has or what else you are running on your computer, you can set the
-j value higher or lower.
Note: If you run into build issues while using
make -j 4, it can be difficult to trace where the error occurred because the steps will print out of order. Upon error, retry the build using just
make with no
-j argument, which will give you a clearer indication of where the error occurs.
Running Sanity Tests
After the build has successfully completed, you can validate that the code compiled correctly using
When the tests run successfully, you’ll see the following:
What Will You Contribute?
With an OmniSci Core GPU-enabled build environment in-place, the only thing left is to contribute! OmniSci Core is an Apache 2.0 licensed project and we welcome contributions of all kinds, from documentation improvements to the lowest-level code updates.
To contribute to the OmniSci Core codebase, stop by our GitHub repo and submit a pull request. Although not required, if you are planning to submit a large pull request, submitting an issue before you start working will give our internal engineers an idea of what you are working on and allow them to guide you through the submission process.