Data Querying
Published in

Data Querying

Improving Hue’s developer productivity with some Continuous Integration

Tooling and automation has been simplified dramatically with technologies like Docker and the Cloud. On top of this, continuous integration companies like Circle CI and Travis CI have added one more layer of simplicity.

Background

Hue’s scope has been pretty wide (ranging from SQL parsers to CSS to scheduling jobs or querying any type of databases…) and the increase of external contributions has made the previous Development process outdated. It was becoming frequent to brake the master branch with errors or not making easy for developer to contribute by asking to manual run the test suite. It was time to modernize the development experience!

Goals

  • Automate all the test runs and remove the manual steps burden of the contributors
  • Forbid to push code that breaks tests
  • Keep the overall runs below 10min and closer to 5min (the less friction, the better)
  • Keep it simple

Picking up a CI infrastructure

Continuous Integration (CI) is a mean to automate checks on developer changes. The team looked at three separate systems:

  • Build our own in Docker images and Jenkins
  • Travis CI
  • Circle CI

The goal was to minimized the time spent on the setups while providing a maximum of automation to the developers. Influential open source projects like Kubernetes were also looked at for inspiration. We played with Travis CI and Circle CI and found Circle CI free open source plan clear and interface pretty easy to use (literally 5 minutes to get a first pipeline started). The documentation was good even if mostly only needed for more advanced automation points (e.g. automatically pushing commits to the master branch). The paying plans seemed also fair, even if we would not need it as Hue is open source.

Integrating the CI to Github

On the Github side, these steps below were added to the project configuration. Circle CI was also authorized to access the repository.

  • Protect master branch
  • Require Circle CI check
  • Require pushed branch to be up to date with master

On the Circle side:

For historical reason, traditional pip install of requirements is not possible yet in Hue. For the record, this will be cleaner after the migration to Python 3 support HUE-8737. In the meantime, the Hue docker base image is pulled, the repository is checked out and the Python and JavaScript code changes are applied to the image. Here is the Circle CI configuration.

On the Hue side, here are the current checks:

Python tests

Ran 641 tests in 131.800s

JavaScript tests

2289 specs, 0 failures, 10 pending specs

Finished in 5.627 seconds

JavaScript linting

Finished in 15.678 seconds

Pull requests

Pull requests (PR) automatically gets the test suite and code analysis ran for them. It shows up directly within the PR page:

Automatic pushes

First the process required developers to push to a branch, then push again to the master branch. This was cumbersome as it was easy to forget to push to master, was a redundant steps and easier to get into conflicts as someone might push some other commits to master in the meantime.

Via Circle CI workflows, and only on some dedicated branches, the commits are now automatically pushed to master if the test passes. Regular test branches can still be used and we also took the opportunity to disable the duplicate re-run of the test on the master branch:

Results

So far the time invested (~1 week) is already paid back after less than a month. Developer likes the ease of use and the lead does not need to police and revert commits breaking the master branch. Accepting external contribution has been facilitated too. Developer are even asking for integrating more checks like linting and code coverage.

As the core is now there, it is easy to iteratively add more functionalities to save even more time going forward. For example by adding steps to build Docker images, check Python 2 and 3 compatibility, add Python test coverage checks, start having integration tests or publish documentation changes automatically. Also with some more tweaks on the image we could shave off 1 or 2 minutes of run time.

To finish, here is an example of a CI run. If you also want to play with SQL, just open-up demo.gethue.com.

What is your favorite CI process? Any feedback? Feel free to comment here or on @gethue!

Originally published at http://gethue.com on June 12, 2019.

--

--

--

Modernizing & Simplifying how to Query Data

Recommended from Medium

Netsuite Suitescript Release Process and Guidelines

Ansible playbook — Java and Docker installation

Why Product Owner needs courage

Gontroller: a Go library to create reliable feedback loop controllers

Enabling LDAC Codec Support in Fedora Linux for Bluetooth Headsets

Women of Silicon Roundabout 2021 — Live attendance

Aligning Developer Incentives with the Business

Stay On Target By Defining Software Requirements Using Events

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Romain Rigaux

Romain Rigaux

Adventuring & Smart Data Querying | getromain.com

More from Medium

Bitbucket — Reduce large repository size on Local and Remote

Enhance Your Contract Lifecycle Management With ITAM Software

The Dream New Mac Development setup, Part 1

<Open Source Round 1: Read other people’s code> Week 1 learnings