Project Boards are GitHub’s take on Kanban Board-like project management feature. Quorum engineers foster a strong collaboration and code review culture through Git and GitHub, and we recently decided to migrate our project management tool from ZenHub to GitHub’s Project Board. While ZenHub was a pleasure to use, it was not free and didn’t come out of the box, which makes it harder to scale as we grow and add more awesome people to our team. As we scoped out our needs, we decided that the new project management service has to:
- Feature Trello-like user-friendly functionalities, like dragging and dropping issues between columns.
- Come with the ability to create new projects and close old ones programmatically using Ansible, our primary deployment framework.
- Be easy to setup and explain during the new hires on-boarding process.
- Offer organization-level and/or individual repo’s projects so we can follow good security practice, particularly privilege separation. While it’s invaluable to loop in our Business Development and especially Customer Success teams what issues were resolved in the latest deployment, it’s not the best practice to give everyone at Quorum full access to the codebase.
All things considered, choosing GitHub Project Boards as our primary project management tool was a no-brainer.
GitHub API V3
At Quorum, we create a Project Board for every deployment, using the release number as the unique identifier. Since deployment can happen multiple times a day, it quickly becomes tedious for our developers to create and close Project Boards manually. We believe that what can be automated, should be automated. Having leveraged Ansible in our day-to-day DevOps and deployment process, we understand that it’s essential to add the ability to programmatically manipulate Project Boards to our deployment playbook.
Fortunately, GitHub does offer a REST API for creating and closing projects programmatically, as well as manipulating project’s columns and cards. To continue following this tutorial and trying out the code yourself, it’s highly recommended that you create an Access Token if you don’t have one already. Head over to your GitHub’s Profile Developer Settings and select Generate access tokens under Personal access tokens. Voila!
For your convenience, here is the GitHub Project Boards API docs.
Now that we have obtained a token, let’s try to send a simple GET request to list all Projects in your organization using curl . 💪
curl --user USERNAME:TOKEN https://api.github.com/orgs/YOUR-ORG-NAME/projects --header "Accept: application/vnd.github.inertia-preview+json"Replace USERNAME with your GitHub’s username, TOKEN with your access token, and YOUR-ORG-NAME with your organization’s name. While the authentication using --user is self-explanatory, we also explicitly request the preview version of GitHub’s Projects API through the Accept: header. You should get a JSON response that looks similar to this:
[
{
"owner_url": "https://api.github.com/orgs/YOUR-ORG-NAME",
"url": "https://api.github.com/projects/PROJECT-ID",
"html_url": "https://github.com/orgs/YOUR-ORG-NAME/projects/ORG-SPECIFIC-PROJECT-ID",
"columns_url": "https://api.github.com/projects/PROJECT-ID/columns",
"id": PROJECT-ID,
"state": "open",
"created_at": "2017-10-10T12:42:20Z",
},
]For brevity, we omit a few entries like the name and body of the project, as well as the project creator’s metadata. We saw that it was possible to list all projects programmatically through a GET request. We can generalize this into creating a new project and a standard set of columns with a POST request or updating/closing a project at the end of deployment with a PATCH request.
We’ll see soon that Ansible has a module for interacting with web services and APIs instead of using curl through the shell / command module, but let’s first take a step back and brainstorm how we can version control our playbook without exposing our GitHub API access token.
Ansible Vault
So far, we’ve created an access token and successfully sent a GET request to the GitHub’s Projects API. While we can start putting our curl command into an Ansible playbook, there are a couple problems with that approach:
- We’ll want to version control our playbook, which means our token will be stored in plaintext in a central Git repository, unless we take an extra step to encrypt/hide it in some way.
- We’ll likely want to make these tasks reproducible and executable on different hosts. Are we following good software engineering practice by copying the same
curlcommand everywhere and changing the command line flags or arguments?
Let’s tackle one problem at a time, starting first with encrypting the secret. There are many possible solution to this, including but not limited to:
- Add the secret to a file not tracked in Git (add such file to .
gitignore) andlookupthe content - Use environment variables
- Leverage
git-cryptand encrypt the relevant inventory files using a Git hook.
The opportunities are endless. However, just like how we at Quorum prefer Project Boards because they come out of the box with GitHub itself, we lean towards a solution that is self-contained within Ansible. Fortunately, said solution does exist and it is called Ansible Vault. There are already a lot of excellent tutorials on Vault; here we’ll attempt to introduce a new feature in Ansible 2.3 that might have flown under your radar, namely the ability to have both encrypted and unencrypted variables in the same inventory file.
On your terminal, run
ansible-vault encrypt_string YOUR-SECRET-TOKEN --ask-vault-passYou’ll be prompted to enter a password that will be used to encrypt and decrypt YOUR-SECRET-TOKEN . Copy the output and place it in the relevant inventory files among the unencrypted variables. How cool is that?
github_api_proj: https://api.github.com/orgs/YOUR-ORG-NAME/projects
github_api_username: YOUR-USER-NAME
github_api_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
663864396532363364626265666530633361646639663032313639346535613639
6431626536303530376336343832656537303632313433360a6264383463363533
626563616536303732316136626339623162336339363961653864396165333539
3430613539666330390a3137363232656564323662366333303139633263656539
346237313766646231343834633162656434363434386232666239656363633261Then, when you run the playbook, simply append --ask-vault-pass to be prompted for the same password earlier. Ansible Vault will then use the decrypted secret (the actual value of github_api_password) in execution.
ansible-playbook deploy.yml -i path/to/inventory --ask-vault-pass
Optionally, you can set the flag --vault-password-file /path/to/vault or the environment variable $ANSIBLE-VAULT-PASSWORD-FILE. Both should point to a file not tracked in git which contains the plaintext password to open the vault. Either option does make automation even simpler (for example, by putting the Vault’s password file under a shared directory all authenticated SSH users have access to), but does come at a higher risk comparing to entering the password “on-demand” using --ask-vault-pass. Of course, we can go one (or several) step(s) further by encrypting the Vault password because you can never have enough security. Let’s not go down that route in this article, however.
We took a detour to make sure our access token is properly encrypted and not exposed to the world. Let’s now tackle the other problem: how do we properly scale and reuse our curl command across playbooks without copy-pasta?
Introducing the uri module
Once again, Ansible’s built-in functionalities and modules come to the rescue. According to the Ansible docs, the module uri does exactly what we want, which is to “interact with HTTP and HTTPS web services and support Digest, Basic and WSSE HTTP authentication mechanisms.”
Equipped with this new-found knowledge, let’s try to recreate the same curl command using the uri module.
- name: Send a GET request to list all projects
uri:
url: '{{ github_api_proj }}'
headers:
Accept: application/vnd.github.inertia-preview+json
method: GET
user: '{{ github_api_username }}'
password: '{{ github_api_password }}'
force_basic_auth: yes
status_code: 200
register: projects_list_json_responseNotice the use of templated variables, such as '{{ github_api_proj }}', which comes from the inventory file where we place encrypted variables among unencrypted ones. Running the playbook containing this task with either --ask-vault-pass , --vault-password-file , or by setting the environment variable, you should get the same exact JSON response from running the curl command earlier. Magical!
By making our deployment-related playbooks all use the same inventory file, we can now easily generalize tasks that use the GitHub Projects API without copying the same curl command and repeating it everywhere. Moreover, we gain the flexibility of moving these set of Github Projects tasks into an Ansible role, as well as listing the options to the uri module in a human-readable and maintainable way.
We’ll now show you two more tricks before wrapping up this article.
Magic Tricks, or Why Ansible is Powerful
Earlier, we mentioned the ability to close projects with a PATCH request. While that sounds easy if we know the Project’s ID ahead of time, how can we programmatically determine the relevant ID and pass that to a uri task that PATCH-es the relevant API call with body: '{"state": "closed"}' ?
Solution: we use the same list call earlier to get all projects, grab the relevant one we want to close, extract its ID, and pass that to the PATCH call.
This sounds super easy to do in Python: we’ll call json.loads() on the return json to load it as a Python dictionary. We then iterate over the list of results, look for the one that matches a certain criteria, break out, perform basic string manipulation to retrieve the ID, and return it. Sounds like it’s time to write a custom filter!
… But wait? Why do we have to do all of that? Shouldn’t Ansible’s template engine have something to solve this problem? We do like self-contained things at Quorum, remember?
Ansible not only ships with many amazing filters, but also the ability to leverage Jinja 2’s built-in filters, many of which follow the map-filter-reduce functional programming paradigm. By chaining the right set of filters, we can map a function over our list of projects to get all the IDs, filter on the ones that meet a certain criteria, and return them. The Jinja2 filters relevant to us here are map and selectattr (and to a lesser extent, join to concatenate the final result, the same way Python’s str.join() works).
Remember our projects_list_json_response variable that was registered earlier? We’ll now apply the magic on it.
vars_prompt:
- name: version_number
prompt: Enter the release's version number
confirm: yestasks:
- name: Set the relevant project ID as a fact
set_fact:
project_id: >
'{{ projects_list_json_response.json |
selectattr("name", "equalto", version_number) |
map(attribute="id") |
join("") }}'
Here we ask the user (one of our engineers who’s deploying) to enter the release’s version number. By naming our GitHub’s Project Boards using the release number, we can match the user’s input with the relevant board using selectattr . We then call map to retrieve the ID field, and (optionally) join the result together using nothing but an empty string because we know there should be only one result. Should we retrieve two or more results and want them comma-separated, we would slightly change the filter to |join(",") .
Sending a PATCH request with uri using the resulting project_id is now trivial.
- name: Send a PATCH request to close a deployed project
uri:
url: 'https://api.github.com/projects/{{ project_id }}'
headers:
Accept: application/vnd.github.inertia-preview+json
method: PATCH
user: '{{ github_api_username }}'
password: '{{ github_api_password }}'
force_basic_auth: yes
status_code: 200
body_format: json
body: '{"state": "closed"}'One magic trick down, just one more to go. 👌
Another thing we’d also like to automate is the ability to create columns programmatically for new projects. Like many Kanban boards, we employ certain columns including: Backlog, To Do, Pending Review, Merged into Release, and Tested on Staging. While we could duplicate 5 or more uri tasks to create each column, Ansible magic comes to the rescue again thanks to with_items.
In our inventory file, we declare a YAML list containing the names for the columns we want to create.
github_project_column_names:
- Backlog
- To Do
- Pending Review
- Merged into Release
- Tested on StagingWe then send a POST request through uri to the Columns API to programmatically create all columns in one task without duplicating code.
- name: Send a POST request to create new columns
uri:
url: 'https://api.github.com/projects/{{ project_id }}/columns'
headers:
Accept: application/vnd.github.inertia-preview+json
method: POST
user: '{{ github_api_username }}'
password: '{{ github_api_password }}'
force_basic_auth: yes
status_code: 201
body_format: json
body: '{"name": "{{ item }}"}'
with_items: '{{ github_project_column_names }}'We’ve written a lot of code in this article! Here they are again in Github Gists for your convenience.
Summary
We’ve learned a few things through migrating to Github Project Boards and writing up this article.
- What can be automated, should be automated.
- Similarly, what can be done in a self-contained way, should be done in a self-contained way (re: Ansible and built-in Jinja2’s filters).
- Secrets should always be encrypted, regardless of whether the repository is private or not (re: Ansible Vault).
- Even though we are a fast-moving team, we strongly value code readability, maintainability, and reproducibility. It’s much harder to read code than to write it. If our DevOps developer is out sick or is on vacation, we want the team to continue functioning as normal. Ansible and its developer-friendly syntax make it very easy to train new developers on old code and allow us to continue growing and scaling.
Interested in working at Quorum? We’re hiring!
