GitHub PR + Heroku + Runscope = Peace of Mind

tl;dr : check out the code to run your own mini GitHub CI server

We use Runscope to test our backend APIs (and Ghost Inspector for our front-end) and these run as triggers in our Heroku pipeline whenever we deploy to our staging and prod environments. However, for our “Review App” within our Heroku Pipeline, it is a bit kludgy. We’ve setup the “postdeploy” hook in our app.json so that we trigger a Runscope test after the app is up. However, what we really want is Runscope tests to block a PR from being merged just like we have with our Semaphore unit tests. This also is the crucial missing step for us to have true continuous delivery. If we had a better way to run a suite of tests against our API and UI, we can feel a lot more confident about pushing to prod directly.

I really wanted to avoid building it myself

Why waste time writing it if someone else has it working? The easiest way was to just hook it up into Semaphore. Run a “test” that just is a blocker to execute the Runscope test and then we can use the Semaphore CI integration with GitHub to our advantage. However, I ran into a major issue. You can’t determine the PR number during a Semaphore run. In fact, after confirming with Semaphore support, this information is just not available, (unless its a PR from a fork), and the fact of the matter is, Semaphore doesn’t even know a build was triggered by a PR. So onto option 2, try to use Zapier to be the middle man. If I could get Zapier to receive requests and then funnel them to Runscope, that would be easier. This didn’t work because we couldn’t really have two way communication so Zapier could receive a PR from GitHub and send a “webhook” but then there’s no way for it to send a GitHub status update to “block’ the PR.

I get to learn how the GitHub status works!!

I’ll admit, it felt good to finally understand how the GitHub PR status mechanism works. I read this example about how to build a simple CI server and read this about the GitHub status API. It was fairly straightforward so I decided to take the example and write it as an Elixir/Phoenix app.

I first created a simple phoenix app without ecto and html

$ mix phoenix.new --no-ecto --no-html github-ci

Next, I built a simple router.ex (I just did screenshots, if you want the code, just go to the repo)

and event_handler_controller.ex

This was fairly straight forward. Notice that you can pull the req_headers pretty easily and then just match that the “x-github-event” is a “pull_request”.

Next, I had to setup the webhook on GitHub and inputted the URL and then just selected it to receive one event, the “Pull Request”

Now back to the code, per the GitHub CI example, I first had all the processing code in the event_handler_controller, but decided to create a separate “lib/connector.ex” file which would let me organize the GitHub method calls. Here’s a snippet of the calls to GitHub

Note, I used tentacat, a fantastic SDK for the GitHub API. A couple of notes, parse_params looks for 3 key components you’ll need for the GitHub API, the owner (in our case “Brightergy”), the repo (github-ci) and the sha, which is the hash of the PR request in question. This will be required to look up the deployment. The other is, tentacat actually doesn’t have a way to use the deployment API to filter out based on the sha (I filled an issue), so we have to get a list of deployments and then just find the one associated with the PR.

The reason why we even need the deployment is simple, I want to wait for the Heroku deployment to finish before running the Runscope test. Here’s the meat of the code that actually handles that logic

So the “status in-progress/failed/complete” in your GitHub PR is all controlled by that simple data chunk, with state, description, and context. The context is the unique identifier in GitHub, so whatever you use, will be registered as a separate state.

Now, the sweetness of Elixir comes into play with

{:ok, pid} = Task.start(fn -> wait_for_status(“heroku”, context, params, pids) end)

This will use “Task”, which will kick off an erlang process to run the method “wait_for_status”, which handles checking the deployment status.

In wait_for_status, we do a simple receive/after loop. I am still a noob when it comes to erlang processes and things like GenServer and async calls, so this part got me a little. I just wanted a process to wait in a loop to check to see if the deployment status was valid. I originally wanted to use Process.sleep, but that’s only available in Elixir 1.3, and its not recommended as it would block. So, the best way is to just wait for a message to come in, and if it doesn’t match, it will wait “15_000” or 15 seconds, and then call the wait_for_status again. Notice its a simple send self(), {dep, parse_status(status} which is just saying, send this erlang process a chunk of data, and then only process the block of code if it had the {:ok, found} block, which will only happen if the deployment status on GitHub was a success, fail or inactive.

Now the deployment has passed, I need to call the Runscope trigger. Now if you noticed earlier in the handle code, we did a process_tests call, which returned a set of pids.

As you can see here, it actually also calls create_status, so we can block the PR until the specific type passes/fails (in this case Runscope). It also loops through and creates a test_runner which is the pid that was sent to the wait_for_status call. As you can see in this line

Enum.map(pids, fn(pid) -> send(pid, {dep, found}) end)

It’s looping through all the test_runner pids, and then sends the dep which is the deployment data as well as the found or an array of statuses. The test_runner code then receives that data, and will kick off the actual tests.

Below, you will see what GitHub PR while its waiting for the two processes

Below is the Runscope logic to kick off the trigger (only if the test_runner received the proper response from the deployment checker)

As you see in test_exec, it will call the trigger, while passing in the url parameter, which is a variable in my Runscope test which tells it which url to test against, and in this case, we use the “environment” value from the deployment, which would look something like brighterlink-api-stage-pr-433

The same type of waiting logic I wrote for deployments, is implemented for Runscope test

A simple note here, the reason why I wrote two wait_for_status is because for some reason, if you call the Runscope status api to close to test triggering, it returns 404, so the first, we “sleep” for 3 seconds, and then go into the loop (except in the case if the response was a success in the first api call) and then the subsequent rounds will fail if it gets a 404 or other errors.

Finally, the logic for parsing the Runscope status response is below

Here, you’ll notice a simple matching on the result value within the data field. Then it will return a simple message, which will be used as the description for the GitHub status.

And for the final triumph, passing tests!!

There’s actually a few other items related to getting it to run in Heroku, but I won’t cover it here as I’ve rambled long enough, but suffice to say, this was a really fun exercise which gave me a much better understanding of how CI works with GitHub and hopefully with the above examples, it’ll be easier for you to understand how you can build a simple GitHub CI system for yourself!

One thing to note is, it was a bit harder to debug all this as I need to trigger events from GitHub, but the use of ngrok, which will allow you to setup GitHub to call your local server. The second is to use the GitHub webhook section to re-issue the same event data without having to constantly open new PRs. I definitely abused the “Redeliver” button!