The advent of Change… API

FT Product & Technology
May 20 · 9 min read

By Nikita Lohia

Change is inevitable, change is constant..

Someone famous said that. All knowing WWW seems to be ambiguous on who said it. But finding out the origin of this quote isn’t the goal of this blog.

We, the reliability engineering team, had a far more pressing problem to solve:

TL;DR : How to track various code releases, happening multiple times a day, pushed by various different teams in a completely random order, which may or may not affect your service in a forever increasing microservices architecture software development environment?

The Problem

We want to track the changes happening in all of our systems.

We have attempted to solve it before. Our legacy Change Request API, fondly called CRAPI (/ˈkræpɪ/) was tracking releases, both automated and manual, and storing them in a third party change management tool. There were a few reasons why CRAPI didn’t work today:

  • It was written 4 years ago in Java which no one in the team quite understood anymore.
  • It was synchronous and did not handle server side errors gracefully, resulting in a failure rate of over 50%. This discouraged developers to add it to their production pipelines because it would constantly break/delay their builds.
  • It was too complicated to adopt: one had to create an API key by calling another API, the request payload was bulky and unclear, and there was a lack of documentation.
A screenshot of the old API with large numbers of complicated fields to supply
A screenshot of the old API with large numbers of complicated fields to supply

Meanwhile, during production incidents, our operations and development teams were forever trying to figure out which change in our wide and varied microservice estate could have possibly broken a system.

Gif of people being pointed on to the next person and the next person and so on
Gif of people being pointed on to the next person and the next person and so on

The need for having a central log of changes was evident.

However, between various feature requests of building cool new things and supporting our existing services and products, Change Management often got deprioritized. We realised that we needed a smarter approach to solving this problem, and quickly.

So a group of us gathered around a white board, handed a marker to the one who could draw and write on the whiteboard the most legibly amongst us, and we got to work. External auditors, compliance team, dev teams, Project Managers, and our Operations support team: they were all potential users of Change Management. However, we decided to start with one user group to get something out the door quickly — our developer teams.

The Journey

We used the following guiding principles when building a solution.

Someone at a laptop surrounded by pictures of the things they need to keep in mind
Someone at a laptop surrounded by pictures of the things they need to keep in mind

Users First

The main one is simple, but often forgotten — keep the user at the centre of it all. Whatever we build must be easy for the user to adopt and use. Always always always, think from the user’s perspective when building new things.

Learn from the past

While it’s important to know what to build, it’s equally important to know what not to build. Use the lessons you have learnt from the past to guide you. In our case, we knew that one of the reasons CRAPI wasn’t hugely popular was because it was slow and synchronous. It took a user’s request, validated it, waited for a 3rd party to respond, log it and then finally send a response back to the user. By adding so many dependencies to it, CRAPI was bound to be unreliable. We decided to make the replacement solution asynchronous. As long as the user sends a minimal valid payload, we respond with a 202 accepted and let the user be on their merry way. All of the other processing happens asynchronously.

After all, a server side error should not really be a user’s problem.

Decoupling

We decided to track microservice changes with a microservice application. We made our solution highly decoupled. In the first iteration of building a new Change API, we kept it simple. When a user sends us a change log, we validate it, send an “Accepted” response immediately upon success and send a notification on Slack asynchronously. To make this happen, all that was needed was a couple of AWS lambda functions and an API gateway with a real-time data streaming service — in our case, AWS Kinesis.

Architecture diagram showing the components of our MVP
Architecture diagram showing the components of our MVP

By default, we sent all of our “production” change logs to a single Slack channel. This was already hugely helpful.

We went from

Screen shot of users manually typing and cross posting release notifications on slack channels
Screen shot of users manually typing and cross posting release notifications on slack channels

To

Screen shot of automated Change API notifications in a slack channel
Screen shot of automated Change API notifications in a slack channel

Have a futuristic outlook

One of my favourite quotes is from Arthur C. Clarke

“Any sufficiently advanced technology is indistinguishable from magic

Change API in beta, it was pretty basic. We then did extensive user research and took that feedback to build on it. We added lots of “enrichments” to Change API.

  • Change API would take the Git commit details (commit hash, repository name etc.) from the available CircleCI environment variables and helpfully add a link to the GitHub Pull Request/Release into the Change notification on Slack without requiring any effort from the user.
  • It uses the “systemCode” ( a unique identifier for every system) supplied by the user and converts that into a clickable link to the relevant troubleshooting guide (we call them Runbooks) — really handy if something were to go wrong after the release.
  • Users can also optionally add further key:value metadata with the Change log notification, for example our Memberships team add additional changeSummary and changeDescription fields to give more context.
Alt text: Screenshot showing the enhanced notifications surfacing key:value metadata supplied by the Membership team
Alt text: Screenshot showing the enhanced notifications surfacing key:value metadata supplied by the Membership team
  • It adds a nice little “Last Release Timestamp” next to the system’s monitoring page to make it super easy for people to correlate a monitoring alert with a recent change to the system.
Screenshot showing the Last release date information being surfaced on the monitoring page for the system
Screenshot showing the Last release date information being surfaced on the monitoring page for the system
Screenshot showing a monitoring tile saying that there was a recent release of that system
Screenshot showing a monitoring tile saying that there was a recent release of that system

Each of these “enrichments” runs as its own lambda function, asynchronous and independent of each other, without costing our developer teams any additional time or effort. Moreover, thanks to principle 3, “decoupling”, any future enrichments can be built easily, by anyone, by adding them as a new lambda consumer on the existing Kinesis stream which is logging changes.

Tooling

Now that Change API was more mature, we decided to put it in high gear and make its adoption easier. We identified the most common user patterns within the FT for software deployment. The majority of us use GitHub for code control with CircleCI as our CI/CD whilst other teams use Heroku or Jenkins.

To cater to the majority of the dev population, we decided to build a CircleCI orb for Change API with some sensible defaults.

CircleCI Orbs, for those who don’t know, are a nice way to share common CircleCI config. By creating one for Change API, we essentially abstracted away all the shared configuration for Change API, leaving the user to only supply the system-specific information in a clear, concise YAML format, instead of writing complicated bash commands in a circle config.

To support some of the other common deployment patterns, we also supplied a plug and play script for Heroku integration with Change API.

Finally, we integrated Change API with existing tooling. One such example is the `n-gage` NPM module which our developers of Ft.com use extensively in their repos. We simply added it to this package once and then all of the repos which use n-gage adopted Change API with a simple version upgrade.

Gif of Barack Obama saying “for the world has changed, and we must change with it”
Gif of Barack Obama saying “for the world has changed, and we must change with it”

The result

By applying all of the principles above, we now have a mature Change API product which is widely used across our Technology department.

Graph showing a curve increasing steadily in height over a period of several months
Graph showing a curve increasing steadily in height over a period of several months

Change API now:

Screenshot of our dashboard of stats for the Change API
Screenshot of our dashboard of stats for the Change API

We are clocking around 150–200 daily code releases.

Even during this COVID-19 pandemic, it seems like we are pushing our code, business as usual:

Graph showing that the number of releases in March and April has stayed pretty similar to February
Graph showing that the number of releases in March and April has stayed pretty similar to February

We are tracking most production releases in a single place, which makes debugging really easy.

One such example of Change API in action is below

Screenshot of a slack conversation where someone reacts to alerts by heading to the change log channel
Screenshot of a slack conversation where someone reacts to alerts by heading to the change log channel

We are now in the mindset that whenever an alert fires for a system, we look in the change-log channel to see what has been released recently and whether that could have caused the alert to fire. We rollback, make sure user impact has been reduced or all together removed and then do root cause analysis.

Feedback from another colleague on the CircleCI Orb:

“On a semi-related note the circleci orb integration is awesome and it’s super easy to integrate! Thanks Nikita”
“On a semi-related note the circleci orb integration is awesome and it’s super easy to integrate! Thanks Nikita”

One completely unexpected but seemingly very useful effect of Change API was identified by another colleague:

“Thanks for making it easy to get Big Numbers for Conference Talks!”
“Thanks for making it easy to get Big Numbers for Conference Talks!”

Another area of tracking changes we have long struggled with has been DNS. With our old DNS solution, we didn’t have an easy way to track changes being made to DNS records. We recently moved to Route53 as our main DNS provider and with that, moved to Infrastructure as Code. You can probably see where I am going with this…

Thanks to Change API, we now immediately know exactly what changed in DNS and by whom.

A Change API notification for a DNS change via Route 53
A Change API notification for a DNS change via Route 53

We can proudly say that the Change API codebase has been touched by almost every secondee* we have had since its creation. Till today, at least 16 different people have made code contributions to this project.

What’s next

While Change API has been extremely successful, as with any other piece of tech, there are areas for improvement. While some of the decisions we made helped its success and encouraged a wide user-base, we also made some things difficult for ourselves.

We currently don’t have a very reliable monitoring check for Change API. This is a common problem for any microservice-based architecture. We have individual functional tests in place, but we still need to implement an end-to-end testing strategy which will alert us when any part of the pipeline is broken.

Secondly, because Change API is essentially a bunch of lambdas, it can be a nightmare to debug if a specific part of it isn’t behaving as expected. We have tried to solve that problem in part by central and structured logging, but it’s still a far cry from ideal. What makes it worse is the fact that by design, we never throw any server side errors, so catching a bug real time can be very difficult — as we are learning.

We are also looking to add more features to Change API — for example, there have been feature requests to also track package releases, and AWS secrets and key management tracking.

However, because it was built by a temporary feature team which is now disbanded, we need to find a way to make these happen for our users.

One thing is for certain, Change API is far from done.

Wanna talk Change Management? Have any questions, suggestions, or feedback on this blog? Reach me at nikita.lohia@ft.com or @NikitaLohia on twitter

* The Reliability Engineering team runs a secondment program via which people from teams across our Engineering department can work with us temporarily on something different than their usual projects. This encourages collaboration and knowledge sharing and reduces developer fatigue. Get in touch if you want to know more about this!

FT Product & Technology

A blog by the Financial Times Product & Technology…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store