Photo by You X Ventures on Unsplash

At Invoca, our engineering team has worked hard to transition to a decentralized service ownership structure. Each team has services and features that they own, from ideation all the way to deployment and monitoring. The autonomy of the teams to define and prioritize their respective roadmaps has been effective at ensuring reliability and stability while continuing to deliver new features. This structure gets difficult, however, when there are efforts that span multiple teams and require tight coordination. We use working groups as a tool when working on cross team initiatives that are otherwise having difficulty gaining momentum.

Wikipedia defines a…

A Kibana dashboard showing requests over time by controller as well as top average latency

We have a lot of metrics that power dashboards and alerts around our production infrastructure. This includes our Rails frontend servers. We believe visibility is critical to being able to anticipate and act on any reliability or performance issue. Something we were lacking though was a complete picture of requests by Rails controllers and actions (in aggregate), that also included the average and upper percentile response latency.

For those familiar with Rails, you know about the stats that show up at the end of each request in the standard Rails logger:

As part of our Journey to Continuous Deployment at Invoca, one of the roadblocks to more frequent and flexible deploys was database schema migrations. We had to lock users out during a deploys that contained migrations because when users read or wrote to tables that were being migrated they would get errors. While effective at reducing schema incompatibility errors, our goal was to deploy daily and at any time of day. We needed a way to migrate our databases without having a maintenance window.

We solved the migration problem by using, and later extending, a tool for MySQL from Percona…

Nick Burwell

Lead Architect at Invoca, Inc.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store