Automation at TouchBistro — Part 1: A Boilerplate for Testable and Observable Services

Photo by Birmingham Museums Trust on Unsplash

As TouchBistro continues to grow, the engineering organization has been moving away from a single monolithic web application and towards smaller domain-services that are owned and deployed by smaller working teams, each focused on specific areas of our business (Eg: payments, new customer experience, menu management).

This helps teams move faster and allows us to end-of-life some legacy systems that were written in the company’s earlier, scrappier days. While that code helped the organization to grow to where it is today, it has become increasingly costly to operate, support and extend. Moving to a services-oriented architecture isn’t without its own drawbacks. For us that included:

1. Duplication of cross-cutting functionality across services meant that fixing a bug or making an upgrade required changing code in several repositories, which was tedious and prone to human error.
2. Allowing each team to identify their own best practices meant that some projects were missing key pieces of infrastructure that were present in other projects like structured logging or CI configuration.

Recognizing these challenges, we took a step back and began to define a standard architecture and project structure for all our services to meet certain goals:

• The bulk of developer-facing application code in each service should reflect the functionality that is unique to the service. Developers should not spend their time reproducing the functionality that is common between services, including, but not limited to: effective monitoring, CI, artifact creation, deployment.
• All services should meet our baseline requirements around observability and testing.
• Our services should behave in a similar way when it comes to deploying, migrating, and other common operational tasks.

We have been able to achieve these goals by abstracting and reusing as much cross-cutting code as possible both as npm packages and a standard boilerplate service that is used as a template for all other services. As we move into 2020, almost all of our services are derived from this template. The development teams responsible for individual services gain the following benefits before writing a single line of code:

• Database configuration
• Health check endpoints
• Application performance monitoring, request and custom logging, and metrics
• Standardized Exception reporting
• Externalized configuration and secret management
• Best in class javascript tooling (eslint, typescript, prettier, husky, etc)
• Unit and Integration test tooling
• CI configuration
• One-click deployments and rollbacks
• OpenAPI documentation and validation (for services that use REST and OpenAPI)
• Feature flags
• Convenient build targets for linting, testing, running database migrations, running local servers with code watching, and other common developer tasks.
• Easy integration with tb, our beloved local-development tool

There are four major components that are captured as part of our boilerplate service:

• the GitHub template
• a standard project architecture
• shared libraries
• standard configuration dotfiles

Boilerplate Template: A Good Start

All our Node.js services are extensions of a base service which we call the boilerplate service.

We already had a number of new services that we had created over the preceding year to learn from. The team extracted all the useful and common code into a single place — consolidating it into shared packages wherever possible — and then imported and bootstrapped those packages in a GitHub template repository.

This boilerplate service serves both as a living example of our best engineering practices at TouchBistro and a testbed for trying out new tools and improvements before rolling them out across the wider organization. We treat the boilerplate service the same way we do all our other services. There is a full CI pipeline, including automated tests and production deployments. We ensure that all changes to the boilerplate remain fully compatible with all existing tooling so that it can be safely adopted by other teams.

Sometimes it is the Developer Acceleration team that updates the boilerplate because we decide to integrate a new tool for all our services to adopt; most recently that was the addition of feature flags to support staggered rollouts.. But often it is other working teams who have made some useful process or tooling improvement. They can submit a pull request to the boilerplate so that the wider organization will benefit from their innovation.

Common Structure and Architecture

Many companies doing SOA or Microservices leave it up to each team to choose whatever language, stack, or application architecture they prefer. We took the opposite approach. Following inspiration from Spring Boot, Ruby on Rails, and other popular “batteries included” backend frameworks, we decided to limit the choices our developers have to make to a subset that actually matters.

At TouchBistro we use Typescript whenever possible and go a step further by defining a common project directory structure and yarn targets to which every app must adhere. We also specify common packages like express for request handling and knex for database access. This is easy with Github’s template repository feature, which allows us to generate new repositories derived from the structure of our boilerplate.

Teams are permitted to deviate from these standards and experiment with different approaches, but we encourage teams that discover these improvements to bring them back to the boilerplate and then roll them out to our existing services. A recent example was our integrations team having greater success with TypeScript over Facebook’s Flow, which had been the standard at the company for a few years. After sharing their experience with the rest of the engineering team, we incorporated their changes to the boilerplate and transitioned the organization to TypeScript.

In creating and extending the boilerplate service, the team has identified some principles to guide that work:

• Separate data access and HTTP handling from business logic (the classic MVC pattern with a strong business layer that does not know about express or knex).
• Push cross cutting concerns (like logging, authentication) to a lower level of the stack, such as middleware or initialization libraries that can be shared between every service that needs them.
• Create contracts that allow higher level tooling to make assumptions about any service without knowing the internals. All services are started with yarn serve. All services respond to a GET /ping request and return, in a standard header HTTP, the current application version.

Libraries: The monorepo

All our cross cutting code is packaged as node.js packages hosted in a private npm repository. As of January 2020, we have approximately 40 of these packages. Whenever our developers write code that performs some self contained and useful function — like uploading and downloading files to S3 or parsing and verifying a JWT from a header — we extract it into a shared package to promote reuse and standardization.

Some of our shared packages are small frameworks onto themselves, like jobseeker, a database backed job scheduler inspired by delayed_job. But many of our shared packages are wrappers over existing tools. For example, we have a utility called node-config-fns that provides an API for reading environment variables from a file in a development and test environment, and from a cloud secret store like AWS Secret manager in production. Is this code we expect all our devs to be able to write themselves if they had to? Absolutely. But what a waste of time that would be. And what a nightmare it would be for our Site Reliability Engineers if every service were to handle secrets and environment variables in its own unique way. Having this mindset encourages our developers to separate general purpose code from application specific code.

Once the number of our shared packages got into the double digits we moved them all into a single monorepo and adopted lerna to simplify the management, versioning and publishing of these packages.

We enforce our highest coding standards for these shared libraries in terms of documentation and test coverage, and use semantic versioning to communicate to our internal consumers how much work an upgrade is going to be. To promote the habit of keeping our dependencies up to date, we recently adopted Dependabot, a free GitHub tool.

TouchBistro would not exist without open source software. So we will be paying it forward by open sourcing some of our packages soon.

Standard Configuration

The boilerplate will also take care of most of the standard configuration we use across all of services. Much of this is the dotfiles for common javascript tooling like linters, type checkers and code formatters. Some of this is configuration for our in-house tools, like service_graph.yml and gehen.yml — for service ownership and deployment respectively.

Whenever possible, we try to minimize the duplication of these dotfiles across our different services. For example, we moved CircleCI configuration into a shared orb, and have recently moved our eslint configuration into a shared package that is imported.

This did present some challenges when we needed to update the standard configuration in the boilerplate after it had already been adopted in a multitude of services. (for example, our Dockerfiles), To assist us we created golang CLI, cannon that allows us to make the same source code changes across multiple repositories, automatically opening pull requests for each one. You can read more about cannon here or check out the repository in GitHub.

What’s next

One of the things that adopting these shared standards has allowed us to do is to create some powerful tools that leverage the common configuration we embed in the boilerplate. In our next blog post, we are going to talk about tb, a cli tool we created that makes it easy to spin up and develop against whichever of the dozens of services you need for local development.

We have plans to extend our boilerplate even further in 2020. Our roadmap includes:

• Cleaning up some abstractions in our shared middleware so that it is easier for developers who want to contribute to common infrastructure to understand what happens in that layer.
• Finding a convenient way for people to make some optional choices when bootstrapping a new service. We have a lot of shared libraries that some, but not all services need. Right now, developers would need to be aware those common services were available and add and configure those dependancies manually. We are taking inspiration from frontend tools like yeoman and vue-cli that allow a developer to customize the bootstrapping of their project.
• Automating the first-time provisioning of any required infrastructure for new services. At the moment, this is done manually by an operations team.
• Adopting the boilerplate mindset to our front-end projects, with a focus on shared components that are performant, usable, and accessible by default. We hope to share a post about our front-end efforts in this area later in the year.

If any of this sounds like fun to you and you are interested in working together, check out some of the open roles at https://www.touchbistro.com/careers/#open-positions.