tb CLI — Simplifying development in a complicated microservices world
At TouchBistro we use a microservices approach for new products that we build. You can read more about how and why we do this in this blog post. This has allowed each team to own their code and services without disrupting other teams.
One challenge we have come across with microservices is the complexity it adds to local development. It is very tedious to recreate a microservices ecosystem on a developer’s machine because each service has to be built individually. This requires dealing with dependencies, different languages, different databases and a million tiny little details. Once all the services are built they still need to be configured so that they can talk to each other. This might not be a big deal if you only need one or two supporting services. It is annoying but not unmanageable to just manually run those services. However it becomes a real pain when the services you need may also depend on a web of services. If you are trying to test an end-to-end feature you can easily end up with a service dependency graph that includes 5–10 services. It is not reasonable to expect every developer to manually run 10 services on their machine by hand. For one team, in order to get a full end-to-end environment running on their machine they would need to:
- Clone or pull the latest versions of 10 git repositories
- Figure out how to build them (ruby on rails, node 10, node 12, dotnet)
- Figure out what other ancillary services need to be run (postgres, mssql, redis)
- Configure them all to talk to each other
- Run startup scripts on each services, to do things like trigger database migrations
Something had to be done and the Developer Acceleration team saw this as a prime opportunity to significant speed up delivery across a wide number of teams
In 2020, developers at TouchBistro run a single command that takes care of all of the following:
- Install any development dependencies needed that are missing, like brew, the awscli, and node/yarn
- Log into third party services like ECR (AWS’ docker registry) and NPM to pull private resources
- Clone all the latest versions of each required git repository
- Pull the latest version of any required artifacts (docker images), as well as any services they need like postgres, redis, etc. By default, we pull images of the latest version of all of our services.
- Run database migrations and seeds or other setup scripts
- Present a nice GUI with all the logs from docker containers
- And more!
Why not use bash and docker?
Docker containers are great because they make it easy to create reproducible builds. Our services were already using docker for production deployment and CI so we decided to leverage this for local development as well. We quickly realized though that there’s a lot more to standing up services than just building and running docker containers and decided to automate the numerous steps required.
Early on, one team at their wit’s end created a tool called core-devtools. It was a few small Bash scripts that ran a couple services using docker-compose. This was a game changer as now developers could run all the services they needed with a single command: bin/up. They could focus on writing new code for their services rather than worrying about running other team’s services and keeping up to date with production. Over time more and more teams began to use core-devtools and added their features to automate additional tasks including: logging into various services, interacting with databases and the containers themselves, and updating docker containers. Life was good for a while, until it wasn’t.
While core-devtools was a very helpful tool, it had its fair share of issues. We were maintaining numerous bash scripts and configuration files to support all this automation. Another issue was that the user experience was not great. Logs were very noisy and hard to decipher. And often core-devtools didn’t work as expected. It became so hard to troubleshoot and debug that when an error was encountered the most common advice was “nuke the whole thing and try running it again”. We attempted to educate other teams about how docker and docker-compose work so they could troubleshoot issues themselves more easily. However, we found that this was not a viable solution because many developers found it too steep of a learning curve. Docker isn’t the easiest thing to understand and there are a lot of weird quirks that can trip people up. We also found that docker-compose was way too low level for most developers and it would be better to have a higher level abstraction to simplify usage.
As core-devtools grew so did the number of people using it. Our QA, front-end, and iOS teams started to benefit from the automation. Because of this we started to accumulate more and more use cases for the tool. We wanted core-devtools to work out of the box for most people. That meant it needed to to install dependencies, log into services, etc; all with little to no hassle. Next, teams asked if they could run only a subset of the available services. So we introduced the ability to run collections of services defined in text files. Sometimes developers wanted to build docker containers while other times they wanted to pull from a remote registry. Sometimes people wanted to run historical versions of our services to investigate a regression, so we bolted on a configuration format. Gradually this became harder and harder to maintain and troubleshoot. Developers began to get frustrated with the frequent issues and our inability to solve some of them. We needed to change our approach.
It was time to build a real tool in a proper language (sorry Bash). core-devtools 2.0 would be a Unix style CLI; simple to use but easy to customize. Like most Unix commands, we wanted a short and simple name that was easy to type. Thus tb was born. We chose to write it in Go because the language is easy to work with while also being fast and safe. In large part due to the success of tb Go has become the defacto standard for development of infrastructure tools at TouchBistro.
We started by recreating the existing core-devtools feature set in tb using Go. With a basic replacement for core-devtools, we began to address some of the issues that proved difficult before.. We added proper logging and error messages to help troubleshoot issues. tb’s design made it easier to add new functionality over time, which allowed us to meet the needs of a growing development team. We were also able to invest in the developer experience; providing command autocompletion, help, user customization, and more.
We migrated existing users from core-devtools to tb and it was instantly a hit. Today tb is an essential tool and has simplified the development process. Let’s review some of the powerful features of tb that make it such a useful tool.
The Magic of tb
A fundamental concept in tb is the playlist. A playlist is a collection of services that can be run in a single command.
For example, say I had a list of services like this:
- A legacy MSSQL database
- A legacy .NET API
- A new Node.js/TypeScript microservice
- A PostgreSQL database the microservice talks to
- A Node.js adapter of the legacy API
This is a lot to keep track of and most of the time developers don’t care about every single service, especially if they are transitive dependencies for the service they are working on. To simplify things I created a playlist called online-ordering. Now, anytime I need to run these services I can simply run tb up --playlist online-ordering and tb will handle the heaving lifting to start all of the services I need to work on online ordering. This is especially useful for anyone who doesn’t know the details of each of our services; particularly QA, product managers or mobile developers. We provide playlists to run that make it easy for them to test features without needing to know the whole dependency graph. Developers can also run this command over and over again and they will get the latest versions of each service without needing to pull git repos, build the services themselves or update any configuration. We deploy our services dozens of times every day and embrace the CI/CD lifestyle. By using tb you can easily stay up to date with what is in production.
A big goal for tb was to make it simple to use and easy to customize. Command line flags is one way we accomplished this. You can use the command tb up --playlist online-ordering --no-service-prerun to skip the pre-run step for each service (where we typically seed our databases). Further customization can be done using the ~/.tbrc.yml configuration file. Users can modify the way tb runs by defining their own playlists, or changing the properties of a service.
An example of this is changing the docker tag for a service. All our services have a CI job that builds a docker image, tags it with the branch name and commit SHA, and then pushes to AWS ECR, our remote docker registry. To make it easy for any developer to run any branch/commit of a service with tb they can override the service’s tag field in ~/.tbrc.yml. To use the branch new-cool-feature for venue-provisioning-service I could write the following YAML:
overrides:
TouchBistro/tb-registry/venue-provisioning-service:
remote:
tag: new-cool-feature
~/.tbrc.yml configuration fileWhen I run this service, either with tb up -s venue-provisioning-service or using my playlist tb up --playlist online-ordering, tb will automatically pull the new-cool-feature tag and run that version of the image. This makes it straightforward for developers (or QA) to test another developer’s work, which is something we recommend in code reviews.
This only scratches the surface of what is possible with tb and the types of workflows it enables. For example, you can easily jump into a shell in a service by doing tb exec <service> bash. You can also jump into a service’s database, whether it’s postgres, mysql, or mssql, with tb db <service>.
Hopefully this post gives you some idea of what led us to creating tb and how you might be able to use it for your own projects. In a future blog post we’ll talk about how we use tb to run iOS apps in the simulator by passing a branch name.
Journey to open source
tb has been immensely valuable to us here at TouchBistro and we wanted to share it with the wider community since the problems we are solving are not unique to TouchBistro.
This seemed a big undertaking since tb was originally designed to automate toil encountered by developers TouchBistro. We needed to figure out how to generalize its functionality and refactor it in a way that could easily be extended by others. Luckily, we found that what we built was general enough to work with any services that can be run as docker containers and live in a docker registry. The nature of having to support a huge number of services and technologies and technologies had already forced us to treat something like “call `yarn run db:migrate` to run migrations before starting the server” to an instance of a generic “preRun” command that each service can define.
First, we implemented the strategy pattern to generalize logging into services such as NPM and AWS. We recognized that these were specific to our use case and made it possible to use different registries, and log into other services if needed.
We took inspiration from homebrew taps and created the concept of a tb registry which is a git repository with configuration files that tb uses to figure out which services and playlists are available to run. This allowed us to decouple tb from our specific services and tools and make it a generic distributed package manager for docker services and more.
tb is available for both macOS and Linux. If you think tb could make development easier for you and would like to know more check out the GitHub repository to learn about how to use it and all its available features. We would love to receive feedback on it and contributions.
Does TouchBistro sound like somewhere you might like to work? We are always looking for talented new team members. Check out https://www.touchbistro.com/careers/#open-positions to find the role that is right for you.
