Building Services at One Medical
By Jessie Chen and Wen Li
Working for a growing company has many benefits, including an expanding codebase. As the engineering organization grows however, owning a large shared codebase is often at odds with smooth continuous integration and delivery, which is key to rolling out new features. Having a monolithic codebase also limits the ability to scale isolated parts of the application that are receiving more traffic. At One Medical, our teams are organized by autonomous business domains that currently share a common monolithic architecture that has served us well up to this point. Some of the cool features we’ve developed in the past include on-demand video chat with providers, automatic symptom follow up, and machine learning (ML) procedure detection.
In order to alleviate some of the scaling challenges of working with a monolith, we are encouraging our engineering teams to look for opportunities to build a service and expand outside the monolith. However, this initiative is not without its own challenges. Most of our engineers are focused on working in a business domain — how would they know where to start in creating a service? What to work on first? Even if they get one off the ground, another engineer will encounter a similar set of service creation challenges and choose a different approach and setup with their own process and naming conventions. This can keep happening until we are supporting many unique service setups that are aiming to achieve the same goals. Ramp-up time for a new team member would be expensive even if they came from another engineering team. Each setup would also need its own planning session with infrastructure, site reliability engineering (SRE) and security teams sign-off. This all leads to a scenario that does not scale, and made us think about how we could make service development accessible and scalable for our product-focused engineering teams.
Build a Production Quality Example
We started by building service templates to reduce the friction from starting a project, define best practices, and allow product teams to move fast with consistent scaffolding. Instead of each team facing a paradox of choice when embarking into new territory, the same repeated problems of setup are taken care of with an audited set of tools and configuration. Without shared plumbing, it’s likely that each team would come up with a different solution to the same problems and we would need to support all of them. One of our core principles at One Medical as a company is to continuously improve and eliminate duplicative work. We constantly ask ourselves how to save time and effort on repetitive tasks so we can focus on delivering value to our patients.
To build the template, we researched what other companies have done, and analyzed what is required to build a service and what’s the fastest and safest way for developers to start writing code in order to deliver business value. Then, we took the results, along with our One Medical standard tool sets, to build a production quality hello-world application focusing on the following areas: lifecycle, state, networking, binding, and beyond. While the application only responds to 200 OK health checks on the surface, it does a variety of interesting things under the hood.
For example, the application provides the infrastructure-as-code module to spin up cloud services easily, so that a given application can be deployed on a computing instance. It promotes the approved framework and language to power the application. It surfaces naming conventions, scaling and inconsistency issues that we end up solving as a group and build them in the example. It wires up with the delivery pipelines so that the team can follow along the standard way to do continuous integration and delivery. The example also shows how an application reads environment variables out of the secret store. It demonstrates how a service can connect with the rest of the system and spell out the inter-service communication in code.
Since everything is in code and is live and testable in various environments, it further allows other teams to build even more standard examples with it, such as how we log and instrument the application and how we raise the alarm when there is an issue. Teams are set up with built-in observability plugins that are already tested and documented. Though it was coincidental, we learned our approach was similar to the process described by Thoughtworks on creating tailored service templates.
So what happened after we built the hello-world example? Was it a success? There are still lots of configuration and granting of permissions by hand in order to successfully boot up the application in various environments. We started brainstorming the idea to build an application registry/onboarding system that offers the capability for applications when the integration is defined and the right permissions are given to allow the service to run. This application registry/onboarding system saves us lots of time and effort in configuration and since it’s written in code, engineers can build more integrations within it in various iterations. Since we launched the application registry, we have seen an upward trajectory in the number of services in our portfolio.
Evolution and Adoption
The user and creator feedback loop is something we rely on heavily to develop commonly used and well-loved features in our application. The product engineering teams in this case are the users for the service tools. In each case, there are self-selected early adopters that want to test out the tools before they’re ready to be more mainstream, oftentimes helping to shape the details and provide fast, valuable feedback. At One Medical, the product development team is encouraged to join working groups and guilds, results-oriented cross-team and cross-functional inclusive groups that work on goals outside of the standard product development team feature development lifecycle.
The Service Architecture Guild is an example where knowledge is shared about the various initiatives impacting service architecture. The members are cross-functional engineers from multiple teams across the organization who want to share and learn more from each other on service architecture, and help bring innovation to the company. It also gives a chance for engineers to share best practices downstream to individual teams and to look for opportunities on their teams to incorporate and experiment on the infrastructure that is being built. Part of the scalability of service creation conventions is that we as a group can help enforce the path forward for newer services presented at the guild. We’ve also made it a practice to incorporate the initiatives from these guilds into quarterly planning for teams to make a conscious effort in investing in the long-term engineering strategy initiative. This structure is still evolving as we look to make improvements to our process every quarter to deliver results.
Conclusion
With the success of rolling out hello-world and application registry/onboarding as well as continuing participation in the Services Architecture guild, we saw decent growth in the last couple of years in the One Medical services portfolio. If you are interested in joining us in our effort to create tools and services, check out the engineering career page.
