How we build our APIs
My experience building APIs at Vendasta.
I joined Vendasta as a developer around ten months ago. Since joining, I’ve been impressed by how we build our APIs. Specifically, the workflow and the developer tools for building APIs.
I’m writing this post as a user of our developer tools at Vendasta, and as someone who works on some of our APIs. These tools were built by our Infrastructure team. Vendasta’s R&D organization have been using these developer tools for some time before I joined.
This post assumes that you have a basic understanding of Protocol Buffers and gRPC. Not required but I recommend reading the introduction if you’re interested.
You will find this helpful if you are:
- Interested to join Vendasta and wants to get a general idea of how we build our APIs.
- Curious about how growing R&D or software engineering organizations can efficiently build APIs.
Common inefficiencies around building APIs
From my experience, building APIs within larger organizations can be inefficient. Some common causes are:
- Lack of a streamlined process when defining APIs. It is time-consuming to go back and forth with other teams to agree on an API contract while building the API at the same time.
- No single source of truth of API definitions.
- Boilerplate code. You’ll find yourself writing the same code that could have been generated.
- Silos between backend and client teams. This is commonly an organizational design problem. Establishing systems to prevent silos may help.
How are we solving these inefficiencies at Vendasta?
Our approach
Driven by two of our core values at Vendasta: Innovation and Agility, we’re solving these inefficiencies by using a combination of open-source software, internal tools, and a streamlined process.
Innovation: We make the most of what’s already available by using open-source software. We then build the tools that we need on top of it.
Agility: We follow a standard workflow when building APIs. We start by defining the API. We automate most of the tedious parts of building APIs. Reduce the time we spend writing boilerplate code by generating them. Our R&D teams are capable of building an entire feature end-to-end. Which prevents silos between “backend” and “frontend” teams.
About our client and API tech stack
We build our backend services (microservices) using Golang. We build our web apps using the Angular framework. We manage and run our infrastructure on Google Cloud.
We do have other programming languages and tech. However, I’m going to keep the focus of this post around what we commonly use in our APIs and clients.
Breakdown of our workflow
Below are the main activities involved in our building APIs. Steps 1 and 3 are done manually, while steps 2 and 4 are automated through a build process and a CLI tool respectively.
1. Define the API contract
We use Protocol Buffers (proto3 syntax) to serialize the payload that we send between our services. We maintain a single git repository that keeps all of our API’s protocol buffer files.
If I want to build a new API, I start by defining the proto files in our branch in the API definitions repository e.g. request, response, and service. This gives me a chance to gather feedback around my API design before starting the implementation. Conversations of the decisions around the API design are all kept in the GitHub pull request as much as possible. These conversations can be used for future reference.
The API definitions repository is also the source of truth for our APIs. I can use it as a reference if I want to use an existing API.
2. Generate the gRPC server and client code
Every time I push a commit to my branch in the API definitions repository, the build pipeline will lint and build my updated proto files. It will then generate the Golang server and client code for every commit. And finally, the build will automatically commit the generated code to our Generated Code repository using the same branch name in the API definitions repository. I will use the generated code to implement the API interface in my microservice.
3. Implement the API
At this point, I can start implementing and testing my API without merging the API definitions branch to master. I only need to pull the generated code from my branch into my microservice’s repository using go get <generated-git-repo-url>@mybranch.
Generate the standard packages
We also have an internal CLI tool that generates the standard backend code for a new API. For example, I want to build a new CRUD API called User from scratch. I can start by writing my User schema, run a CLI command, and the Golang packages that I will need for my API are generated for me. I get the repository, service, access control packages, and model helpers for free! Saving me from writing hundreds of lines of code.
4. Generate the SDK
Using SDKs ensures that we are consistent in implementing API method calls in our clients. They also come with free commonly used functions like authentication and retry policies.
I can generate my SDK before implementing my API, as long as I have the generated gRPC client and server code from the updated API definitions file. The SDK will be installed as a library in the client that will use the API. The client can either be a web application (TypeScript) or a microservice (Golang).
We have an internal CLI tool that generates the SDK for us. If I want to generate an SDK that will be used in the web application, I will run for example generate-sdk --language=typescript. The SDK code has its folder in our API repository.
The SDK is automatically published as an npm library by our build every time a git commit is pushed. After it is published, the npm URL will be posted as a comment in my Pull Request. Using that npm URL, I will do an npm install in my web application’s repository. After installation, the API call methods are now available in my web application’s repository.
Using stubs
We use stubs so we don’t potentially block any client-side work that is dependent on the API. That way, Step 3. Implementing the API does not impede 4. Generate the SDK. Since the API contract is already defined after steps 1 and 2. I can start generating my SDK before I implement the API. I can generate the SDK with the stubs so any frontend-related work can be implemented in parallel.
Closing thoughts
Our tools and process around building APIs are force multipliers for us at Vendasta. They help us automate repetitive tasks. They help set coding standards and consistency, enabling developers to focus on solving problems.
Building and maintaining these tools require time and commitment. Stakeholders need to support the initiative, while the developers need to participate, use the tools, provide feedback and contribute back bug fixes or features.
I’m lucky to be part of companies like Vendasta that invest in improving developer productivity. It’s one of the many reasons why I enjoy working here.
We’re hiring!
Build APIs and web apps with us. Check out our careers page.
Thanks to Corey Hickson and Dustin Walker for reviewing my draft.
