How We Build gRPC Services At Namely

Over the past year, we’ve been transitioning two applications into gRPC services: a monolithic Ruby on Rails application and a .NET application. Along the way we’ve experimented with best practices, built tools to help with the process, and learned a lot about what works and what doesn’t. In this post, we’re going to talk about:

• Managing protocol buffer definitions
• Generating the appropriate language files from the definitions
• Deploying these services

What we’re NOT going to talk about (so you can turn back now if you want):

• Setting up secure connections for our services
• Request tracing
• How we implement the services themselves

Why gRPC?

In a word: structured. Although building a JSON API is simple to do, you have a higher risk of breaking service consumers because of its unstructured nature. You can achieve a variance of structure using a format such as Swagger or a Consumer Driven Contract. But it’s not a guarantee.

gRPC and protocol buffers promise a certain structure that will be sent without exception. gRPC also defines the protocol in which the data is transferred and the libraries for it. JSON is usually sent over HTTP, but gRPC doesn’t leave this up for debate. gRPC also provides the framework for error codes, authentication, and streaming messages.

Overall, we’re in the business to deliver value to our customers, not discuss which JSON standard to use. JSON-API? HAL? Hypermedia at all (shivers)? We just want to ship. gRPC lets us do that easily.

How We Manage Protocol Buffer Files

Our first approach attempted to keep protocol buffer files (.proto) within the same project they were implemented. However, we found this to be cumbersome because whenever a new service consumer was being built, the developer would have to copy the generated files out of of the service itself into their new project.

Our philosophy shifted to keeping the definitions and the implementations separate because it stays true to the idea of protocol buffers: describe what your service will do and abide your service to that contract.

After realizing that we can kill two birds with one stone, we moved our protocol buffers into a separate repository on Github called the protorepo:

This new approach allowed us to gain many advantages. First off, our new single repository for protocol buffers acts as an inventory of all Namely’s services using gRPC. This grants us the pleasantry of knowing what those are at a quick glance.

Now that the definitions and implementations were separated, team collaboration was more easily focused on how services communicate with each other rather than getting distracted by implementation details.

Developers were now only notified for definition changes, rather than spam alerts from pull requests such as “move spacing 2 pixels to the right”. The notifications from a chosen repository became more meaningful. This spurred more conversations with members from multiple teams chiming in on service proposals, proto changes, etc.

By having a single repository for all service definitions, it allowed us to localize all of our build tools into one place.

How we generate our language specific files

One of the great things about gRPC and protocol buffers is the amount of languages it supports. As of writing, it currently supports 10 languages(!). Although, a problem with this is installing the tool to generate the specific languages files from the proto definitions can be a pain, and frankly, why do something a computer can do it for you automatically?

Our folder structure in the protorepo is standardized to enable automatic generation of language specific files. For example, if you have a service called "companies", you'd have a folder called the same thing:

$ pwd
# /Sites/namely/protorepo
$ tree
├── | companies
│   ├── .protolangs
│   └── companies.proto

Let's focus on a single file here: .protolangs. This file informs our build tools which languages need to be built for this service. For our companies service, it looks like this:

$ cat .protolangs
go
csharp

Each language is separated with a new line. When changes are pushed to the repository, our tool runs through each folder, reads the .protolangs file and runs the appropriate generator for all of the .proto files in the directory.

You might be thinking, “what awesome tool did you make for this?”. It’s not fancy by any means. In fact, it’s just long bash script. If you dare, here is a link to it.

After you’ve added all of the languages you care about to the.protolangs file, you'll need to create a new repository in the format protorepo-{service-name}-{language}. So if you have a project called “heimdall” that uses the Go language, your new repository would be “github.com/namely/protorepo-heimdall-go”. These repositories are where we push all of the newly generated protocol buffer files once any commit is made to the protorepo repository.

Here’s a rundown of our generation process:

• Change into a service specific directory
• Read every language the .protolangs file declares
• Run a docker image relevant for each language
  (We support most languages — if you’re interested, check out our docker-protoc project: https://github.com/namely/docker-protoc)
• Take the newly outputted files and move them into the language specific repository
• Do a simple git diff to see if anything changed
• If the project did change, commit the changes, and push
• Rinse and repeat for every folder

This process has made it incredibly easy to write a service definition using the interface definition language provided by protocol buffers and generate the files you want quickly. But where do we go from here?

How We Deploy Our gRPC Services

We’re huge fans of containers at Namely, specifically the Docker toolchain. We have now adopted a system that uses Kubernetes on AWS for a majority of our deployment and orchestration.

When a team is building a service, it’s required that the build process includes building a Docker image and pushing it to our private Docker repository (we use quay.io internally). Whenever a commit is pushed to a project, we take the first 8 characters of the commit’s SHA and use that as the Docker image’s variant.

For example, if you push a new commit that the first 8 characters are bc79d9cb and your project is called companies, then your Docker tag will look like:

registry.ourinternaldns.com/namely/companies:bc79d9cb

It is natural to be thinking “Yikes, but isn’t that using a ton of space?”. Short answer: yes.

The reason we do this is because we allow developers to deploy any version of their code to a development Kubernetes cluster. This means any branch, feature, etc. can be tested by our QA team and run a blackbox test suite against it. Everyone gets a clean environment on demand (out of scope for this post, but it’s damn cool).

However, the process is a little different once we actually deploy to our customers. Instead, we use release tags by using the releases feature on Github.

When a tag is created on a repository, our CI pipeline builds a Docker image using the new release tag as the variant. By using the companies service above as an example, the new Docker tag would be:

registry.ourinternaldns.com/namely/companies:v1

Once we have our newly built image, we modify another monorepo full of Kubernetes manifests for each project (similar to our protorepo):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: companies-prod
  namespace: production
  labels:
    name: companies-prod
    component: companies
spec:
  replicas: 3
  template:
    metadata:
      labels:
        name: companies-prod
        component: companies
    spec:
      containers:
      - name: companies
        image: "registry.ourinternaldns.com/namely/companies:v1"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 50051

This process is still manual, e.g. someone modifies these yaml files by hand. But! We are considering ways to improve it. For example, we’re using Helm for our development cluster (the one developers and QA use) but we haven’t yet switched to using it for production usage.

There are many ways to organize services, and while ours is far from perfect, it has mitigated the pain and confusion around building services with protocol buffers and gRPC. There’s no longer a mystery of where definitions are stored and how the language specific files are generated. It is handled automatically for you!

I hope this post was interesting and would love to chat and answer any questions you might have about it. Cheers!