Golang Microservices Challenge: Designing an extensible, easy-to-use, and testable HTTP client

As a software engineer, backend developer and being part of the development team at Augury, we continuously face challenges when working with Microservices architecture and Kubernetes Orchestration.

Augury, as the leading company in the Machine Health category, provides its solutions in the IoT sector to its customers as a platform, and this platform is hosted on cloud services and consists of many distributed services that talk to each other. This modern architecture spawns many challenges.

Why did we need an HTTP Go client?

Today Augury counts 30+ microservices that depend on and communicate with each other. Each microservice can communicate with others through a special SDK library made for each microservice.

Making HTTP requests using the SDK can be done in various ways and conventions, for unification and convention we use our own custom HTTP library/layer, this library is a wrapper for the standard net/http Go library.

The Challenge

Digging into the details, some of our microservices were missing a graceful shutdown, which means that whenever one of the microservice pods was removed, due to K8s scaling, all running requests were dropped.

Consequently, the calling microservice that initiated the request, got no response.

To directly solve this issue we only needed to add a graceful shutdown to our microservices, but if you think about it, this issue is a special case of a known challenge in microservices communication — transient faults that occur due to service unavailability for a few seconds.

In order to solve such an issue, we needed to have transient fault tolerance capabilities in our microservices, or in our microservices communication layer/proxy. How did we achieve that?

Brainstorming

After a few discussions in the Backend Guild Team, which is the team that gathers all backend developers in the company and discusses new technologies and initiatives, we decided to add a Request Retry Mechanism in our HTTP requests between microservices.

To achieve that we decided to build a Golang HTTP library, to fulfill our logical needs and implement the retry mechanism. One of the main reasons we decided to implement the retry at the application level and in this library, is because in this case you can pass the full control of this behavior to the specific developer who initiates the request.

The big question

What makes a library a good one?

One of Augury’s conventions is that we all use the same customized HTTP client that is used by all our developers and across all Golang microservices. This library was missing some important aspects of a “good” developer’s library. The library was hard to extend, we needed to make it Extensible, so we could easily add more features to it in the future.

Another problem we had is that this library couldn’t be Mockable, one of the most powerful things that I learned in Augury from other Augurians Devs, that may be trivial to others, is that we need to be able to test every code unit/package/library/service everywhere. So this HTTP client needs to have mocks, so we can test in all microservices when testing flows that have calls to other microservices.

In the HTTP client library case, we found these aspects to be very important to optimizing and improving our code to support future challenges.

How did we achieve that?

Keeping the developers’ experience in mind, using an HTTP client can be simple or complicated.

One of the commonly used and elegant patterns I ran into when I started using Go, is the Builder pattern.

The main idea is to create a “Request” struct with “RequestOptions” as one of its properties, with a request options customization ability — override or add options — in order to pass full control to the developer over the HTTP request that is being initiated.

type Request struct {
   oauthToken string
   method string
   endpoint string
   body interface{}
   response interface{}
   options RequestOptions
   withRetry bool
   client api
   headers map[string]string
}

Using builder pattern in the HTTP client library seemed to be nice, first of all, as a developer and as the librarys’ “user” we want to build a request struct step-by-step, which is in each step we can add a new option or override the existing one on the request, like “RetryInterval” and “Timeout”.

request := api.NewRequest(http.MethodGet, endpoint, machine)
 .Timeout(TIMEOUT)
  .Retry()
   .RetryInterval()
    .MaxRetries(3)

As a developer, and keeping the developer’s experience in mind, this one line of code is very clear, readable, understandable, and clean. Everyone who reads this line can understand what is happening and what will happen in runtime.

Interfaces in Golang in my opinion is a very powerful concept and “tool”, we can define multiple types with the same behavior, consequently decoupling and separating implementation from concepts and design.

type IRequest interface {
  Retry() 
  IRequestMaxRetries(n int) 
  IRequestRetryInterval(n int) 
  IRequestTimeout(timeout *time.Duration) 
  IRequestExec() error
}

All the properties of the Request struct are internal/private means that outside the Request package we cannot access these properties. Adding an IRequest interface can expose to the user all the possible functionality that the request has, Retry(), MaxRetries(), and Exec().

func (req *Request) RetryInterval(n int) IRequest {
  req.options.RetryOptions.RetryInterval = n
  return req
}

Pointer receivers are another Go feature that comes to play a crucial role in this implementation. The Request type will implement each of the IRequest interface funcs, and at the end of each func we will return the struct pointer so we can chain calls on the Request instance.

This implementation gives us some sort of flexibility around this layer, the IRequest struct can be easily extended in the future by adding more properties, and the new property can be implemented as a Receiver of the Request type.

Wrapping up & Inner Thoughts

Selecting the best solution can be challenging because every solution creates its own tradeoffs, cons and pros.

Challenges in the microservices world are many, some are general and common, and some can be very specific to the organization’s environment and architecture. Before selecting the solution mentioned above we did investigate some alternatives where the main discussion was on “in which layer are we going to add the retry mechanism?” would it be on the communication layer level — K8 containers? Or would it be on the internal proxy layer? Do we need to implement a Service Mesh?

If you think you have some great ideas that can enrich this challenge solution, please leave a comment or contact me.