Crafting Robust HTTP Requests: Building a Requester in Go
In the world of making programs and applications, sending API requests to other services is super important. Go is great for this. But what if we could make our own tool to do it even better?
In this guide, we’ll create a special tool in Go that can send API requests really well. Before we get started, let’s think about what makes a tool like this great and why we should make one.
A good tool helps us send API requests just the way we want. It also understands when things go wrong and can keep our secrets safe. Knowing this, we’ll see why building our own tool is a big help for our projects.
Whether you’re new to Go or already know a bit, this guide will teach you how to make your own tool for sending API requests. This tool will make your work with the internet easier and better.
What is a requester?
At its core, a requester is a component or tool responsible for making HTTP requests to external services, APIs, or web servers. These requests serve as a means of communication between your application and remote resources on the internet. Requesters play a pivotal role in retrieving data, sending data, and interacting with various online services.
Here are some key aspects that define a requester:
- HTTP Requests: Requesters are primarily used to construct and send HTTP requests. These requests can take various forms, such as GET requests to retrieve data, POST requests to submit data, PUT and DELETE requests to update or remove data, and more.
- Versatility: A well-designed requester is versatile, allowing you to tailor each request to your specific needs. You can customize the request headers, set query parameters, define the request method, and include a request body, among other options. This flexibility is vital for interacting with a wide range of APIs, each with its unique requirements.
- Handling Responses: After sending a request, the requester awaits a response from the remote server. It can then process this response, which typically includes important information like status codes, response headers, and response bodies. On the other hand, you don’t handle the response and may give all response control to the developer.
- Error Handling: Requesters are equipped to detect and manage errors that may occur during the request-response cycle. These errors can range from network issues and timeouts to server errors and authorization problems. Handling errors gracefully is essential for maintaining the stability and reliability of your application.
- Performance: To optimize performance and resource utilization, requesters often incorporate features like concurrency, rate limiting, and caching. These mechanisms help manage the flow of requests and responses efficiently, preventing overloading of servers and minimizing response times.
Boosting Your Requester’s Reliability
Retrier: A Second Chance for Requests
Sometimes, when your app talks to other services on the internet, things can go wrong temporarily. Maybe the service is busy, or there’s a tiny glitch in the network. A Retrier is like giving your request a second chance.
Imagine your app sends a request that doesn’t work the first time. Instead of giving up, a Retrier says, “Let’s try again a few times.” It keeps retrying until it either works or it decides it’s time to stop. This can be super helpful since the internet isn’t always perfect.
Circuit Breaker: Protecting Your App
Sometimes, the services your app talks to have a bad day and don’t work well at all. If your app keeps asking for things from a broken service, it can make things worse. A Circuit Breaker is like a smart switch.
When a Circuit Breaker sees that a service is acting up, it says, “Okay, let’s not ask that service for a while.” It stops your app from bothering the broken service for some time. This way, your app stays safe, and it can look for other ways to get the job done.
Timeout: Don’t Wait Forever
Imagine you’re waiting for a friend, but they never show up. You don’t want to wait forever, right? The same goes for your app when it’s waiting for a response from a service. A Timeout is like setting a timer.
When your app sends a request, it says, “I’ll wait for a response, but only for this long.” If the response doesn’t come back in time, your app stops waiting and moves on. This helps your app stay snappy and not get stuck waiting endlessly.
Building a Requester in Go
In this section, we’ll explore the code for building a custom requester module in Go, designed to make HTTP requests to external services and APIs more reliable. This module incorporates key features like retrying, circuit breaking, and timeouts to enhance the requester’s robustness.
Package and Imports
This is the package declaration and the imports section. Notably, we use the github.com/slok/goresilience package to implement resilience patterns like retrying and circuit breaking.
We decided to use the github.com/slok/goresilience package in our custom requester module in Go for a simple reason — it helps make our HTTP requests to external services and APIs more reliable. Think of it as a safety net for our code. This package allows us to handle temporary errors, like when a service is temporarily unavailable, by retrying the request or “breaking the circuit” to prevent further requests if there’s a problem. It’s like having a backup plan in case things don’t go smoothly. Plus, using this package makes our code easier to maintain and understand, ultimately giving us a more robust and dependable requester module that can handle tricky situations in distributed systems.
Requester Interface and Structs
To create our requester, we first declare an interface called Requester. This interface outlines the methods that any requester should implement. Additionally, we define RequestEntity to hold information related to the HTTP request and Request to manage the requester’s configuration, including timeout and middleware setup.
Separating Request and RequestEntity makes working with HTTP requests easier by organizing the specific request details (in RequestEntity) separately from the overall request behavior and settings (in Request). This helps keep your code clean and understandable, lets you reuse common request setups, and simplifies testing and maintenance, making your work with HTTP requests more manageable.
HTTP Methods
Next, we implement HTTP methods for GET, POST, PUT, and DELETE requests. These methods are responsible for constructing the HTTP requests, applying headers, and processing the responses.
The sendRequest function serves as an internal utility for sending HTTP requests. It encapsulates the logic for creating requests, setting headers, and handling errors. This function also integrates resilience patterns based on the middleware added to the requester.
Middleware Functions
We provide middleware functions such as WithRetry, WithCircuitbreaker, and WithTimeout to configure and add resilience patterns to our requester. These functions enable us to fine-tune our requester’s behavior, including retry attempts, circuit breaker settings, and timeout durations.
Load Function
The Load function is responsible for loading all the configured middlewares into the runner, making our requester ready for action.
In the upcoming sections, we will delve deeper into how to use these components effectively to create a resilient requester in Go, capable of handling real-world scenarios with ease.
Using Our Custom Requester Module
Now that we’ve introduced our custom requester module, let’s put it to use with a practical example. We’ll demonstrate how to create a requester, configure it with various resilience features, and make an HTTP request.
Creating a Requester
To get started, we create a new requester using the NewRequester() function:
This initializes a new requester with default settings.
Configuring Resilience
One of the key advantages of our requester module is its resilience features. Let’s configure some of these features for our requester:
Retry Configuration
We can configure the requester to retry failed requests multiple times with a specific wait time between retries:
In this example, we set up a retry mechanism with a base wait time of 200 milliseconds and a maximum of 3 retry attempts.
Circuit Breaker Configuration
To protect our application from continuous failures, we can configure a circuit breaker:
Here, we configure the circuit breaker to open after 3 consecutive failures, require at least 1 successful attempt during the half-open state, and wait for 5 seconds in the open state before attempting to make requests again.
Timeout Configuration
We can also set a timeout for our requests to prevent them from waiting indefinitely:
In this example, we set a timeout of 30 seconds for our requests.
Load the Configurations
We need to load all our middleware configurations.
Additionally, you can use these middleware configurations in the chain pattern.
Making an HTTP Request
With our requester configured, let’s make an HTTP GET request to a sample endpoint:
In this example, we create a RequestEntity with the target endpoint and use the Get method of our requester to send an HTTP GET request. We then handle the response and any errors that may occur.
Conclusion
Our custom requester module simplifies the process of making HTTP requests in Go while adding resilience features to handle various scenarios gracefully. With the ability to configure retries, circuit breakers, and timeouts, your application can communicate with external services and APIs more reliably, ensuring a smoother user experience and better error management.
If you’d like to explore the full code and implementation of the custom requester module in Go and a collection of other useful Go modules, you can find them in our open-source github.com/useinsider/go-pkg. Don’t hesitate to be a contributor to this repository and join our community of developers working together to build better software.
I hope you enjoyed this article. If you have any questions, please feel free to contact me on LinkedIn or comment below.
Follow us on the Insider Engineering Blog to read more about our AWS solutions at scale and engineering stories. Here are more stories you may enjoy.
