Native HTTP/S client programming in Go (without any external libraries)
Hi, everyone. In this article, I present working with the built-in Go net/http library to create HTTP clients without using external libraries like restyor requests .
Go 1.22 adds many new features to allow developers to write modular code while writing HTTP clients and servers. In this article, we are more interested in writing different clients using native Go libraries.
Let’s jump into action by building a simple HTTP client.
Note: Please note I interchange words object and instance for struct instances.
HTTP Go Client & Variations
An HTTP client makes an HTTP request to an HTTP server. Curl is a well-known, language-agnostic HTTP client used by hundreds of thousands of developers world wide. Let’s see different variations of HTTP clients from hereon.
A Hello World client — variant 1
Go already ships a well-designed client in the standard library, and we can make an HTTP Get request to an endpoint using http.getfunction.
The prog_01.goprogram imports the net/http package and uses a function called Get to make an HTTP request to the domain: https://httpbin.org to a path /get. This function is quick and easy, but in the real world HTTP, clients are complex. They need to configure HTTP headers, set a timeout, or configure query parameters, etc. We will see options available in Go to make real-world HTTP requests.
Improved client with a custom timeout — variant 2
To improve the previous client, let’s take a different approach. Create a *http.Client instance and add a timeout of 5 seconds. This is to avoid the client indefinitely hanging on the HTTP server. This cannot be done with a simple client. We should use http.Client struct and call a Get method.
This version of the client is slightly different from the previous version. Here, we are instantiating a struct called http.Client and configuring properties on it. One of the properties is Timeout which specifies the time.Duration type. Because duration is not a native Go type (int32, int64), we need to convert the integer to duration by multiplying with time.Duration. This client has the same functionality as the hello-world one but with a time out.
Why set timeout on a HTTP request ?
Setting a timeout on a client request is a good practice and provides a quick feedback loop to client if a server is overloaded or protected by a firewall. You can checkout this great article by Zalando to know the importance of setting timeouts.
https://engineering.zalando.com/posts/2023/07/all-you-need-to-know-about-timeouts.html
Quoting the relevant suggestion from article:
The default timeout is your enemy, always set timeouts explicitly!
As you saw with the `Timeout`, you need to set the properties of the client. Let’s go a level deeper to customize a HTTP request.
Advance the client with configurable Requests — variant 3
We can even configure a client at the request level. There is a constructor function called http.NewRequest to create a *http.Requestand pass it to the client do method like the below. We can also attach HTTP headers on that request with the `req.Header.add` method.
The returned response object will have a reference to the original request.
In this variant, we are manually creating an HTTP Request, and using the client’s do method to request on-demand. This means we can create a request at a prior time, before doing an actual request. Also, we can:
- Modify the client & requests on-demand
- Keep client fixed, and change requests on demand
Anyway, this provides greater flexibility to the initial version of http.Get. We can go a level deeper to control how idle connections are handled for a given client. This is helpful to create a pool of connections for a given client to utilize for multiple requests.
Make HTTP requests with a configurable transport — variant 4
A Transport defines how idle connections are handled for a given HTTP client like the number of maximum idle connections allowed, Maximum idle connections per host, or idle Connection Timeout, etc.
These transport options can be defined once on a client to reuse them across multiple requests. We can create a new transport directly using *http.Transport struct (no constructor required). Let’s see how to make the same request as the previous one but using a Transport.
In prog_04.go, we create and plug an HTTP Transport into a Client instance. The rest is the same as the previous example. As you can see from line numbers: 12–14, transport is defining hard limits to `keep-alive` connections spawned by client. Next. let’s see how to make a CA certificate verification in a Go client.
Setting client TLSConfig for trusted CA verification — variant 5
TLS (Transport Layer Security) is a security protocol that powers HTTPS encryption. All HTTPS servers use SSL certificates issued by a standard CA (Certificate Authority). For example, httpbin.org got its certificate signed by Amazon Root Certificate Authority.
We can enhance client security and stop middle-man attacks by verifying the server CA certificate ourselves. For that, export the certificate of https://httpbin.org server to httpbin.cer and place it in your client program directory. You can also load it over the network instead of a local filesystem (for dynamic certs).
We can create a quick function that parses an x509 certificate (Public key) and returns a certificate pool.
The getCertPool function uses the crypto/x509 package to parse a given certificate file and adds the parsed certificate to the pool. If we need to verify multiple hosts, we can add those host certificates to the same pool.
This pool should be wrapped into a TLS config object and passed to the TLSClientConfig struct field of an http Transport (Line No: 48 in prog_05.go)
Verify Private CA certificates generated for a local server leveraging the above technique.
This concludes the 5 variations of using a built-in net/http package to make HTTP requests. The Do method allows one to perform different types of methods (GET, POST, PUT, PATCH, HEAD, DELETE, and OPTIONS)
Once again, the Libraries like restyand requests are built to reduce boilerplate in Go code but can abstract important details about how things work underneath.
This article is one such attempt to visualize how the `net/http` package can be used to create different types of HTTP/S requests in Go.
Now lookup source code of any third-party library and everything will make sense 😎 with knowledge of net/http . That’s another way of learning the package.
