Sending Content-Length in Go

TLDR

Use *bytes.Buffer, *bytes.Reader, or *strings.Reader if you would like Content-Length being set in the HTTP client request sent.

Background

Last week, we have encountered an issue when we were trying to enable HTTP/2 in our firewall. The issue happened to one of the API calls that was made against the application behind the firewall. The HTTP response code that was being received by the application was 411 Length Required. Since then, HTTP/2 was reverted and we have decided to take a closer look at what could be happening.

For those who are unfamiliar with what HTTP 411 Length Required is, here is an explanation from MDN.

411 Length Required

Based on this explanation, it seems to be pretty clear on what the client applications need to do to correct this error.

Begin Investigation

First, we need to check if whether our client application is sending the Content-Length header. Since there was never such a need for it before, it wasn’t surprising that we have not set such a header explicitly in our application. So we went to look at the Go’s net/http documentation. In the documentation, it is mentioned that

For client requests, certain headers such as Content-Length and Connection are automatically written when needed and values in Header may be ignored. See the documentation for the Request.Write method.

So this means that we shouldn’t need to explicitly set the Content-Length header.

Second check

Second, looking at our client applications, not all the API calls to the server application have failed. There is one particular code path that was having an issue. So that must be something unique in that path and it turned out to be true. When we were constructing the HTTP request, we pass ioutil.NopCloser as the body instead of a usual *bytes.Buffer , *bytes.Reader or *strings.Reader .

But does this matters? Turns out it does, in the same Go’s documentation, it is also mentioned that

If body is of type *bytes.Buffer, *bytes.Reader, or *strings.Reader, the returned request’s ContentLength is set to its exact value (instead of -1), GetBody is populated (so 307 and 308 redirects can replay the body), and Body is set to NoBody if the ContentLength is 0.

Code verification

So how can we test it? In Go, we have net/http/httptest and net/http/httputil that we can use to easily test this.

With ioutil.NopCloser (error omitted for clarity)

If you run this example in Go’s Playground, this is output that you will see

2009/11/10 23:00:00 Response Body:
POST / HTTP/1.1
Host: 127.0.0.1:2
Transfer-Encoding: chunked
Accept-Encoding: gzip
User-Agent: Go-http-client/1.1
2
{}
0

And by changing line 19 to reqBody := bytes.NewBufferString(`{}`) , this is the output that you will see

2009/11/10 23:00:00 Response Body:
POST / HTTP/1.1
Host: 127.0.0.1:2
Accept-Encoding: gzip
Content-Length: 2
User-Agent: Go-http-client/1.1
{}

Conclusion

So, there we are. We now know how to solve this issue. However, we are still finding out why the Content-Length header is required when HTTP/2 is being turned on from the firewall.

Thank Emmanuel T Odeke for showing me a quicker way to prove this behaviour in the GitHub issue.