WTF Dial: HTTP API
Now that we have our storage layer finished, we need some way for users to access and manipulate that data remotely. The standard approach seems to be JSON over HTTP so that’s what we’ll be using. It’s simple and it’s easy.
In the Go world there seems to be a lack of consensus on how to write an HTTP API. You can see this in the vast number of HTTP frameworks written for Go. I’m no different and I have my own particular way of writing HTTP API servers in Go.
You can follow along with the code for this post by looking at this pull request. As always, feedback and questions are welcome!
Advice: Stick with the standard library (mostly)
The purpose of a framework is to abstract away the underlying complexity of a system. However, the net/http package is relatively simple once you get used to it so a framework just adds cognitive overhead.
One place where it’s useful to add library support is if you are using REST-style routes. We’ll be using Julien Schmidt’s httprouter package to handle this part.
Integrating HTTP into the domain interfaces
Our HTTP API provides a way to find, create, and update dials. Coincidentally, that’s exactly what our wtf.DialService is for too. However, the service declares functions to implement and our server provides an HTTP handler. It can’t implement the interface on its own.
We can resolve this by providing a Client in our http package which implements the wtf.Client interface and translates those function calls to HTTP calls. That allows us to use our HTTP dial service interchangeably with any other dial service implementation.
Let’s dive in and look at some code.
I use subpackages to abstract away dependencies entirely — that means that users of the package shouldn’t ever use the package it wraps. In our case, “wtf/http” wraps net/http so it should handle everything relevant to HTTP.
We need to handle the entire process of serving our API so we’ll create a Server type to do this:
We’ll also need to encapsulate the opening and closing of our Server so that the caller doesn’t need to know about the internals:
That’s the whole server. It’s only responsibility is to open the socket and serve the handler.
NOTE: Because the handler is served in a goroutine, your main() function which calls Open() must hang the process itself so that the process does not close. For example, this can be done by waiting on a signal using os/signal.
Since the server is managing the socket then we need a Handler to actually process each request. We could do this with a single Handler type but I like to break them up into a handler per service just so that they don’t get too large and unwieldily.
We’ll create a parent Handler to delegate requests to each service handler:
This handler embeds our DialHandler and then checks the URL path prefix for each request to delegate.
The Dial Handler
Our DialHandler is where the meat of the code is. Because it is part of the http layer, it’s sole responsibility is to translate incoming requests to our model and then translate results from our model back to HTTP responses.
First let’s look at the type and the constructor:
Our handler is composed of 3 parts:
- An httprouter.Router to manage REST-style routing.
- A DialService for that we’re translating HTTP calls into.
- A log.Logger for logging an errors.
The most important part to note is that we are just wrapping a DialService. This means we can inject different implementations for different purposes:
- Use mock.DialService for testing.
- Use bolt.DialService to persist data.
- Use http.DialService to build a proxy.
This abstraction allows us to not care what the underlying implementation is. We could even build a wrapper DialService implementation that logs metrics for each call.
Our first route we’ll look at is “POST” which will create a new dial.
Let’s break this code down:
- L5: We need to decode our incoming JSON request. I like to add an unexported request type inside my http package to structure it.
- L6: I use a generic Error() function to handle all errors so that they’re uniform and logged appropriately. It’s structure is inspired by the standard library’s http.Error() function.
- L9–11: We don’t allow Token to be encoded in the wtf.Dial struct for security reasons. It wouldn’t be good if we accidentally returned that field to the user. Because of that we need to pass token as a separate field in the request and copy it to the Dial in the handler.
- L14: Now that we’ve translated our HTTP request we simply call the CreateDial() function on our DialService.
- L16: If there was no error then we can simply encode our response with our updated dial object. We’ll use the encodingJSON() utility function to handle logging if there is an encoding error. We’ll cover the utility functions in just a minute.
- L17–18: If our service reports that we have invalid data then we return an HTTP 400 (Bad Request) status to the user.
- L19–20: If the dial already exists then we return an HTTP 409 (Conflict).
- L21–22: Finally, if another unexpected error occurs then we use the generic HTTP 500 (Internal Server Error). This could occur if we’re using bolt.DialService and there’s a disk error. We’ll look at 500 errors in more detail when we look at the Error() utility function later.
Now that we can create dials, we need to add a “GET” route to fetch them.
- L3: The router handles parsing of the “:id” field in the route we created in the constructor. It’s made available to us by using the ps parameters object.
- L6: We’ll invoke the Dial() function on DialService to retrieve the dial.
- L7–8: We don’t have any defined errors that we expect from Dial() so all errors will be returned as a 500.
- L9–10: A nil returned dial should return an empty response with a 404 code so we use a NotFound() helper utility function. We’ll look at this further later.
- L12: If we successfully retrieved a dial then we can encode the response back to the body. Again, we can use the encodeJSON() helper here.
Finally, we need to be able to set the level of an existing dials so we’ll add a “PATCH” route. We’re using PATCH instead of PUT because we’re just changing part of our dial resource and not overwriting the whole thing.
- L4–8: Again, we’re decoding our JSON request body into a struct.
- L11: Here we set the level of the dial and pass in our Token for authorization.
- L12–13: If no error occurs then we can encode an empty response which will also set the status code to 200 OK.
- L14–15: Because we are expecting the dial to exist we need to handle the error case where it isn’t found. If there’s no dial then we’ll return a 404 Not Found.
- L16–17: We also need to handle the fact that the token might not match. In this case we’ll return a 401 Unauthorized.
- L18–19: Finally, if an unexpected error occurs then we’ll log a general 500 error.
Utility function: Error()
There are a handful of small helper functions I’m using to simplify the main code path. The most common one is the Error() function:
This starts by logging the error locally. Next, if we have an internal error then we want to hide it from the end user. For example, it may contain sensitive information about your system such as file paths.
Finally, we’ll set the HTTP response code and write out the error using a generic error response. Since all our responses include an Err field we can encode this generic version and all our responses will be able to decode from it.
Utility function: NotFound()
The NotFound() is very simple but it’s common enough that I like to have a function for it. It’s used when an object is fetched but not found. This isn’t necessarily an error, in my opinion, so I just return an empty response with no error set.
Utility function: encodeJSON()
JSON APIs need to deal with encoding to the http.ResponseWriter constantly. If it fails then it’s either because the value could not be serialized to JSON or the writer stream closed. Either way there’s nothing we can do so we just need to log an error. That’s exactly what this helper does:
I like to add a client to my HTTP package so that it can implement my service interfaces. It also makes it so code that uses my HTTP API doesn’t have to worry about the details of HTTP.
Our wtf.Client interface is simply a container for attached services. Our http.Client will do the same thing except that it’ll hold some additional HTTP related data — namely the URL that it’s connecting to.
There are 2 important points to note:
- The dialService is embedded in Client so we just need to return the pointer to it in our DialService() function.
- We are attaching a pointer to Client.URL to dialService.URL. This means that once we create a new Client we can update the URL and all the services will have the updated URL since they are just pointers.
Our DialService struct itself is very small:
Notice that there’s no reference to the parent client. It can simply be used as a standalone object.
First we’ll look at creating a new dial from the Client. I intentionally avoided helper functions in the client to help make the code clearer. Unfortunately, this makes it a bit longer too.
- L3–6: First we’ll validate that a dial object was passed in. We’re operating on the dial later so we need to make sure it’s not nil.
- L8–9: We’ll copy the URL we have set on our service and then update the Path. It’ll retain the scheme and host without changing the original on the Client or DialService.
- L14–18: We need to encode our request body to JSON. This request includes our dial object and token. Again, the token is separate because we disabled encoding in the wtf.Dial struct.
- L20–25: Next we’ll post our request and use bytes.Reader to stream the in-memory byte slice of our encoded request.
- L27–30: We’ll then decode our response and verify no decoding error occurred.
- L31–33: We’ll also check the Err field on the response to see if a wtf.Error occurred. Since our errors are all constant strings, we can convert this string to a wtf.Error and it will act just like any other declared error.
- L36–37: Finally we’ll copy the returned dial back to our Dial argument. This will update the modified date just like the bolt.DialService does.
Next we’ll look at fetching a dial. This is simpler because we are not encoding a JSON request.
- L3–4: Again, we copy the URL and update the path. We’re using url.QueryEscape() to handle escaping any odd characters in the ID.
- L6–11: Here we simply fetch the URL using a GET method.
- L13–19: We’ll decode the body and we’ll check the Err response field.
- L20: If no errors occurred then we’ll return our dial object.
Finally, let’s look at updating the level of an existing dial. This looks a lot like our CreateDial() method except that we’re using PATCH instead of POST.
I’ll just highlight the main differences from CreateDial():
- L12–16: The http package doesn’t have a built-in helper method for PATCH so we need to build the http.Request by hand here.
- L18–23: We can use the default HTTP client’s Do() method to process our hand built request object.
The standard library provides (nearly) everything you need to build a REST-style HTTP API. We can avoid the complexity of frameworks by embracing the abstractions that already exist in net/http.
By implementing our own http package to translate HTTP into our domain model we now have a pluggable service that we can use anywhere. This gives us huge flexibility if we ever wanted to build additional layers in front of our services such as a caching layer.
In the next post we’ll tie everything together and build our wtf binary for serving this data.
Questions or feedback? Find me at @benbjohnson on Twitter.
If you liked this, click the💚 below so other people will see this here on Medium.