Let’s sit down and write a new feature for a website: a visitor counter. This is just a small API service, that saves a single number in a database, and serves/increments it via a single endpoint. So, we sit down and hammer out something like this:
It’s succinct, it works, so we’re done, right? Not quite. No good code is done without testing, so let’s do that.
Where do we even get started? This is a mess!
- A bunch of the logic is in the
main()function, which can’t be tested without launching a fully-fledged web server. Can we even count on being able to launch a server (or even open a net port) on any machine we’re testing this on?
- The state of our routing is contained in global variables in Go’s
net/httppackage, which is rather inconvenient.
- Code behavior depends on environment variables, which might be already set during test runtime. How much care do we have to dedicate to making sure they don’t accidentally use “production” values?
- The MVC separation of concerns is hopelessly messed up, with database code mixed all up in the HTTP request handler. That’s going to be one complex test function.
On top of all that, the code already has some smells: global state/variables, hidden unhandled failure cases (what happens when
PORT is unset?), no logging, and some hard dependencies. Yuck!
Let’s do it again, the right way.
Disclaimer: Egregious over-engineering below. No gophers were harmed in the making of this article.
Separation of concerns via single-purpose modules
The concept of structuring your code into single-purpose modules is nothing new, which repeats at many layers of abstraction: from a Java package, to a Python file, to the underpinnings of frameworks like Spring or Angular. The idea is to define a high level organization of your code in each module has a well defined “contract” of behavior.
This contract has two parts: the requirements (or, “dependencies”) a module needs, and the outputs that the module creates. What is actually inside the module is irrelevant to anyone using it — all that matters is that it does its job as long as it has its dependencies. So, let’s redesign our visitor counter sever into some modules with separate concerns:
- Configuration module: has no dependencies; produces a port integer and a database URI string
- Database connection module: depends on a database URI; produces a database connection (in Go land, probably a
- Database counter module: depends on a database connection; produces a counter accessor — an abstract object providing
incrementCounter()and handling all the SQL interactions
- Counter controller module: depends on a counter accessor; produces a web counter handler — an object that receives a HTTP request for the counter, executes the proper control code, and writes the HTTP response (in Go land, an implementation of
- Web server module: depends on the port integer, and the web counter handler; produces a function that starts the webserver — to be called by the main function, this sets up the routing, registers the web counter handler at
/counter, and starts the server
Note that this is all at a conceptual level. What actually constitutes a “module” in the code depends on your language, framework, etc. More on how to modularize Go this way in a bit…
This might seem excessive, but it’s actually the bare minimum in my opinion of what it takes to get a properly tested, modular web service. How? For instance…
Suppose your counter accessor looked like this:
And the counter handler looked like this:
The most important outcome:
CounterHandler has no idea what database (if any) backs the CounterAccessor it has; all it knows is, there’s an
Increment() function that it can use. That means that a test for
CounterHandler can look like this:
That’s a good test! It covers the branching logic of receiving a result vs. receiving an error, has no external effects or implications, and does not actually depend on the
CounterHandler having any sort of database access. Plus, 100% coverage. Who doesn’t like that?
What’s the actual point of all this?
This is a modular, testable, reusable, and Go-idiomatic way to build a complex Go application. It is of course overkill for a simple “counter” application — you could remove a couple layers of abstraction and still be fine. Consider the possibility of an API with many tens of endpoints, though. Should all of them include mocking a
*sql.DB for their handlers? Or, consider an API that has tens of data models in its database; should they all be individually responsible for opening connections, or would that cause issues? Should their tests similarly emulate database connections for all of them, or can something simpler — but equally effective at testing — be arrived at?
Using interface abstraction in this way is a great way to get tested and reusable modular code within Go.
But wait, there’s more! Dependency injection!
To anyone who has worked in languages other than Go — particularly those familiar with Spring MVC in Java — this might seem very familiar. This modularity that I am describing has a well established design pattern: dependency injection. In Spring, this is accomplished using a complex collection of compile time and runtime annotations plus a healthy helping of object reflection. Testing is done in a similar way — frameworks like Mockito let you completely mask types behind mock objects.
Go unfortunately tends to be less flexible than that. There are some dependency injection frameworks that work similarly to Spring, but my preference is for a lightweight framework called Wire.
Instead of hooking up all the types, instantiations, values, etc at runtime, Wire does so at compile-time. It does this by using a pattern that is very similar to the concept of “modules” as I defined them earlier: provider functions. A provider function is a Go function that has a signature that looks like one of:
func ProvideWumpus(foo Foo, bar Bar) Wumpus
func ProvideWumpus(foo Foo, bar Bar) (Wumpus, error)
func ProvideWumpus(foo Foo, bar Bar) (Wumpus, func(), error)
Each of these is recognized as a “provider” that has dependencies of types
Bar, and produce a value of type
Wumpus. The latter two forms can produce an error (if the initialization failed somehow), and a cleanup function (to close resources or otherwise clean things up neatly when needed). A less abstract example, providing a database connection, might look like this:
Note that the
DBConnection types are aliases of real values. This aliasing matters, since it lets Wire distinguish between, say, multiple strings. Variable/function names don’t matter, but type names do. We’re also not using the global function
sql.Open itself. By assigning it to a variable before using it, it lets our unit tests override the variable in order to avoid the external effect of interacting with a database (which would always happen if we called
Here’s another example of the
CounterHandler from above, as written in Wire. Note that the accessor is saved for later use in the handler itself:
Also note that Wire itself does not play into any of this code. Wire is not a runtime library. It does nothing and should not be part of the code bundled in your executable. Instead, Wire lets you write special files in which you can specify collections of provider functions and the signature of an “injector” that your code can actually call into. It then generates the code of said injector to contain all the construction and deconstruction, in appropriate order, of all the providers.
So is this just a stump for Wire?
Yeah. Sort of. Not really, no.
It’s a stump for using Wire or a similar framework, plus good code organization, to structure a robust Go web API. Taking the “counter” example to its conclusion (and a little past that, for good measure), I came up with this:
What is this? This is a simple webserver that just serves a visitor counter and its uptime seconds. It is built using…
It’s structured in a way that should be easy to amend with any number of new providers/modules, so it can serve as a starting point for a variety of interesting projects — beyond simple visitor counters, and perhaps even beyond HTTP servers. Lastly, just for fun, here’s a diagram (similar to the above) of the providers and their relationships in the sample application:
Why the extra complications? Because every good service needs standard stream logging, a health check endpoint, and graceful shutdown when it receives an interrupt signal! Also, why not?
I hope that this journey through setting up a dependency injection backed Go web service has been more than just informative. I know I will be using the code in the above repository as a reference point for structuring some of my own future projects. Happy coding!