Build a Website Using Only the Go Std Library

There are quite a few web frameworks available within the Go ecosystem. Yet the advice that is often stated is that the std lib net/http is more than capable and that it’s best to start with that library until you grow out of it. So how easy is the net/http library to use when building a website? Is it possible to use the standard library exclusively?

Photo by Markus Spiske on Unsplash

Objective

We’ll be building (from scratch) a riff on the type of website that is synonymous with beginner tutorials, the Todo app. What follows is by no means the right/best way of coding a site, but it will provide a working example of how things can be stitched together.

The site will provide simple CRUD abilities to an in-memory dataset. We’ll look to utilise the Go standard library and the first steps will be to provide a multi-page application. We’ll move on to how we use the standard library to define a JSON REST API on top of this.

Scaffold

Let’s start by scaffolding out a basic app structure. Replace [your_github_user] with your Github username. Or initialise the go modules with whichever VCS or module structure you wish.

mkdir -p mystdhttp/cmd && cd $_
touch main.go runHttp.go
cd ../
go mod init github.com/[your_github_user]/mystdhttp
echo “# Builds\nbin” > .gitignore
touch makefile

We’ll be using make, but you can just build and run the binary yourself if needed. Add the following to the makefile:

HTTP Server

Fill out the cmd/main.go file as follows to create our entry point.

The runHttp function can then be created in the runHttp.go file:

If we run make run now the HTTP server will run on port 8000, but it will return a 404 on everything.

Note that most web frameworks will wrap the http.Server themselves (or their own version of an HTTP server) and provide a different way of starting ListenAndServe. All will follow this API and will be blocking and return any error created by the server.

Routes

Our web server has no routes, and currently utilises the DefaultServemux from the http package. We’ll take control and build our own router/mux using the http package that we will define our routes on, but there are lots of other routers that could be used with the net/http server, as well as those provided/wrapped by other frameworks.

Create a router folder in the app root, and then a router.go file within that.

Now update the server to use this new router.

s := http.Server{
  Addr:    listenAddr,
  Handler: router.NewRouter(), // Our own instance of servemux
}

We now have a working HTTP server that will respond ok to every URL path. Note how we haven’t specified the HTTP verb here. This server will respond with ok on all methods. If we wanted this to respond to just the root path, and when the method was GET, then we have to add in extra logic into our existing handler to deal with this situation ourselves.

mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
  if r.URL.Path != "/" || r.Method != http.MethodGet {
    http.NotFound(w, r)
    return
  }
  w.WriteHeader(http.StatusOK)
  w.Write([]byte("ok"))
})

Here, we are manually invoking the generic http.NotFound handler. We could create our own bespoke handler if we wanted to and call accordingly, but another item to be aware of is that the http.Server does not provide a way of setting the not found handler. Other frameworks can provide the ability to set the notfound handler as a global item and, in addition, will call this automatically as if no routes match from those defined for specific methods.

Html/Template

Our site needs a webpage, some html to be presented to the user. We’ll look to utilise the html/template package from the standard library, which some frameworks also utilise. Let’s first create the folder structure that we need to hold our templates.

mkdir -p web/templates

We will create a base template which defines a basic HTML layout, and then this can be reused by multiple pages. Within the templates directory create a base.gohtml file with the following contents. N.B I’ve chosen the gohtml extension here, but they can just be .html files as well. I feel that .gohtml shows that there will be some other syntax involved in the file, and it helps with some IDE extensions.

And then create an index.gohtml file with the following:

Within the router folder create a template.go file which will hold some helper functions we need. This is merely planning ahead for a few things later on.

defaultFuncs are some global functions we might need in the templates. We provide one function which will return the title of the app if nothing is provided.

templateFiles is a global variable which allows us to define the paths to all of our layout files. At the moment we only have the single base.gohtml but in the future we can expand on this as we need to add more files that form our base layout.

tmplLayout is a helper function which will take a list of filepaths (which should point to template files) and return these concatenated with the base templateFiles.

Now to utilise these and show this working, update the router/router.go file as follows:

It’s worth noting here that we parse the template at the beginning of the application…

mux.HandleFunc("/", taskListPage())
...
func taskListPage() http.HandlerFunc {
  // Anything here is executed once at setup
  return func(w, r) {
    // This is our request handler that has access to items in the above scope.
  }
}

…and the return of taskListPage is the anonymous function which is executed at every request. This means that we are doing the intensive work of parsing the template only once rather than at each request. There is a side-effect though; during development we are going to have to restart the application each time we make a change to the template files. It would be possible to move the parsing of the files into the anonymous function to stop this.

To render the template to the client, we process the ExecuteTemplate into a bytes.Buffer, just in case there is any other issue whilst executing the template. If we had executed directly on the http.ResponseWriter then there’s a chance that it would have already written the HTTP headers and when we try to respond, instead of some form of error response, the runtime will throw another error. This might happen not just by bad template syntax, but also by how the template is accessing the data being passed in.

It would also be possible to change the declaration of the buffer to use a sync.Pool which would ensure that allocated buffers could get reused, and thus reduce the amount of buffers being allocated by multiple requests. The package documentation has an example of this. I’ve left this out of this example and note that a few of the frameworks I’ve looked at stay away from using a sync.Pool here as well. I guess there is an overhead?

You can now run this server using make run and navigate to http://localhost:8000 to see the webpage being displayed.

The initial task site with dummy data

Err Page

Navigating to any of the Edit or Delete links on the tasks, or to any other routes provides just a simple text response of 404 page not found. Let’s create a helper function and a custom not found template file:

All of a sudden we can start to see repetition, and the rendering of a template can make our handler funcs a bit messy. We could refactor out the rendering of a template, but it would also make sense to refactor out the parsing of the templates in a similar fashion to some of the helper functions we put in earlier in the router/template.go file. The other frameworks can provide this level of abstraction resulting in a simpler API.

We now have a fairly simple starter template for a website built completely with net/http. But we’ve yet to really test the default mux, which is where it becomes difficult to handle with multiple routes.

Basic starter · chilledoj/mystdhttp@e38eb69

Static files

Let’s look at static file delivery so that we can look to add some javascript to our pages.

mkdir -p web/static/js
echo "console.log('working');" > web/static/js/tasks.js

Add the file server provided by the net/http library into the router, below the existing root path. N.B. this is pretty much a copy+paste from the package docs.

/* router/router.go */
func NewRouter() http.Handler {
  mux := http.NewServeMux()
  mux.HandleFunc("/", taskListPage())
  mux.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("./web/static"))))
  return mux
}

Run the server again and check that the js file can be accessed by navigating to http://localhost:8000/static/js/tasks.js and checking that the text of the file is returned. Note that if you try to navigate to a file that doesn’t exist the default net/http not found handler is used instead of our custom one. We cannot override this behaviour, but in the context of serving a file directory, under a specific path (/static), then this can be accepted.

Most frameworks provide their own mechanism for serving static files such that they have control over the ‘not found’ behaviour. The API’s might be simpler, but I think I’d end up doing the same copy+paste from any of their documentation.

JSON API

At this point we could create other paths within the app, render some html and potentially even handle some form posting. But instead let’s look at how we might implement some REST routes for CRUD operations with a JSON API.

As we start to ramp up the number of routes, the first thing that we realise we need to handle is providing a response in JSON format. We should create a function that can serialise arbitrary data out as JSON to the ResponseWriter. Most frameworks provide this, but it is fairly simple to add and we can use the encoding/json package from the standard library.

If we are sending our data as JSON, shouldn’t our errors also be responding in JSON? We can create a struct which acts as a container for the actual error and a more end-user friendly message.

Rest Routes

Our objective would be to provide the following routes:

GET     /api/tasks/     ->  List all tasks 
POST    /api/tasks/     ->  Create a task
GET     /api/tasks/:id  ->  Read a task by ID
PUT     /api/tasks/:id  ->  Update a task
DELETE  /api/tasks/:id  ->  Delete a task

If we create a single struct to represent the Tasks API group and service, we can create methods on the struct for each of the different handlers. But our first step would be to create a group router for each of our routes. We can achieve this by making our struct meet the http.Handler interface, and implement a ServeHTTP method. The separate routes can then be further methods on our service struct.

We can then instantiate a new variable containing our struct which can be used as the handler for the /api/tasks path within the servemux. Note that here I use the StripPrefix Handler as this will mean that our struct will only receive the path from after the tasks bit of the path, which can make other things (path parameters) a bit easier to use — more on that in a sec.

th := NewTasksHandler()
mux.Handle("/api/tasks/", http.StripPrefix("/api/tasks", th))

Most routing frameworks provide ways of creating a sub router with which to group common routes, very similar to what we are doing here, just with a simpler API. The sub routers usually then have the same simpler HTTP verb methods meaning the exact routes handlers can be specified. The handlers (or really HandlerFuncs) that are further methods on our service struct (th.deleteTaskHandler etc.), are themselves then responsible for handling the request further and therefore acting as another subrouter.

Path Parameters

When defining routes in most frameworks we can usually provide a placeholder for an item we want to extract as a parameter. e.g.

router.Get("/tasks/:id"). -> /tasks/abc  &  id="abc"

The standard library doesn’t provide a way of doing that and so you need to go back to basics and extract the various parameters yourself.

e.g. If we take the /api/tasks/:id path, our handler can utilise a function like the following.

It can then be utilised in our handlers.

Middleware

One last (brief) thing to look at is middleware. We can apply middleware at any point within the web service, but the most common pattern is to apply some to the main router/mux (logging, recover etc), and then have some others set on either a specific route or API group (auth, content-negotiation, header checks).

There are libraries that assist with chaining middleware that can use the net/http package and use http.Handler, so that ultimately we should be able to write middleware once and share.

The web frameworks which are more feature-rich tend to have bespoke middleware as a result of having a different API for the handler functions. These functions tend to take a single “context” variable which has reference to the underlying request and connection, and has all of the helper methods attached. Having said that, the coverage of either provided middleware, or community written packages is usually very good.

The repo shows how everything we’ve discussed so far could be pulled together into a (simple) website. It’s not the cleanest code but it should show how to setup and use what I’ve outlined above…and there’s no external modules!

chilledoj/mystdhttp

Summary

When I first started coding Go I had come from a background of Node and ExpressJS, which has an API that is common amongst many web frameworks across various languages. The standard library provides a fantastic baseline which allows web sites/services to be developed but its API can be viewed as a little basic or confusing.

The various web frameworks out in the wild do provide a simpler API, and can provide a more performant web server. The helper methods they provide, like HTTP verb methods for specifying the exact route, json and error responses and the ability to render a template, do reduce the amount of boilerplate that is required compared to when starting from scratch using the net/http package. They allow you to concentrate on writing the application rather than the nuts & bolts.

For some reason, I feel a sense of achievement when I use the net/http package over these larger frameworks. I’d never be able to work at a lower level on the raw TCP connection at all, but I do feel that I’m able to reason about what’s going on a lot better with the standard library, without any hidden magic. Mainly because I would have had to spend time writing/copying repetitive code for what I need!

And so I do tend to reach for whichever framework falls upon my search. I can’t provide any recommendations on which of the various frameworks out there to use; there are too many and they all ultimately achieve the same thing in very similar ways. I think it will just come down to personal preference as to whether a repo has a lot of support and fits your needs…

…But don’t dismiss the std library net/http. It’s more than capable!