Structuring Applications in Go

How I organize my applications in Go

Ben Johnson
Jul 6, 2014 · 7 min read

Overview

For me, the hardest part of learning Go was in structuring my application. Prior to Go, I was working on a Rails application and Rails makes you structure your application in a certain way. “Convention over configuration” is their motto. But Go doesn’t prescribe any particular project layout or application structure and Go’s conventions are mostly stylistic.

1. Don’t use global variables

The Go net/http examples I read always show a function registered with http.HandleFunc like this:

package mainimport (
 “fmt”
 “net/http”
)func main() {
    http.HandleFunc(“/hello”, hello)
    http.ListenAndServe(“:8080", nil)
}func hello(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, “hi!”)
}
type HelloHandler struct {
    db *sql.DB
}func (h *HelloHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    var name string    // Execute the query.
    row := h.db.QueryRow(“SELECT myname FROM mytable”)
    if err := row.Scan(&name); err != nil {
        http.Error(w, err.Error(), 500)
        return
    }    // Write it back to the client.
    fmt.Fprintf(w, “hi %s!\n”, name)
}
func main() {
    // Open our database connection.
    db, err := sql.Open(“postgres”, “…”)
    if err != nil {
        log.Fatal(err)
    }    // Register our handler.
    http.Handle(“/hello”, &HelloHandler{db: db})
    http.ListenAndServe(“:8080", nil)
}
func TestHelloHandler_ServeHTTP(t *testing.T) {
    // Open our connection and setup our handler.
    db, _ := sql.Open("postgres", "...")
    defer db.Close()
    h := HelloHandler{db: db}    // Execute our handler with a simple buffer.
    rec := httptest.NewRecorder()
    rec.Body = bytes.NewBuffer()
    h.ServeHTTP(rec, nil)
    if rec.Body.String() != "hi bob!\n" {
        t.Errorf("unexpected response: %s", rec.Body.String())
    }
}

2. Separate your binary from your application

I used to place my main.go file in the root of my project so that when someone runs “go get” then my application would be automagically installed. However, combining the main.go file and my application logic in the same package has two consequences:

  1. I can only have one application binary.
camlistore/
  cmd/
    camget/
      main.go
    cammount/
      main.go
    camput/
      main.go
    camtool/
      main.go

Library driven development

Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library. I find this helps me make a cleaner abstraction of what code is for my core logic (the library) and what code is for running my application (the application binary).

adder/
  adder.go
  cmd/
    adder/
      main.go
    adder-server/
      main.go
$ go get github.com/benbjohnson/adder/...

3. Wrap types for application-specific context

One trick I’ve found especially helpful is realizing that some generic types should be wrapped to provide application-level context. A great example of this is wrapping the DB and Tx (transaction) types. These types can be found in the database/sql package or other database libraries such as Bolt.

package myappimport (
    "database/sql"
)type DB struct {
    *sql.DB
}type Tx struct {
    *sql.Tx
}
// Open returns a DB reference for a data source.
func Open(dataSourceName string) (*DB, error) {
    db, err := sql.Open("postgres", dataSourceName)
    if err != nil {
        return nil, err
    }
    return &DB{db}, nil
}// Begin starts an returns a new transaction.
func (db *DB) Begin() (*Tx, error) {
    tx, err := db.DB.Begin()
    if err != nil {
        return nil, err
    }
    return &Tx{tx}, nil
}
// CreateUser creates a new user.
// Returns an error if user is invalid or the tx fails.
func (tx *Tx) CreateUser(u *User) error {
    // Validate the input.
    if u == nil {
        return errors.New("user required")
    } else if u.Name == "" {
        return errors.New("name required")
    }

    // Perform the actual insert and return any errors.
    return tx.Exec(`INSERT INTO users (...) VALUES`, ...)
}

Transactional composition

Another benefit to adding these functions to your Tx is that it allows you to compose multiple actions in a single transaction. Need to add one user? Just call Tx.CreateUser() once:

tx, _ := db.Begin()
tx.CreateUser(&User{Name:"susy"})
tx.Commit()
tx, _ := db.Begin()
for _, u := range users {
    tx.CreateUser(u)
}
tx.Commit()

4. Don’t go crazy with subpackages

Most languages let you organize your package structure however you’d like. I’ve worked in Java codebases where every couple of classes get stuffed into another package and these packages would all include each other. It was a mess!

  1. Organize the most important type at the top of the file and add types in decreasing importance towards the bottom of the file.
  2. Once your application starts getting above 10,000 SLOC you should seriously evaluate whether it can be broken into smaller projects.
bucket.go
cursor.go
db.go
freelist.go
node.go
page.go
tx.go

Conclusion

Code organization is one of the hardest parts about writing software and it rarely gets the focus it deserves. Use global variables sparingly, move your application binary code to its own package, wrap some types for application-specific context, and limit your subpackages. These are just a few tricks that can help make Go code easier and more maintainable.

    Ben Johnson

    Written by

    Writing databases and distributed systems in Go.