Open in app

Sign in

Write

Sign in

Go Testing Technique: Testing JSON HTTP Requests

Aldo "xoen" Giambelluca
6 min readApr 24, 2016

Let’s say you have some code that make an HTTP request with some JSON data. What do you test and how? I’m sure you run into something similar.

I tried to keep this example as toy as possible without making it useless.

Don’t worry too much about the exact details about the specific JSON format, URL, etc…what’s important is that your code is making an HTTP request and you want to test it.

I’ll walk you through the following example.

Example

You need to publish an NSQ message in your application-specific format:

Publish(nsqdUrl, msg string) error

This function will:

  1. Wrap the msg string in a JSON string
  2. Make a POST request to a given URL
  3. Return an error if the response is not acceptable

What to test

  1. You want to test that the format of the POSTed data is as expected. This is the example format:
{
  "meta": {
    "lifeMeaning": 42
  },
  "data": {
    "message": "%{MSG}"
  }
}

(where %{MSG} is the string passed)

2. You want to test that Publish() is making the right HTTP request.
In this example is a POST %{nsqdUrl}/pub?topic=meaningful-topic.

3. You also want to test that the request was successful and returned the right HTTP status, in this example would be a 200 OK.

Go testing basics

Go provides a testing package which makes writing basic tests very easy.

Let’s assume the Publish() function is part of a nsqpublisher package in the nsqpublisher.go file. By convention your tests will be in a nsqpublisher_test.go file like this:

package nsqpublisherimport (
 “testing”
)func TestPublishUnreachable(t *testing.T) {
  nsqdUrl := “http://localhost:-41"
  err := Publish(nsqdUrl, “hello”)
  if err == nil {
    t.Errorf(“Publish() didn’t return an error”)
  }
}

A few interesting things to note here:

  • You’re in the same package you’re testing. So you have access to everything in the package
  • The test function starts with Test*.
  • The test function receive a *testing.T which you can use to make the test fail by calling Errorf() on it.

And you can run the tests by $ go test

This test will fail because there is no Publish() function yet. Let’s write one.

A first (broken) implementation

Let’s start with a broken implementation of Publish():

package nsqpublisherimport (
  "net/http"
)func Publish(nsqdUrl, msg string) error {
  _, err := http.Get(nsqdUrl)  return err
}

With this in place that test will pass: `Publish()` makes an HTTP request, it fails and returns an error.

An HTTP test server

The first test is useful, but how do you test that you’re making a real HTTP request without the need to run the NSQ servers locally? (or whatever server you’re making requests to)

The answer to this question is the Go httptest package. This package makes the creation of a test HTTP server very easy.

Let’s have a look at how by writing a test for when the nsqd server response wasn’t a 200 OK:

import (
  "testing"
  "net/http"
  "net/http/httptest"
)func TestPublishWrongResponseStatus(t *testing.T) {
  ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusServiceUnavailable)
  }))
  defer ts.Close()  nsqdUrl := ts.URL
  err := Publish(nsqdUrl, "hello")
  if err == nil {
    t.Errorf("Publish() didn’t return an error")
  }
}

OK, this is a bit more complex but very few things happen here:

  • We create a test server by using httptest.NewServer
  • We get the URL of this server by using ts.URL
  • This server will respond with a 503 SERVICE UNAVAILABLE by using w.WriteHeader()
  • The server also has access to the request r which is an *http.Request (not used in this test)
  • This server will be closed when the test will finish because of the defer ts.Close()

If you run this test it will fail: Publish() makes a request to the test server but it doesn’t check the response’s status code. Easy to fix:

package nsqpublisherimport (
  "net/http"
  "fmt"
)func Publish(nsqdUrl, msg string) error {
  resp, err := http.Get(nsqdUrl)
  if err != nil {
    return err
  }
  if resp.StatusCode != http.StatusOK {
    return fmt.Errorf(“nsqd didn’t respond 200 OK: %s”, resp.Status)
  }  return nil
}

Testing the request

Now that we have a test server is easy to test that Publish() makes the right request:

func TestPublishOK(t *testing.T) {
  ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)if r.Method != "POST" {
      t.Errorf(“Expected ‘POST’ request, got ‘%s’”, r.Method)
    }
    if r.URL.EscapedPath() != "/pub" {
      t.Errorf("Expected request to ‘/pub’, got ‘%s’", r.URL.EscapedPath())
    }    r.ParseForm()
    topic := r.Form.Get("topic")
    if topic != "meaningful-topic" {
      t.Errorf("Expected request to have ‘topic=meaningful-topic’, got: ‘%s’", topic)
    }
  }))
  defer ts.Close()  nsqdUrl := ts.URL
  err := Publish(nsqdUrl, "hello")
  if err != nil {
    t.Errorf("Publish() returned an error: %s", err)
  }
}

Run the tests and you’ll get something like this:

$ go test — — FAIL: TestPublishOK (0.00s)
         nsqpublisher_test.go:35: Expected ‘POST’ request, got ‘GET’
         nsqpublisher_test.go:38: Expected request to ‘/pub’, got ‘/’
         nsqpublisher_test.go:44: Expected request to have ‘topic=meaningful-topic’, got: ‘’FAILexit status 1

Wow, this is very useful. We now have a test which is telling us exactly what Publish() is doing wrong. Yes, it makes an HTTP request but to the wrong path, with the wrong verb and it’s not passing the topic param as it should.

How does it work?

There are a few key things to notice here:

In order to make this test pass change Publish() to make a POST request with the right URL:

package nsqpublisherimport (
  "net/http"
  "fmt"
)const (
  TOPIC = "meaningful-topic"
)func Publish(nsqdUrl, msg string) error {
  url := fmt.Sprintf("%s/pub?topic=%s", nsqdUrl, TOPIC)  resp, err := http.Post(url, "application/json", nil)
  if err != nil {
    return err
  }
  if resp.StatusCode != http.StatusOK {
    return fmt.Errorf("nsqd didn’t respond 200 OK: %s", resp.Status)
  }  return nil
}

Tests will pass again. Try to break them by changing the URL or topic.

Testing we’re posting the correct JSON

As it stand, our Publish() function is essentially broken. It doesn’t POST any data. The request body should be JSON and with the expected format.

How do we test it? You probably guessed that we’ll need to read the request body in the test server, unmarshal the JSON string and test it has the right properties.

I will use the go-simplejson package for it. It makes manipulating/traversing JSON a breeze and it has a nice interface:

func TestPublishOK(t *testing.T) {
  msg := "Test message"  ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)    if r.Method != "POST" {
      t.Errorf("Expected ‘POST’ request, got ‘%s’", r.Method)
    }
    if r.URL.EscapedPath() != "/pub" {
      t.Errorf("Expected request to ‘/pub’, got ‘%s’", r.URL.EscapedPath())
    }    r.ParseForm()
    topic := r.Form.Get("topic")
    if topic != "meaningful-topic" {
      t.Errorf("Expected request to have ‘topic=meaningful-topic’, got: ‘%s’", topic)
    }    reqJson, err := simplejson.NewFromReader(r.Body)
    if err != nil {
      t.Errorf("Error while reading request JSON: %s", err)
    }
    lifeMeaning := reqJson.GetPath("meta", "lifeMeaning").MustInt()
    if lifeMeaning != 42 {
      t.Errorf("Expected request JSON to have meta/lifeMeaning = 42, got %d", lifeMeaning)
    }
    msgActual := reqJson.GetPath("data", "message").MustString()
    if msgActual != msg {
      t.Errorf("Expected request JSON to have data/message = ‘%s’, got ‘%s’", msg, msgActual)
    }
  }))
  defer ts.Close()  nsqdUrl := ts.URL
  err := Publish(nsqdUrl, msg)
  if err != nil {
    t.Errorf("Publish() returned an error: %s", err)
  }
}

The key things here are:

  • We build a simplejson’s object from the request body reader: reqJson, err := simplejson.NewFromReader(r.Body)
  • We traverse the JSON’s properties by using simplejson’s GetPath(“key”, “subkey”)
  • We convert these values into an int/string by using MustInt()/MustString()

Run the tests again and they will fail:

$ go test — — FAIL: TestPublishOK (0.00s)
         nsqpublisher_test.go:52: Error while reading request JSON: EOF
         nsqpublisher_test.go:56: Expected request JSON to have meta/lifeMeaning = 42, got 0
         nsqpublisher_test.go:60: Expected request JSON to have data/message = ‘Test message’, got ‘’

Publish() is not posting any data. Let’s sort this out.

Posting the right JSON

To make the tests pass, Publish() will need to POST the JSON in the right format:

package nsqpublisherimport (
  "fmt"
  "net/http"
  "strings"
)const (
  LIFE_MEANING = 42
  TOPIC        = "meaningful-topic"
)func Publish(nsqdUrl, msg string) error {
  url := fmt.Sprintf("%s/pub?topic=%s", nsqdUrl, TOPIC)
  payload := fmt.Sprintf(`
    {
      "meta": {
        "lifeMeaning": %d
      },
      "data": {
        " “message”: "%s"
      }
    }`, LIFE_MEANING, msg)  resp, err := http.Post(url, "application/json", strings.NewReader(payload))
  if err != nil {
    return err
  }
  if resp.StatusCode != http.StatusOK {
    return fmt.Errorf("nsqd didn’t respond 200 OK: %s", resp.Status)
  }  return nil
}

NOTE: In this simple example I’m just using fmt.Sprintf() to build the request JSON, but if the JSON is more complex I would prefer to use simplejson.

With this in place, Publish() will now make a POST request to the right URL and with the right body. Tests will pass.

Conclusion

As you’ve seen, it’s very easy to test that your code is making the correct HTTP requests in Go.

Thanks to its httptest package your tests can spawn a simple HTTP server which you can use to inspect these requests.

I hope this is useful and will help you to write better code. Let me know if you’re testing this in different/better ways.

Golang
Http
Json

--

--

Aldo "xoen" Giambelluca

Written by Aldo "xoen" Giambelluca

87 Followers

Software Engineer. Son, brother, grandchild, dog lover. I believe in readable, clean code that speak easy to you (with a slightly italian accent)

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams