Ginkgotchas (yeh also Gomega)

In my last post, I wrote about some effective Ginkgo and Gomega practices; there were a few more suggestions that came to mind as I was writing but they felt like they were less “effective” and more “where did the last three hours go debugging this crap?” — A fairly arbitrary distinction but it’s my post so…

Like the previous post, familiarity with Ginkgo and Gomega is expected before reading. With that in mind, here’s some pain-points you might want to take care to avoid when writing tests using Ginkgo and Gomega.

Not initialising a variable in the closest BeforeEach

This must be the most pervasive gotcha for people new to writing Ginkgo tests, but it definitely catches old hands from time to time. Look at this code:

This looks simple enough right? We declare a variable and initialise it to 10, test that the value is 10, then we have a block that modifies number to 42 and tests it is 42. What could go wrong?

If you come from a jUnit background, you probably know that each test gets its own instance of the test class to avoid test pollution. In Ginkgo, test lifecycle is bounded by the test processes (1 to n depending on whether you are using the -p parallel flag). If you turn on --randomizeAllSpecs (see my previous post for why you should do this right now) there is no guarantee of the order these examples will run in. As such, for the previous snippet, the first example sets the value of number to 42 and the next example asserts it to be 10 causing the failure.

The correct way to resolve this is to explicitly set a default value in the BeforeEach closest to variable declaration:

This gotcha is particularly frustrating to debug when the -p and --randomizeAllSpecs flags are set (which they should be) because the test pollution can take many forms.

TIP: Each ginkgo run prints the seed it uses to shard and order the examples across nodes, and you can pass this with --seed to Ginkgo to replay a previous run!

Not passing a function to Eventually or Consistently

More like Consistently a pain in my ass. Ok, so the asynchronous matchers in Gomega are pretty nifty to use, they allow you to pass something which will be checked periodically against some particular matcher.

You can pass a channel:

Eventually(ch).Should(Receive())

You can pass a structs that satisfies some interface required by the matcher (in the case of Say, something that has a Buffer method:

Eventually(session).Should(Say("something"))

You can even pass a function which will be periodically called and asserted against:

var i int = 0
func maybe() bool {
  if i > 10 {
    return true
  }
  i++  
  return false
}
Eventually(maybe).Should(BeTrue())

But what you absolutely should not do, is pass the result of an evaluated function:

var i int = 0
func maybe() bool {
  if i > 10 {
    return true
  }
  i++
  return false
}
Eventually(maybe()).Should(BeTrue())

The function will be evaluated once, eagerly, and you’ll be left wondering where you went wrong in life.

Who designed this thing?

Believing Async Should and ShouldNot are opposites

This one seems fairly obvious when you take a moment to think about it but it’s caught me and many others in a moment of weakness. When you are writing a standard assertion:

Expect(foo).To(Equal(something))

and you decide you actually want to assert the opposite, you can simply flip To to NotTo:

Expect(foo).NotTo(Equal(something))

If you try to do the same thing with async assertions (Eventually and Consistently), you are in for a world of hurt. Let’s reuse the same example from above, assuming that foo is a function that will return some expected value after some amount of time.

Eventually(foo).Should(Equal(something))

Flipping the intent, that foo should not ever equal something you might reasonably try to say:

Eventually(foo).ShouldNot(Equal(something))

You made a huge mistake

This looks like it should work, but remember that the function foo won’t return the unexpected value for some time, Eventually will evaluate the function, see that it passes the matcher and declare success!

Likely, the intent is better captured by additionally changing Eventually for Consistently:

Consistently(foo).ShouldNot(Equal(something))

Much better.

Leaking a goroutine

In the last post, I wrote that you should use goroutines with care because Ginkgo doesn’t really play nice. Consider this snippet below:

You can mostly ignore the sleeps, they exist only to orchestrate an overlap between the examples to demonstrate this failure. Our first example creates a goroutine (it even uses GinkgoRecover() like a good little example), and calls Fail. The second example makes an assertion that cannot possibly fail, that true is equal to true!

You’re gonna have a bad time

The second example failed! true is no longer equal to true and Pi is exactly 3!

What happened here is that the goroutine outlived the lifetime of its spec and when Fail was called, Ginkgo determined that the currently running example was the second one. It’s important to make sure your goroutines lifetimes are carefully handled in your examples, typically coordinated via channels.

Yeh, it sucks

Asserting errors against stdout

I’ll keep this one short. If you’ve used the gexec.Say matcher to test stdio of some running process you’ll recognise this:

Eventually(session).Should(Say("something"))

This asserts that the string "something" has appeared on some output. More specifically, it asserts that the string has appeared on stdout. If you want to test output on stderr, this mistake will leave you running in circles.

The Say matcher will call the Buffer method on session to find the contents to assert against. Unfortunately, this returns only the stdout buffer, rather than an amalgamation of both. To test against stderr you need to write:

Eventually(session.Err).Should(Say("something"))

TIP: You may find it more expressive to intentionally use .Out and .Err everywhere to avoid confusion.

Accidentally regex testing with Say

This gotcha is brought to you by the letters [A-Za-z]. Good joke right. Let’s take the following snippet of code that uses the gbytes package:

This looks pretty simple right? There’s a buffer containing the string map[string] and then we assert the buffer contains the string map[string]!

Absolute nightmare fuel

The Say matcher accepts a regular expression as an argument, so the [ and ] characters are tripping it up so we’ll have to escape them with \ characters.

TIP: If you use backticks instead of double quotes to surround your argument to Say, this gives a better indicator to your reader that there’s something special about the string.

Final thoughts!

This list of gotchas is certainly not exhaustive, but they are the most common ones I’ve come across. Despite some of the frustration I’ve experienced over the years at these, in my experience it doesn’t take long for people to recognise these patterns and avoid them. Hopefully this post helps capture some of that tribal knowledge so that others avoid the same issues in the first instance.

If you’ve got any thoughts on any of these, or have any more to add, let me know.

Thanks!