ARTICLE

Writing Fuzz Tests in Elm

Manning Publications

From Elm in Action by Richard Feldman

__________________________________________________________________

Save 37% off Elm in Action. Just enter fccfeldman into the discount code box at checkout at manning.com.
__________________________________________________________________

This article deals with writing fuzz tests in Elm.

Writing Fuzz Tests

When writing tests for business logic, it can be time-consuming to hunt down edge cases -those unusual inputs which trigger bugs that never manifest with more common inputs.

In Elm, fuzz tests help us detect edge case failures by writing one test which verifies a large number of randomly-generated inputs.

DEFINITION: Elm’s fuzz tests are tests which run several times with randomly-generated inputs. Outside of Elm, this testing style is sometimes called fuzzing, generative testing, property-based testing, or QuickCheck-style testing. elm-test went with fuzz because it's concise, suggests randomness, and it's fun to say.

Figure 1 shows what we’ll be building towards.

Figure 1 Randomly generating inputs with fuzz tests

A common way to write a fuzz test is to start by writing a unit test and then converting it to a fuzz test to help identify edge cases.

Let’s dip our toes into the world of fuzz testing by converting our existing unit test to a fuzz test. We’ll do this by randomly generating our JSON instead of hardcoding it; this way we can be sure our default title works properly no matter what the other fields are set to!

Converting Unit Tests to Fuzz Tests

Before we can switch to using randomly-generated JSON, first we need to replace our hardcoded JSON string with some code to generate that JSON programmatically.

Building Json programmatically with Json.encode

As we use the Json.Decode module to turn JSON into Elm values, we can use the Json.Encodemodule to turn Elm values into JSON. Let’s add this to the top of PhotoGrooveTests.elm, right after import Json.Decode exposing (decodeString).

import Json.Encode as Encode

Because JSON encoding is the only type of encoding we’ll be doing in this file, that as Encodealias lets us write Encode.foo instead of the more verbose Json.Encode.foo. While we’re at it, let’s give our Json.Decode import the same treatment, and change it to this:

import Json.Decode as Decode exposing (decodeString)

Json.encode.value

Whereas the Json.Decode module centers around the Decoder abstraction, the Json.Encode module centers around the Value abstraction. A Value (short for Json.Encode.Value) represents a JSON-like structure. In our case we'll use it to represent JSON, but later we'll see how it can be used to represent objects from JavaScript as well.

We’ll use three functions to build our {"url": "fruits.com", "size": 5} JSON on the fly:

  • Encode.int : Int -> Value
  • Encode.string : String -> Value
  • Encode.object : List ( String, Value ) -> Value

Encode.int and Encode.string translate Elm values into their JSON equivalents. Encode.object takes a list of key-value pairs; each key must be a String, and each value must be a Value.

Table 1 shows how we can use these functions to create a Value representing the same JSON structure as the one our hardcoded string currently represents.

Table 1 Switching from String to Json.Encode.Value

Json.Decode.decodevalue

Once we have the Value we want, there are two things we could do with it.

  1. Call Encode.encode to convert the Value to a String, then use our existing decodeString photoDecoder call to run our decoder on that JSON string.
  2. Don’t bother calling Encode.encode, but instead swap out our decodeString photoDecoder call for a call to decodeValue photoDecoder.

Let’s start by editing our import Json.Decode line to expose decodeValue instead of decodeString. It should end up looking like this:

import Json.Decode as Decode exposing (decodeValue)

Then let’s incorporate our new encoding and decoding logic into our test’s pipeline.

Listing 1 Using programmatically created JSON

decoderTest : Test
decoderTest =
test "title defaults to (untitled)" <|
\_ ->
[ ( "url", Encode.string "fruits.com" )
, ( "size", Encode.int 5 )
]
|> Encode.object
|> decodeValue PhotoGroove.photoDecoder ❶
|> Result.map .title
|> Expect.equal (Ok "(untitled)")

❶ We now call decodeValue instead of decodeString here

from test to fuzz2

Now we’re building our JSON programmatically, but we’re still buiding it out of the hardcoded values "fruits.com" and 5. To help our test cover more edge cases, we'll replace these hardcoded values with randomly generated ones.

The Fuzz module helps us do this. Add this after import Expect exposing (Expectation):

import Fuzz exposing (Fuzzer, list, int, string)

We want a randomly generated string to replace "fruits.com" and a randomly generated integer to replace 5. To access those we'll make the substitution shown in Table 2.

Table 2 Replacing a Unit Test with a Fuzz Test

We’ve done two things here.

First, we replaced the call to test with a call to fuzz2 string int. The call to fuzz2 says that we want a fuzz test which randomly generates two values. string and int are fuzzers which specify that we want the first generated value to be a string, and the second to be an integer. Their types are string : Fuzzer String and int : Fuzzer Int.

DEFINITION: A fuzzer specifies how to randomly generate values for fuzz tests.

The other change was to our anonymous function. It now accepts two arguments: url and size. Because we've passed this anonymous function to fuzz2 string int, elm-test will run this function one hundred times, each time randomly generating a fresh String value and passing it in as url, and a fresh Int value and passing it in as size.

NOTE: Fuzz.string doesn't generate strings completely at random. It has a higher probability of generating values which are likely to cause bugs: the empty string, extremely short strings, and extremely long strings. Similarly, Fuzz.int prioritizes generating 0, a mix of positive and negative numbers, and a mix of extremely small and extremely large numbers. Other fuzzers tend to be designed with similar priorities.

Using the randomly-generated Values

Listing 2 Our first complete fuzz test

decoderTest : Test
decoderTest =
fuzz2 string int "title defaults to (untitled)" <|
\url size -> ❶
[ ( "url", Encode.string url ) ❶
, ( "size", Encode.int size ) ❶
]
|> Encode.object
|> decodeValue PhotoGroove.photoDecoder
|> Result.map .title
|> Expect.equal (Ok "(untitled)")

❶ url and size come from the string and int fuzzers we passed to fuzz2

TIP: For even greater confidence, we can run elm-test --fuzz 5000 to run each fuzz test function five thousand times instead of the default of one hundred times. Specifying a higher --fuzz value covers more inputs, but it also makes tests take longer to run. Working on a team can get us more runs without any extra effort. Consider that if each member of a five-person team runs the entire test suite ten times per day, the default --fuzz value of of one hundred gets us five thousand runs by the end of the day!

Next we’ll turn our attention to a more frequently invoked function in our code base: update.

Testing update functions

All Elm programs share some useful properties which make them easier to test.

  1. The entire application state is represented by a single Model value.
  2. Model only ever changes when update receives a Msg and returns a new Model.
  3. update is a plain old function, and we can call it from tests like any other function.

Let’s take a look at the type of update:

update : Msg -> Model -> ( Model, Cmd Msg )

Because this one function serves as the gatekeeper for all state changes in our application, all it takes to test any change in application state is to:

  1. Call update in a test, passing the Msg and Model of our choice
  2. Examine the Model it returns

Testing Clickedphoto

Let’s use this technique to test one of our simplest state changes: when a SlidHue message comes through the application. For reference, here's the branch of update's case-expression that runs when it receives a SlidHue message.

SlidHue hue -> 
applyFilters { model | hue = hue }

This might seem like a trivial thing to test. It does little! All it does is update the model’s hue field, right?

Listing 3 shows a basic implementation, which combines several concepts.

Listing 3 Testing SlidHue

slidHueSetsHue : Test
slidHueSetsHue =
fuzz int "SlidHue sets the hue" <|
\amount ->
initialModel ❶
|> update (SlidHue amount) ❷
|> Tuple.first ❸
|> .hue ❹
|> Expect.equal amount ❺

❶ Begin with the initial model

❷ Call update directly

❸ Discard the Cmd returned by update

❹ Return the model’s hue field

❺ The model’s hue should match the amount we gave SlidHue

Tuple.first takes a tuple and returns the first element in it. Because update returns a ( Model, Cmd Msg ) tuple, calling Tuple.first on that value discards the Cmd and returns only the Model -which is all we care about in this case.

Let’s run the test and… whoops! It didn’t compile!

Exposing variants

 port module PhotoGroove exposing (Model, Msg(..), Photo, initialModel, main, photoDecoder, update)

The (..) in Msg(..)means to expose not only the Msg type itself (for use in type annotations such as Msg -> Model -> Model), but also its variants. If we'd only exposed Msg rather than Msg(..), we still wouldn't be able to use variants like SlidHue in our test! In contrast, Photo is a type alias, and writing Photo(..)yields an error; Photo has no variants to expose!

NOTE: We could also write port module PhotoGroove exposing (..) instead of separately listing what we want to expose. It's best to avoid declaring modules with exposing (..), except in the case of test modules such as PhotoGrooveTests.

Because we’re using SlidHue without qualifying it with its module name-which we could have done by writing PhotoGroove.SlidHue instead-we'll need to expose some variants in our import declaration to bring them into scope. Let's do that for Msg, Photo, Model, and initialModel by changing the import PhotoGroove line in PhotoGrooveTests like this:

import PhotoGroove exposing (Model, Msg(..), Photo, initialModel, update)Unfortunately, elm-test doesn’t currently support testing commands directly. You can work around this if you’re willing to modify your update function. First, make a custom type which represents all the different commands your application can run. In our case this is:type Commands = FetchPhotos Decoder String | SetFilters FilterOptions Then change update to have this type: update : Msg -> Model -> ( Model, Commands )Next, write a function which converts from Commands to Cmd Msg.toCmd : Commands -> Cmd Msg toCmd commands = case commands of FetchPhotos decoder url -> Http.get url decoder |> Http.send GotPhotos Setfilters options -> setFilters optionsFinally, we can use these to assemble the type of update that programWithFlags expects:updateForProgram : Msg -> Model -> ( Model, Cmd Msg ) updateForProgram msg model = let ( newModel, commands ) = update msg model in ( newModel, toCmd commands )Now we can pass updateForProgram to programWithFlags and everything will work as before. The difference is that update returns a value we can examine in as much depth as we like, meaning we can test it in as much depth as we like!This technique is useful, but it’s rarely used in practice. The more popular approach is to hold off on testing commands until elm-test supports it directly.

Excellent! If you re-run elm-test, you should see two passing tests instead of one.

Creating multiple tests with one function

One way to add tests for the other two is to copy and paste the SetHue test two more times, and tweaking the other two to use SlidRipple and SlidNoise. This is a perfectly fine technique! If a test doesn't have other tests verifying its behavior, the best verification tool is reading the test's code. Sharing code often makes it harder to tell what a test is doing by inspection, which can seriously harm test reliability.

In the case of these sliders, though, we’d like to share code for a different reason than conciseness: they ought to behave the same way! If in the future we changed one test but forgot to change the others, which would almost certainly be a mistake. Sharing code prevents that mistake from happening!

Grouping tests with describe

Listing 4 shows how we can use the Test.describe function to make a group of slider tests.

Listing 4 Testing SlidHue

sliders : Test
sliders =
describe "Slider sets the desired field in the Model" ❶
[ testSlider "SlidHue" SlidHue .hue ❶
, testSlider "SlidRipple" SlidRipple .ripple ❶
, testSlider "SlidNoise" SlidNoise .noise ❶
]


testSlider : String -> (Int -> Msg) -> (Model -> Int) -> Test
testSlider description toMsg amountFromModel =
fuzz int description <| ❷
\amount ->
initialModel
|> update (toMsg amount) ❸
|> Tuple.first
|> amountFromModel ❹
|> Expect.equal amount

❶ Group this List of Test values under one description

❷ Use testSlider’s description argument as the test’s description

❸ (toMsg : Int -> Msg) will be SetHue, SetRipple, or SetNoise

❹ (amountFromModel : Model -> Int) will be .hue, .ripple, or .noise

The Test.describe function has this type:

describe : String -> List Test -> Test
Figure 2 Failure output after using describe

Returning tests from a custom function

The testSlider function is a generalized version of our slidHueSetsHue test from earlier. Table 3 shows them side by side.

Table 3 Comparing slidHueSetsHue and testSlider

Have a look at the type of testSlider:

testSlider : String -> (Int -> Msg) -> (Model -> Int) -> Test testSlider description toMsg amountFromModel =

Its three arguments correspond to what we want to customize about the SlidHue test:

  1. description : String lets us use descriptions other than "SlidHue sets the hue"
  2. toMsg : Int -> Msg lets us use messages other than SetHue
  3. amountFromModel : Model -> Int lets us use model fields other than .hue
describe "Slider sets the desired field in the Model" 
[ testSlider "SlidHue" SlidHue .hue
, testSlider "SlidRipple" SlidRipple .ripple
, testSlider "SlidNoise" SlidNoise .noise
]

This compiles because the SlidHue variant is a function whose type is SlidHue : Int -> Msg, which is what the toMsg argument expects, and because the .hue shorthand is a function whose type is .hue : Model -> Int, which is what the amountFromModel argument expects.

Running the Complete tests

Let’s take them out for a spin! If you re-run elm-test, you should still see four happily passing tests.

TIP: Notice how elm-test always prints " to reproduce these results, run elm-test --fuzz 100 --seed " and then a big number? That big number's the random number seed used to generate all the fuzz values. If you encounter a fuzz test which is hard to reproduce, you can copy this command and send it to coworker. If they run it on the same set of tests, they'll see the same output as you; fuzz tests are deterministic given the same seed.

We’ve now tested some decoder business logic, confirmed that running a SlidHue message through update sets model.hue appropriately, and expanded that test to test the same logic for SlidRipple and SlidNoise by using one function that created multiple tests. Next we'll take the concepts we've learned this far and apply them to testing our rendering logic as well!

That’s all for this article. If you want to learn more about the book, check in out on liveBook here and see this slide deck.


Originally published at https://freecontent.manning.com.

Manning Publications

Written by

Follow Manning Publications on Medium for free content and exclusive discounts.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade