Testing and Test Driven Development in Snap (Part 2)

In our previous post, we described the new testing taxonomy that we’re adopting for the Snap test framework, the transition from our older, legacy tests to the newer small, medium, and large tests, and what that transition means for contributors to the Snap framework. In this post, we will continue that discussion by showing a detailed example of the refactoring and code changes that were necessary to convert the legacy tests for a single package in the Snap framework into a new set of small tests that provide adequate coverage of that package. Along the way we’ll discuss the changes that we made to the existing legacy tests that were refactored into a set of small tests as well as the corresponding changes that we had to make to the underlying code from the Snap framework that we were testing to ensure that we could construct appropriate mock objects in those small tests. Our hope is that this detailed example will help contributors by showing them how to construct a similar set of small tests for their own contributions to the Snap framework.

Understanding the value of each test type (image inspired by this post)

Construction of test “types” in the Snap codebase

For those not familiar with the go test command, construction of a test suite that contains multiple categories (like those described in our previous post) is a relatively simple process. The go test command actually picks up and runs the tests contained in any file in the targeted package (or packages) with a name that looks like *_test.go (here, we’re using a glob-style pattern for the filename). For example, the scheduler package in the current Snap framework includes the following legacy test files:

Unfortunately, the original tests in the Snap framework were not separated out by type. Instead, as you can see from the list shown above, the original tests for the various modules that make up the Snap framework were all placed into a common set of files with names that were based purely on the functionality that was being tested. After looking through the current version of the Snap framework, our first impression is that most of the original tests in the framework most likely would be classified as medium tests in our new taxonomy. That being the case, a significant amount of work will be necessary to refactor those tests into a new set of small, medium, and large tests that provide sufficient coverage of the code in the Snap framework before the existing tests can be phased out.

But, how exactly should we indicate that a given test file in the Snap framework contains small, medium, or large tests? As it turns out, tagging a file as containing tests from one of those categories is a relatively simple process in Go. We simply add a single line to the beginning of each file that provides a build tag describing the type of tests that are contained in that file. For example, a line like this

// +build small

would identify the file in question as one containing small tests, while a line like this

// +build medium

would identify that file as a file containing medium tests. That being said, simply adding a build tag to the test file probably is not enough. While this satisfies the constraints from a go test perspective, it’s not obvious to the outside observer which files contain which types of tests. To make it obvious, we decided to extend the file naming convention for test files in the Snap framework to include the build tag value for each file as part of the filename. A file containing small tests would look like *_small_test.go, while a file containing medium tests would look like *_medium_test.go. By adopting this file naming convention we can make it very clear to developers and users which type of tests are contained in each of the test files in the Snap framework.

Once build tags have been added to the new test files that we are adding to the Snap framework, it is easy to run tests of a specific type by adding a -tags [TAG] command-line flag to the go test command (where the [TAG] value is replaced by one of our test types). For example, the following command will run all of the tests in the current working directory or any of it’s subdirectories that have been tagged as small tests:

$ go test -v -tags=small ./...

It should be noted here that the ./… path from the go test … command shown above is actually a package reference; it refers to the package in the current working directory and all packages contained under that directory. If we wanted to run the same set of tests from outside of that directory, we could replace the first . in that package reference with the complete name of one of our packages. For example, the following command will run all of the small tests in a local clone of the Snap repository:

$ go test -v -tags=small github.com/intelsdi-x/snap/...

While the changes described above provide us with a strategy for adding test files to the Snap framework containing tests from our new taxonomy, additional changes will be necessary in order to fully separate our tests by type. This is a result of how the go test command handles untagged tests when a -tags [TAG] command-line flag is provided as part of the command. It turns out that if there are any untagged tests (tests that don’t have a build tag value assigned to them) in the package or packages that are referenced in the go test command, those untagged tests will also be run, even if a -tags [TAG] command-line flag was provided.

To get around this issue (and completely separate out our tests) we added a new legacy build tag to all of our original tests from the Snap framework. We also modified the current test matrix in the appropriate TravisCI and shell script files so that both the existing legacy tests and the new small tests that we are developing would be run on any pull requests received from the community, at least until such time that we feel that there is sufficient code coverage provided by those small tests. Once the maintainers feel that the new small tests that are being developed provide sufficient code coverage, the legacy tests will be either phased out or used to construct a set of medium and large tests for the Snap CI/CD toolchain.

Generating new ‘small’ tests from the existing ‘legacy’ tests

So, with the approach outlined above in mind, the first step in this process of migration from the existing tests to a set of tests that follow the new taxonomy we are adopting will be to create a new set of small tests that can be used to replace the existing legacy tests over time. Fortunately, the procedure we need to follow to accomplish this is relatively simple.

The first step is to generate an equivalent *_small_test.go file from one of the existing *_test.go files and tag that new *_small_test.go file as a file containing small tests by adding a small build flag to the start of that file.

// +build small

Once the new file has been tagged as a small test file, the next step is to determine which tests, if any, from the existing legacy tests that we have copied over can be used as small tests “as is” and which will need to be modified before they can be used as small tests. Ideally, the individual tests in any given small test file will test the functionality of one, and only one function or method from the Snap framework. If there is significant coverage in any given test of multiple functions or methods, or if those tests rely on access to external resources (access to local filesystem, for example), then we have identified an interaction in the underlying codebase that we will have to mock in our small tests in order to test the components of the Snap framework separately as part of our new small test suite.

Fortunately, identifying these interactions is a relatively straight forward process: all we have to do is generate a coverage map for the tests that we are converting. This is accomplished by using a combination of the go test … and go cover … commands. For example, here’s the combination of commands that will generate a coverage map (on a function-by-function basis) for the TestWorkerManager tests, which can be found in the scheduler/work_manager_test.go file in the Snap framework:

For clarity, we’ve used a grep command to filter out the functions from the scheduler package that are not covered at all by that TestWorkerManager test set (those with 0.0% coverage). Ideally, the output of this command would only show coverage of a single function from the work_manager.go file (the file that is being tested by our TestWorkerManager tests in the work_manager_test.go file), but as you can see in this example the current tests also cover functions and methods from the job.go, queue.go, and worker.go files. The methods called from these other files (those not in the work_manager.go file) would be our primary target for functionality that needs to be mocked up in order to turn this set of tests into a set of truly small tests.

However, before we discuss the changes that we need to make, perhaps it would be best if we first discuss how mocking works in Go. Once the concept of mocking in Go is clearly described, we can return to our step-by-step example.

Mocking in Go

Those familiar with the concept of mocking in other languages might think that the best approach would be to search for an appropriate mock framework or dependency injection framework that we could use to auto-generate the mock objects that we will need in order to mock up the expected responses from methods in our Go codebase for use in our small tests. However, our approach here is to actually write the mock objects we need for those tests as native Go code (defined in the test files themselves) instead. Some might question this decision, however it was made consciously and with a good deal of forethought. We’d like to explain some of that reasoning behind that decision here before we move on with our example.

The real issue is that regardless of how you’re mocking responses within your small tests (natively, as mock objects in the tests themselves, or through some sort of external framework that auto-generates those mock objects for you) the first step is always the same: defining a set of interfaces in your project’s source code for the behavior that you would like to mock within your small tests. Interfaces are a powerful feature in Go (or any other language that supports this concept) that let you define the API for the components that make up your application without being very specific about the actual implementation details of that API. As such, the interactions between the components in your application should be defined as interfaces. Once those interfaces are properly defined in your application it’s a fairly easy matter to actually mock up a response for any of those components for use in your test framework. However, without those interface definitions, it’s not possible to define such mocks in your test framework, regardless of the whether those mock objects are defined in native code within your test framework or are auto-generated by an external library or solution.

Conversely, auto-generation of a set of mock objects using an external framework has a couple of fairly serious drawbacks. First, it creates another tool or layer in your project that any potential contributor must understand before they can actually contribute to your project. Second, and perhaps more importantly, it has been our experience that any component in a project that is initially auto-generated by an outside framework or tool is a component that becomes very difficult to manage over time as the project evolves. Given these drawbacks, we have consciously chosen to implement our mock objects in native Go code here rather than attempting to formally adopt a mocking or dependency injection framework that auto-generates the mock objects used in the Snap test framework for us.

With those constraints in mind, we’d like to return to our step-by-step example: the refactoring of the existing set of legacy tests of the code used by Snap to load it’s global configuration file (in JSON or YAML format) into a new set of small tests that adequately cover that same code. Our hope is that you will find this detailed example useful as a when it comes time to construct your own small tests (or when you decide to help with the process of converting the existing legacy tests in the Snap framework into small tests).

Step 1: adding a mock that mimics reading of a configuration file

The first step in the process, after creating the small test file from one of the existing legacy test files, involves the removal of any dependencies from the set of small tests that we’re constructing on external systems or resources. The issue here is that the code in the pkg/cfgfile/cfgfile.go file that we are testing with our new set of small tests actually makes a call to the ReadFile(…) method from the io/ioutil package in order to load the named JSON or YAML configuration file into the interface that is passed as an argument to that function. This dependency on the underlying filesystem can be seen in the coverage map for the small tests that we’re constructing:

In this example, we’ve added the -coverpkg command-line flag to our go test … command and passed in a comma-separated list of the packages for which we would like to receive coverage information. As you can clearly see, there are a number of dependencies in this test on functions from the io/ioutil package, all of which are related to the call to the ioutil.TempFile(…) method from within our new set of small tests and the call made to the ioutil.ReadFile(…) method from within the cfgfile.Read(…) function that we are testing.

For reference, here is a snippet of code that shows how this call to the ioutil.ReadFile(…) method looked prior to our refactoring of the pkg/cfgfile/cfgfile.go file from the Snap codebase:

For completeness, here’s a snippet from our newly created small test file that shows how these tests actually write out YAML and JSON files to the temporary filesystem and then pass the resulting filenames for each of those files into the cfgfile.Read(…) method in order to test its functionality.

In this example, the calls to the writeYaml(tc) and writeJson(tc) methods on the third and fourth lines of this snippet actually create the two temporary files in the /tmp directory on the localhost’s filesystem. Those files are then read back into the local config instances from those temporary files by the Read(…) calls on the second line of both of the Convey code blocks. Finally, here’s an example of one of those two write*(…) methods from the same small test file for reference.

As you can clearly see, this example takes the input testConfig object, marshals it as a YAML byte slice, and then writes those bytes to the temporary file created by the call to the ioutil.TempFile(…) method. The first return value from the ioutil.TempFile(…) method is a pointer to the underlying os.File instance that it creates, and the filename of that file is returned to the calling routine and passed as an argument to the cfgfile.Read(…) function that we are testing.

To turn this set of tests into a set of tests that really are small tests, we need to refactor our new tests to remove this dependency on the creation (and reading) of those two temporary files. To accomplish this, we need to refactor the existing cfgfile.go file (the file containing the functionality we’re testing here) so that we can easily override the call to the ioutil.ReadFile(…) method that it contains in any small tests of that code. Once that refactoring is complete we can then go back and add a mock object to our small tests that overrides the call to this method from the Read(…) function in the cfgfile.go file. This mock object will ensure that the contents are read from memory in those small tests instead of being retrieved from an actual file on the filesystem.

The first step in this refactoring process is to wrap the underlying call to the ioutil.ReadFile(…) method in the cfgfile.go file so that we can easily override the call to this method in any of our small tests. This involves:

• creation of a reader interface
• creation of a new cfgReaderType; a struct that implements that interface
• creation of a cfgReader instance variable that is of that interface type, and
• initialization of that instance variable via a new init() method which will run the first time that the cfgfile.go file is loaded

Once those changes are made, we can then modify the existing Read(…) function so that it calls to our new cfgReader.ReadFile(…) method instead of calling the ioutil.ReadFile(…) method directly. When these modifications are made, the snippet that we showed above ends up looking something like this:

Now that we’ve modified our cfgfile.go file, the next step is to modify our small tests so that they define a mock object that overrides the default behavior of the cfgReader.ReadFile(…) method that is shown above. To accomplish this, we need to:

• create a mockReader that will override that behavior
• bind a new ReadFile(…) method to that mockReader type
• create an instance of our new mockReader type for the two tests that were shown above (one to test “reading” a YAML “configuration file” and the other to test “reading” of a JSON “configuration file”), and
• assign that instance to the same cfgReader instance variable

Once our refactoring of our new small test file is complete, the small tests in that file will use the ReadFile(…) method associated with our mockReader instead of the ReadFile(…) method associated with the cfgReader interface that is defined in the cfgfile.go file. The end result of this refactoring of our small test file looks something like this:

As you can see from this example, the modifications made to these two files are relatively simple, and the end result is a pair of small tests that don’t rely on access to the localhost’s temporary filesystem.

Step 2: adding a mock for additional calls to other functions

While we’ve made a good start in separating out our small tests, when we re-generate our coverage map using the newly refactored small tests we can see that there is another issue that we will have to deal with before our refactoring is complete:

While we’ve removed our external dependency on the iotuil.ReadFile(…) method, our testing of the Read(…) function actually invokes the associated ValidateSchema(…) function. This dependency on the underlying ValidateSchema(…) function in our TestReadConfig tests shows up as 50% code coverage of that associated function by the tests we’ve written to test the Read(…) function. That invocation can be seen in this snippet from our newly refactored cfgfile.go file:

To remove this interdependency between these two method calls and convert our TestReadConfig tests into a set of small tests that only test the Read(…) function we’ll have to do a bit more refactoring.

As was the case with the refactoring we did to remove the dependency on the underlying iotuil.ReadFile(…) method, we’ll:

• define a new schemaValidator interface that wraps the existing ValidateSchema(…) function in the cfgfile.go file as a method within that new interface type
• create a new schemaValidatorType, a struct that implements that interface: as part of this step in the process we’ll move the code from the existing ValidateSchema(…) function, making it the implementation of this method in our new schemaValidatorType
• add a cfgValidator instance variable that is of that interface type, and
• initialize that instance variable in the init() method that we added in the previous round of refactoring we performed on this file

The last change we’ll make in this round of refactoring is to modify the Read(…) function in our refactored cfgfile.go file so that it calls our new cfgValidator.ValidateSchema(…) method instead of calling the old ValidateSchema(…) function directly. The following snippet shows the result of that refactoring:

With these changes in place, we can then create a second mock object in our small test file and use that mock object to override the default schema validation process. Our modified test will look something like this:

and the result of running the same coverage test that we showed above, after those changes are made, is as follows:

As you can see, we’ve now completely removed the interdependency between the TestReadConfig test set and the underlying ValidateSchema(…) function call. What is left is a set of tests that only test the functionality of a single function call, the Read(…) function. So with the changes shown above we have finally constructed a set of true, small tests.

Step 3: completing our refactoring

While the refactoring shown in the previous two steps in this process is a good start to the refactoring we need to do, we’re not quite done. As you can see from the output of the go test … && go tool cover … command shown above, the current set of tests in that test set only cover 53.8% of the Read(…) method that we are testing. To provide adequate coverage of this function we’ll have to add more tests to our TestReadConfig tests. We also need to add tests to our test suite that test the functionality of the ValidateSchema(…) method independently, currently there are none.

The refactoring involved in this step of the process is actually much simpler since it will only require adding new tests to our cfgfile_small_test.go file in order to test alternate paths through the functions we are testing from the cfgfile.go file. However, before we can add those additional tests of the Read(…) function we’ll need to add some additional behavior to our mockReader so that it can return more than two responses. To accomplish this, we will:

• create an enumeration of the response types we want to receive back from our mockReader
• modify our mockReader instance so that it responds with one of those pre-defined responses

This will be accomplished using a test table that maps the enumerated values for each of our expected responses into a set of entries that contain the []byte and error values that should be returned for that particular scenario. We can then modify the mockReader.ReadFile(…) method so that it returns the expected output and error values from the specific test table entry represented by that mockReader instance. This snippet shows the result once that part of our refactoring is complete:

Similarly, we need to modify our mockSchemaValidator to add a bit of state so that it can return a result indicating that an invalid schema was found. This snippet shows the result of that refactoring:

Now we can add some additional tests to the TestReadConfig test set that will test the alternate code paths through the cfgfile.Read(…) function that we are testing:

As you can see, we’re now testing not only for our ability to successfully parse both JSON and YAML files, but also for our ability to catch the errors that result when: the requested file cannot be found, the YAML/JSON that is read is invalid, the schema that is passed into the Read(…) method is invalid or the YAML/JSON that is read is not valid according to the schema that was passed in.

With those changes in place, our code coverage has now improved significantly over the previous test run:

While we are at it, we need to do a bit more refactoring in order to add a new set of tests that can be used to test the cfgValidator.ValidateSchema(…) method from our cfgfile.go file directly:

and then generate a coverage map for that set of tests to see how well our new tests cover the cfgValidator.ValidateSchema(…) method

We’re now done with our refactoring. Our new set of tests and mock objects can be used to obtain 100% coverage of both the Read(…) function and the cfgValidator.ValidateSchema(…) method from our refactored pkg/cfgfile/cfgfile.go file.

Step 4: Validate our refactoring

Before we commit our changes, there is one more step that we should take to ensure that we have refactored the pkg/cfgfile/cfgfile.go file properly. To ensure that our refactoring has not broken anything, we should run the existing legacy tests one more time and check their output.

Since our newly refactored code still passes the existing legacy tests, we can rest assured that our refactoring did not break any of the existing functionality in that code and we know that we can safely commit our changes to our fork of the Snap repository and submit those changes as a pull request to the Snap project.

In closing

Hopefully this detailed example helps you, as a potential contributor to the Snap framework, understand the strategy that we’re following to build out a new set of small tests that adequately cover part of the Snap framework’s codebase from the existing set of legacy tests.

As we said in the closing comments in our previous post, this transition from the existing set of legacy tests in the Snap framework to a new set of small, medium, and large tests is a process that we are just starting and that will take some time to complete. We certainly welcome feedback and/or contributions from the Snap community as part of this process, and encourage you to reach out to us directly via the Snap Gitter channel at https://gitter.im/intelsdi-x/snap if you would like to participate.