Building Nimoy: The Test Runner

Documenting the process of building Nimoy — Part 2.

Now that we’ve decided on the foundations of Nimoy, let’s get down to business.

Run, Run, Run

First, a test runner should be able to run tests.
Let’s try to break down the requirements for this behavior. Nimoy needs to:
1. Auto-discover specification files
2. Read the text contents of every specification file
3. Transform each specification file to valid, executable Python code
4. Execute the specifications in a context of a unittest suite

Eureka!

Let’s look at the context in which Nimoy discovers tests. We run Nimoy from a certain location on the filesystem, say your project’s home directory. We’ll name this the working directory. On top of that, the user may suggest locations of other Specification files. These locations could point to directories that contain Specification files. They could also point to specific files. We’ll name these the suggested locations.

It is possible that there will be no suggested locations. In this case we’ll default and search for Specifications within the working directory.

Let’s write some code to support this! We’ll start from top to bottom with the SpecFinder class:

The constructor accepts a working_directory parameter. This parameter represents the file system location in which Nimoy was executed.

The class has a single entry point — the find method. find accepts a single parameter — suggested_locations. This is a list of file system locations. It might be an empty one. They may be relative or absolute and may point to a directory or a Specification file.

To implement the default fall back, we test suggested_locations. If it’s empty, we append the working directory to the list.

Now we can scan the suggested_locations for Specification files by calling _find_specs_in_suggested_locations:

Iterating over every suggested_location we first must normalize each one. This is because the suggested location may be relative or absolute. So we call _normalize_suggested_location.

By convention, Nimoy will treat any Python file ending with spec.py as a Specification file. If our normalized suggested location ends with spec.py we can add it the list of results and carry on. If it doesn’t, we treat it as a directory and call _find_specs_in_directory.

Let’s take a look at how we normalize each path:

Using Python’s os.path.isabs method we can learn whether the suggested location is an absolute path or not. If it is, no problem! Return it. If it’s a relative path, we’ll join it to the working_directory to make it absolute. We join paths using Python’s os.path.join method.

And searching for Specifications in a directory is super as easy:

The easiest way to recursively find all files in a directory is to use Python’s os.walk. Given a directory path it’ll return the names of all the directories and files it contains. But we can’t be sure that the suggested directory contains only Specification files, so we reduce the list of files using fnmatch.filter. The filter method receives a pattern and returns all strings that match it.

Listen, Learn, Read On

So we’ve got our hands on all relevant Specifications, let’s read their contents. To read the Specification files, we’ll create a simple SpecReader class:

The constructor accepts a single parameter — resource_reader. The read method returns a generator rather than one big list. A generator lets us read any number of Specifications without loading them all to memory at once.

The SpecReader should not care if it reads the resource from a file system or a socket. This is why it uses an abstraction for reading each Specification. It reads and aggregates. The resource_reader accepts the location and takes care of the technicalities. With this abstraction we can substitute content sources with minimal side effects. A simple file system resource reader would look like:

Transformers Assemble!

The Specification syntax which is invalid Python code. A chain of transformers process it and manipulates it into valid Python code. This is Nimoy’s “magic” and the next step of our test runner.

For this purpose we’ll create the SpecLoader class:

The constructor accepts a single parameter — ast_chain. This is a single entry point for all transformations that need to take place.

To transform the Specification, we first must parse it into an AST structure. We do this using the ast.parse function. It takes the text and returns an AST node object that we can the manipulate.

The AST nodes represent the source code. As a result of that, each node maintains the line and column number of every expression. This is due to the fact that Python is whitespace sensitive. Because we manipulated the AST nodes, their line and column numbers are no longer valid. Calculating the new numbers would be a pain in the ass. Luckily we have a helper method already fixes them for us. So after having all manipulations take place, we call ast.fix_missing_locations.

Let’s turn these AST nodes into actual executable code! To do this we can invoke Python’s built-in compile method.

Last but not least — load each compiled Specification as a module using Python’s built-in exec method.

The Last Mile

We’re almost done! All that’s left is to execute all Specifications with the SpecExecutor class:

We’ve made the decision to execute all Specifications using unittest. Yet, a pluggable execution framework might still prove useful. To achieve this the SpecExecutor delegates the execution to an execution_framework.

We then initialize a test suite, append every Specification to it and run the suite.

The implementation of the unittest execution_framework is simple as well:

Sticking Together

Now we have all 4 building blocks required for running our Nimoy Specifications! Huzzah!