Building Nimoy: The Base
I’ve decided to document my process of building Nimoy — The testing and specification framework for Python. This is the first installment.
“So you’ve decided to build a testing framework”
And the first thing a testing framework should be able to do is run tests.
Nimoy is a different type of testing framework. It’s not a fully fledged testing framework but a wrapper around an existing one.
From the start I knew that Nimoy should be a wrapper rather than a rewrite. This realization came as Python already has great testing frameworks and Nimoy can improve on one, rather than replace one.
“There’s so many to choose from!”
So which existing framework should I use as Nimoy’s base?
Here are my requirements:
1) Lightweight
2) Modern
3) Programmatic APIs for running tests
And here are the candidates:
unittest (Formerly PyUnit)
Is (I think) the eldest of them all. Some argue is the worst for the reason that it’s a copy of Java’s JUnit and isn’t “Pythonic”.
Is it lightweight? Yes. It’s bundled with Python. Starting with Python 3 it even comes bundled with the Mock framework. There’s no need for any extra dependencies when using unittest.
Is it modern? Fairly, yes. Though features and fixes are not quick as it’s tied to Python releases.
Does it have programmatic test running APIs? Yes, and it’s simple too!
Say you’ve got a test class
You can programmatically create a unittest suite to execute it
py.test
Is the most popular and well known testing framework for Python. Probably the best framework as well.
Is it lightweight? No, it’s a behemoth of a module. Fully featured and well matured, it’s the django of testing frameworks.
Is it modern? Yes. It sees constant development and as it’s an independent module, it has its own release cycle. Features and fixes can be pushed quickly.
Does it have programmatic test running APIs? No. There’s an internal API which can be fed locations of testable modules but it isn’t reliable enough.
nose2
A reboot of the popular but abandoned nose framework.
Is it lightweight? Yes, it’s actually based on unittest and relies on a plugin system for its features.
Is it modern? Sadly, no. Looking at the github project, commits are being done but the project hasn’t seen a release for over a year.
Does it have programmatic test running APIs? Similar to py.test, no. You can try but results may be hacky.
The Verdict
I really wanted to use py.test for its richness, but the lack of programmatic APIs has put me off. I’d rather rely on a stable, public API.
Nose’s abandonment issues and lack of programmatic API are deal breakers as well. I don’t want to rely on unmaintained code.
unittest has modern features, is regularly maintained and has programmatic APIs. I hesitated to use unittest because of its Java-ish flavour but the fact that Nimoy is a DSL that wraps the framework makes this issue negligible.
unittest is the winner!
