pytest: Save full error output to file

I recently had a problem with some rare occurring failures in tests, they happened at a rate of about 1 time per 5000 full suite runs. These were property-based tests so input-data was automatically generated.

As to make matters worse the entire test suite froze after the failure before it was unable to print what went wrong, at that point Travis-CI would kill the test-runner before we had a chance to catch it.

The natural next step was to start logging failures to a file while running the test suite continuously on a machine set up for the purpose. I used a snippet from the pytest site. At that point I let the tests run in an infinite loop and the next week I hit my first failure. Turns out I hadn’t looked at the snippet to closely, it only logs which tests failed and no details about which params were used or what went wrong.

So I had to tweak the code some and ended up with the following snippet.

Logs full failures including asserts and fixtures, save as conftest.py in your pytest root directory

At this point, I started running my tests again and this time I were rather lucky, only about 30 hours later I got my first failure with a proper falsifying, repeatable example.

At this point, we could finally fix the bug. I hope other Python developers using pytest can find this useful to achieve the same results.