Stop Intermittent Test Failures that Slow the Continuous Delivery Pipeline
Intermittent failures ruin the whole premise of continuous delivery (CD). In fact, they undermine any software QA process, since they break the fundamental assumption that if your test suite passes, you are good to deploy. Automated testing can run orders of magnitude more tests than manual testing but may result in an ever growing backlog of tests that fail intermittently for reasons that aren’t understood. Whether this is embarrassing, annoying, or scary depends on your sensitivity to risk. No matter the level of your pipeline’s sophistication, you need to reproduce the defect before you can begin to address an intermittent failure. Greg Law describes how capturing the actual execution of your tests makes this possible in a way that logging cannot. Debugging an exact replica of the failure allows you to go back in time to any line of code that executed and to inspect the full program state at that point. This makes test failures totally reproducible, allows them to be fixed, and restores trust in the CD process.