Why we automate tests

Recently, I was asked by a friend to name a few reasons to automate tests. In this context unit or integration tests—the kind of tests that should be written as part of the actual implementation—don’t count. Rather, we’re talking about test automation done either after the code’s been written, or in parallel with the implementation work. After having pondered this for a while, I came up with the following reasons.

Please keep in mind that most test automation isn’t really about automating tests, but checks. Testing usually refers to a process more creative than the work of a computer running the same verifications over and over again. This said, automation can be used to test something to a degree. I’ll get to that later.

We automate tests…

…to prevent regressions and instill courage

The more tests we can run frequently and repeatedly, the safer we feel about making changes to the code. An extensive suite of unit tests can make us feel quite safe, but a battery of automated end-to-end tests, performance, and system integration tests lets us make major code changes in complex systems without the fear and anxiety that normally accompanies refactoring of non-trivial code.

…to make any kind of iterative development work

This is an extension of the previous argument. Suppose that our team builds the software in iterations (or sprints, if you will). In iteration one, it’s able to deliver three tested stories: A, B, and C. In iteration two, it manages to deliver two more stories, D, E, while ensuring that A, B, and C still work. Iteration three adds yet another three stories, F, G, and H. Now, had there been no automated tests up until this point, regression testing would take more and more of our team’s time and its velocity would drop iteration by iteration. In fact, a team’s velocity can become more or less negative in cases where a small change results in days or weeks of regression testing, or avoiding the change altogether.

In conclusion, it’s crucial that tests are automated in teams that aim to maintain steady or increasing velocity iteration after iteration. Or we can just write “agile teams”.

…to implement continuous delivery

Taking the argument one step further would be stating that continuous delivery, or continuous deployment, just cannot work without all tests being automated. Would you release your system to production several times a day if only its unit tests were automated?

…to mobilize more brainpower and to engage

Test automation is mostly developer work. It requires non-trivial infrastructure, provisioning, and system architecture. On the other hand, if test automation is left solely to developers, without input from testers, some interesting test cases may be left out because of creator’s bias. Therefore, it’s only natural that testers and developers collaborate around test automation, which results in more brains on the problem and more buy-in from everybody.

…to make the system testable

Just as writing unit tests will make basic program elements testable, so will creating higher-level tests to larger components. Developers who know that they’ll end up writing automated tests (checks) for their system will obviously design it to accommodate that. A trivial example is putting in proper ids in web pages, so what WebDriver testing becomes a breeze, but other examples are easy to find.

…to verify the acceptance criteria of a story

Yes, acceptance test-driven development (ATDD), behavior-driven development (BDD), and specification by example (pick your flavor) are mostly about collaboration and shared understanding. That being said, tests that emerge from the specifications based on concrete examples provide a phenomenal opportunity to formalize the act of meeting a story’s acceptance criteria. If the team decides that a certain story has three major acceptance criteria, one developer may start implementing the tests for them, while another starts implementing the actual functionality. For a while, the tests will be red, but once they turn green, there should be indisputable proof that the story has indeed been implemented according to plan.

…to find defects in permutation-intensive scenarios

Many tests—both unit tests and higher level tests—are just examples that confirm something we know about the code. However, there are types of automated tests that can actually find new defects. True, this only applies during their first execution (after which they become normal regression tests), but still.

What tests am I talking about? Generative tests and tests based on models (MBT), of course. Such tests operate given a set of constraints, rather than exact scenarios, and are able to produce states that our brilliant tester mindset hasn’t been able to foresee.

…to find defects 🙂

Finally, if you’re a tech-savvy tester, or a developer in test, why even bother with manual tests? To be usable in the long run, the need to go into the regression test suite anyway. Therefore, why not automate them from the start?

Well, that’s that: my quick thoughts on automating tests. Comments, rebuke, and feedback welcome.