Developer Testing and the Micro Service Architecture

Micro service architectures where each service is maintained by a single team present some challenges to testing; challenges that are partly outside the realm of developer testing, but that can be tackled with developer testing techniques. The classic problem is this: The system made up of services A, B, and C, is for practical purposes observed and controlled by the UI. In addition, service B depends on service C. Now, how do you deploy and test that system with confidence? And who does it?

It’s not unheard of that the great honour is bestowed upon testers, who may and may not employ test automation to avoid the tediousness. And while the testers may or may not have a deep enough knowledge to test the entirety of the system during while preparing the deployment, they usually neither have the skills, time, or permissions to perform the actual deployment. Somebody else does. This setup leads to frustration, long testing and development cycles, and bugs slipping through the cracks.

It’s hard to provide a silver bullet solution to a problem of this complexity, but here’s an attempt to address the majority of common setups.

First, note the similarity between the hierarchy of the services and the test pyramid.

To utilize the analogy, let’s remind ourselves that unit tests constitute the bulk of (developer) tests. This is analogous to having the feature teams employ enough testing to ensure that their services work in isolation and means that each team is responsible for writing unit tests, integration tests, and any other tests needed to ensure that their service is working and can be modified. This includes managing the test data for their services. In the figure above, team B also needs to write tests that ensure that the integration with service C works. When all teams have achieved full coverage of their individual services, we can view this as full “unit test” coverage.

Next, the services need to be integrated with the UI. Before looking at pixels on the screen, we must acknowledge that the user interface seems to be some kind of orchestrator, which means that integration tests are needed for whatever its orchestrating. Such tests should be maintained by whatever team that deploys the system. After all, in this setup, it’s their job to make sure that the deployed system works as a whole. One might argue that it’s cumbersome/unfair/impractical to have that team own the tests, but keep in mind that feature teams A-C have done their homework by ensuring that their services and integrations have been tested and work as expected. Plainly speaking, the team that performs the integration needs a way to ensure that it works, and automated tests are the tool, just like in the lower layers.

As soon as the team starts developing tests that cover the entire system, they realize that they need to know all ins and outs of the individual services to come up with good test data, or do they? As a matter of fact, they don’t. It would be unreasonable to have a single team own the test data generation for an entire service-based system. It’s probably just as unreasonable to expect that such a team would know all business rules. To solve this, teams A through C should publish APIs for test data generation; test data that’s correct for, and relevant to, a single service. The exact specifics are context- dependent, but the goal is to simplify for the caller of each service.

So, for example, let’s say service B is the dull customer service which is prevalent throughout most of the average systems. Its team would publish a library or API endpoint tailored to creating one or multiple customers with reasonable default values, which would mean that the caller would be alleviated of the burden of knowing the specifics of postal addresses, username generation, or some context-specific flags that are typically attached to each customer. If all teams do this, the job of the integrating team is much easier. In a world of good engineering, creating these test data APIs should be very inexpensive, because the teams need to generate the same kind of data for their own tests anyway.

Given that the “integration” layer has been solved, the pixels on the screen part becomes relatively simple. The integrating team can crank out some end-to-end tests that run through the UI. Since the test data problem has been solved, it should be relatively easy. Manual testing should also be quite straight-forward at this point.

To sum this up, there are two key points:

  1. There is a team that’s responsible for deploying the system and ensuring that its components work together to form a whole. This might sound very reasonable but may require merging a testing team with an ops-heavy team and possibly adding a developer or two.
  2. Feature teams publish APIs or libraries that populate the test versions of their services with good data. Not rocket science, but not standard procedure in the industry either.

Why we automate tests

Recently, I was asked by a friend to name a few reasons to automate tests. In this context unit or integration tests—the kind of tests that should be written as part of the actual implementation—don’t count. Rather, we’re talking about test automation done either after the code’s been written, or in parallel with the implementation work. After having pondered this for a while, I came up with the following reasons.

Please keep in mind that most test automation isn’t really about automating tests, but checks. Testing usually refers to a process more creative than the work of a computer running the same verifications over and over again. This said, automation can be used to test something to a degree. I’ll get to that later.

We automate tests…

…to prevent regressions and instill courage

The more tests we can run frequently and repeatedly, the safer we feel about making changes to the code. An extensive suite of unit tests can make us feel quite safe, but a battery of automated end-to-end tests, performance, and system integration tests lets us make major code changes in complex systems without the fear and anxiety that normally accompanies refactoring of non-trivial code.

…to make any kind of iterative development work

This is an extension of the previous argument. Suppose that our team builds the software in iterations (or sprints, if you will). In iteration one, it’s able to deliver three tested stories: A, B, and C. In iteration two, it manages to deliver two more stories, D, E, while ensuring that A, B, and C still work. Iteration three adds yet another three stories, F, G, and H. Now, had there been no automated tests up until this point, regression testing would take more and more of our team’s time and its velocity would drop iteration by iteration. In fact, a team’s velocity can become more or less negative in cases where a small change results in days or weeks of regression testing, or avoiding the change altogether.

In conclusion, it’s crucial that tests are automated in teams that aim to maintain steady or increasing velocity iteration after iteration. Or we can just write “agile teams”.

…to implement continuous delivery

Taking the argument one step further would be stating that continuous delivery, or continuous deployment, just cannot work without all tests being automated. Would you release your system to production several times a day if only its unit tests were automated?

…to mobilize more brainpower and to engage

Test automation is mostly developer work. It requires non-trivial infrastructure, provisioning, and system architecture. On the other hand, if test automation is left solely to developers, without input from testers, some interesting test cases may be left out because of creator’s bias. Therefore, it’s only natural that testers and developers collaborate around test automation, which results in more brains on the problem and more buy-in from everybody.

…to make the system testable

Just as writing unit tests will make basic program elements testable, so will creating higher-level tests to larger components. Developers who know that they’ll end up writing automated tests (checks) for their system will obviously design it to accommodate that. A trivial example is putting in proper ids in web pages, so what WebDriver testing becomes a breeze, but other examples are easy to find.

…to verify the acceptance criteria of a story

Yes, acceptance test-driven development (ATDD), behavior-driven development (BDD), and specification by example (pick your flavor) are mostly about collaboration and shared understanding. That being said, tests that emerge from the specifications based on concrete examples provide a phenomenal opportunity to formalize the act of meeting a story’s acceptance criteria. If the team decides that a certain story has three major acceptance criteria, one developer may start implementing the tests for them, while another starts implementing the actual functionality. For a while, the tests will be red, but once they turn green, there should be indisputable proof that the story has indeed been implemented according to plan.

…to find defects in permutation-intensive scenarios

Many tests—both unit tests and higher level tests—are just examples that confirm something we know about the code. However, there are types of automated tests that can actually find new defects. True, this only applies during their first execution (after which they become normal regression tests), but still.

What tests am I talking about? Generative tests and tests based on models (MBT), of course. Such tests operate given a set of constraints, rather than exact scenarios, and are able to produce states that our brilliant tester mindset hasn’t been able to foresee.

…to find defects 🙂

Finally, if you’re a tech-savvy tester, or a developer in test, why even bother with manual tests? To be usable in the long run, the need to go into the regression test suite anyway. Therefore, why not automate them from the start?

Well, that’s that: my quick thoughts on automating tests. Comments, rebuke, and feedback welcome.