Category Archives: testing

Flaky test Kessler syndrome

I have been working within a monorepo (that is, a gigundo Git repository with lots of separate projects in it) and participating in maintaining its test health. This includes unit tests.

  • After every commit to the repository, all of the unit tests across the whole thing are run by our CI system. The idea is that the entire monorepo should always build successfully.
  • Also, for every pull request (proposed change), all of the unit tests across the whole thing are run by our CI system. Same idea as above.

It isn’t feasible for individual contributors to do this themselves, as the monorepo is just too big and varied. I don’t even try to compile the entire thing myself, let alone run all of its unit tests. All I do, and expect from others, is work within the projects that are relevant to the task at hand (and their dependencies), including making sure that existing and new tests pass.

We really need existing tests to be reliable. A flaky test that fails randomly once in a while crops up in random test runs in the CI system. For a PR test, it’s annoying because that test failure seldom has anything to do with the proposed change. For a post-commit test, it’s just noise. In both circumstances, someone tasked to investigate test failures has to spend time discovering that the flaky test is at fault – a false alarm. Additionally, someone proposing a PR has to re-run tests so that, hopefully, that flaky test doesn’t flake out again. This all produces friction in the development process.

Say you have multiple flaky tests. Now, a test run may fail if one or more of them fails. If there are only a few flaky tests, the effect isn’t much worse than having one. But, as the number of flaky tests grows, probability starts to be your enemy.

Continue reading

Chunking through unit tests

Ever since the advent of multi-tier application architectures (which seems forever ago now), I’ve commonly had to work on tasks that span multiple levels of abstraction. You probably have too. For example, say you need to implement a new REST API call in your classic web application. You have to:

  • Define the endpoint itself, with its request and response formats.
  • Implement some business logic for it.
  • Update the data layer to persist and/or query what needs it.

That’s just your three classic tiers, and sure, once you’ve done this sort of work enough, it’s almost straightforward. But, as time goes on and your application gets complicated, more aspects creep in. Security. Audit logging. Retries and resiliency. And then there’s all those newfangled microservices to deal with. The neat layer cake ends up looking more like monkey bread. How can you keep track of all of this and get your work done?

One way is to use something you should be doing anyway: writing unit tests.

Continue reading