Flaky test Kessler syndrome

I have been working within a monorepo (that is, a gigundo Git repository with lots of separate projects in it) and participating in maintaining its test health. This includes unit tests.

  • After every commit to the repository, all of the unit tests across the whole thing are run by our CI system. The idea is that the entire monorepo should always build successfully.
  • Also, for every pull request (proposed change), all of the unit tests across the whole thing are run by our CI system. Same idea as above.

It isn’t feasible for individual contributors to do this themselves, as the monorepo is just too big and varied. I don’t even try to compile the entire thing myself, let alone run all of its unit tests. All I do, and expect from others, is work within the projects that are relevant to the task at hand (and their dependencies), including making sure that existing and new tests pass.

We really need existing tests to be reliable. A flaky test that fails randomly once in a while crops up in random test runs in the CI system. For a PR test, it’s annoying because that test failure seldom has anything to do with the proposed change. For a post-commit test, it’s just noise. In both circumstances, someone tasked to investigate test failures has to spend time discovering that the flaky test is at fault – a false alarm. Additionally, someone proposing a PR has to re-run tests so that, hopefully, that flaky test doesn’t flake out again. This all produces friction in the development process.

Say you have multiple flaky tests. Now, a test run may fail if one or more of them fails. If there are only a few flaky tests, the effect isn’t much worse than having one. But, as the number of flaky tests grows, probability starts to be your enemy.

Continue reading

Software commitments guarantee poor quality

You’ve heard of the iron triangle, right? Of course you have, I think they teach it in coder kindergarten now. Since I couldn’t find any freely-licensed images of it in less than a minute, I drew my own.

As a refresher, you can only pick two of the three points of this triangle. The third one is therefore chosen for you. Well, that’s how it’s supposed to work.

Continue reading

Controversial programming opinions

The software engineering Stack Exchange has a mostly dead (not judging) community blog associated with it. I found this article on controversial programming opinions there, via Hacker News. Because opinions are like, um, other things that everyone has one of, here are my takes on the list.

Each opinion is presented in italics, and I’ll respond assuming that you’ve read the original argument for the opinion.

Continue reading

Three Rules of Leadership

These rules are my own idea for how a good leader does their job. I don’t claim they are “The” Three Rules, just a set of them that I like.

Because I am a nerd, they are structured like Asimov’s Three Laws of Robotics. You can break later rules in order to follow an earlier one.

Rule 1. Don’t be stupid.

Above all else, don’t make moronic decisions or take dumb actions. This is essentially unforgivable. A leader has to be intelligent or, some might say, “smart”.

Rule 2. Be fair, unless that would be stupid.

Even if you have to displease somebody, if it’s a fair decision, they should eventually at least understand and accept it. Everyone really cares about fairness, even the very young.

However, it’s a mistake to do stupid things in the name of fairness. See Harrison Bergeron. It helps to remember that fair does not mean equal.

Rule 3. Don’t be a jerk, unless that’s needed to be fair and not stupid.

To be effective, those you lead must at least not hate you. When a leader acts, they should be compassionate and understanding, or at least not hostile.

However, if push comes to shove, and you can’t effect change in a fair and intelligent way otherwise, then you’ll have to stop being nice, and get the job done. Can’t please everybody.


I don’t want to expound on these ideas, because, well, that could take a while, and detract from their nice, simple form. I’ve considered them for quite some time, and I think they hold up well when considering what real-world leaders do, and whether they are doing a good job or not.

Implicit no

I’ve felt busier at work this year than usual. (Hence why it’s been a year since my last post – yikes!) I’m probably not actually busier as measured by, I dunno, theoretical units of work effort. It’s that there is a wider variety and breadth of what I have to deal with each day than, I think, ever. Rather than concentrating on fewer items, my time and attention are spread across many items.

I don’t like working like this, and I’m also not used to it, compared with diving deeply into technical efforts. I don’t have the mental habits in place to work well this way (at least by my own assessment).

So, I’ve been learning and reading about this conundrum. While researching one topic, how to make effective requests, I ended up reflecting on how I respond to requests. I try to take care of everything on my plate, regardless of who placed it there, but this year I’ve seen just how finite the size of that plate is. I can’t respond to everything that either I want to or others want me to.

For this common work scenario, you’ll hear the recommendation to learn to say no. I’ve realized how very rarely, nay, even seldom, I out-and-out say no. I mean, I totally buy into the idea, but when it comes time … that little word barely ever passes between my lips. Just thinking about saying that to a coworker feels bad to me when I imagine it.

The problem is that, because of my reluctance to explicitly say no, I am instead implicitly saying no, now more than I used to. That’s even worse.

Continue reading

Chunking through unit tests

Ever since the advent of multi-tier application architectures (which seems forever ago now), I’ve commonly had to work on tasks that span multiple levels of abstraction. You probably have too. For example, say you need to implement a new REST API call in your classic web application. You have to:

  • Define the endpoint itself, with its request and response formats.
  • Implement some business logic for it.
  • Update the data layer to persist and/or query what needs it.

That’s just your three classic tiers, and sure, once you’ve done this sort of work enough, it’s almost straightforward. But, as time goes on and your application gets complicated, more aspects creep in. Security. Audit logging. Retries and resiliency. And then there’s all those newfangled microservices to deal with. The neat layer cake ends up looking more like monkey bread. How can you keep track of all of this and get your work done?

One way is to use something you should be doing anyway: writing unit tests.

Continue reading

In praise of incremental development

A good traveler has no fixed plans and is not intent on arriving. – Laozi

The other day I was asked about whether an encryption feature in a product I work on supports user-supplied keys. I had to look back in Jira, years into the past, for the answer, which was: nope. It supports default keys, but we hadn’t had time for supporting user-supplied ones.

While this is a little disappointing, it still was the right thing for us to do. That’s the paradox of incremental development, and why I think it’s hard for developers to fully embrace it.

Continue reading