In praise of incremental development

A good traveler has no fixed plans and is not intent on arriving. – Laozi

The other day I was asked about whether an encryption feature in a product I work on supports user-supplied keys. I had to look back in Jira, years into the past, for the answer, which was: nope. It supports default keys, but we hadn’t had time for supporting user-supplied ones.

While this is a little disappointing, it still was the right thing for us to do. That’s the paradox of incremental development, and why I think it’s hard for developers to fully embrace it.

In agile development, the reason that you have sprints is not to dice up bigger tasks into bite-size pieces simply for the sake of doing so. It’s because, in order to be “agile”, you need to be able to quickly switch gears, and that’s hard when in the midst of a task that takes, say, a few months. You don’t want to have to wait until that task is fully completed before you can do something else.

So, agile methodologies have you break up those bigger tasks into self-contained, smaller work items that can be completed individually. The end state of each of those items is supposed to be shippable; that is, you could still release the product and it’ll work fine. This lets you freely jump to whatever the latest top priority is.

That’s what happened in this case. Basically, there were two pieces: first, basic encryption with default keys; second, user-supplied key support. We bailed on the effort after the first piece, because something else mattered more. And, it turned out, we never needed to get back to it, for years. (Maybe we still don’t, because there are workarounds!) Therefore, we avoided spending time on a feature we didn’t really need as much. Win.

This is difficult for some developers to accept, though. There’s this strong urge to complete tasks, even the big ones, in one fell swoop. After all, if half of it is done, we can’t just leave the other half missing, right? Where’s your sense of craftsmanship? Pride in your work?

I have a favorite quote (NFSW language) from Pulp Fiction about pride. In the end, it’s what customers want that you need to build. That’s what you should take pride in. There’s a line between what’s required by, you know, the people who pay you (your own company, and indirectly your customers) and what you feel needs to be there for a finished job.

Sometimes it all lines up. Often it doesn’t, and that’s when you negotiate, bringing your expertise and in-depth knowledge to the planning table. When that happens, you’ll want the flexibility that incremental development, which is what I’m advocating for here, gives you.

Suppose that the proposal is to stop work on A in order to work on B, and A is partway done. Your range of options are threefold:

  • fully stop A and start B
  • split time between A and B
  • stay on A and put off B

If you’ve been working on A incrementally, then all of those options can be pursued. If you haven’t, then you’ll be stuck in a situation where you have to argue for only the third option, putting off B, because A is this big, uninterruptible monster.

This irritates managers, because you’re intransigent. They don’t care about your arguments about how it’s not practical to just flip over to B right now, that stuff is A is broken and half-finished, so you can’t just drop it. (Why did you let it get stuck like this, they’ll ask. Who’s in charge here, anyway, they’ll wonder.)

Meanwhile, you’re annoyed at them because once again, they’ve changed their minds on what they need, and it’s messing up your progress. Well, about that.

First, they’re always going to change their minds. You just need to get over that. The important needs for a software project are often simply unpredictable, and managers have perspectives outside your own that informs them on their positions.

More importantly, though, you’ll find it’s much less annoying if you’re in a position where you can oblige them without a lot of pain. It might not be your preference, sure, but at least you won’t feel irresponsible dropping work on A, because what is already there is solid and tested and working. That all makes you a lot nicer to deal with.

So far I’ve been arguing for incremental development as a way to improve the quality of your work life. However, you can argue, correctly, that it is less efficient. First off, you’ll have to switch contexts – your whole team might need to – and that slows you down and takes mental effort. Also, there’s a continual overhead in making each bite-sized piece of a large task self-contained. You can’t leave dangling ends of interfaces, for example, to simply pick up tomorrow; known bugs can’t linger. The extra effort to neatly wrap up each little task takes time, more than doing the same work for several tasks at once. Therefore, you can’t deliver the final end product as quickly. Surely management won’t like that, right?

overhead

There’s more overhead with incremental development (top row) over the alternative. The last block on the bottom row represents the lost time / inefficiency.

As ever in software development, it’s a trade-off. You choose to spend extra effort in practicing incremental development in order to buy agility. You need to say to yourself, “I will intentionally work slower in order to deliver features in smaller, working pieces instead of all at once … because there are benefits.” Once you use those benefits a couple of times to satisfy your stakeholders sooner, you may find it’s worth the price.

This shouldn’t be a hidden decision, if you can help it. Let all stakeholders understand why you are working incrementally. I myself am very open with the process my team follows. This gives my management a more complete view of what they’re managing – which really it’s fair for them to have, I think – and also lets them manage. They might tell me that they actually like this slightly inefficient way of working, because it helps them do their job.

All of this implies a certain maturity in the development organization, by the way. If meta-thinking about the business of creating software isn’t accessible to management, you’ll get wildly inconsistent prioritization and no appreciation for the actual tradecraft you’re trying to apply. It won’t matter, then, what your process is. See CMM Level 0. Dealing with immature organizations is a whole ‘nother post.

But, hopefully you’re not in that situation. So, I encourage you to set aside your fears and embrace incremental development. Beyond all the other benefits I’ve outlined, there’s one more I prize: that feeling of satisfaction when you finish something. Even if it’s a small something, it counts as a victory. When you pile up victories, delivering newer and better software, well, it means you’re doing a good job. You can even be proud of that.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s