Chunking through unit tests

Ever since the advent of multi-tier application architectures (which seems forever ago now), I’ve commonly had to work on tasks that span multiple levels of abstraction. You probably have too. For example, say you need to implement a new REST API call in your classic web application. You have to:

  • Define the endpoint itself, with its request and response formats.
  • Implement some business logic for it.
  • Update the data layer to persist and/or query what needs it.

That’s just your three classic tiers, and sure, once you’ve done this sort of work enough, it’s almost straightforward. But, as time goes on and your application gets complicated, more aspects creep in. Security. Audit logging. Retries and resiliency. And then there’s all those newfangled microservices to deal with. The neat layer cake ends up looking more like monkey bread. How can you keep track of all of this and get your work done?

One way is to use something you should be doing anyway: writing unit tests.

I generally avoid analogies when talking about computery things, because they inevitably fall apart. (Especially for cars, gawd.) But here, I’ll take a risk and liken building a complex application to constructing a brick building, in one specific aspect: If the bricks that you are using are of low quality, then the building will also be low quality, no matter what else you do. By analogy, then, a complex software product will have low quality unless its individual pieces – its bricks – have high quality. This is pretty much why unit tests exist, anyway – to check quality at a very basic level.

When working with bricks, a bricklayer doesn’t think about how the bricks themselves will hold up. The assumption is that the bricks are fine, and the bricklayer can concentrate on the higher order task of constructing, say, a wall with them. If a bricklayer had to verify their work by not only testing their wall as a whole, but also by testing each brick, it would get pretty difficult. They’d probably wonder why they aren’t using a different, continuous material like adobe, because they aren’t taking advantage of the primary benefit of bricks – individual, proven bits of structure.

In the world of software, our bricks could be libraries, or classes, or even functions – somewhat discrete hunks of logic (that also happen to be reusable, unlike bricks – see, the analogy already starts to break down!). To make life tolerable, you want to know that those hunks are of high quality. For libraries, you generally rely on their reputation: who built them, how commonly used they are, their development history, and so on. For your own stuff, you need to do the quality checks yourself, and again, that’s unit tests.

Once you have done that, you get a benefit beyond just knowing “this component has good quality”. You also are free to no longer think about its innards. It has become a “unit” that you know works as advertised, so when you go on to use it, you don’t worry about whether they’ll hold up. There is less for you to keep track of, mentally.

In psychology, this is called chunking. The idea is to group individual pieces of information into larger, coherent collections. Sayeth Wikipedia: “It is believed that individuals create higher order cognitive representations of the items within the chunk.” You can use these technique to more easily remember things like phone numbers, but what I care more about is that “higher order” part. If you can take a complex system and chunk it out, you gain the ability to think in a more advanced state about it. And as we know, when creating software, you need all the advanced thinking you can get.

The bricklayer wields each brick as a chunk to create the higher order structure we call a wall. You, dear software developer, can wield each software “unit” as a chunk to create the higher order structure we call an application. Let’s revisit my three-tier example, by reversing the order and adding unit tests.

  1. Update the data layer to persist and/or query what needs it [some data]. Create unit tests to ensure the data layer works.
  2. Implement some business logic for it [that data]. Use the data layer as it’s designed. Create unit tests to ensure the business logic works, mocking the data layer because it’s known to work correctly.
  3. Define the endpoint itself, with its request and response formats. Use the business logic as it’s designed. Create unit tests to ensure the endpoint works, mocking the business logic because it’s known to work correctly.

The brick analogy is stretched thinly at this point. For example, maybe the endpoint will hit the data layer directly, or maybe work on the business logic exposes problems in the data layer that need to be fixed. Still, hopefully you get the idea.

The important benefit, though, is the reduced complexity – at least on the happier paths – of steps 2 and 3. When creating the business logic, for example, the data layer is just there as a unit, with some defined usage patterns but otherwise without any further complexity to add. You can concentrate on the immediate work more easily. That’s the higher order thinking.

The feature of software that enables this to work, as you may have guessed by now, is encapsulation. A well-designed, encapsulated software unit hides its internals and has a carefully tailored interface. Among other benefits, this makes it easier for users of the unit to create a chunk of the unit, which may as well be a good definition for “ease of use”.

The authors of the book Individual and Social Influences on Professional Learning: Supporting the Acquisition and Maintenance of Expertise, which I totally didn’t find by just googling for “chunking memory encapsulation”, talk about this.

Encapsulation is closely related to chunking processes that have been identified many years ago as crucial components of expertise. Experts reorganise their knowledge according to domain-specific principles so that larger meaningful units emerge. When encountering new domain-specific problems, experts, thus, are able to perceive larger information units, and they extract relevant information quicker than novices.

Sound good? Ready for some higher-order, expert-level thinking? Well, you know what to do, which is what you should have been doing all along: write those unit tests!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s