Don’t build your tests when you run your tests

You’ve got a Jenkins job set up to run your tests automatically, like you should! Good job. 5 points to Ravenclaw.

git clone
cd awesome-test-framework
mvn -DskipTests install
java awesome-test-framework/awesome-test.jar -d where-my-tests-at run

Wait, what have you done. I’m very disappointed. 10 points from Ravenclaw.

The purpose of your test is to … run your test, not build the tools needed to run the test. You can do that anytime. When you build testing tools as part of the test:

  • you waste time – the build of the test tool will come out the same every time
  • you add an extra source of failure – if building the test tool fails, the test fails, but not because of anything wrong in what you want to test
  • you make the test more difficult to run – because the test run also needs the requirements for building the test tool
  • you make the test longer
  • you reduce test stability – if you build from a branch subject to change, like, say, master
  • kittens die

What should you do instead? You should treat the tool like the project it is, and that means creating builds for it, tagging those builds as releases, and publishing them for download. Then, tests can simply install the tool and use it. This runs quicker, has less prerequisities, and is more stable.

curl -O http://repo/awesome-test-1.2.3.jar # BOOM
java awesome-test-1.2.3.jar -d where-my-tests-at run

The obvious counter to this is that it’s an awful lot of work to devote to a test tool that isn’t going to be released to anybody; by always building from source, you always get the latest version of the tool! I’m sorry, but this doesn’t hold up under scrutiny.

  • If you build from source every time, how do you know what the tool is doing? If someone inadvertently breaks the tool, your test will fail on you and it won’t be your fault. If someone changes how the tool is used, your test will fail on you and it won’t be your fault.
  • By now, we all now refactoring is a vital part of software development. If every test is building a tool, that work must be refactored into one build pass, which means it should just be built ahead of any test. DRY.
  • You work in an organization with lots of other people, and when they use the tool, they are the customers of those who wrote the tool. Don’t those customers, your colleagues, deserve the same respect and devotion of effort that you’d give to others? Wouldn’t you love it if someone gave you a one-line installation of a cool tool?
  • Calculate the time that it takes to build a tool. Say it’s one minute. If you run a test hourly that uses the tool but builds it itself, you waste over 15 days of compute time per year that you could use for actually running tests.
  • Test tools need to be rock solid, or else you can’t be sure that your test results are valid. Did everything pass legitimately or is the tool busted and passes when it shouldn’t. You need to work from known good states of the tool, and that’s what releases are for.

Don’t give in to laziness. Set up binary releases for test tools and install them during tests like Maven, Git, or any other utility you need to run a test.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s