Testing: Duplicate Code in Your Tests

Luke Kanies recently left a comment on Using Stubs to Capture Test Essence:

Now if you would only see the wisdom of applying DRY to your whole code base, rather than just the functional bit. :)

Sadly, DRY has become a philosophy that is dogmatically and blindly applied to every aspect of programming. I'm a big fan of firing silver bullets but if you go that path your number one priority needs to be finding where a solution does not fit. In fact, those who have applied DRY excessively have found that there are situations when DRY may not be advantageous.

Three of the points from the Wikipedia entry relate to test setup code.

In some small-scale contexts, the effort required to design around DRY may be far greater than the effort to maintain two separate copies of the data.
There's no question that designing around DRY for tests adds additional effort. First of all, there's no reason that tests need to run in the context of an object; however, the most common way to use a setup is to create instance variables and reuse those instance variables within a test. The necessity of instance variables creates the requirement that a test must be run within a new object instance for each test.

Ensuring that a setup method only creates the objects that are necessary for each test adds complexity. If all tests don't require the same data, but you create a setup that creates everything for every test, digesting that code is painful and unnecessary. Conversely, if you strictly adhere to only using setup for objects used by all methods you quickly limit the number of tests you can put within one test case. You can break the test case into multiple test cases with different setup methods, but that adds complexity. Consider the situation where you need the foo instance variable for test 1 and 2 and the bar instance variable for test 2 and 3. There's no straightforward way to create foo and bar in a DRY way using a setup method.

Of course, there's also readability impacts to consider. When a test breaks that relies on setup and teardown I have to look in 3 places to get the full picture of what's going on. When a test is broken I want to fix it as quickly as possible, so I prefer looking in one place.

The crux of the issue is whether a test case is one context or the individual tests are small-scale multiple contexts. While a test case is conceptually one object, we rarely concern ourselves with a test cases as a whole. Instead testing is very much about many individual executable pieces of code. In fact, it's a testing anti-pattern to have any tests depend on other tests. We write tests in isolation, fix tests in isolation and often run tests in isolation; therefore, I consider them to be many small independent entities that do completely isolated jobs. The result of those jobs is aggregated in the end, but ultimately the different entities never (or should never) interact. Each test is and runs in it's own context.

Imposing standards aimed at strict adherence to DRY could stifle community involvement in contexts where it is highly valued, such as a wiki.
Tests are often written by one, but maintained by many. Any barrier to readable, reliable, and performant tests should be removed as quickly as possible. As I've previously discussed, abstracting behavior to 3 different places reduces readability, which can reduce collaboration. Additionally, setup methods can cause unexpected behavior for tests that are written by an author other than the creator of the setup method. This results in less reliable tests, another obvious negative.

Human-readable documentation (from code comments to printed manuals) are typically a restatement of something in the code with elaboration and explanation for those who do not have the ability or time to read and internalize the code...While tests aren't code comments or printed manuals they are very much (or should be) Human-readable documentation. They are also restatements of something in the code with elaboration and explanation. In fact the setup and teardown logic is usually nothing more than elaboration and explanation. The idea of setup and teardown is to put the system into a known state before executing an assertion. While you should have the ability to understand setup and teardown, you don't have the time. Sure, you can make the time, but you shouldn't need to.

The problem is that it's much easier to understand how to apply DRY than it is to understand why.

If your goal is to create a readable, reliable, and performant test suite (which it should be) then there are better ways to achieve that goal than by blindly applying DRY.

Where there's smoke, pour gasoline --Scott Conley

Duplicate code is a smell. Setup and teardown are deodorant, but they don't fix the underlying issue. Using a setup method is basically like hiding your junk in the closet. There are better ways.

Pull the data and the behavior out of the setup and teardown methods and move it to the tests, then take a hard look.

Unimportant behavior specification
Setup methods often mock various interactions; however, the ultimate goal of the interactions is a single result. For example, you may have designed a Gateway class that takes a message, builds a request, sends the request and parses the results of the request. A poorly designed Gateway class will require you to stub the request building and sending the request. However, a well designed Gateway class will allow you to stub two methods at most and verify the behavior of the parse method in isolation. The poorly designed Gateway class may require a setup method, the well designed Gateway class will have a simple test that stubs a method or two in a straightforward manner and then focuses on the essence of the behavior under test. If your domain model requires you to specify behavior which is has nothing to do with what you are testing, don't specify that behavior in a setup method, remove the need to specify the behavior.

Loosely couple objects
I've always disliked the ObjectMother pattern. ObjectMother is designed to build complex hierarchies of objects. If you need an ObjectMother to easily test, you probably actually need to take another look at your domain model. Loosely coupled objects can be created without complicated graphs of dependencies. Loosely coupled objects don't require setting up multiple collaborators. Fewer collaborators (or dependencies) result in tests that don't rely on setup methods to create various collaborators.

The idea is to write tests that only specify what's necessary and verify one thing at a time. If you find that you need to specify things that are unrelated to what you are trying to verify, change the domain model so that the unnecessary objects are no longer required.

Solve problems globally or locally, but not between
Sometimes there are things that need to happen before and after every test. For example, running each test in a transaction and rolling back after the test is run is a good thing. You could add this behavior at a test case level, but it's obviously beneficial to apply this behavior to every test in the entire test suite.

There are other cases where it can make sense to solve issues on a global level. For example, I once worked on a codebase that relied heavily on localization. Some tests needed to verify strings that were specified in the localization files. For those tests we originally specified what localization file to use (in each test), but as we used this feature more often we decided that it made sense to set the localization file for the entire test suite.

Putting everything within a test is the best solution for increasing understanding. However, applying behavior to an entire test suite also aides in comprehension. I would expect any team member to understand that reference data is loaded once before the test suite is run, each test is run in a transaction, and localization uses the test localization file.

Understanding the behavior of an entire test suite is valuable and understanding the behavior of a test is valuable, but understanding a test case provides almost no benefit. Rarely do you run a single test case, rarely does an entire test case break, and rarely do you edit an entire test case. Since your primary concerns are the test suite and the tests themselves, I prefer to work on those levels.

I wouldn't go too far with the global behavior approach, and I definitely wouldn't start moving conditional logic into global behavior, but applied judiciously this can be a very valuable technique.

Bringing it all together
The core of the idea is that it should be easy to understand a test. Specifying everything within a test makes understanding it easy; however, that solution isn't always feasible. Setup and teardown are not the solution to this problem. Luckily the techniques above can be used to make specifying everything within a test a realistic and valuable solution.