Testing: The Value of Test Names

Test names are glorified comments.

def test_address_service_recognizes_post_code_WC1H_0HT # Test::Unit
test "address service recognizes post code WC1H 0HT" # Dust
it "should recognize post code WC1H 0HT" # RSpec
should "recognize post code WC1H 0HT" # Shoulda

No matter what form is used a test name is a description, a required comment.

I used to think comments were a really good thing. I always tried to think about why when I was writing a comment instead of how. Having your comments describe why was a best practice.

Then, Martin Fowler and Kent Beck showed me the light.

Don't worry, we aren't saying that people shouldn't write comments. In our olfactory analogy, comments aren't a bad smell; indeed they are a sweet smell. The reason we mention comments here is that comments often are used as a deodorant. It's surprising how often you look at thickly commented code and notice that the comments are there because the code is bad. -- Fowler & Beck, Refactoring

Fast forward a few years...

I used to think test names were a really good thing. I always tried to think about why when I was writing a test name instead of how... I think you see where this is going.

Once I recognized that test names were comments I began to look at them differently. I wondered if all the things I disliked about comments could also be said about test names.

Just like comments, their accuracy seemed to degrade over time. What once explained why is now a distracting bit of history that no longer applies. Sure, better developers should have updated them. Whatever. The problem with test names is that they aren't executable or required. Making a mistake doesn't stop the test suite from running.

While test names can be greatly important, they are never essential to execution. There's no guard to ensure accuracy other than conscience and diligence. One persons essential comment is another persons annoying noise, which complicates things even more. Stale test names make me uneasy, but I wonder if test names have bigger problems. For example, it would be poor form to put the why in the name instead of in the code. [C]omments are there because the code is bad, could the same be said of your test names and their tests?

In fact, the entire section in Refactoring on comments may apply to test names also.

Comments lead us to bad code that has all the rotten whiffs we've discussed in [Chapter 3 of Refactoring]. Our first action is to remove the bad smells by refactoring. When we're finished, we often find that the comments are superfluous.

Comments lead us to bad code...remove the bad smells by refactoring. When we're finished, we often find that the comments are superfluous. With this in mind I began to toy with the idea that I could write tests that made test names unnecessary.

In fact, writing tests without names was fairly easy. I focused very heavily on creating a domain model that allowed me to write tests that required no explanation. Eighty percent of the time I found I could remove a test name by writing better code. Sometimes coming back to a test required a bit of time to see what was going on, but I didn't take this as a failure, instead I took the opportunity convert the code to something more maintainable.

If you need a comment to explain what a block of code does, try Extract Method. If the method is already extracted but you still need a comment to explain what it does, use Rename Method. If you need to state some rules about the required state of the system, use Introduce Assertion.

To be clear, I don't think you should be extracting or renaming methods in your tests, I think test names can indicate that those things are needed in the code you are testing. Conversely, Introduce Assertion could apply directly to your test. Instead of stating your assumptions in the test name, introduce an assertion that verifies an assumption. Or better yet, break the larger test up into several smaller tests.

A good time to use a comment is when you don't know what to do. In addition to describing what is going on, comments can indicate areas in which you aren't sure. A comment is a good place to say why you did something. This kind of information helps future modifiers, especially forgetful ones.

I agree with Fowler and Beck, comments are valuable when used correctly. Wrapping legacy code with a few tests may result in tests where you know the results, but you don't quite understand how they are generated. Also, higher level tests often benefit from a brief description of why the test exists. These are situations where comments are truly valuable. However, I found I could handle those situations with a comment, they didn't require a test name.

I started sharing the idea of making test names superfluous with friends and got very mixed reviews.

John Hume asks for a test suite with examples of how I put this idea into practice. The first draft of this entry didn't include examples because I think the idea isn't tied to a particular framework, and I didn't want to give the impression that the framework was the important bit. However, I think John is right, the entry is better if I point to examples for those interested in putting the idea into practice.

The expectations.rubyforge.org testing framework is designed with anonymous tests as a core concept. I have used expectations on my last 3 client projects and all the open source projects I still work on. You can browse the following tests for plenty of examples.

• Expectations tests: http://expectations.rubyforge.org/svn/trunk/test/
• Validatable unit tests: http://validatable.rubyforge.org/svn/trunk/test/unit/
• Arbs tests: http://arbs.rubyforge.org/svn/trunk/test/

It's probably worth mentioning that RSpec also supports testing without a description.

John also likes code folding at the test name level, which is obviously not possible without test names. Truthfully, I don't have an answer for this. If you use code folding, you'll need to take that into account.

Pat Farley asks: So when do method names get the ax?
Method names are actually an interesting comparison. The reality is that methods are already available nameless. Blocks, procs, lambdas, closures, anonymous functions, or whatever you prefer to call them are already essentially methods (or functions) that have no name.

Ola Bini probably had the best response when it came to method names.

A method selector is always necessary to refer to the method from other places. Not necessarily a name, but some kind of selector is needed. That's not true for test names, and to a degree it's interesting that the default of having test names comes from the implementation artifact of tests usually being represented as methods. Would we still have test names in all cases if we didn't start designing a test framework based around a language that cripples usage of abstractions other than methods for executable code?

Prasanna Pendse pointed out that if you use test names for anything you'll lose that ability. The only usage we could think of was creating documentation from the names (RSpec specdoc).

Dan North and I stumbled onto the idea that test names may be more or less valuable based on your workflow. I find that I never care near as much about the test names as I do about individual tests. Dan seems to use test names to get a high level understanding of how a class works. I'll probably explore this idea in a later post after I do a bit more to mature the thoughts. For now it's sufficient to say, if you rely heavily on test names for understanding, you might not be at a point where dropping them makes sense.

Mike Mason likes test names because they can act as a boundary for what the test should do. The basic rules I always followed were that a test can never contain a "and" or an "or". I think Mike is correct that there is a benefit to requiring people to stick to tests that can be explained in a short sentence. However, I like to think the same thing can be achieved with other self imposed constraints such as limiting each test to One Assertion per Test or One Expectation per Test.

Of course, now I'm suggesting one assertion per test and no test names, so if you aren't ready to adopt both, it might not be time to adopt nameless tests.

Ben Butler-Cole provided some anonymous tests written in Haskell that were quite readable. He made the observation that [T]he terser your language the more likely you will be able to make your tests sufficiently comprehensible. I actually find this to be a key in advocating anonymous tests. The test must be easily readable and maintainable, otherwise the effort necessary to work with a test would warrant a description.

A common thought (that I agree with) is that using anonymous tests is something you'll want to try only if you have a talented team. If you're new to testing or have a fair amount of junior programmers on your team, you should probably stick to conventional thinking. However, if you have a team of mostly masters and journeyman, you may find that anonymous tests result in better tests and a better domain model, which is clearly a good thing.

Thanks to Ben Butler-Cole, Chris Stevenson, Dan North, John Hume, Mike Mason, Ola Bini, Prasanna Pendse, Ricky Lui and anyone else who provided feedback on the rough draft.