Tuesday, May 27, 2008

Testing: The Value of Test Names

Test names are glorified comments.

def test_address_service_recognizes_post_code_WC1H_0HT # Test::Unit
test "address service recognizes post code WC1H 0HT" # Dust
it "should recognize post code WC1H 0HT" # RSpec
should "recognize post code WC1H 0HT" # Shoulda

No matter what form is used a test name is a description, a required comment.

I used to think comments were a really good thing. I always tried to think about why when I was writing a comment instead of how. Having your comments describe why was a best practice.

Then, Martin Fowler and Kent Beck showed me the light.
Don't worry, we aren't saying that people shouldn't write comments. In our olfactory analogy, comments aren't a bad smell; indeed they are a sweet smell. The reason we mention comments here is that comments often are used as a deodorant. It's surprising how often you look at thickly commented code and notice that the comments are there because the code is bad. -- Fowler & Beck, Refactoring
Fast forward a few years...

I used to think test names were a really good thing. I always tried to think about why when I was writing a test name instead of how... I think you see where this is going.

Once I recognized that test names were comments I began to look at them differently. I wondered if all the things I disliked about comments could also be said about test names.

Just like comments, their accuracy seemed to degrade over time. What once explained why is now a distracting bit of history that no longer applies. Sure, better developers should have updated them. Whatever. The problem with test names is that they aren't executable or required. Making a mistake doesn't stop the test suite from running.

While test names can be greatly important, they are never essential to execution. There's no guard to ensure accuracy other than conscience and diligence. One persons essential comment is another persons annoying noise, which complicates things even more. Stale test names make me uneasy, but I wonder if test names have bigger problems. For example, it would be poor form to put the why in the name instead of in the code. [C]omments are there because the code is bad, could the same be said of your test names and their tests?

In fact, the entire section in Refactoring on comments may apply to test names also.
Comments lead us to bad code that has all the rotten whiffs we've discussed in [Chapter 3 of Refactoring]. Our first action is to remove the bad smells by refactoring. When we're finished, we often find that the comments are superfluous.
Comments lead us to bad code...remove the bad smells by refactoring. When we're finished, we often find that the comments are superfluous. With this in mind I began to toy with the idea that I could write tests that made test names unnecessary.

In fact, writing tests without names was fairly easy. I focused very heavily on creating a domain model that allowed me to write tests that required no explanation. Eighty percent of the time I found I could remove a test name by writing better code. Sometimes coming back to a test required a bit of time to see what was going on, but I didn't take this as a failure, instead I took the opportunity convert the code to something more maintainable.
If you need a comment to explain what a block of code does, try Extract Method. If the method is already extracted but you still need a comment to explain what it does, use Rename Method. If you need to state some rules about the required state of the system, use Introduce Assertion.
To be clear, I don't think you should be extracting or renaming methods in your tests, I think test names can indicate that those things are needed in the code you are testing. Conversely, Introduce Assertion could apply directly to your test. Instead of stating your assumptions in the test name, introduce an assertion that verifies an assumption. Or better yet, break the larger test up into several smaller tests.
A good time to use a comment is when you don't know what to do. In addition to describing what is going on, comments can indicate areas in which you aren't sure. A comment is a good place to say why you did something. This kind of information helps future modifiers, especially forgetful ones.
I agree with Fowler and Beck, comments are valuable when used correctly. Wrapping legacy code with a few tests may result in tests where you know the results, but you don't quite understand how they are generated. Also, higher level tests often benefit from a brief description of why the test exists. These are situations where comments are truly valuable. However, I found I could handle those situations with a comment, they didn't require a test name.

I started sharing the idea of making test names superfluous with friends and got very mixed reviews.

John Hume asks for a test suite with examples of how I put this idea into practice. The first draft of this entry didn't include examples because I think the idea isn't tied to a particular framework, and I didn't want to give the impression that the framework was the important bit. However, I think John is right, the entry is better if I point to examples for those interested in putting the idea into practice.

The expectations.rubyforge.org testing framework is designed with anonymous tests as a core concept. I have used expectations on my last 3 client projects and all the open source projects I still work on. You can browse the following tests for plenty of examples.It's probably worth mentioning that RSpec also supports testing without a description.

John also likes code folding at the test name level, which is obviously not possible without test names. Truthfully, I don't have an answer for this. If you use code folding, you'll need to take that into account.

Pat Farley asks: So when do method names get the ax?
Method names are actually an interesting comparison. The reality is that methods are already available nameless. Blocks, procs, lambdas, closures, anonymous functions, or whatever you prefer to call them are already essentially methods (or functions) that have no name.

Ola Bini probably had the best response when it came to method names.
A method selector is always necessary to refer to the method from other places. Not necessarily a name, but some kind of selector is needed. That's not true for test names, and to a degree it's interesting that the default of having test names comes from the implementation artifact of tests usually being represented as methods. Would we still have test names in all cases if we didn't start designing a test framework based around a language that cripples usage of abstractions other than methods for executable code?
Prasanna Pendse pointed out that if you use test names for anything you'll lose that ability. The only usage we could think of was creating documentation from the names (RSpec specdoc).

Dan North and I stumbled onto the idea that test names may be more or less valuable based on your workflow. I find that I never care near as much about the test names as I do about individual tests. Dan seems to use test names to get a high level understanding of how a class works. I'll probably explore this idea in a later post after I do a bit more to mature the thoughts. For now it's sufficient to say, if you rely heavily on test names for understanding, you might not be at a point where dropping them makes sense.

Mike Mason likes test names because they can act as a boundary for what the test should do. The basic rules I always followed were that a test can never contain a "and" or an "or". I think Mike is correct that there is a benefit to requiring people to stick to tests that can be explained in a short sentence. However, I like to think the same thing can be achieved with other self imposed constraints such as limiting each test to One Assertion per Test or One Expectation per Test.

Of course, now I'm suggesting one assertion per test and no test names, so if you aren't ready to adopt both, it might not be time to adopt nameless tests.

Ben Butler-Cole provided some anonymous tests written in Haskell that were quite readable. He made the observation that [T]he terser your language the more likely you will be able to make your tests sufficiently comprehensible. I actually find this to be a key in advocating anonymous tests. The test must be easily readable and maintainable, otherwise the effort necessary to work with a test would warrant a description.

A common thought (that I agree with) is that using anonymous tests is something you'll want to try only if you have a talented team. If you're new to testing or have a fair amount of junior programmers on your team, you should probably stick to conventional thinking. However, if you have a team of mostly masters and journeyman, you may find that anonymous tests result in better tests and a better domain model, which is clearly a good thing.

Thanks to Ben Butler-Cole, Chris Stevenson, Dan North, John Hume, Mike Mason, Ola Bini, Prasanna Pendse, Ricky Lui and anyone else who provided feedback on the rough draft.


  1. I disagree.

    I work with Q, a programming language descended from APL. People often suggest that comments are unnecessary for Q because of the terseness of the code, but quite often, you can walk through the body of a function with a name like z and have no clue why the hell it exists or what inspired its design choices.

    You say that comments are unnecessary as a matter of refactoring? I say you'll never get to the step of refactoring if those comments aren't there in the first place, because you'll spend all your time trying to figure out what was in the head of the stupid developer that came before you. One way or another, you're going to need a diligent developer to do his job properly and either update the comments, or update the code artifacts to have as much semantic meaning as possible.

    I have seen far too many comments about how something works when I can clearly see that, right there in the damn code. No, I still need the why.

    As for removing test names from specification tests? Yes, terrific idea, lets take away the big feature that gives business value to spec testing: documenting the guarantees of the program.

    Let me make something clear though, I understand that there is a difference between piles of comments and a few accurately targeted descriptions of what's going on. I spend a lot more time trying to write good variable and function names (we don't have classes) than I do writing comments. But I think that the suggestions that comment existence is in and of itself an indicator of code smell and the corollary that test names are unnecessary are flawed.

  2. Anonymous6:29 AM

    Hi Dan, thanks for your thoughts.

    "you can walk through the body of a function with a name like z"

    I don't think that's a argument for comments, it's an argument for firing developers that think "z" is an acceptable name for a method.

    I also think comments are valuable when used correctly. Step 1 for me is to make the code as clear as possible. Step 2 is to create a comment explaining any why's that the code does not convey. Sometimes I don't get to step 2, but sometimes I do, which is why I left the bit about valuable comments in.

    "lets take away the big feature that gives business value to spec testing: documenting the guarantees of the program."

    Why do I need to document the guarantees of the system? Is this a business requirement? Do they read the documents? I've never been in that situation, but if you are then you'll probably not be happy with removing descriptions. I mentioned that you'd need to consider this when I said you'd lose the ability to use specdoc. I'm not really a believer in letting the business read descriptions that could be incorrect. I also believe there are other ways to ensure the validity of the system that require less effort from the development team.

    I believe Fowler and Beck's assertion that comments generally mark bad code is true. Why else do you need the comment? In some way the code cannot convey enough information so something more (comments) is needed. Sometimes it's unavoidable, but that doesn't remove the fact that comments are still taking the place of what we'd all prefer: code that conveyed intention on it's own.

  3. Anonymous8:33 AM

    The point I agree most about is extracting methods out of comments.

    I'm now actually getting accustomed to writing comments with the intent of naming a method after them on later refactorings. So instead of writing a comment like

    // Assert that inventory quantity in shelf S3 is more than 5

    I write

    // bool CheckInventory(Shelf X, int requestedQuantity)

  4. "I don't think that's a argument for comments, it's an argument for firing developers that think "z" is an acceptable name for a method."

    It's a culturally ingrained thing in the APL programming communities. They'd get rid of ASCII characters if they had a chance and every function name would be a unicode character :(

    http://www.nsl.com/papers/style.pdf (Jump to page 8)

    "I mentioned that you'd need to consider this when I said you'd lose the ability to use specdoc"

    Sorry, I must've glossed over that earlier. I read this post right after I got up.

    "I'm not really a believer in letting the business read descriptions that could be incorrect. I also believe there are other ways to ensure the validity of the system that require less effort from the development team."

    If the descriptions are incorrect, then you're implying that you have a developer that didn't go to the effort of updating the test descriptions or fixing the test in the first place (that is the test no longer validates the guarantee made or is now superfluous or otherwise incorrect).

    I just don't think that Intentional code is ever going to be a perfect reality, if only because you'd have to have hard AI to be able to validate intent. Perhaps I came out too strong in my wording earlier (again, I wrote my comment first thing after I'd woken up), because I mostly agree with the general point that code should reflect intent as much as possible. I just think that, especially for something like a test, where you're already double checking your work, it doesn't hurt to double check your meaning (seriously, I can type at like 100wpm. How many words go into a good test description? 8 at the max?)

  5. Anonymous11:05 AM


    I said "I also believe there are other ways to ensure the validity of the system..."

    I should have stated what I think is a better way.

    RSpec's story runner is an example of how I think customers are better served by tests. In that case I don't need to show them a description, hopefully they can write the tests themselves, or I can write tests in their domain language.

    That way the tests written by me for me can be as terse as necessary, and tests written from an acceptance perspective can be written with a domain expert in mind.

    Thanks again for the comments.

  6. Anonymous3:26 AM

    Not bad idea indeed! I'll try go with your way and see how it works with rspec.

    I looked your examples and they are readable and understandable. I also feel quite often spec descriptions are too noisy for me and prefer to read spec code instead of descriptions.

  7. Anonymous12:03 PM

    Hi Jay,

    My name is Wei-Ling Chen, I'd like to talk to you about our MVB program. Could you shoot me a email at weiling@dzone.com please.

    Thanks much, look forward to read more on your blog :).

    Wei-Ling Chen

  8. Anonymous6:02 PM

    I quite regularly use unnamed specs in rspec. I have a few simple rules for when a spec can (and should) have no explicit description:

    Only one expectation/assertion is allowed. The code that does it reads very very easily. And the rspec expectation must generate a readable and correct specdoc.

    And when I leave out the description, I always use specify rather than it, to create the example. I find that it reads very poorly.

    For example:

    specify { @it.should reject("").for(:name).with("cannot be blank) }

    specify { @it.should accept("John Doe").for(:name) }

    specify { @it.should put_the_lotion_in_the_bucket }

    specify { @it.should put_the_lotion_in_the_bucket }

    and so on. (These examples don't match with your testing style of never using setup, but it could certainly be adapted to match that as well.)

    The ability for rspec matchers to dynamically generate specdoc is one of my favorite aspects of rspec. :-)

  9. Good answer, I am looking for the solution of the same question. Find the movies or mp3 you are looking for at your-download.org the most comprehensive source for free-to-try files downloads on the Web


Note: Only a member of this blog may post a comment.