Maintainability and Expect Literals

Recently, Stephen Schaub asked the following on the wewut group:

Several of the unit test examples in the book verify the construction of both HTML and plain text strings. Jay recommends using literal strings in the assertions. However, this strikes me as not a particularly maintainable approach. If the requirements regarding the formatting of these strings changes (a very likely scenario), every single test that verifies one of these strings using a literal must be updated. Combined with the advice that each test should check only one thing, this leads to a large number of extremely brittle tests.

Am I missing something here? I can appreciate the reasons Jay recommends using literals in the tests. However, it seems that we pay a high maintainability price in exchange for the improved readability.

I responded to Stephen; however, I've seen similar questions asked a few times. Below are my extended thoughts regarding literals as expected values.

In general, given the option of having many similar strings (or any literal) vs a helper function, I would always prefer the literal. When a test is failing I only care about that single failing test. If I have to look at the helper function I no longer have the luxury of staying focused on the single test; now I need to consider what the helper function is giving me and what it's giving all other callers. Suddenly the scope of my work has shifted from one test to all of the tests coupled by this helper function. If this helper function wasn't written by me, this expansion in scope wasn't even my decision, it was forced upon me by the helper function creator. In the best case the helper function could return a single, constant string. The scope expansion becomes even worse when the helper function contains code branches.

As for alternatives, my solution would depend on the problem. If the strings were fairly consistent, I would likely simply duplicate everything knowing that any formatting changes can likely be addressed using a bulk edit via find and replace. If the strings were not consistent, I would look at breaking up the methods in a way that would allow me to verify the code branches using as little duplication as possible, e.g. if I wanted to test a string that dynamically changed based on a few variables, I would look to test those variables independently, and then only have a few tests for the formatting.

A concrete example will likely help here. Say I'm writing a trading system and I need to display messages such as

"paid 10 on 15 APPL. $7 Commission. spent: $157"
"paid 1 on 15 VTI. Commission free. spent: $15"
"sold 15 APPL at 20. $7 Commission. collected: $293"
"sold 15 VTI at 2. Commission free. collected: $30"

There's quite a bit of variation in those messages. You could have 1 function that creates the entire string:
confirmMsg(side, size, px, ticker)

However, I think you'd end up with quite a few verbose tests. Given this problem, I would look to break down those strings into smaller, more focused functions, for example:

describeOrder(side, size, px, ticker)
describeCommission(ticker)
describeTotal(side, size, px, ticker)

Now that you've broken down the function, you're free to test the code paths of the more focused functions, and the test for confirmMsg becomes trivial. Something along the lines of

assertEquals("paid 10 on 15 APPL",
  describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("sell 15 APPL at 10",
  describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))

assertEquals("$7 Commission", 
  describeCommission({tickerName:"APPL",commission:"standard"}))
assertEquals("Commission free", 
  describeCommission({tickerName:"APPL",commission:"free"}))

assertEquals("spent: $157", 
  describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("collected: $143", 
  describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("spent: $150", 
  describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"free"}))
assertEquals("collected: $150", 
  describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"free"}))

assertEquals("order. commission. total", 
  confirmMsg("order", "commission", "total"))

I guess I could summarize it by saying, I should be able to easily find and replace my expected literals. If I cannot, then I have an opportunity to further break down a method and write more focused tests on the newly introduced, more granular tests.

Testing: Frameworks are evolving

Test-Driven Development was introduced to me by way of xUnit frameworks.

In the beginning, all xUnit frameworks looked very much the same. All tests lived within a class that inherited from an abstract test case class, and all tests were defined by creating a method that begin with the word test. However, as time progressed the xUnit frameworks began to take advantage of features specific to individual languages.

Martin Fowler's bliki entry Xunit contains a description of how NUnit matured.

The first version of NUnit even had a "isVAforJava" method which originated in special handling for Visual Age for Java. Others have been more sophisticated: NUnit 2.0 was praised by Anders Heljsberg for its use of attributes in C#.

Utilizing language specific features was the first step; however, as testing methods mature it's becoming more clear that the features of xUnit also need to mature.

The first framework I noticed that decided to start breaking rules was RSpec. The 1.0 version of RSpec provided a new syntax for defining test cases and for defining assertions. The following example shows how you would write the same test in Test::Unit (a traditional xUnit framework) and RSpec.

# RSpec 
describe Bowling do
  before(:each) do
    @bowling = Bowling.new
  end

  it "should score 0 for gutter game" do
    20.times { @bowling.hit(0) }
    @bowling.score.should == 0
  end
end

# Test::Unit
class BowlingTest < Test::Unit::TestCase
  def setup
    @bowling = Bowling.new
  end
  
  def testShouldScoreZeroForGutterGame
    20.times { @bowling.hit(0) }
    assert_equal 0, @bowling.score
  end
end

I do believe that the RSpec version is more semantically pleasant. However, given a programmer who practices TDD, when they maintain tests they use an xUnit clone, then I'm not sure the semantic benefit is worth learning a new framework. At least, I wasn't at first.

Another new xUnit framework is xUnit.net, announced recently on James Newkirk's blog. xUnit.net excites me more than any other piece of software that I'm likely to never use. I'll probably never use it since I don't plan on going back to .net development, but the features are exciting nonetheless. James et al have taken their test-driven development experiences and created a framework that guides them towards writing better tests. James' entry gives the full details (which I suggest reading even if you never plan on doing any .net work), but a few features I like are the removal of setup and teardown, and the aspect-like functionality. You may be less impressed since Ruby provides aspect-like functionality very easily, but I believe it's nice to see a xUnit framework built with extension points in mind.

These new frameworks are exciting because they are incorporating lessons learned from the past few years of practicing test-driven development. As our experience with TDD grows, so should our tools.