Friday, September 21, 2007

Testing: Frameworks are evolving

Test-Driven Development was introduced to me by way of xUnit frameworks.

In the beginning, all xUnit frameworks looked very much the same. All tests lived within a class that inherited from an abstract test case class, and all tests were defined by creating a method that begin with the word test. However, as time progressed the xUnit frameworks began to take advantage of features specific to individual languages.

Martin Fowler's bliki entry Xunit contains a description of how NUnit matured.
The first version of NUnit even had a "isVAforJava" method which originated in special handling for Visual Age for Java. Others have been more sophisticated: NUnit 2.0 was praised by Anders Heljsberg for its use of attributes in C#.
Utilizing language specific features was the first step; however, as testing methods mature it's becoming more clear that the features of xUnit also need to mature.

The first framework I noticed that decided to start breaking rules was RSpec. The 1.0 version of RSpec provided a new syntax for defining test cases and for defining assertions. The following example shows how you would write the same test in Test::Unit (a traditional xUnit framework) and RSpec.

# RSpec 
describe Bowling do
before(:each) do
@bowling =

it "should score 0 for gutter game" do
20.times { @bowling.hit(0) }
@bowling.score.should == 0

# Test::Unit
class BowlingTest < Test::Unit::TestCase
def setup
@bowling =

def testShouldScoreZeroForGutterGame
20.times { @bowling.hit(0) }
assert_equal 0, @bowling.score

I do believe that the RSpec version is more semantically pleasant. However, given a programmer who practices TDD, when they maintain tests they use an xUnit clone, then I'm not sure the semantic benefit is worth learning a new framework. At least, I wasn't at first.

Another new xUnit framework is, announced recently on James Newkirk's blog. excites me more than any other piece of software that I'm likely to never use. I'll probably never use it since I don't plan on going back to .net development, but the features are exciting nonetheless. James et al have taken their test-driven development experiences and created a framework that guides them towards writing better tests. James' entry gives the full details (which I suggest reading even if you never plan on doing any .net work), but a few features I like are the removal of setup and teardown, and the aspect-like functionality. You may be less impressed since Ruby provides aspect-like functionality very easily, but I believe it's nice to see a xUnit framework built with extension points in mind.

These new frameworks are exciting because they are incorporating lessons learned from the past few years of practicing test-driven development. As our experience with TDD grows, so should our tools.


  1. Anonymous2:28 PM

    It seems to me that of all these frameworks, TestNG is still the one that's innovated the most compared to all the xUnit-based ones. Interestingly, xUnit frameworks in the .Net world seems to be more inclined to follow TestNG than the one in the Java world...

  2. One of the things I miss from Perl when I'm using other languages is its really rather excellent testing framework.

    It as a common testing protocol (TAP - Test Anything Protocol) that you can use to communicate test success/failure to the test harness coupled with a _whole_ bunch of test modules that allow you to pick the best strategy for the task at hand.

    So I can have a test suite that has some tests in the xUnit style (using Test::Class), some data driven tests (Test::Data), a nice DLS for web testing (Test::WWW::Declare), etc. all playing nicely together. Dead handy.

    The OneBestTool approach that seems so common in other languages seems such a pain in comparison.


Note: Only a member of this blog may post a comment.