Ubiquitous Assertion Syntax

One thing that really bothers me about testing is having various different assertion syntaxes. Take a look at the following JUnit examples. (don't worry, I'll be picking on Ruby and .net frameworks as well)

@Test public void simpleAdd() {
    assertEquals(2, 1+1);
}

@Test(expected= IndexOutOfBoundsException.class) public void empty() { 
    new ArrayList<Object>().get(0); 
}

In both tests I want to verify something; however the two tests use different mechanisms for verification. This adds pain for anyone reading or maintaining the test. When determining what a test is verifying you need to look for assertions as well as expected exceptions in the annotation.

When you start adding behavior based tests the situation gets even worse.

Mockery context = new Mockery();

@Test public void simpleAdd() {
    assertEquals(2, 1+1);
}

@Test(expected= IndexOutOfBoundsException.class) public void empty() {
    new ArrayList<Object>().get(0);
}

@Test public void forWorksCorrectly() {
    final List<Integer> list = context.mock(List.class);

    context.checking(new Expectations() {{
        one(list).add(1);
        one(list).add(2);
    }});

    for (int i=1; i < 3; i++) {
        list.add(i);
    }
}

Testing 1.0
3 different tests, 3 different ways to verify your code does what you expect it to. To make matters worse the mock expectations live in the body of the test, so there's no guarantee that when looking for assertions you only need to look at the last few lines of the test. Every time you encounter a test you must spend time looking at the test, top to bottom, and determine what is actually being tested.

The tests above are simple, it's a much bigger problem when a test uses a few different forms of verification.

Mockery context = new Mockery();

@Test(expected= IndexOutOfBoundsException.class) public void terribleTest() {
    final List<Integer> original = new ArrayList<Integer>(asList(1, 2));
    assertEquals(2, original.size());
    final List<Integer> list = context.mock(List.class);

    context.checking(new Expectations() {{
        one(list).add(1);
        one(list).add(2);
    }});

    for (Integer anOriginal : original) {
        list.add(anOriginal);
    }

    original.clear();
    original.get(0);
}

Okay, it would be terrible to run into a test that bad. You might work with people who are smart enough not to do such a thing, but it's very common to see tests that have both assertions and methods that are mocked. You may not consider mocked methods to be a form of verification, but if one of those methods isn't called as you specified it should be called, your test will fail. If it can cause your test to fail, it's a form of verification in my book.

In Java, there are at least 3 ways to define expected behavior, and often a test mixes more than one. This is not a good situation for a test reader or maintainer.

In the Ruby and .net worlds, the story isn't much better. The state based assertions are specified differently than the behavior based assertions; however, both Ruby and .net have a superior way of handling expected exceptions: assert_raises and Assert.Throws.

Below are similar tests in test/unit with mocha (Ruby), with an example of assert_raises for handling the expected exception.

def testState
  assert_equal 2, 1+1
end

def testError
  assert_raises(NoMethodError) { [].not_a_method }
end

def testBehavior
  array = []
  array.expects(:<<).with(1)
  array.expects(:<<).with(2)
  [1, 2].each do |number|
    array << number
  end
end

And, for the .net crowd, here's the NUnit + NMock version of the 3 types of tests. As previously mentioned, NUnit provides an assertion for expecting exceptions, but there's still a mismatch between the state based and behavior based assertions. (full disclosure, I don't have Visual Studio running on my Mac, so if there's a typo, forgive me)

private Mockery mocks = new Mockery();

[Test]
public void Add()
{
    Assert.AreEqual(2, 1+1);
}

[Test]
public void Raise()
{
   Assert.Throws<ArgumentException>(delegate { throw new ArgumentException() } );
}

[Test]
public void Behavior()
{
    IList list = mocks.NewMock<IList>();

    Expect.Once.On(list).Method("Add").With(1);
    Expect.Once.On(list).Method("Add").With(2);

    for (int i=1; i < 3; i++) {
        list.Add(i);
    }
    mocks.VerifyAllExpectationsHaveBeenMet();
}

Testing 2.0
In each language there have been steps forward. Both Java and C# have moved in the right direction regarding placement of behavior based expectations. Ruby still suffers from setup placement of mock expectations, but at least the behavior based expectations follow the same assertion syntax that the state based assertions use.

In Java, the Mockito framework represents a step in the right direction, moving the mock expectations to the end of the test.

@Test public void forRevisited() {
    List list = mock(List.class);

    for (int i=1; i < 3; i++) {
        list.add(i);
    }

    verify(list).add(1);
    verify(list).add(2);
}

Likewise, in .net, Rhino Mocks allows you to use "Arrange, Act, Assert Syntax (AAA)". The result of AAA is (hopefully) all of your tests put their assertions at the end.

public void ForRevisited()
{
    var list = MockRepository.GenerateStub<List>();

    for (int i=1; i < 3; i++) {
        list.Add(i);
    }

    list.AssertWasCalled( x => x.Add(1));
    list.AssertWasCalled( x => x.Add(2));
}

These steps are good for the big two (Java & C#); however, I'd say they are still a bit too far away from ubiquitous assertion syntax for my taste.

RSpec represents forward progress in the Ruby community.

it "should test state" do
  2.should == 1+1
end

it "should test an error" do
  lambda { [].not_a_method }.should raise_error(NoMethodError)
end

it "should test behavior" do
  array = []
  array.should_receive(:<<).with(1)
  array.should_receive(:<<).with(2)
  [1, 2].each do |number|
    array << number
  end
end

RSpec does a good job of starting all their assertions with "should", (almost?) to a fault. It's not surprising that RSpec was able to somewhat unify the syntax, since the testing framework provides the ability to write both state based and behavior based tests. Looking at the tests you can focus on scanning for the word "should" when looking for what's being tested.

Unfortunately the unified "should" syntax results in possibly the ugliest assertion ever: lambda {…}.should raise_error(Exception). And, as I previously stated RSpec still suffers from setup placement of the mock expectation "should" methods. Perhaps RSpec could benefit from AAA or some type of test spy (example implementation available on pastie.org).

# this doesn't work, but maybe it should....

it "should test behavior" do
  array = Spy.on []
  [1, 2].each do |number|
    array << number
  end
  array.should have_received << 1
  array.should have_received << 2
end

Next Generation Testing (Testing 3.0?)
Eventually I expect all testing frameworks will follow RSpec's lead and include their own mocking support, though I'm not holding my breath since it's been two years since I originally suggested that this should happen. Today's landscape looks largely the same as it did 2 years ago. When testing frameworks take that next step, the syntax should naturally converge.

In the Ruby world you do have another option, but one with serious risk. I've been looking for ubiquitous assertion syntax for so long that I rolled my own framework in Ruby: expectations.

Expectations standardizes on both the location of your assertions (expectations) and how you express them. The following code shows how you can expect a state based result, an exception, and a behavior based result.

expect 2 do
  1 + 1
end

expect NoMethodError do
  [].no_method_error
end

expect Object.new.to.receive(:ping).with(:pong) do |obj|
  obj.ping(:pong)
end

As a result of expectations implementation you can always look at the first line of a test and know exactly what you are testing.

Full disclosure: Before you go and install expectations and start using it for your production application, you need to know one big problem: there's no support for expectations. I'm no longer doing Ruby full-time and no one has stepped up to maintain the project. It's out there, it works, and it's all yours, but it comes with no guarantees.

In the .net and Java worlds, the future of testing looks less... evolved. In the .net world, the ever focused on testing Jim Newkirk has teamed up with Brad Wilson to create xUnit.net. xUnit.net represents evolution in the .net space, but as far as I can tell they haven't done much in the way of addressing ubiquitous assertion syntax. In Java, I don't see any movement towards addressing the issue, but can it even happen without closures (anonymous methods, delegates, whatever)?

I'm surprised that more people aren't bothered by the lack of ubiquitous assertion syntax. Perhaps we have become satisfied with disparity in syntax and the required full test scan.