Jay Fields' Thoughts

I'm very opinionated about unit testing. I believe that you should only have one assertion per test which includes having only one expectation per test. I also believe that tests should be as clear as possible, and doing things such as inlining setup and expecting literals will result in more maintainable tests.

I've been an Xunit fan for years. I can do everything I want using Xunit frameworks, but at times the framework gets in my way. For example, I generally don't need a test name, which is really nothing more than a glorified comment. I also dislike having different syntaxes for state based and behavior based tests. Most importantly it enables other people to make questionable decisions.

On Christmas 2007 I released my opinionated unit testing framework: expectations.

Expectations has one syntax for both state based and behavior based tests. Expectations has no test name, if you need a comment, you add a real comment, if you don't you aren't required to. Expected values are specified outside the scope of the test, which encourages you to use literals whenever possible. Expectations has no support for setup or teardown, it forces you to duplicate code or write more loosely coupled code that requires less setup -- which is a good thing.

Some tests aren't a good fit for expectations, that's where a functional test suite comes into play. I've used both test/unit and RSpec for functional testing, both have benefits and provide you the ability to break every suggestion expectations provides.

I've been using expectations on my current project and all my open source projects for about 3 months. The framework has grown in many ways in those 3 months, the rest of this post is about the features that are currently supported.

Expectations supports traditional state based tests. The result of executing the block is compared with the expected value and if they are equal the test passes. The resulting tests are very easy to follow. The expected value is obvious in the first line, all but the last line in the block are setting up the test and the last line is the actual value.

expect 2 do
  1 + 1
end

Expectations also supports behavior based tests by setting expectations on objects. The object can be a concrete class, a stub or a mock, it doesn't matter, they all support behavior based expectations (with the same syntax).

# Behavior based test using a traditional mock
expect mock.to.receive(:dial).with("2125551212").times(2) do |phone|
  phone.dial("2125551212")
  phone.dial("2125551212")
end

# Behavior based test using a stub
expect stub.to.receive(:dial).with("2125551212").times(2) do |phone|
  phone.dial("2125551212")
  phone.dial("2125551212")
end

# Behavior based test using a stub_everything
expect stub_everything.to.receive(:dial).with("2125551212").times(2) do |phone|
  phone.dial("2125551212")
  phone.dial("2125551212")
end

# Behavior based test on a concrete mock
expect Object.to.receive(:deal) do
  Object.deal
end

Expectations also supports asserting an exception will be thrown.

expect NoMethodError do
  Object.no_method
end

Expectations generally uses == internally to test equality; however, expectations also supports case equality (===) for Ranges, Regexps, and Modules.

# State based test matching a Regexp
expect /a string/ do
  "a string"
end

# State based test checking if actual is in the expected Range
expect 1..5 do
  3
end

# State based test to determine if the object is an instance of the module
expect Enumerable do
  []
end

Every expectation that uses case equality also uses regular equality, so the following tests also pass.

# State based test matching a Regexp
expect /a string/ do
  /a string/
end

# State based test checking if actual is in the expected Range
expect 1..5 do
  1..5
end

# State based test to determine if the object is an instance of the module
expect Enumerable do
  Enumerable
end

Expectations also introduces a fluent interface for asserting boolean values. The following tests verify that the attribute is set to true or false, based on the expectation definition.

# this is normally defined in the file specific to the class
klass = Class.new do
  attr_accessor :started, :finished
end

# State based fluent interface boolean tests using to be
expect klass.new.not.to.have.started

expect klass.new.to.be.started do |process|
  process.started = true
end

# State based fluent interface boolean test using to have
expect klass.new.not.to.have.finished

expect klass.new.to.have.finished do |process|
  process.finished = true
end

The above example also shows expectations that exist without using a block. These are fully functional and can be run individually within TextMate (snippet for running individual expectations is in the README).

If you have opinions like mine, you may want to give expectations a shot. If you disagree, that's cool too, to each their own.

Labels: expectations, ruby, testing, unit testing

In February I wrote about removing test noise. 10 months later, I finally took the time to write the unit testing framework I've been wanting for the past year: expectations

expectations is a lightweight unit testing framework. Tests (expectations) can be written as follows

  expect 2 do
    1 + 1
  end

  expect NoMethodError do
    Object.invalid_method_call
  end.

expectations is designed to encourage unit testing best practices such as

• discourage setting more than one expectation at a time
• promote maintainability by not providing a setup or teardown method
• provide one syntax for setting up state based or behavior based expectation
• focus on readability by providing no mechanism for describing an expectation other than the code in the expectation. Since there is no description, hopefully the programmer will be encouraged to write the most readable code possible.

A few things probably come to mind right away: sometimes setup is good, sometimes a description is a good thing, sometimes creating an object is so painful that I need to be able to add multiple assertions. All of those things are true, but generally I don't run into those situations while unit testing. Those situations generally pop up with functional testing, and while functional testing you are better off using RSpec or Test::Unit.

The more I write tests the more I believe that there doesn't need to be a silver bullet testing framework. I think expectations is a good solution for unit testing, and I think RSpec or Test::Unit are good solutions for functional testing, and I plan to use both expectations and RSpec on all my projects moving forward.

I'm currently testing several of my projects using expectations, so I do believe it's in decent shape for using, but it is still very new. As always, patches are welcome.

Mocking is done using Mocha

Here's a few more examples of how easy it is to test with expectations

Expectations do
  
  # State based expectation where a value equals another value
  expect 2 do
    1 + 1
  end
  
  # State based expectation where an exception is expected. Simply expect the Class of the intended exception
  expect NoMethodError do
    Object.no_method
  end
  
  # Behavior based test using a traditional mock
  expect mock.to.receive(:dial).with("2125551212").times(2) do |phone|
    phone.dial("2125551212")
    phone.dial("2125551212")
  end

  # Behavior based test on a concrete mock
  expect Object.to.receive(:deal).with(1) do
    Object.deal(1)
  end
  
end

Labels: expectations, unit testing

Consider the following expectations:

expect 4 do
  Math.plus(2,2)
end

expect ActiveRecord::Base.to_recieve(:execute).with("insert ...") do
  Person.save
end

In the past I've written about the absence of mocks in xUnit frameworks. I still believe this is a limitation and it leads to the following additional limitation: You have to set up behavioral expectations in a different way than you set up state based assertions. A side effect of this is that it often leads to tests that contain both behavioral and state based requirements, which are generally brittle. I believe that having two ways to express a test requirement increases noise in tests. The above example attempts to address the disconnect.

There's another spot where test noise can creep in: the name of the test. This idea is very controversial because it goes against what can work best. The problem is, what can work best, rarely does.

For example, when you encounter a test name such as def test_should_send_email_when_the_order_is_placed, how often does the test actually verify that an email is sent following an order being placed? Furthermore, how often do you even find test names that are that descriptive?

In my experience, I generally find tests such as def test_can_save. This is annoying, but it's better than when I find def test_should_verify_that_the_service_is_called_to_verify_a_credit_card and the actual body of the test does something entirely different. This generally happens on larger code bases that have been in development for a bit of time. When the description was originally written it was (hopefully) valid; however, at some point the test broke when a developer was trying to check in. The developer came to the test, didn't understand it and did what they had to do to make it work again. This cycle continued a few times until I found it (also when the tests broke). By the time I found it, no one had a full picture of what the test should be doing and I often end up deleting the test since even the original author has no idea what it's doing with all the patches that other developers have put on it.

The above scenario describes a larger problem: lack of communication. When the tests break the team members should talk to each other to ensure that the test continues to provide value; however, this simply doesn't happen. You could argue that the developers need to be beaten with a stick; however, I prefer a solution that plays to what developers desire: the ability to change the tests themselves.

While the above example does highlight a lack of communication, it isn't simply in the form of team members talking to each other. The test itself did not clearly communicate it's intent and was hard for the maintainers to comprehend.

The original example takes into account that the test name is rarely valuable and replaces it with what is important, the expectation of the test. By reducing the noise in the test maintainability is increased as well as the likelihood that the test will continue to provide value.

If you are a fan of Dave Astels one assertion per test guideline you will notice that the original example also guides you to follow the guideline. However, one assertion per test is simply a guideline so I expect you could also add additional expectations in the block where necessary.

In the Validatable work I've been doing lately I started using:

expect [expected value] do
  ....
end

This is working well, but I haven't taken the time to implement the mock syntax. I expect that using Mocha it would be fairly easy; however, I think the long term solution is to create a new testing(expectation?) framework.

Labels: expectations, TDD, test names, testing