Tuesday, December 25, 2007

Ruby: expectations gem

In February I wrote about removing test noise. 10 months later, I finally took the time to write the unit testing framework I've been wanting for the past year: expectations

expectations is a lightweight unit testing framework. Tests (expectations) can be written as follows
  expect 2 do
1 + 1
end

expect NoMethodError do
Object.invalid_method_call
end.
expectations is designed to encourage unit testing best practices such as
  • discourage setting more than one expectation at a time
  • promote maintainability by not providing a setup or teardown method
  • provide one syntax for setting up state based or behavior based expectation
  • focus on readability by providing no mechanism for describing an expectation other than the code in the expectation. Since there is no description, hopefully the programmer will be encouraged to write the most readable code possible.
A few things probably come to mind right away: sometimes setup is good, sometimes a description is a good thing, sometimes creating an object is so painful that I need to be able to add multiple assertions. All of those things are true, but generally I don't run into those situations while unit testing. Those situations generally pop up with functional testing, and while functional testing you are better off using RSpec or Test::Unit.

The more I write tests the more I believe that there doesn't need to be a silver bullet testing framework. I think expectations is a good solution for unit testing, and I think RSpec or Test::Unit are good solutions for functional testing, and I plan to use both expectations and RSpec on all my projects moving forward.

I'm currently testing several of my projects using expectations, so I do believe it's in decent shape for using, but it is still very new. As always, patches are welcome.

Mocking is done using Mocha

Here's a few more examples of how easy it is to test with expectations

Expectations do

# State based expectation where a value equals another value
expect 2 do
1 + 1
end

# State based expectation where an exception is expected. Simply expect the Class of the intended exception
expect NoMethodError do
Object.no_method
end

# Behavior based test using a traditional mock
expect mock.to.receive(:dial).with("2125551212").times(2) do |phone|
phone.dial("2125551212")
phone.dial("2125551212")
end

# Behavior based test on a concrete mock
expect Object.to.receive(:deal).with(1) do
Object.deal(1)
end

end

30 comments:

  1. Liking it a lot... will have to take it for a spin.

    ReplyDelete
  2. Why "not providing a setup or teardown method" is a best practice?

    Kind Regards

    ReplyDelete
  3. Marcos: http://blog.jayfields.com/2007/06/testing-inline-setup.html

    People whom I regard as authority, such as Jim Newkirk (an original author of NUnit), are also moving away from setup. The new testing framework XUnit.net does not include a setup method for reasons similar to those I list in the entry linked above.

    ReplyDelete
  4. I've been using my own homegrown helper like this when doing testing. I didn't know other people liked this approach too! ;)

    ReplyDelete
  5. I installed 0.02 and I'm seeing something odd with respect to the time Expectations is reporting it is taking to finish.

    I am seeing simple, single-expectation, tests with no additional requires report between 3-5s to finish when it is obvious to me that a fraction of a second of real time has elapsed.

    If I run my test with ruby-prof instead of ruby I see a (more likely sounding) 0.0002s reported to finish the test.

    Does anyone else see this?

    I'm using Ruby 1.8.6p110, gems 1.0.1, on OSX10.5.1 on a 2.4GHz MBP.

    Regards,

    Matt

    ReplyDelete
  6. Hi Matt,

    I'm seeing the same problem. I simplified the issue to this:

    require 'benchmark'
    benchmark = Benchmark.measure { }
    benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "3.09944"
    benchmark.real # => 3.09944152832031e-05

    benchmark = Benchmark.measure { }
    benchmark.real # => 9.05990600585938e-06
    benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "9.05990"

    benchmark = Benchmark.measure { sleep 1 }
    benchmark.real # => 1.00011682510376
    benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "1.00011"

    benchmark = Benchmark.measure { sleep 0.5 }
    benchmark.real # => 0.50015115737915
    benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.50015"

    benchmark = Benchmark.measure { sleep 0.25 }
    benchmark.real # => 0.250133991241455
    benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.25013"

    benchmark = Benchmark.measure { sleep 0.05 }
    benchmark.real # => 0.0501019954681396
    benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.05010"

    It seems that the Benchmark::Tms instance reports incorrectly when the test runs very quickly.

    I could compare two time instances to get the run time instead, but I'm not convinced that is the best plan.

    Thoughts?

    Cheers, Jay

    ReplyDelete
  7. Hi Jay.

    The problem seems to be that the call to benchmark.real.to_s is returning time formatting in scientific notation for small values so that stripping off the leading value as seconds yields an unexpected result.

    Unless sub-tenth-of-a-second resolution is important I might just report anything under 0.1 seconds as "<0.1s" and only do the #to_s.gsub for larger values.

    Regards,

    Matt

    ReplyDelete
  8. Hi Matt,

    I'm not convinced that it is the call to benchmark.real.to_s.

    Here's more examples where I never call to_s on real, yet the variable (still a number) is reporting larger than expected times.

    require 'benchmark'
    benchmark = Benchmark.measure { }
    benchmark # => #<Benchmark::Tms:0x17764 @label=, @stime=0.0, @real=3.00407409667969e-05, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

    benchmark = Benchmark.measure { }
    benchmark # => #<Benchmark::Tms:0x17264 @label=, @stime=0.0, @real=9.05990600585938e-06, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

    benchmark = Benchmark.measure { sleep 1 }
    benchmark # => #<Benchmark::Tms:0x16d78 @label=, @stime=0.0, @real=1.00014090538025, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

    benchmark = Benchmark.measure { sleep 0.5 }
    benchmark # => #<Benchmark::Tms:0x1688c @label=, @stime=0.0, @real=0.500118017196655, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

    benchmark = Benchmark.measure { sleep 0.25 }
    benchmark # => #<Benchmark::Tms:0x163a0 @label=, @stime=0.0, @real=0.250134944915771, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

    benchmark = Benchmark.measure { sleep 0.05 }
    benchmark # => #<Benchmark::Tms:0x15eb4 @label=, @stime=0.0, @real=0.0501132011413574, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

    Am I missing something?

    Cheers, Jay

    ReplyDelete
  9. Hi Jay.

    I made an assumption that you are familiar with scientific (exponent based) notation.

    In the first two cases where the benchmark case is an empty block the "real" ivar contains the values:

    3.00407409667969e-05

    and

    9.05990600585938e-06

    respectively which are in scientific notation where the e-05 means "times ten to the power of minus 5" and e-06 to the power of minus 6 and so on.

    So the first example is equivalent to:

    0.000030040740966769

    I think the problem here is that Float#to_s does not give you any control over whether results are displayed in the familiar fixed-decimal or scientific notation.

    Regards,

    Matt

    ReplyDelete
  10. Matt,

    Hah, that makes perfect sense. Sorry, it's been awhile since I've done anything with scientific notation.

    I'll try to put a fix in today.

    Cheers, Jay

    ReplyDelete
  11. Hello Matt,

    I made the change and put out version 0.0.3

    ReplyDelete
  12. Hi Jay.

    Looking good here ;-)

    Matt.

    ReplyDelete
  13. Hi Jay.

    Another unexpected behavior in results output. Here is my simple test case:

    require 'rubygems'
    require 'expectations'

    module A
    module B
    end
    end

    Expectations do

    expect true do
    A::B::C
    end

    end

    The reported error is:

    error <uninitialized constant A;;B;;C<

    I see that suite_results.rb:42 is gsubbing ":" to ";" which is, I guess, causing this but is it intentional?

    Matt.

    ReplyDelete
  14. I'm not sure if I've missed something obvious but I can't figure out how to expect "not nil", e.g. how I would write:

    def test_should_return_something
    assert_not_nil some_expression
    end

    as an expectation? Or have I got the wrong end of the stick about expectations?

    Regards,

    Matt

    ReplyDelete
  15. Hi Matt,

    It is intentional. I gsub because TextMate seems to treat any string with three or more colons as a link. So, in an effort to subvert TextMate I was subbing the colons with semi colons. Not the best solution.

    Thoughts on what's better? I can just ignore TextMate, but seems like a bad option.

    ReplyDelete
  16. I haven't seen the problem (because I am not running my tests via mate) so I'm not sure how bad it is. Is there a way to fix the Mate bundle that is doing this?

    I guess I don't have a real problem with changing ":" to ";" beyond the fact that it looks weird every time I see it ;-)

    ReplyDelete
  17. Matt,
    instead of expecting not nil you could write the test in terms of what you do expect.

    expect Integer do
    some_expression.class
    end

    Or

    expect true do
    some_expression != nil
    end

    I'm not convinced that expectations needs a not nil, but I'm open to adding something if it makes it more usable.

    Cheers Jay

    ReplyDelete
  18. I think I am happy enough with

    expect true do
    some_expr != nil
    end

    however I offer:

    expect do
    some_expr
    end

    as a suggested sugar.

    Regards,

    Matt

    ReplyDelete
  19. Matt, I just put out 0.0.4 which doesn't need the semi-colons. I think I found another issue with mock expectations not being cleared correctly, but after I take care of that I'll spend some more time thinking about the not nil syntax sugar.

    Thanks for all the feedback, it's very helpful.

    ReplyDelete
  20. I just released version 0.0.5, which fixes a bug concerning mocking concrete classes. Concrete mocks were failing because their expectations were never being cleared. Thus, Object.expects(:class).returns(Foo) would verify on every test, instead of the test that defined the expectation. All expectations are now cleared from mocks and only the expectation set in the expect will be verified.

    ReplyDelete
  21. I like it. A lot. I'll try to use it on an upcoming project and let you know how it works out. Also, +1 for Matt's "not nil" syntactic sugar. Asserting not nil has always struck me as a testing antipattern, but it's just one of those things you seem to need every once in a while.

    ReplyDelete
  22. Brian.

    I am interested in why you think asserting not nil is an anti-pattern. Please do not read that as an argument, I'm genuinely interested because I've never considered the point before.

    Matt.

    ReplyDelete
  23. Mike--Two reasons, neither of them particularly good. First, assuming your method doesn't purposefully return nils, there's almost always something more useful you can assert about the return value of that method than whether or not it was nil. And second, assuming your method does purposefully return nils, there's almost always something more useful it could be returning.

    ReplyDelete
  24. Looks like there's a copy & paste typo in the README. It refers to "arbs" instead of "expectations".

    ReplyDelete
  25. Thanks James.

    I uploaded a fixed version

    ReplyDelete
  26. Hi Jay,
    the following fix for Suite#expect will make Expectations work on Windows as well:

    def expect(expected, &block)
    expectations << Expectations::Expectation.new(expected, *caller.first.match(/\A(.+):(\d+)\Z/)[1..2], &block)
    end

    The original code gives the wrong source and line number when an expectation fails (because ':' is included in a Windows path)

    ReplyDelete
  27. Thanks, I've checked in the change.

    Cheers, Jay

    ReplyDelete
  28. Umm... okay, I'm not sure if this a dumb question or not... but how can I do equality assertions where it's an identity-based equality?

    eg: how would an Exceptions test look like for something like this:

    def test_only_winners_are_returned
    player1 = Player.new(:Rock)
    player2 = Player.new(:Scissor)
    player3 = Player.new(:Rock)
    game = Game.new(player1, player2, player3)
    assert_equal([player1, player3], game.play)
    end

    In Expectations, the problem I'm running into is how to expect for player1 and player3 when they don't yet exist (since there is no setup, and an inline-setup will be in the block to be executed after the expectation is set).

    ReplyDelete
  29. One of the principles of expectations is that you want to expect literals. This forces you to write your tests differently, but I believe the result is often better.

    There's plenty of ways to rewrite the test, which you choose will be up to your preferences. You could have play return nothing and then use querying methods to determine that two players won. You could expect :rock, and have game.play return that. You could expect an array of [:rock, :rock] and map the results of game.play (e.g. game.play.map(&choice))

    Lots of options, it just depends how you want to do it.

    Cheers, Jay

    ReplyDelete
  30. Hi Jay, thanks for your response. I gave the example more thought and finally came up with something like this:

    expect true do
    player1 = Player.new(:Rock)
    player2 = Player.new(:Scissor)
    player3 = Player.new(:Rock)
    game = Game.new(player1, player2, player3)
    [player1, player3] == game.play
    end

    I didn't go with any of your alternatives for the following (personal) reasons:

    "You could have play return nothing and then use querying methods to determine that two players won."

    Umm... that won't directly tell me who exactly won. It could incorrectly be player1 and player2.

    "You could expect :rock, and have game.play return that."

    I found this suggestion interesting. It'd lead me to creating a new class though that would have to loop over all the players, figure out who said :Rock, and then choose those as winners... which is what I set out to be Game's responsibility in the first place. And I may end up breaking encapsulation. (Right now, Game#play simply calls Player#against(other) to find who wins among two players, and 'choice' is a protected field).

    But still, I liked that expectations made me think of any missing classes.

    "You could expect an array of [:rock, :rock] and map the results of game.play (e.g. game.play.map(&choice))"

    Nah, similar reasons as above. I don't want to be making getters (reader methods) for internal stuff like 'choice' just 'cos the test needs it. This sorta test is probably fine for ActiveRecords though (they already expose everything, so why not use them for test purposes).

    Overall, I liked that Expectations restricts us to one assertion at a time. Regarding expecting literals, I like that it discourages unnecessary variables for expected values... but I'm afraid that it'll probably push some people to indulge in Primitive Obsession. If you're careful not to stray though, I like that it'll make you think of Value Objects. But again here, I'd be wary of not making Entity Objects into Value Objects just for testing purposes.

    Bottomline: Expectations is making me think about all that, which is a good thing; and I'd presume, also a goal behind creating Expectations. :-)

    ReplyDelete

Note: Only a member of this blog may post a comment.