expectations is a lightweight unit testing framework. Tests (expectations) can be written as follows
expect 2 doexpectations is designed to encourage unit testing best practices such as
1 + 1
end
expect NoMethodError do
Object.invalid_method_call
end.
- discourage setting more than one expectation at a time
- promote maintainability by not providing a setup or teardown method
- provide one syntax for setting up state based or behavior based expectation
- focus on readability by providing no mechanism for describing an expectation other than the code in the expectation. Since there is no description, hopefully the programmer will be encouraged to write the most readable code possible.
The more I write tests the more I believe that there doesn't need to be a silver bullet testing framework. I think expectations is a good solution for unit testing, and I think RSpec or Test::Unit are good solutions for functional testing, and I plan to use both expectations and RSpec on all my projects moving forward.
I'm currently testing several of my projects using expectations, so I do believe it's in decent shape for using, but it is still very new. As always, patches are welcome.
Mocking is done using Mocha
Here's a few more examples of how easy it is to test with expectations
Expectations do
# State based expectation where a value equals another value
expect 2 do
1 + 1
end
# State based expectation where an exception is expected. Simply expect the Class of the intended exception
expect NoMethodError do
Object.no_method
end
# Behavior based test using a traditional mock
expect mock.to.receive(:dial).with("2125551212").times(2) do |phone|
phone.dial("2125551212")
phone.dial("2125551212")
end
# Behavior based test on a concrete mock
expect Object.to.receive(:deal).with(1) do
Object.deal(1)
end
end
Liking it a lot... will have to take it for a spin.
ReplyDeleteWhy "not providing a setup or teardown method" is a best practice?
ReplyDeleteKind Regards
Marcos: http://blog.jayfields.com/2007/06/testing-inline-setup.html
ReplyDeletePeople whom I regard as authority, such as Jim Newkirk (an original author of NUnit), are also moving away from setup. The new testing framework XUnit.net does not include a setup method for reasons similar to those I list in the entry linked above.
I've been using my own homegrown helper like this when doing testing. I didn't know other people liked this approach too! ;)
ReplyDeleteI installed 0.02 and I'm seeing something odd with respect to the time Expectations is reporting it is taking to finish.
ReplyDeleteI am seeing simple, single-expectation, tests with no additional requires report between 3-5s to finish when it is obvious to me that a fraction of a second of real time has elapsed.
If I run my test with ruby-prof instead of ruby I see a (more likely sounding) 0.0002s reported to finish the test.
Does anyone else see this?
I'm using Ruby 1.8.6p110, gems 1.0.1, on OSX10.5.1 on a 2.4GHz MBP.
Regards,
Matt
Hi Matt,
ReplyDeleteI'm seeing the same problem. I simplified the issue to this:
require 'benchmark'
benchmark = Benchmark.measure { }
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "3.09944"
benchmark.real # => 3.09944152832031e-05
benchmark = Benchmark.measure { }
benchmark.real # => 9.05990600585938e-06
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "9.05990"
benchmark = Benchmark.measure { sleep 1 }
benchmark.real # => 1.00011682510376
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "1.00011"
benchmark = Benchmark.measure { sleep 0.5 }
benchmark.real # => 0.50015115737915
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.50015"
benchmark = Benchmark.measure { sleep 0.25 }
benchmark.real # => 0.250133991241455
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.25013"
benchmark = Benchmark.measure { sleep 0.05 }
benchmark.real # => 0.0501019954681396
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.05010"
It seems that the Benchmark::Tms instance reports incorrectly when the test runs very quickly.
I could compare two time instances to get the run time instead, but I'm not convinced that is the best plan.
Thoughts?
Cheers, Jay
Hi Jay.
ReplyDeleteThe problem seems to be that the call to benchmark.real.to_s is returning time formatting in scientific notation for small values so that stripping off the leading value as seconds yields an unexpected result.
Unless sub-tenth-of-a-second resolution is important I might just report anything under 0.1 seconds as "<0.1s" and only do the #to_s.gsub for larger values.
Regards,
Matt
Hi Matt,
ReplyDeleteI'm not convinced that it is the call to benchmark.real.to_s.
Here's more examples where I never call to_s on real, yet the variable (still a number) is reporting larger than expected times.
require 'benchmark'
benchmark = Benchmark.measure { }
benchmark # => #<Benchmark::Tms:0x17764 @label=, @stime=0.0, @real=3.00407409667969e-05, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>
benchmark = Benchmark.measure { }
benchmark # => #<Benchmark::Tms:0x17264 @label=, @stime=0.0, @real=9.05990600585938e-06, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>
benchmark = Benchmark.measure { sleep 1 }
benchmark # => #<Benchmark::Tms:0x16d78 @label=, @stime=0.0, @real=1.00014090538025, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>
benchmark = Benchmark.measure { sleep 0.5 }
benchmark # => #<Benchmark::Tms:0x1688c @label=, @stime=0.0, @real=0.500118017196655, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>
benchmark = Benchmark.measure { sleep 0.25 }
benchmark # => #<Benchmark::Tms:0x163a0 @label=, @stime=0.0, @real=0.250134944915771, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>
benchmark = Benchmark.measure { sleep 0.05 }
benchmark # => #<Benchmark::Tms:0x15eb4 @label=, @stime=0.0, @real=0.0501132011413574, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>
Am I missing something?
Cheers, Jay
Hi Jay.
ReplyDeleteI made an assumption that you are familiar with scientific (exponent based) notation.
In the first two cases where the benchmark case is an empty block the "real" ivar contains the values:
3.00407409667969e-05
and
9.05990600585938e-06
respectively which are in scientific notation where the e-05 means "times ten to the power of minus 5" and e-06 to the power of minus 6 and so on.
So the first example is equivalent to:
0.000030040740966769
I think the problem here is that Float#to_s does not give you any control over whether results are displayed in the familiar fixed-decimal or scientific notation.
Regards,
Matt
Matt,
ReplyDeleteHah, that makes perfect sense. Sorry, it's been awhile since I've done anything with scientific notation.
I'll try to put a fix in today.
Cheers, Jay
Hello Matt,
ReplyDeleteI made the change and put out version 0.0.3
Hi Jay.
ReplyDeleteLooking good here ;-)
Matt.
Hi Jay.
ReplyDeleteAnother unexpected behavior in results output. Here is my simple test case:
require 'rubygems'
require 'expectations'
module A
module B
end
end
Expectations do
expect true do
A::B::C
end
end
The reported error is:
error <uninitialized constant A;;B;;C<
I see that suite_results.rb:42 is gsubbing ":" to ";" which is, I guess, causing this but is it intentional?
Matt.
I'm not sure if I've missed something obvious but I can't figure out how to expect "not nil", e.g. how I would write:
ReplyDeletedef test_should_return_something
assert_not_nil some_expression
end
as an expectation? Or have I got the wrong end of the stick about expectations?
Regards,
Matt
Hi Matt,
ReplyDeleteIt is intentional. I gsub because TextMate seems to treat any string with three or more colons as a link. So, in an effort to subvert TextMate I was subbing the colons with semi colons. Not the best solution.
Thoughts on what's better? I can just ignore TextMate, but seems like a bad option.
I haven't seen the problem (because I am not running my tests via mate) so I'm not sure how bad it is. Is there a way to fix the Mate bundle that is doing this?
ReplyDeleteI guess I don't have a real problem with changing ":" to ";" beyond the fact that it looks weird every time I see it ;-)
Matt,
ReplyDeleteinstead of expecting not nil you could write the test in terms of what you do expect.
expect Integer do
some_expression.class
end
Or
expect true do
some_expression != nil
end
I'm not convinced that expectations needs a not nil, but I'm open to adding something if it makes it more usable.
Cheers Jay
I think I am happy enough with
ReplyDeleteexpect true do
some_expr != nil
end
however I offer:
expect do
some_expr
end
as a suggested sugar.
Regards,
Matt
Matt, I just put out 0.0.4 which doesn't need the semi-colons. I think I found another issue with mock expectations not being cleared correctly, but after I take care of that I'll spend some more time thinking about the not nil syntax sugar.
ReplyDeleteThanks for all the feedback, it's very helpful.
I just released version 0.0.5, which fixes a bug concerning mocking concrete classes. Concrete mocks were failing because their expectations were never being cleared. Thus, Object.expects(:class).returns(Foo) would verify on every test, instead of the test that defined the expectation. All expectations are now cleared from mocks and only the expectation set in the expect will be verified.
ReplyDeleteI like it. A lot. I'll try to use it on an upcoming project and let you know how it works out. Also, +1 for Matt's "not nil" syntactic sugar. Asserting not nil has always struck me as a testing antipattern, but it's just one of those things you seem to need every once in a while.
ReplyDeleteBrian.
ReplyDeleteI am interested in why you think asserting not nil is an anti-pattern. Please do not read that as an argument, I'm genuinely interested because I've never considered the point before.
Matt.
Mike--Two reasons, neither of them particularly good. First, assuming your method doesn't purposefully return nils, there's almost always something more useful you can assert about the return value of that method than whether or not it was nil. And second, assuming your method does purposefully return nils, there's almost always something more useful it could be returning.
ReplyDeleteLooks like there's a copy & paste typo in the README. It refers to "arbs" instead of "expectations".
ReplyDeleteThanks James.
ReplyDeleteI uploaded a fixed version
Hi Jay,
ReplyDeletethe following fix for Suite#expect will make Expectations work on Windows as well:
def expect(expected, &block)
expectations << Expectations::Expectation.new(expected, *caller.first.match(/\A(.+):(\d+)\Z/)[1..2], &block)
end
The original code gives the wrong source and line number when an expectation fails (because ':' is included in a Windows path)
Thanks, I've checked in the change.
ReplyDeleteCheers, Jay
Umm... okay, I'm not sure if this a dumb question or not... but how can I do equality assertions where it's an identity-based equality?
ReplyDeleteeg: how would an Exceptions test look like for something like this:
def test_only_winners_are_returned
player1 = Player.new(:Rock)
player2 = Player.new(:Scissor)
player3 = Player.new(:Rock)
game = Game.new(player1, player2, player3)
assert_equal([player1, player3], game.play)
end
In Expectations, the problem I'm running into is how to expect for player1 and player3 when they don't yet exist (since there is no setup, and an inline-setup will be in the block to be executed after the expectation is set).
One of the principles of expectations is that you want to expect literals. This forces you to write your tests differently, but I believe the result is often better.
ReplyDeleteThere's plenty of ways to rewrite the test, which you choose will be up to your preferences. You could have play return nothing and then use querying methods to determine that two players won. You could expect :rock, and have game.play return that. You could expect an array of [:rock, :rock] and map the results of game.play (e.g. game.play.map(&choice))
Lots of options, it just depends how you want to do it.
Cheers, Jay
Hi Jay, thanks for your response. I gave the example more thought and finally came up with something like this:
ReplyDeleteexpect true do
player1 = Player.new(:Rock)
player2 = Player.new(:Scissor)
player3 = Player.new(:Rock)
game = Game.new(player1, player2, player3)
[player1, player3] == game.play
end
I didn't go with any of your alternatives for the following (personal) reasons:
"You could have play return nothing and then use querying methods to determine that two players won."
Umm... that won't directly tell me who exactly won. It could incorrectly be player1 and player2.
"You could expect :rock, and have game.play return that."
I found this suggestion interesting. It'd lead me to creating a new class though that would have to loop over all the players, figure out who said :Rock, and then choose those as winners... which is what I set out to be Game's responsibility in the first place. And I may end up breaking encapsulation. (Right now, Game#play simply calls Player#against(other) to find who wins among two players, and 'choice' is a protected field).
But still, I liked that expectations made me think of any missing classes.
"You could expect an array of [:rock, :rock] and map the results of game.play (e.g. game.play.map(&choice))"
Nah, similar reasons as above. I don't want to be making getters (reader methods) for internal stuff like 'choice' just 'cos the test needs it. This sorta test is probably fine for ActiveRecords though (they already expose everything, so why not use them for test purposes).
Overall, I liked that Expectations restricts us to one assertion at a time. Regarding expecting literals, I like that it discourages unnecessary variables for expected values... but I'm afraid that it'll probably push some people to indulge in Primitive Obsession. If you're careful not to stray though, I like that it'll make you think of Value Objects. But again here, I'd be wary of not making Entity Objects into Value Objects just for testing purposes.
Bottomline: Expectations is making me think about all that, which is a good thing; and I'd presume, also a goal behind creating Expectations. :-)