Jay Fields' Thoughts

Liking it a lot... will have to take it for a spin.
 

Why "not providing a setup or teardown method" is a best practice?

Kind Regards
 

Marcos: http://blog.jayfields.com/2007/06/testing-inline-setup.html

People whom I regard as authority, such as Jim Newkirk (an original author of NUnit), are also moving away from setup. The new testing framework XUnit.net does not include a setup method for reasons similar to those I list in the entry linked above.
 

I've been using my own homegrown helper like this when doing testing. I didn't know other people liked this approach too! ;)
 

I installed 0.02 and I'm seeing something odd with respect to the time Expectations is reporting it is taking to finish.

I am seeing simple, single-expectation, tests with no additional requires report between 3-5s to finish when it is obvious to me that a fraction of a second of real time has elapsed.

If I run my test with ruby-prof instead of ruby I see a (more likely sounding) 0.0002s reported to finish the test.

Does anyone else see this?

I'm using Ruby 1.8.6p110, gems 1.0.1, on OSX10.5.1 on a 2.4GHz MBP.

Regards,

Matt
 

Hi Matt,

I'm seeing the same problem. I simplified the issue to this:

require 'benchmark'
benchmark = Benchmark.measure { }
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "3.09944"
benchmark.real # => 3.09944152832031e-05

benchmark = Benchmark.measure { }
benchmark.real # => 9.05990600585938e-06
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "9.05990"

benchmark = Benchmark.measure { sleep 1 }
benchmark.real # => 1.00011682510376
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "1.00011"

benchmark = Benchmark.measure { sleep 0.5 }
benchmark.real # => 0.50015115737915
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.50015"

benchmark = Benchmark.measure { sleep 0.25 }
benchmark.real # => 0.250133991241455
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.25013"

benchmark = Benchmark.measure { sleep 0.05 }
benchmark.real # => 0.0501019954681396
benchmark.real.to_s.gsub(/(\d*)\.(\d{0,5}).*/,'\1.\2') # => "0.05010"

It seems that the Benchmark::Tms instance reports incorrectly when the test runs very quickly.

I could compare two time instances to get the run time instead, but I'm not convinced that is the best plan.

Thoughts?

Cheers, Jay
 

Hi Jay.

The problem seems to be that the call to benchmark.real.to_s is returning time formatting in scientific notation for small values so that stripping off the leading value as seconds yields an unexpected result.

Unless sub-tenth-of-a-second resolution is important I might just report anything under 0.1 seconds as "<0.1s" and only do the #to_s.gsub for larger values.

Regards,

Matt
 

Hi Matt,

I'm not convinced that it is the call to benchmark.real.to_s.

Here's more examples where I never call to_s on real, yet the variable (still a number) is reporting larger than expected times.

require 'benchmark'
benchmark = Benchmark.measure { }
benchmark # => #<Benchmark::Tms:0x17764 @label=, @stime=0.0, @real=3.00407409667969e-05, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

benchmark = Benchmark.measure { }
benchmark # => #<Benchmark::Tms:0x17264 @label=, @stime=0.0, @real=9.05990600585938e-06, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

benchmark = Benchmark.measure { sleep 1 }
benchmark # => #<Benchmark::Tms:0x16d78 @label=, @stime=0.0, @real=1.00014090538025, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

benchmark = Benchmark.measure { sleep 0.5 }
benchmark # => #<Benchmark::Tms:0x1688c @label=, @stime=0.0, @real=0.500118017196655, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

benchmark = Benchmark.measure { sleep 0.25 }
benchmark # => #<Benchmark::Tms:0x163a0 @label=, @stime=0.0, @real=0.250134944915771, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

benchmark = Benchmark.measure { sleep 0.05 }
benchmark # => #<Benchmark::Tms:0x15eb4 @label=, @stime=0.0, @real=0.0501132011413574, @utime=0.0, @cstime=0.0, @total=0.0, @cutime=0.0>

Am I missing something?

Cheers, Jay
 

Hi Jay.

I made an assumption that you are familiar with scientific (exponent based) notation.

In the first two cases where the benchmark case is an empty block the "real" ivar contains the values:

3.00407409667969e-05

and

9.05990600585938e-06

respectively which are in scientific notation where the e-05 means "times ten to the power of minus 5" and e-06 to the power of minus 6 and so on.

So the first example is equivalent to:

0.000030040740966769

I think the problem here is that Float#to_s does not give you any control over whether results are displayed in the familiar fixed-decimal or scientific notation.

Regards,

Matt
 

Matt,

Hah, that makes perfect sense. Sorry, it's been awhile since I've done anything with scientific notation.

I'll try to put a fix in today.

Cheers, Jay
 

Hello Matt,

I made the change and put out version 0.0.3
 

Hi Jay.

Looking good here ;-)

Matt.
 

Hi Jay.

Another unexpected behavior in results output. Here is my simple test case:

require 'rubygems'
require 'expectations'

module A
module B
end
end

Expectations do

expect true do
A::B::C
end

end

The reported error is:

error <uninitialized constant A;;B;;C<

I see that suite_results.rb:42 is gsubbing ":" to ";" which is, I guess, causing this but is it intentional?

Matt.
 

I'm not sure if I've missed something obvious but I can't figure out how to expect "not nil", e.g. how I would write:

def test_should_return_something
assert_not_nil some_expression
end

as an expectation? Or have I got the wrong end of the stick about expectations?

Regards,

Matt
 

Hi Matt,

It is intentional. I gsub because TextMate seems to treat any string with three or more colons as a link. So, in an effort to subvert TextMate I was subbing the colons with semi colons. Not the best solution.

Thoughts on what's better? I can just ignore TextMate, but seems like a bad option.
 

I haven't seen the problem (because I am not running my tests via mate) so I'm not sure how bad it is. Is there a way to fix the Mate bundle that is doing this?

I guess I don't have a real problem with changing ":" to ";" beyond the fact that it looks weird every time I see it ;-)
 

Matt,
instead of expecting not nil you could write the test in terms of what you do expect.

expect Integer do
some_expression.class
end

Or

expect true do
some_expression != nil
end

I'm not convinced that expectations needs a not nil, but I'm open to adding something if it makes it more usable.

Cheers Jay
 

I think I am happy enough with

expect true do
some_expr != nil
end

however I offer:

expect do
some_expr
end

as a suggested sugar.

Regards,

Matt
 

Matt, I just put out 0.0.4 which doesn't need the semi-colons. I think I found another issue with mock expectations not being cleared correctly, but after I take care of that I'll spend some more time thinking about the not nil syntax sugar.

Thanks for all the feedback, it's very helpful.
 

I just released version 0.0.5, which fixes a bug concerning mocking concrete classes. Concrete mocks were failing because their expectations were never being cleared. Thus, Object.expects(:class).returns(Foo) would verify on every test, instead of the test that defined the expectation. All expectations are now cleared from mocks and only the expectation set in the expect will be verified.
 

I like it. A lot. I'll try to use it on an upcoming project and let you know how it works out. Also, +1 for Matt's "not nil" syntactic sugar. Asserting not nil has always struck me as a testing antipattern, but it's just one of those things you seem to need every once in a while.
 

Brian.

I am interested in why you think asserting not nil is an anti-pattern. Please do not read that as an argument, I'm genuinely interested because I've never considered the point before.

Matt.
 

Mike--Two reasons, neither of them particularly good. First, assuming your method doesn't purposefully return nils, there's almost always something more useful you can assert about the return value of that method than whether or not it was nil. And second, assuming your method does purposefully return nils, there's almost always something more useful it could be returning.
 

Looks like there's a copy & paste typo in the README. It refers to "arbs" instead of "expectations".
 

Thanks James.

I uploaded a fixed version