Testing: Replace Mock with Stub

I often hear the phrase: I remember when I thought Mocks were the silver bullet. In fact, I remember using Mocks like they were the silver bullet. Unfortunately, using Mocks excessively can lead to a test suite that is just as brittle, if not more brittle than using concrete classes as dependencies.

One of the big reasons I began using Mocks in tests was to avoid cascading failures. However, I quickly learned that cascading failures still occur when using mocks, just in another flavor. Any implementation change required each instance of the mock to be updated to reflect the new implementation. My "solution" was just as bad as my problem.

To resolve this issue I started relying on Stubs. In fact, I found that using both Mocks and Stubs together made my tests more expressive. When I wrote that entry I was using C# in my day job; however, I think the concept transcends languages.

I'm going to use an example similar to the one from the original entry, this time in Ruby. It's completely acceptable to write the test using only mocks. (Mocha used for mocking)

class SqlStatementBuilder
  attr_reader :instance, :where
  def initialize(instance, where)
    @instance, @where = instance, where
  end
  
  def create_find_all_statement
    "select * from #{instance.name.to_s} #{where.to_sql}"
  end
end

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    instance = mock
    instance.expects(:name).returns('Foo')
    where = mock
    where.expects(:to_sql).returns("")
    builder = SqlStatementBuilder.new(instance, where)
    assert_equal "select * from Foo ", builder.create_find_all_statement
  end
  
  def test_where_to_string_is_appended_to_end
    instance = mock
    instance.expects(:name).returns('')
    where = mock
    where.expects(:to_sql).returns("where ...")
    builder = SqlStatementBuilder.new(instance, where)
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

While using mocks does verify the behavior of the builder class, that verification is superfluous. In fact, I believe the above test is covering too much. The state based assertion (assert_equals ...) at the end is all that's really necessary. The mocks can be replaced with stubs since the state based assertion is sufficient.

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    builder = SqlStatementBuilder.new(stub(:name => 'Foo'), stub(:to_sql => ''))
    assert_equal "select * from Foo ", builder.create_find_all_statement
  end

  def test_where_to_string_is_appended_to_end
    builder = SqlStatementBuilder.new(stub(:name => ''), stub(:to_sql => 'where ...'))
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

Looking at the above example your first instinct might be to combine the tests. An alternative and possibly superior solution is to continue to test the interactions independently, but only pass in stubs that are necessary for each individual interaction. This is where the stub_everything method from Mocha becomes very handy.

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    builder = SqlStatementBuilder.new(stub(:name => 'Foo'), stub_everything)
    assert_equal "select * from Foo ", builder.create_find_all_statement
  end

  def test_where_to_string_is_appended_to_end
    builder = SqlStatementBuilder.new(stub_everything, stub(:to_sql => 'where ...'))
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

Now imagine that the SqlStatementBuilder needs to be changed to allow a prefix for the table name.

class SqlStatementBuilder
  attr_reader :instance, :where
  def initialize(instance, where)
    @instance, @where = instance, where
  end
  
  def create_find_all_statement
    "select * from #{instance.prefix unless instance.prefix.nil?}#{instance.name.to_s} #{where.to_sql}"
  end
end

Using the tests from the last example only one test will fail. This is because the test_where_to_string_is_appended_to_end test method only cares about the interaction with the where stub. A quick fix has both our tests running again. Again, the quick fix was isolated to the first test.

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    builder = SqlStatementBuilder.new(stub(:name => 'Foo', :prefix => 't.'), stub_everything)
    assert_equal "select * from t.Foo ", builder.create_find_all_statement
  end
  
  def test_where_to_string_is_appended_to_end
    builder = SqlStatementBuilder.new(stub_everything, stub(:to_sql => 'where ...'))
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

A closing quick tip: The stub_everything method also takes a hash.

def test_stub_everything
  a_stub = stub_everything(:int => 1, :str => "string")
  a_stub.int            # => 1
  a_stub.str            # => "string"
  a_stub.anything_else  # => nil
end

Utilizing a stub_everything with a hash you can create stubs that return values you care about and nil for everything else. Used correctly this can also increase the robustness of your test suite.

Testing: When are Unit tests enough?

I recently received the question: When are Unit tests enough. More specifically, when is it unnecessary to create a Functional test.

I think the case is rare, but I believe it's when the method under test executes entirely in isolation. Put another way, if the method under test doesn't interact in any way with dependent classes or instances, it may be sufficient to Unit test it exclusively.

An example of a method that can execute entirely in isolation is the humanize method defined in rails.

  def humanize(lower_case_and_underscored_word)
    lower_case_and_underscored_word.to_s.gsub(/_id$/, "").gsub(/_/, " ").capitalize
  end

The humanize method is so focused that it manipulates a string and returns the result. I see no need in Functional testing a method that executes at this level of isolation.

Again, I do believe this is rarely the case. Much more often methods interact with instances of other classes (dependencies). Any time you are interacting with dependencies it's important to verify the integration between classes.

Testing: Unit Test or Functional Test

Update at bottom.

A common question when adding a new class is: Do I need a Unit test or Functional test?
The answer: Yes.

I'm not sure why, but there seems to be a common misconception that if you have a Unit test a Functional test is not necessary. This simply is not true.

I believe it may stem from the days where we only had one test suite. In those days the entire suite was essentially a Functional test suite. However, we've learned many lessons since then, and some of those lessons have allowed us to create an additional test suite for testing the units of our system in isolation. And, while it's still true that sometimes you only need a Functional test, it's rarely true that you will only want a Unit test.

I love to talk about different strategies concerning Unit Testing. And, often during those discussions holes are pointed out where the unit tests will not fail despite non-functioning code. These holes appear at the class integration points, which are not exposed when testing classes in isolation. This is why creating Functional Tests is crucial to test driven development.

So, if we must write Functional tests anyway, what's the use in having Unit tests? I like to create Unit tests to verify edge cases and slightly different code paths. For example, imagine a phone number class that contains a method for returning formatted string representations of the phone number.

class PhoneNumber < ActiveRecord::Base
  def to_formatted_s type
    case type
      when :en then ...
      when :fr then ...
    end
  end
end

The tests for the to_formatted_s method can be Unit tests that do not hit the database. It's possible to test all the scenarios required by to_formatted_s without the overhead of hitting the database. Avoiding trips to the database wont make a big difference if your entire test suite is 50 tests; however, as test suites grow, often so does the time to execute them.

If I were testing the PhoneNumber class I would create Unit tests for each logical path through the to_formatted_s method and I would create one Functional test that tested the same method, but actually used an instance of the PhoneNumber class that interacted with the database. This strategy would give me many Unit tests that quickly test the method and one Functional test that runs slower, but ensures integration is valid. Working this way should allow me to create a large test suite without creating a long running test suite.

You know you are in a bad situation when you don't want to run the tests because they take too long to execute. If you find yourself in this position, creating a test suite that cuts out trips to external dependencies (such as dbs or web services) is probably a good choice. The trips to the external dependencies are still required, but can be limited by testing as much logic as possible in the Unit tests and only hitting the external dependencies in the Functional tests.

Update
I got this comment that I thought was valuable enough it was worth adding to the post.

Erock said...

Hmm, maybe I don't quite understand, but why are functional tests needed in this case? You've tested all cases in your Unit tests, but then decide that to feel comfortable with your tests, you MUST hit the DB? Wouldn't it be ideal not to hit the db at all? How is a functional test going to give you better results then isolated unit tests? I was interested in this post, but you left more to the imagination than your usual very well thought out posts.

Sorry, I should have been more specific. The method under test, to_formatted_s, is going to format the data from from the area_code, exchange, and station attributes. Those attributes come from the columns that are defined on the phone_numbers table. Therefore, since the to_formatted_s method depends on attributes that are created on the database side it's a good idea to functionally test that the data coming from those columns is formatted correctly. This can prevent you from changing a column name in the database and not having a test that fails because your unit tests stub all the column names.

However, if the attributes of the phone number class didn't come from the database, then I wouldn't necessarily need a functional test for the to_formatted_s method.

The more common case is where methods aren't so simple that they can execute in isolation. Often methods interact with other classes, and those classes should be mocked/stubbed in Unit tests. But those methods should also be tested in a Functional test while collaborating to ensure proper integration.

Testing: Replace Collaborator with Stub

If you've done much testing I'm sure you've been annoyed by cascading failures. Cascading failures are most often a symptom of tests that rely on concrete classes instead of isolating the class under test. A solution for creating more robust tests is to replace the collaborators with stubs.

Example: Testing the AccountInformationPresenter independent of the Account class.

The prefactored test involves testing the presenter by creating the Account ActiveRecord instance and testing that the presenter correctly formats the phone number of the instance.

class AccountInformationPresenterTest < Test::Unit::TestCase
  def test_phone_number_is_formatted_correctly
    Account.delete_all
    Account.create(:phone_number => '1112223333')
    account = Account.find :first
    presenter = AccountInformationPresenter.new(account.id)
    assert_equal "(111) 222-3333", presenter.phone_number
  end
end

The potential issue with the above test is that if the Account class changes this test could fail. For example, if a validation is added to Account that verifies that the area code of the phone number is valid the above test will fail despite the fact that nothing about the presenter changed.

The same behavior of the presenter can be tested without relying on an actual instance of the Account class. The following test takes advantage of the Mocha mocking and stubbing framework. The new version of the test mocks the interaction with the Account class and returns a stub instead of an actual account instance.

class AccountInformationPresenterTest < Test::Unit::TestCase
  def test_phone_number_is_formatted_correctly
    Account.expects(:find).returns(stub(:phone_number => '1112223333'))
    presenter = AccountInformationPresenter.new(:any_number)
    assert_equal "(111) 222-3333", presenter.phone_number
  end
end

The new test will not fail regardless of any changes to the Account class.

Where the refactoring applies
This refactoring is most often applicable to unit tests. The unit test suite is often a good place to test edge cases and various logic paths. However, there is much value in testing collaboration among the various classes of your codebase. For these reasons you wouldn't want to apply this refactoring to your functional tests. If you do apply this refactoring, you should ensure you have proper coverage in the functional suite.

The code for the AccountInformationPresenter remains the same regardless of test implementation and is only shown for completeness.

class AccountInformationPresenter
  def initialize(account_id)
    @account = Account.find(account_id)
  end
  
  def phone_number
    @account.phone_number.gsub(/(\d{3})(\d{3})(\d{4})/, '(\1) \2-\3')
  end
end

Testing: Refactoring Examples

I recently gave a talk at RailsConf 2007 about testing. During the talk I put several ideas on the table and encouraged the audience to ask questions and challenge my ideas. I thought the presentation went well, but I think examples of each idea could provide a clearer picture. Look for more entries tagged with "Testing Refactorings" in the future.

Alternatively, this link should give the full list: http://blog.jayfields.com/search/label/Testing%20Refactorings

TextMate: Required Reading

If you spend much time developing in TextMate I'd highly suggest picking up the TextMate book. I've gotten through about 25% so far and already picked up several things what will make me more efficient.

Here's a quick tip to entice you: Reformat a file
- Command + A (Select All)
- Alt + Command + [ (Reindent Selected)

Ruby: method_missing alternatives

I love the value that method_missing provides; unfortunately, sometimes I get into trouble when I abuse the power Matz has bestowed upon me. If you've done much work with method_missing the following error message probably isn't foreign to you.

SystemStackError: stack level too deep

It's a painful message to see, as it's always problematic to debug. Given the inherent difficultly of debugging method_missing calls I thought it might be valuable to offer a few alternatives.

As an example I'll use the valid_for_[a group] methods that Validatable exposes. The Validatable validations allow you to put your validations into groups.

class User
  include Validatable
  
  attr_accessor :first_name, :last_name, :ssn
  validates_presence_of :first_name, :last_name, :ssn, :groups => :account_creation
  validates_presence_of :ssn, :groups => :persistence
end

Given the above User class you can expect the following behavior.

irb(main):010:0> user = User.new
=> #<User:0x696900>
irb(main):011:0> user.ssn = 111223333
=> 111223333
irb(main):012:0> user.valid_for_persistence?
=> true
irb(main):013:0> user.valid_for_account_creation?
=> false
irb(main):014:0> user.first_name = 'Shane'
=> "Shane"
irb(main):015:0> user.last_name = 'Harvie'
=> "Harvie"
irb(main):016:0> user.valid_for_persistence?
=> true
irb(main):017:0> user.valid_for_account_creation?
=> true

As you can see the User instance exposes the valid_for_account_creation? and the valid_for_persistence? methods. Of course, we want the valid_for_[a group] methods to change based on the groups defined in each class. Additionally, we only want valid_for_something? to be defined on a class if that class contains any validations in the something group. This looks like a classic situation for using method_missing; however, I chose one of the following other solutions.

Delegate Responsibility:
The basic idea behind this solution alternative is to move the method_missing logic into a class whose sole responsibility is the method_missing logic. For example, had I chosen this solution for Validatable I would have made the valid methods work like the following example.

irb(main):010:0> user = User.new
=> #<User:0x696900>
irb(main):011:0> user.ssn = 111223333
=> 111223333
irb(main):012:0> user.valid_for.persistence?
=> true
irb(main):013:0> user.valid_for.account_creation?
=> false

Given the above syntax I could have created a ValidFor class that encapsulated the logic for dynamically executing a subset of validations. A few cons for this solution are the obvious violation of the Law of Demeter and the fact that the stack overflow can still exist in the delegate class. However, I like to use this solution when the methods that are necessary are truly dynamic and cannot be easily predicted (i.e. the find_by_* methods that ActiveRecord::Base exposes). Additionally, the added class removes variables when trying to figure out where the logic went wrong.

Metaprogramming:
I chose this solution because I was able to easily identify which methods were necessary and define then at interpretation time. In fact the implementation for defining the valid methods is quite simple.

def create_valid_method_for_groups(groups) #:nodoc:
  groups.each do |group|
    self.class_eval do
      define_method "valid_for_#{group}?".to_sym do
        valid_for_group?(group)
      end
    end
  end
end

Given the above code a class will define a new valid_for_[a group] method for each group given to any of it's validations. By explicitly defining each method I get much more manageable error messages when things do go wrong.

RailsConf07: Testing Presentation

I don't have a .key (or .ppt) to post from my RailsConf 2007 talk. In fact, the talk was much more of a discussion than a presentation. However, I did go back and tag all the previous entries that contained discussion starting points: http://blog.jayfields.com/search/label/RailsConf2007

Law of Demeter: Canary in a coal mine

I've always been a fan of the Law of Demeter. The Law of Demeter is most often summarized as “Only talk to your immediate friends.”

The fundamental notion is that a given object should assume as little as possible about the structure or properties of anything else (including its subcomponents).
Wikipedia

That's all well and good, but I quite like the Law of Demeter because it's the canary (warning device) in my coal mine (codebase).

I was recently looking over some code with Mike Ward, Shane Harvie, Muness Alrubaie, Martin Fowler, Dan Manges, and David Vollbracht. The code in question had around a hundred lines of def_delegator :object, method_name, new_method_name. No one in the room thought that the 100+ lines of delegation were a good thing, but we all had opinions on how to deal with the ugly code.

Despite the various opinions on solutions, one thing was clear: The Law of Demeter exposed other issues within the codebase. For example, the object in question was taking on far too much responsibility. It had become the DoAnything object of the system. Additionally, methods such as diamond.clarity= were being delegated from ring via the ring.diamond_clarity= method. These methods were required as the attributes of a diamond were changed. However, a possibly better solution would be a method such as ring.replace_diamond that set an entirely new diamond. With the replace_diamond method in place the ring no longer needed to expose the setters of the diamond.

An additional note that came from this conversation: If you are a mockist you will often prefer to follow the Law of Demeter because it makes mocking much easier. However, classicists don't seem to mind as much since they can use train-wreck syntax for state based tests without any pain. Being a mockist or classicist seems to greatly influence a person's opinion on whether or not to follow the Law of Demeter.

Ruby: Unit testing delegations

When working with Rails I tend to use a lot of delegations. Since I'm a big fan of the Law of Demeter I often find myself delegating an_ad_instance.lister_name to the name method of the lister attribute.

Class Ad < ActiveRecord::Base
  extend Forwardable
  def_delegator :lister, name, lister_name
  def lister
    ...
  end
end

Often, I end up writing tests for something even as simple as delegation, since I try to TDD all the code I write.

class AdTest < Test::Unit::TestCase
  def test_lister_name_is_delegated_to_listers_name
    ad = Ad.new(:lister => Lister.new(:name => 'lister_name'))
    assert_equal 'lister_name', ad.lister_name
  end
end

After a few of these tests, the duplication begins to tell me that I can do something simpler with some metaprogramming. So, I stole the idea from Validatable and created some delegation custom assertions.

class AdTest < Test::Unit::TestCase
  Ad.delegates do
    lister_name.to(:lister).via(:name)
    lister_address.to(:lister).via(:address)
    ...
  end
end

The above delegation testing DSL allows me to test multiple validations without the need for many (very similar) tests.

The implementation to get this working is fairly straightforward: I create a subclass of the class under test, add a constructor to the subclass that takes the objec tthat's going to be delegated to, and set the value of the attribute on the delegate object. I also add an attr_accessor to the subclass, since all I care about is the delegation (for this test). It doesn't matter to me that I'm changing the implementation of the lister method because I'm testing the delegation, not the lister method.

class DelegateAssertion
  attr_accessor :desired_method, :delegate_object, :original_method
  
  def initialize(desired_method)
    self.desired_method = desired_method
  end
  
  def to(delegate_object) 
    self.delegate_object = delegate_object
    self
  end
  
  def via(original_method)
    self.original_method = original_method
  end
end

class DelegateCollector
  def self.gather(block) 
    collector = new
    collector.instance_eval(&block)
    collector.delegate_assertions
  end
  
  attr_accessor :delegate_assertions
  
  def delegate_assertions
    @delegate_assertions ||= [] 
  end
  
  def method_missing(sym, *args)
    assertion = DelegateAssertion.new(sym)
    delegate_assertions << assertion
    assertion
  end
end

class Class
  def delegates(&block) 
    test_class = eval "self", block.binding
    assertions = DelegateCollector.gather(block)
    assertions.each do |assertion|
      klass = Class.new(self)
      klass.class_eval do
        attr_accessor assertion.delegate_object
        define_method :initialize do |delegate_object|
          self.send :"#{assertion.delegate_object}=", delegate_object
        end
      end
      test_class.class_eval do 
        define_method "test_#{assertion.desired_method}_is_delegated_to_#{assertion.delegate_object}_via_#{assertion.original_method}" do
          klass_instance = klass.new(stub(assertion.original_method => :original_value))
          begin
            assert_equal :original_value, klass_instance.send(assertion.desired_method)
          rescue Exception => ex
            add_failure "Delegating #{assertion.desired_method } to #{assertion.delegate_object} via #{assertion.original_method} doesn't work"
          end
        end
      end
    end
  end
end

The resulting tests make it hard to justify leaving of tests, even if you are 'only doing simple delegation'.

note: In the example I use an ActiveRecord object, but since I'm adding this behavior to Class any class can take advantage of these custom validations.

RailsConf 2007 Presentation

It looks like I'll be attending and speaking at RailsConf 2007.

My talk is going to cover testing lessons learned from various Ruby/Rails projects. More info is available on the official website.

See you in Portland.

Ruby: Validatable Tests

In the comments for my most recent entry, Validatable 1.2.2 released, Jon Leighton asks: how do you recommend testing validations?

I've had this discussion before when I wrote about testing ActiveRecord validations. I've not been able to come up with a 'best practice', but I've had some fun trying out different solutions.

My latest attempt at a solution alternative is to create assertions for the framework. For example, if I were to create a Validatable framework, that framework would include ValidatableAssertions.

To create validations for the Validatable framework I wrote my ideal syntax and then figured out how to make it work. The following code was the original version for validating that a class contained a presence of validation.

class FooTest < Test::Unit::TestCase
  Foo.must_validate.presence_of :name
end

I originally chose the above syntax because it was expressive enough to convey my intent in the code. I spent a bit of time making this syntax work, but in the end I went with a syntax that I found a bit drier, and easier to implement.

class FooTest < Test::Unit::TestCase
  include ValidatableAssertions
  
  Foo.must_validate do
    presence_of :name
    format_of(:name).with(/^[A-Z]/)
    numericality_of(:age).only_integer(true)
  end
end

The above code creates a test for each line in the block given to must_validate. If the Foo class does not contain a presence of validation for name, an error with the text "Foo does not contain a Validatable::ValidatesPresenceOf for name" will be raised.

Clearly this solution has limitations. Any validates_true_for validation cannot be tested using this DSL style of testing. Furthermore, any validation that uses an :if argument cannot use this DSL, since those validations require an instance to eval the :if argument against. However, for validations that are not validates_true_for and do not rely on an :if argument, the ValidatableAssertions can replace various existing success and failure validation tests.

If you hate this idea, no worries, you can always test your classes the traditional way: creating an instance and calling the valid? method. This DSL is another tool that may or may not help you out based on the context in which you use it.

The ValidatableAssertions are available in gem version 1.3.0 and greater.