Testing: Replace Mock with Stub

I often hear the phrase: I remember when I thought Mocks were the silver bullet. In fact, I remember using Mocks like they were the silver bullet. Unfortunately, using Mocks excessively can lead to a test suite that is just as brittle, if not more brittle than using concrete classes as dependencies.

One of the big reasons I began using Mocks in tests was to avoid cascading failures. However, I quickly learned that cascading failures still occur when using mocks, just in another flavor. Any implementation change required each instance of the mock to be updated to reflect the new implementation. My "solution" was just as bad as my problem.

To resolve this issue I started relying on Stubs. In fact, I found that using both Mocks and Stubs together made my tests more expressive. When I wrote that entry I was using C# in my day job; however, I think the concept transcends languages.

I'm going to use an example similar to the one from the original entry, this time in Ruby. It's completely acceptable to write the test using only mocks. (Mocha used for mocking)

class SqlStatementBuilder
  attr_reader :instance, :where
  def initialize(instance, where)
    @instance, @where = instance, where
  end
  
  def create_find_all_statement
    "select * from #{instance.name.to_s} #{where.to_sql}"
  end
end

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    instance = mock
    instance.expects(:name).returns('Foo')
    where = mock
    where.expects(:to_sql).returns("")
    builder = SqlStatementBuilder.new(instance, where)
    assert_equal "select * from Foo ", builder.create_find_all_statement
  end
  
  def test_where_to_string_is_appended_to_end
    instance = mock
    instance.expects(:name).returns('')
    where = mock
    where.expects(:to_sql).returns("where ...")
    builder = SqlStatementBuilder.new(instance, where)
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

While using mocks does verify the behavior of the builder class, that verification is superfluous. In fact, I believe the above test is covering too much. The state based assertion (assert_equals ...) at the end is all that's really necessary. The mocks can be replaced with stubs since the state based assertion is sufficient.

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    builder = SqlStatementBuilder.new(stub(:name => 'Foo'), stub(:to_sql => ''))
    assert_equal "select * from Foo ", builder.create_find_all_statement
  end

  def test_where_to_string_is_appended_to_end
    builder = SqlStatementBuilder.new(stub(:name => ''), stub(:to_sql => 'where ...'))
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

Looking at the above example your first instinct might be to combine the tests. An alternative and possibly superior solution is to continue to test the interactions independently, but only pass in stubs that are necessary for each individual interaction. This is where the stub_everything method from Mocha becomes very handy.

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    builder = SqlStatementBuilder.new(stub(:name => 'Foo'), stub_everything)
    assert_equal "select * from Foo ", builder.create_find_all_statement
  end

  def test_where_to_string_is_appended_to_end
    builder = SqlStatementBuilder.new(stub_everything, stub(:to_sql => 'where ...'))
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

Now imagine that the SqlStatementBuilder needs to be changed to allow a prefix for the table name.

class SqlStatementBuilder
  attr_reader :instance, :where
  def initialize(instance, where)
    @instance, @where = instance, where
  end
  
  def create_find_all_statement
    "select * from #{instance.prefix unless instance.prefix.nil?}#{instance.name.to_s} #{where.to_sql}"
  end
end

Using the tests from the last example only one test will fail. This is because the test_where_to_string_is_appended_to_end test method only cares about the interaction with the where stub. A quick fix has both our tests running again. Again, the quick fix was isolated to the first test.

class SqlStatementBuilderTests < Test::Unit::TestCase
  def test_instance_name_is_used_as_table_name
    builder = SqlStatementBuilder.new(stub(:name => 'Foo', :prefix => 't.'), stub_everything)
    assert_equal "select * from t.Foo ", builder.create_find_all_statement
  end
  
  def test_where_to_string_is_appended_to_end
    builder = SqlStatementBuilder.new(stub_everything, stub(:to_sql => 'where ...'))
    assert_equal "select * from  where ...", builder.create_find_all_statement
  end
end

A closing quick tip: The stub_everything method also takes a hash.

def test_stub_everything
  a_stub = stub_everything(:int => 1, :str => "string")
  a_stub.int            # => 1
  a_stub.str            # => "string"
  a_stub.anything_else  # => nil
end

Utilizing a stub_everything with a hash you can create stubs that return values you care about and nil for everything else. Used correctly this can also increase the robustness of your test suite.

Testing: When are Unit tests enough?

I recently received the question: When are Unit tests enough. More specifically, when is it unnecessary to create a Functional test.

I think the case is rare, but I believe it's when the method under test executes entirely in isolation. Put another way, if the method under test doesn't interact in any way with dependent classes or instances, it may be sufficient to Unit test it exclusively.

An example of a method that can execute entirely in isolation is the humanize method defined in rails.

  def humanize(lower_case_and_underscored_word)
    lower_case_and_underscored_word.to_s.gsub(/_id$/, "").gsub(/_/, " ").capitalize
  end

The humanize method is so focused that it manipulates a string and returns the result. I see no need in Functional testing a method that executes at this level of isolation.

Again, I do believe this is rarely the case. Much more often methods interact with instances of other classes (dependencies). Any time you are interacting with dependencies it's important to verify the integration between classes.

Testing: Unit Test or Functional Test

Update at bottom.

A common question when adding a new class is: Do I need a Unit test or Functional test?
The answer: Yes.

I'm not sure why, but there seems to be a common misconception that if you have a Unit test a Functional test is not necessary. This simply is not true.

I believe it may stem from the days where we only had one test suite. In those days the entire suite was essentially a Functional test suite. However, we've learned many lessons since then, and some of those lessons have allowed us to create an additional test suite for testing the units of our system in isolation. And, while it's still true that sometimes you only need a Functional test, it's rarely true that you will only want a Unit test.

I love to talk about different strategies concerning Unit Testing. And, often during those discussions holes are pointed out where the unit tests will not fail despite non-functioning code. These holes appear at the class integration points, which are not exposed when testing classes in isolation. This is why creating Functional Tests is crucial to test driven development.

So, if we must write Functional tests anyway, what's the use in having Unit tests? I like to create Unit tests to verify edge cases and slightly different code paths. For example, imagine a phone number class that contains a method for returning formatted string representations of the phone number.

class PhoneNumber < ActiveRecord::Base
  def to_formatted_s type
    case type
      when :en then ...
      when :fr then ...
    end
  end
end

The tests for the to_formatted_s method can be Unit tests that do not hit the database. It's possible to test all the scenarios required by to_formatted_s without the overhead of hitting the database. Avoiding trips to the database wont make a big difference if your entire test suite is 50 tests; however, as test suites grow, often so does the time to execute them.

If I were testing the PhoneNumber class I would create Unit tests for each logical path through the to_formatted_s method and I would create one Functional test that tested the same method, but actually used an instance of the PhoneNumber class that interacted with the database. This strategy would give me many Unit tests that quickly test the method and one Functional test that runs slower, but ensures integration is valid. Working this way should allow me to create a large test suite without creating a long running test suite.

You know you are in a bad situation when you don't want to run the tests because they take too long to execute. If you find yourself in this position, creating a test suite that cuts out trips to external dependencies (such as dbs or web services) is probably a good choice. The trips to the external dependencies are still required, but can be limited by testing as much logic as possible in the Unit tests and only hitting the external dependencies in the Functional tests.

Update
I got this comment that I thought was valuable enough it was worth adding to the post.

Erock said...

Hmm, maybe I don't quite understand, but why are functional tests needed in this case? You've tested all cases in your Unit tests, but then decide that to feel comfortable with your tests, you MUST hit the DB? Wouldn't it be ideal not to hit the db at all? How is a functional test going to give you better results then isolated unit tests? I was interested in this post, but you left more to the imagination than your usual very well thought out posts.

Sorry, I should have been more specific. The method under test, to_formatted_s, is going to format the data from from the area_code, exchange, and station attributes. Those attributes come from the columns that are defined on the phone_numbers table. Therefore, since the to_formatted_s method depends on attributes that are created on the database side it's a good idea to functionally test that the data coming from those columns is formatted correctly. This can prevent you from changing a column name in the database and not having a test that fails because your unit tests stub all the column names.

However, if the attributes of the phone number class didn't come from the database, then I wouldn't necessarily need a functional test for the to_formatted_s method.

The more common case is where methods aren't so simple that they can execute in isolation. Often methods interact with other classes, and those classes should be mocked/stubbed in Unit tests. But those methods should also be tested in a Functional test while collaborating to ensure proper integration.

Testing: Replace Collaborator with Stub

If you've done much testing I'm sure you've been annoyed by cascading failures. Cascading failures are most often a symptom of tests that rely on concrete classes instead of isolating the class under test. A solution for creating more robust tests is to replace the collaborators with stubs.

Example: Testing the AccountInformationPresenter independent of the Account class.

The prefactored test involves testing the presenter by creating the Account ActiveRecord instance and testing that the presenter correctly formats the phone number of the instance.

class AccountInformationPresenterTest < Test::Unit::TestCase
  def test_phone_number_is_formatted_correctly
    Account.delete_all
    Account.create(:phone_number => '1112223333')
    account = Account.find :first
    presenter = AccountInformationPresenter.new(account.id)
    assert_equal "(111) 222-3333", presenter.phone_number
  end
end

The potential issue with the above test is that if the Account class changes this test could fail. For example, if a validation is added to Account that verifies that the area code of the phone number is valid the above test will fail despite the fact that nothing about the presenter changed.

The same behavior of the presenter can be tested without relying on an actual instance of the Account class. The following test takes advantage of the Mocha mocking and stubbing framework. The new version of the test mocks the interaction with the Account class and returns a stub instead of an actual account instance.

class AccountInformationPresenterTest < Test::Unit::TestCase
  def test_phone_number_is_formatted_correctly
    Account.expects(:find).returns(stub(:phone_number => '1112223333'))
    presenter = AccountInformationPresenter.new(:any_number)
    assert_equal "(111) 222-3333", presenter.phone_number
  end
end

The new test will not fail regardless of any changes to the Account class.

Where the refactoring applies
This refactoring is most often applicable to unit tests. The unit test suite is often a good place to test edge cases and various logic paths. However, there is much value in testing collaboration among the various classes of your codebase. For these reasons you wouldn't want to apply this refactoring to your functional tests. If you do apply this refactoring, you should ensure you have proper coverage in the functional suite.

The code for the AccountInformationPresenter remains the same regardless of test implementation and is only shown for completeness.

class AccountInformationPresenter
  def initialize(account_id)
    @account = Account.find(account_id)
  end
  
  def phone_number
    @account.phone_number.gsub(/(\d{3})(\d{3})(\d{4})/, '(\1) \2-\3')
  end
end

Testing: Refactoring Examples

I recently gave a talk at RailsConf 2007 about testing. During the talk I put several ideas on the table and encouraged the audience to ask questions and challenge my ideas. I thought the presentation went well, but I think examples of each idea could provide a clearer picture. Look for more entries tagged with "Testing Refactorings" in the future.

Alternatively, this link should give the full list: http://blog.jayfields.com/search/label/Testing%20Refactorings