Clojure: Mocking

An introduction to clojure.test is easy, but it doesn't take long before you feel like you need a mocking framework. As far as I know, you have 3 options.

1. Take a look at Midje. I haven't gone down this path, but it looks like the most mature option if you're looking for a sophisticated solution.
   


2. Go simple. Let's take an example where you want to call a function that computes a value and sends a response to a gateway. Your first implementation looks like the code below. (destructuring explained)

   (defn withdraw [& {:keys [balance withdrawal account-number]}]
    (gateway/process {:balance (- balance withdrawal)
                      :withdrawal withdrawal
                      :account-number account-number}))

   No, it's not pure. That's not the point. Let's pretend that this impure function is the right design and focus on how we would test it.
   
   You can change the code a bit and pass in the gateway/process function as an argument. Once you've changed how the code works you can test it by passing identity as the function argument in your tests. The full example is below.

   (ns gateway)
   
   (defn process [m] (println m))
   
   (ns controller
    (:use clojure.test))
   
   (defn withdraw [f & {:keys [balance withdrawal account-number]}]
    (f {:balance (- balance withdrawal)
        :withdrawal withdrawal
        :account-number account-number}))
   
   (withdraw gateway/process :balance 100 :withdrawal 22 :account-number 4)
   ;; => {:balance 78, :withdrawal 22, :account-number 4}
   
   (deftest withdraw-test
    (is (= {:balance 78, :withdrawal 22, :account-number 4}
     (withdraw identity :balance 100 :withdrawal 22 :account-number 4))))
   
   (run-all-tests #"controller")

   If you run the previous example you will see the println output and the clojure.test output, verifying that our code is working as we expected. This simple solution of passing in your side effect function and using identity in your tests can often obviate any need for a mock.
   


3. Solution 2 works well, but has the limitations that only one side-effecty function can be passed in and it's result must be used as the return value.
   
   Let's extend our example and say that we want to log a message if the withdrawal would cause insufficient funds. (Our gateway/process and log/write functions will simply println since this is only an example, but in production code their behavior would differ and both would be required)

   (ns gateway)
   
   (defn process [m] (println "gateway: " m))
   
   (ns log)
   
   (defn write [m] (println "log: " m))
   
   (ns controller
     (:use clojure.test))
   
   (defn withdraw [& {:keys [balance withdrawal account-number]}]
     (let [new-balance (- balance withdrawal)]
       (if (> 0 new-balance)
         (log/write "insufficient funds")
         (gateway/process {:balance new-balance
                           :withdrawal withdrawal
                           :account-number account-number}))))
   
   (withdraw :balance 100 :withdrawal 22 :account-number 4)
   ;; => gateway:  {:balance 78, :withdrawal 22, :account-number 4}
   
   (withdraw :balance 100 :withdrawal 220 :account-number 4)
   ;; => log:  insufficient funds

   Our new withdraw implementation calls two functions that have side effects. We could pass in both functions, but that solution doesn't seem to scale very well as the number of passed functions grows. Also, passing in multiple functions tends to clutter the signature and make it hard to remember what is the valid order for the arguments. Finally, if we need withdraw to always return a map showing the balance and withdrawal amount, there would be no easy solution for verifying the string sent to log/write.
   
   Given our implementation of withdraw, writing a test that verifies that gateway/process and log/write are called correctly looks like a job for a mock. However, thanks to Clojure's binding function, it's very easy to redefine both of those functions to capture values that can later be tested.
   
   The following code rebinds both gateway/process and log/write to partial functions that capture whatever is passed to them in an atom that can easily be verified directly in the test.

   (ns gateway)
   
   (defn process [m] (println "gateway: " m))
   
   (ns log)
   
   (defn write [m] (println "log: " m))
   
   (ns controller
     (:use clojure.test))
   
   (defn withdraw [& {:keys [balance withdrawal account-number]}]
     (let [new-balance (- balance withdrawal)]
       (if (> 0 new-balance)
         (log/write "insufficient funds")
         (gateway/process {:balance new-balance
                           :withdrawal withdrawal
                           :account-number account-number}))))
   
   (deftest withdraw-test1
     (let [result (atom nil)]
       (binding [gateway/process (partial reset! result)]
         (withdraw :balance 100 :withdrawal 22 :account-number 4)
         (is (= {:balance 78, :withdrawal 22, :account-number 4} @result)))))
   
   (deftest withdraw-test2
     (let [result (atom nil)]
       (binding [log/write (partial reset! result)]
         (withdraw :balance 100 :withdrawal 220 :account-number 4)
         (is (= "insufficient funds" @result)))))
   
   (run-all-tests #"controller")

In general I use option 2 when I can get away with it, and option 3 where necessary. Option 3 adds enough additional code that I'd probably look into Midje quickly if I found myself writing a more than a few tests that way. However, I generally go out of my way to design pure functions, and I don't find myself needing either of these techniques very often.

Testing: One expectation per test

In a previous entry I discussed why I prefer One Assertion Per Test. In that entry I give a few state based examples, but didn't address the topic of mocks. This entry is going to focus on how you can use mocks while maintaining focus on maintainability.

Let's start with an example.

require 'test/unit'
require 'rubygems'
require 'mocha'

module Expectations
  class SuiteRunner
    attr_accessor :suite
  
    def initialize
      self.suite = Expectations::Suite.new
    end
  
    def self.suite_eval(&block)
      self.new.suite.instance_eval &block
    end
  end
end

class ObjectTests < Test::Unit::TestCase
  def test_suite_eval_evals_the_block_in_the_context_of_the_suite
    suite = mock
    suite.expects(:instance_eval)
    runner = mock
    runner.expects(:suite).returns(suite)
    Expectations::SuiteRunner.expects(:new).returns runner
    Expectations::SuiteRunner.suite_eval {}
  end
end

The SuiteRunner class is fairly straightforward, it's simply delegating the block on to the instance_eval method on the suite attribute, which is an instance of the Suite class. Even though the class isn't doing anything very interesting the test isn't completely simple. Due to the chain of method calls you need to set up 2 mocks and a total of 3 expectations. While the test isn't unmaintainable, if the description (the method name) isn't kept up to date you could easily lose the intent of the test.

A step in the right direction toward making this test more readable is to introduce a few stubs. However, before we decide what to stub we need to decide what this test is trying to verify. Given the description it seems that eval'ing in the context of the suite is what is being tested, thus the suite mock should probably remain. Since the suite mock is the focus of the test, the rest of the mocks are probably better expressed as stubs. The test can be written more concisely now that we've chosen to use stubs.

class ObjectTests < Test::Unit::TestCase
  def test_suite_eval_evals_the_block_in_the_context_of_the_suite
    suite = mock
    suite.expects(:instance_eval)
    Expectations::SuiteRunner.stubs(:new).returns stub(:suite => suite)
    Expectations::SuiteRunner.suite_eval {}
  end
end

With one mock and one expectation the test expresses it's purpose. Code should not only express how it works, but also why it's been written in a particular way. The new test not only executes, but it conveys the intent of the test (even if the description becomes stale). Converting that test was easy, but tests that have behavior expectations and state based assertions can be a bit more complicated to clean up.

require 'test/unit'
require 'rubygems'
require 'mocha'

class ReservationService
  # implementation
end
class MaidService
  # implementation
end
class VipService
  # implementation
end

class HotelRoom
  attr_accessor :booked
  def book_for(customer)
    reservation = ReservationService.reserve_for(customer)
    self.booked = true
    MaidService.notify(reservation)
    VipService.notify(reservation) if reservation.for_vip?
    reservation.confirmation_number
  end
end

class HotelRoomTests < Test::Unit::TestCase
  def test_book_for_returns_confirmation_number
    customer = mock
    room = HotelRoom.new
    reservation = mock
    reservation.expects(:for_vip?).returns true
    reservation.expects(:confirmation_number).returns 1979
    ReservationService.expects(:reserve_for).with(customer).returns(reservation)
    MaidService.expects(:notify).with(reservation)
    VipService.expects(:notify).with(reservation)
    assert_equal 1979, room.book_for(customer)
    assert_equal true, room.booked
  end
end
# >> Loaded suite -
# >> Started
# >> .
# >> Finished in 0.001121 seconds.
# >> 
# >> 1 tests, 7 assertions, 0 failures, 0 errors

There's a lot going on in this test. Test::Unit even reports that 7 assertions have been met. Unfortunately, with that much going on in a test, it's hard to tell what the original intent of the test was. Even worse, it would be hard to change anything in the book_for method without breaking this test. The test is completely tied to the implementation and if that implementation changes you'll break one or more of the assertions within the test. What really bothers me about this test is the ability to break "one or more" assertions. If you change the reserve_for method to also take the hotel as an argument the test will immediately stop executing with the following error.

# >> Loaded suite -
# >> Started
# >> F
# >> Finished in 0.001728 seconds.
# >> 
# >>   1) Failure:
# >> test_book_for_returns_confirmation_number(HotelRoomTests)
# >>     [(eval):1:in `reserve_for'
# >>      -:18:in `book_for'
# >>      -:36:in `test_book_for_returns_confirmation_number']:
# >> #<Mock:0x50f2a8>.reserve_for(#<Mock:0x50fa50>, #<HotelRoom:0x50fadc>) - expected calls: 0, actual calls: 1
# >> Similar expectations:
# >> #<Mock:0x50f2a8>.reserve_for(#<Mock:0x50fa50>)
# >> 
# >> 1 tests, 0 assertions, 1 failures, 0 errors

Knowing that an assertion failed is good, knowing that other assertions might also fail when you fix the first problem is not so good. The problem with this test is that it's verifying 7 different things. This can most likely be resolved by writing several different tests that all focus on one thing. (There are several tests that can be written, here are 5 examples)

require 'test/unit'
require 'rubygems'
require 'mocha'

class ReservationService
  # implementation
end
class MaidService
  # implementation
end
class VipService
  # implementation
end

class HotelRoom
  attr_accessor :booked
  def book_for(customer)
    reservation = ReservationService.reserve_for(customer, self)
    self.booked = true
    MaidService.notify(reservation)
    VipService.notify(reservation) if reservation.for_vip?
    reservation.confirmation_number
  end
end

class HotelRoomTests < Test::Unit::TestCase
  def test_book_for_reserves_via_ReservationService
    room = HotelRoom.new
    ReservationService.expects(:reserve_for).with(:customer, room).returns(stub_everything)
    MaidService.stubs(:notify)
    room.book_for(:customer)
  end
  
  def test_book_for_notifys_MaidService
    reservation = stub_everything
    MaidService.expects(:notify).with(reservation)
    ReservationService.stubs(:reserve_for).returns(reservation)
    HotelRoom.new.book_for(:customer)
  end
  
  def test_book_for_notifys_VipService_if_reservation_if_for_vip
    reservation = stub_everything(:for_vip? => true)
    VipService.expects(:notify).with(reservation)
    MaidService.stubs(:notify)
    ReservationService.stubs(:reserve_for).returns(reservation)
    HotelRoom.new.book_for(:customer)
  end
  
  def test_book_for_sets_booked_to_true
    room = HotelRoom.new
    MaidService.stubs(:notify)
    ReservationService.stubs(:reserve_for).returns(stub_everything)
    room.book_for(:customer)
    assert_equal true, room.booked
  end
  
  def test_book_for_returns_confirmation_number
    reservation = stub_everything
    reservation.stubs(:confirmation_number).returns 1979
    ReservationService.stubs(:reserve_for).returns(reservation)
    MaidService.stubs(:notify)
    assert_equal 1979, HotelRoom.new.book_for(:customer)
  end
end
# >> Loaded suite -
# >> Started
# >> .....
# >> Finished in 0.002766 seconds.
# >> 
# >> 5 tests, 5 assertions, 0 failures, 0 errors

The above tests take more lines of code than the original, but they are far more maintainable. You can change the implementation of the book_for method in various ways and only break the tests that are relevant to the change. The tests have become more robust. They've also become more readable because they express what their focus is. The tests that have a mock with an expectation are written to test a behavior that is expected. The tests that have an assertion are written to statefully verify that the object has been set as expected. When the implementation does inevitably change the tests can communicate on their own what the original author intended.

Writing tests in this way is actually quite easy if you follow a few simple suggestions.

• If your test has an assertion, do not add any mock expectations (instead use stubs for any methods that need to be changed for the test).
• If you add a mock expectation you probably don't need it to return a meaningful value (since you won't be verifying that value).
• If you already have a mock expectation you should use the stubs method for any other method where you don't want to use the implementation.
• Do not use .with when using a stub unless you need to. If you are stubbing, you don't need to verify the arguments.
• When returning stubs, prefer stub_everything so that additional calls to the stub will not cause unnecessary exceptions to be raised.

The above suggestions focus on creating tests that assert (or expect) one thing at a time and specify as little other implementation as possible. Every expectation, .with, and .returns statement is specifying implementation. The less implementation you express in a test, the more that test becomes.

I consider each call to stub (or stub_everything), stubs, mock, expects, with, and returns to be 1 implementation specification point. Any test that has an implementation specification score of 5 or more has the high implementation specification smell. As with all smells, everything might be okay, but I'm going to take the extra time to see if anything can be broken up by moving responsibilities or creating smaller methods.

By limiting your tests to one assertion or one expectation you create tests that express their intent. By creating tests that specify as little implementation as possible you reduce the amount of noise taking away from the intent of the test. An additional consequence of creating tests that focus on intent is that they specify less and are therefore more robust.

Ruby: Mocks and Stubs using Mocha

Update: Older versions of Mocha didn't warn when a stub was never called. The newer versions will; therefore, it makes more sense to prefer stub since it's less fragile and more intention revealing. I no longer feel as I did when I wrote this entry, but I've left it for historical reasons. Related reading here.

I've previously written about using Mocks and Stubs convey intent. When I was using C#, I believed this was the best solution for creating robust tests. These days, all my code is written in Ruby. Making the switch to Ruby provided another example that reinforces an assertion I've heard before: Best Practices are so context dependent it's dangerous to use the term.

Here's the reason I no longer feel as I did when I created the above entry: When using Mocha to mock or stub behavior, I can't think of a reason I would ever want to use SomeObject.stubs(..) instead of SomeObject.expects(..). The closest I could come to a reason was that stubs will allow an arbitrary number of calls to the same method. However, I don't believe that's a good enough reason since I can also use SomeObject.expects(:a_method).at_least_once.

The problem with using SomeObject.stubs is that it's almost the same as using SomeObject.expects, except if it's no longer necessary it doesn't cause a test to fail. This can lead to tests that unnecessarily stub methods as the application's implementation changes. And, the more methods that require stubbing the less the test can concisely convey intent.

I'm not asserting that you should never use SomeObject.stubs, but I do believe it's smarter to favor SomeObject.expects if you are concerned with keeping your test suite well maintained.

Of course, none of this is related to creating actual stubs for your tests. I still believe it's wise to create stub object instances for classes that are required for a unit test, but are not the class under test. For example, if I were testing a mapper class, I would use a stub for the objecting being mapped.

test "an order's id is mapped to the request object" do
  request = OrderServiceMapper.create_request(stub(:id=>1, ...))
  assert_equal 1, request.order_id
end

I use a stub in the above example because I don't want to couple the test to the implementation of an order. I believe this creates a more robust, maintainable test