Ruby: Validatable Tests

In the comments for my most recent entry, Validatable 1.2.2 released, Jon Leighton asks: how do you recommend testing validations?

I've had this discussion before when I wrote about testing ActiveRecord validations. I've not been able to come up with a 'best practice', but I've had some fun trying out different solutions.

My latest attempt at a solution alternative is to create assertions for the framework. For example, if I were to create a Validatable framework, that framework would include ValidatableAssertions.

To create validations for the Validatable framework I wrote my ideal syntax and then figured out how to make it work. The following code was the original version for validating that a class contained a presence of validation.

class FooTest < Test::Unit::TestCase
  Foo.must_validate.presence_of :name
end

I originally chose the above syntax because it was expressive enough to convey my intent in the code. I spent a bit of time making this syntax work, but in the end I went with a syntax that I found a bit drier, and easier to implement.

class FooTest < Test::Unit::TestCase
  include ValidatableAssertions
  
  Foo.must_validate do
    presence_of :name
    format_of(:name).with(/^[A-Z]/)
    numericality_of(:age).only_integer(true)
  end
end

The above code creates a test for each line in the block given to must_validate. If the Foo class does not contain a presence of validation for name, an error with the text "Foo does not contain a Validatable::ValidatesPresenceOf for name" will be raised.

Clearly this solution has limitations. Any validates_true_for validation cannot be tested using this DSL style of testing. Furthermore, any validation that uses an :if argument cannot use this DSL, since those validations require an instance to eval the :if argument against. However, for validations that are not validates_true_for and do not rely on an :if argument, the ValidatableAssertions can replace various existing success and failure validation tests.

If you hate this idea, no worries, you can always test your classes the traditional way: creating an instance and calling the valid? method. This DSL is another tool that may or may not help you out based on the context in which you use it.

The ValidatableAssertions are available in gem version 1.3.0 and greater.

Rails: ActiveRecord Unit Testing part II

Back in December I wrote about unit testing ActiveRecord models. Following that post, we started using that pattern to unit test our models. It quickly became obvious that the ActiveRecord::ConnectionAdapters::Column.new("digits", nil, "string", false) statements were largely static and we wrapped this behavior in a fluent interface.

This worked, but some developers on the team didn't like the (arguably) unnecessary additional syntax. These developers argued that since none of our Unit Tests hit the database we should simply override the new method and handle stubbing the columns method.

You can argue that adding magic to the new method could be confusing, but our team has the standard that no unit test can hit the database. Therefore, everyone is very aware that, despite the standard syntax, we are injecting our own column handling mechanism.

The code to make this work is fairly straightforward. The usual alias method gotchas apply so I chose to use an alternative. Adding the following code will allow you to create disconnected models in your unit tests simply by calling new.

class << ActiveRecord::Base

  new_method = instance_method :initialize
  define_method :initialize do |*attributes|
    attributes = attributes.empty? ? nil : attributes.first
    self.stubs(:columns).returns(no_db_columns(attributes))
    new_method.bind(self).call attributes
  end

  def no_db_columns attributes
    return [] if attributes.nil?
    attributes.keys.collect do |attribute|
      sql_type = case attributes[attribute].class
        when " " then "integer"
        when "Float" then "float"
        when "Time" then "time"
        when "Date" then "date"
        when "String" then "string"
        when "Object" then "boolean"
      end
      ActiveRecord::ConnectionAdapters::Column.new(attribute.to_s, nil, sql_type, false)
    end
  end

end

Rails: Unit Testing ActiveRecord Validations

There are at least 2 easy ways to unit test the validations of an ActiveRecord::Base subclass.

The most straight-forward way is to create a model and use the valid? method to determine if it is valid or not. For an example I'm going to reuse the PhoneNumber class I defined in a previous entry on ActiveRecord Unit Testing. The following code shows the three tests I've written to ensure that I've correctly defined a validates_presence_of for digits.

class PhoneNumberTest < Test::Unit::TestCase
  Column = ActiveRecord::ConnectionAdapters::Column
  
  test "digits are required to be valid" do
    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])
    number = PhoneNumber.new(:digits => "1234567890")
    assert_equal true, number.valid?
  end
  
  test "invalid with no digits " do
    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])
    number = PhoneNumber.new()
    assert_equal false, number.valid?
  end
  
  test "errors include digits on invalid digits" do
    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])
    number = PhoneNumber.new()
    number.valid?
    assert_equal "can't be blank", number.errors.on('digits')
  end

end

After adding the validates_presence_of call to the PhoneNumber class, all the above tests pass.

class PhoneNumber < ActiveRecord::Base
  validates_presence_of :digits
end

This all works fine, but I'm not sure it's necessary to actually cause a validation to occur just to test that one has been defined.

An alternative test implementation could mock the validates_presence_of call and load the file. If the validates_presence_of is defined correctly the class methods will be invoked.

class PhoneNumberTest < Test::Unit::TestCase
  Column = ActiveRecord::ConnectionAdapters::Column

  test "validates_presence_of is defined for digits" do
    PhoneNumber.expects(:validates_presence_of).with(:digits)
    load "#{RAILS_ROOT}/app/models/phone_number.rb"
  end

end

This method of testing doesn't require actually creating a PhoneNumber instance, instead it tests that the PhoneNumber validations are defined correctly.

The load call is a bit messy; however, we can clean that up by adding another method to ActiveRecord::Base.

class << ActiveRecord::Base
  def standard_path
    File.expand_path("#{RAILS_ROOT}/app/models/#{self.name.underscore}.rb")
  end
end

Following the above code change the test can be rewritten to the code shown below.

class PhoneNumberTest < Test::Unit::TestCase
  Column = ActiveRecord::ConnectionAdapters::Column

  test "validates_presence_of is defined for digits" do
    PhoneNumber.expects(:validates_presence_of).with(:digits)
    load PhoneNumber.standard_path
  end

end

Which implementation is better? Both implementations provide pros and cons. I suggest trying both out to determine which works best for you (or your team).

Rails: ActiveRecord Unit Testing

There's been a fair amount of information posted recently on the topic of unit testing ActiveRecord::Base subclasses. Of all the information, I think the most valuable was James' observation that a better data model would result in better tests with less dependencies.

That said, sometimes it would be nice to truly unit test an ActiveRecord::Base subclass. When I say unit test, I mean no dependency on the database. For example, I may store a phone number in the database as a 10 digit string, but I may want to expose that phone number with formatting. I may also want to expose the area code, exchange, and station as methods of the PhoneNumber class. To test this behavior I wrote the following tests that do hit the database.

class PhoneNumberTest < Test::Unit::TestCase

  test "to_formatted_s returns US format" do
    number = PhoneNumber.new(:digits => "1234567890")
    assert_equal "(123) 456-7890", number.to_formatted_s
  end
  
  test "area code returns first 3 numbers" do
    number = PhoneNumber.new(:digits => "1234567890")
    assert_equal "123", number.area_code
  end

end

If this syntax looks bizarre at all, you might want to read about how these tests take advantage of the test class method.

The PhoneNumber class has the following implementation.

class PhoneNumber < ActiveRecord::Base
  
  def to_formatted_s
    '(' + area_code + ') ' + exchange + '-' + station
  end
    
  def area_code
    digits[0..2]
  end
  
  def exchange
    digits[3..5]
  end

  def station
    digits[6..9]
  end
  
end

As the implementation shows, the digits are stored in the database. Splitting the digits up or formatting them is handled by methods on the model.

Now that we have a few tests we'll add the code that disallows database access from unit tests and rerun the tests. As expected the tests fail with the following error.

1) Error:
test_area_code_returns_first_3_numbers(PhoneNumberTest):
ArgumentError: You cannot access the database from a unit test

Looking at the stack trace provided by the error you can track the database access to the columns method of ActiveRecord::Base.

If you want to unit test a model the columns method is a good one to stub. I tried a few different options and the best one I found was stubbing the columns method with a method that returns an array of ActiveRecord::ConnectionAdapters::Column instances. Since I only needed the digits attribute for these tests, it was the only column I needed to return in the array. The following tests test my PhoneNumber class without requiring a trip to the database.

class PhoneNumberTest < Test::Unit::TestCase
  Column = ActiveRecord::ConnectionAdapters::Column

  test "to_formatted_s returns US format" do
    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])
    number = PhoneNumber.new(:digits => "1234567890")
    assert_equal "(123) 456-7890", number.to_formatted_s
  end
  
  test "area code returns first 3 numbers" do
    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])
    number = PhoneNumber.new(:digits => "1234567890")
    assert_equal "123", number.area_code
  end

end

Test::Unit test creation

Creating a unit test in Ruby is very similar to Java: create a method that begins with test_. In a recent discussion with some colleagues, Zak Tamsen and Muness Alrubaie, Zak described his distaste for this common convention. He continued by pointing out that using .NET Attributes was a superior approach for identifying tests.

This began a discussion on what a superior approach to writing tests could be when using Ruby. We implemented several examples before deciding on our preference.

test
def add_returns_the_sum_of_two_numbers
  assert_equal 4, add(2, 2)
end

test 'add returns the sum of two numbers'
assert_equal 4 do
  add(2,2)
end

Each implementation contained limitations. In the end we chose to go with a very simple implementation that is similar to how RSpec defines specifications.

test 'add returns the sum of two numbers' do
  assert_equal 4, add(2, 2)
end

To accommodate this syntax we added the test class method to Test::Unit::TestCase.

class << Test::Unit::TestCase
 def test(name, &block)
   test_name = :"test_#{name.gsub(' ','_')}"
   raise ArgumentError, "#{test_name} is already defined" if self.instance_methods.include? test_name.to_s
   define_method test_name, &block
 end
end

An additional benefit we were able to include in our change was the ability to raise an error when a test was defined with a duplicate name (or description in our case).

However, moving to this new style of test definition can cause issues. The first noticeable issue was that TextMate would no longer run individual tests. We quickly fixed this issue by looking at the code in the Ruby Bundle. The fix was to change the regex that selects the method you are testing. The code below shows how you can capture either the previous call to the test class method or the previously defined test method.

File.open(ENV['TM_FILEPATH']) do |f|
   f.read.split("\n")[0...n].reverse.each do |line|
       if line =~ /^\s*test\s+"([_a-z][_a-z0-9 ]*[\?!]?)"\s+do\s*$/i
           print "test_#{$1.gsub(' ', '_')}"
           break
       elsif line =~ /^\s*def ([_a-z][_a-z0-9]*[\?!]?)/i
           print $1
           break
       end
   end
end

We also created a TextMate snippet in the Ruby Bundle to help us create tests in our new style. The following code is the snippet for easily creating tests.

test "${1:test name}" do
   $0
end

After these quick changes we are quite happy working with our new format for creating tests.

Ruby on Rails Unit Tests

Updated following multiple comments requesting examples and questions on where to put AR::Base subclass tests.

Everyone (who reads this blog anyway) knows that you should not cross boundaries while unit testing. Unfortunately, Ruby on Rails seems to believe otherwise. This is evident by the fact that the test:units rake task has the pre-requisite db:test:prepare. Additionally, if you use script/generate to create a model, it creates a [model].yml fixture file and a unit test that includes a call to the fixtures class method. Rails may be opinionated, but that doesn't mean I have to agree with it.

With a minor modification you can tell Rails not to run the db:test:prepare task. You should also create a new test helper that doesn't load the additional frameworks that you will not need. I found some of the code for this from reading a great book, Rails Recipes, by Chad Fowler.

You'll need to add a .rake file to /lib/tasks. The file contains one line:

Rake::Task[:'test:units'].prerequisites.clear

Additionally, you'll need to create a new helper file in /test. I named my file unit_test_helper.rb, but the file name is your choice.

ENV["RAILS_ENV"] = "test" 
require File.expand_path(File.dirname(__FILE__) + "/../config/environment") 
require 'application' 
require 'test/unit' 
require 'action_controller/test_process' 
require 'breakpoint'

class UnitTest
  def self.TestCase
    class << ActiveRecord::Base
      def connection
        raise InvalidActionError, 'You cannot access the database from a unit test', caller
      end
    end
    Test::Unit::TestCase
  end
end

class InvalidActionError < StandardError
end

As you can see, the unit_test_helper.rb only requires what is necessary; however, it also changes ActiveRecord::Base to throw an error if you attempt to access the connection from a unit test.

I included this test in my codebase to ensure expected behavior.

require File.dirname(__FILE__) + '/../unit_test_helper'

class AttemptToAccessDbThrowsExceptionTest < UnitTest.TestCase
  def test_calling_the_db_causes_a_failure
    assert_raise(InvalidActionError) { ActiveRecord::Base.connection }
  end
end

Update (Again):
We have been using this style of testing for several months now and have over 130 tests at this point and our tests still run in less than a second.

This decision does carry some trade-offs though. First of all, it becomes a bit more work to test ActiveRecord::Base subclasses in your unit tests. I'm comfortable with the small amount of extra work since it results in a significantly faster running test suite.

Also, if you need to use a AR::Base class as a dependency for another class, you will need to mock or stub the AR::Base class. This generally requires using Dependency Injection or a framework such as Stubba. For example, if you have a method that returns an ActiveRecord::Base subclass you can mock the new call and return a stub instead.

class AccountInformationPresenter

  def account
    Account.new
  end

end

class AccountInformationPresenterTest

  def test_account_returns_a_new_account
    Account.expects(:new).returns(stub(:name=>'jay'))
    AccountInformationPresenter.new.account
  end

end

In the above code, mocking the new call on Account prevents an unnecesary database trip.

For an example of what our unit tests look like here are some tests and the classes that the tests cover.

require File.dirname(__FILE__) + '/../../../unit_test_helper'

class SelectTest < Test::Unit::TestCase 
  def test_select_with_single_column
    assert_equal 'select foo', Select[:foo].to_sql
  end

  def test_select_with_multiple_columns
    assert_equal 'select foo, bar', Select[:foo, :bar].to_sql
  end
  
  def test_date_time_literals_quoted
    date = DateTime.new(2006, 1, 20, 13, 30, 54)
    assert_equal "select to_timestamp('2006-01-20 13:30:54', 'YYYY-MM-DD HH24:MI:SS')", Select[date].to_sql
  end
  
  def test_select_with_symbol_and_literal_columns
    assert_equal "select symbol, 'literal'", Select[:symbol, 'literal'].to_sql
  end
  
  def test_select_with_single_table
    assert_equal 'select foo from foo', Select[:foo].from[:foo].to_sql
  end
    
  def test_select_with_multiple_tables
    assert_equal 'select column from bar, foo',
                 Select[:column].from[:foo, :bar].to_sql
  end
end 

require File.dirname(__FILE__) + '/../../unit_test_helper'

class TimeTest < Test::Unit::TestCase 
  def test_to_sql_gives_quoted
    t = Time.parse('2006/05/01')
    assert_equal "to_timestamp('2006-05-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS')", t.to_sql
  end
  
  def test_to_pretty_datetime
    d = Time.parse("05/10/2006")
    assert_equal "05-10-2006 12:00 AM", d.to_pretty_time
  end
end 

class Select < SqlStatement
  class << self
    def [](*columns)
      unless columns.select { |column| column.nil? }.empty?
        raise "Empty Column in #{columns.inspect}"
      end
      self.new("select #{columns.collect{ |column| column.to_sql }.join(', ')}")
    end
  end
  
  def from
    @to_sql += " from "
    self
  end
  
  def [](*table_names)
    @to_sql += table_names.collect{ |table| table.to_s }.sort.join(', ')
    self
  end
  
end

class Time
  def to_sql
    "to_timestamp('" + formatted + "', 'YYYY-MM-DD HH24:MI:SS')"
  end
  
  def to_pretty_time
    self.strftime("%m-%d-%Y %I:%M %p")
  end

  private 
  
  def formatted
    year.to_s + "-" + month.pad + "-" + day.pad + " " + hour.pad + ":" + min.pad + ":" + sec.pad
  end
end

More Test Driven Development observations

In part one I described some of the observations I've made while following Test Driven Development practices. In part two I'll describe other observations that I have previously lightly touched on or haven't documented at all.

If your object graph will not permit testing each concrete class individually, your object graph requires refactoring. Look to introduce Dependency Injection to resolve the issue.

ObjectMother is an artifact, a legacy testing pattern. In the early days of TDD, simply writing tests was a large step. However, testing has evolved to separated unit and functional tests. Unit tests should test individual classes; therefore, ObjectMother is clearly not an option. Functional tests should use actual implementations so ObjectMother could be a potential fit. However, unit tests should test boundary cases and functional tests should only test the classes interactions. Therefore, there should be a minimal amount of functional tests. Since the number of functional tests should remain low, there should not be enough duplication to warrant use of ObjectMother. FluentInterface (or the special case of FluentInterface, InitializationChain) is often a superior alternative to ObjectMother.

In an object oriented system, classes collaborate to achieve a larger goal. However, tests are not object oriented. In fact, having tests depend on other tests is an anti-pattern. Tests are procedural in nature and should be treated as such. Each test method should encapsulate the entire behavioral test. Tests should have very specific goals which can easily be identified by looking only at the method body, private method calls, and stubs if applicable.

Using patterns, for example Template Method, in tests does reduce duplication but at the expense of readability. Due to the loss in readability, setup is considered harmful. The superior solution is to create private methods that are executed when called.

Removing duplication within a system reduces the complexity of the entire system. However, the entire scope of a test is the single method body. Therefore, abstracting duplication should not be taken lightly. I rarely find that my tests contain duplication. When my tests do contain duplication, the solution is often refactoring a poorly designed piece of the object graph, not extracting a private method. For example, duplication can occur in setting up an array for testing. But, the actual problem is the getter returning an array, not the duplicate test code. When duplicate code is extracted to a private method, be aware that you are coupling two entities that in general should be decoupled. This is acceptable, but only for a minimal amount of special cases. In fact, even when a special case occurs, it is preferable to leave the duplication as is if it increases readability

Supporting classes, stubs for example, should be included as private subclasses of the test class. This ensures that to read and understand an entire test you only need to look at one file. It additionally ensures that the stub will only be used within the context of the test class. This is very valuable in keeping the tests robust.

'Magic' introduced in a test (such as using the test class as a stub) may appear clever; however, it reduces readability and should be immediately refactored to something more maintainable. Save the effort and write the test clearly in the first place.

Always use strings or numbers as the 'expected' value of an assertion instead of constants or calculated values. Nothing is more meaningless than seeing a test that asserts expected_error_message == customer.save. The major problem here is the question of which is wrong, expected_error_message or the result of customer.save. Additionally, the maintainability suffers because the values of both arguments must be obtained when debugging.

Use assert_equal (or Assert.AreEqual, etc) the majority of the time. Asserting equality results in highly readable test failures. If I see a failure similar to <"select * from customers"> expected but was <"select from customers"> I may know what the issue is without rereading the test. While the others such as assert, assert_not_null, etc have their place, the error messages are not near as meaningful. <false> is not true is very clear, but I have no idea what was false. Finding the true issue requires consulting the test.

I believe that is a fairly comprehensive list my major observations. Look for follow-ups in the future as I continue to learn more lessons.

Some Test Driven Development observations

Testing code is hard. There is some material available, but most of the examples are trivial. When you actually sit down to do strictly Test Driven Development (TDD) the complications appear quickly.

The first tripping point for me was cascading failures. Simple changes to one class often caused several test failures. Sometimes the fix was a one-liner, but often it was not. I began to find the solution when I first read Martin's paper on Mocks and Stubs. After I read that paper I began to use Mocks religiously, but that didn't really solve the problem. Using an excessive amount of mocks is just as fragile as using the actual implementations.

To me, the next distinct progression of my testing skills was following the one assertion per test guideline. Using only one assertion per test allowed me to write very focused tests that generally had less dependencies. By removing dependencies my tests became more robust.

Understanding Dependency Injection was the next major step in improving my testing skills. By using Dependency Injection I was able to further abstract my dependencies to ensure robust tests.

At this point I felt my tests had improved significantly; however, my test suite was becoming a bit of a burden to execute. The major problem was the excessive network traffic between my test suite and the database. To me, long build times are a serious problem because it represents lost developer time, several times a day. I quickly understood the distinction between Unit tests and Functional (or Integration) tests.

The next logical step was to improve test performance by not crossing boundaries while unit testing. Utilizing Dependency Injection I began to mock the data access code in my unit tests which significantly improved test processing time.

Further increasing the robustness of my tests was achieved by testing one concrete class at a time. Again, Dependency Injection helped provide the solution. By using Dependency Injection the code I wrote was significantly decoupled compared to the previous code I was writing. Injecting mocks or stubs for dependencies of my class under test did increase the robustness of the tests; however, I still had the issue of dealing with fragile mock set up code.

A common practice is to specify method names using strings when setting up expectations of a mock. This is obviously fragile when performing constant refactoring. However, using stubs always seemed like more trouble than it was worth because it required implementing an entire interface when often I only cared about a few methods. With no clear answer on the 'best' choice I used both excessively in different places to determine which approach I preferred. In the end, I found that neither should be treated as the silver bullet. In fact, using both mocks and stubs it's possible to make very clear what your test is accomplishing.

Another gray area in testing concerns what visibility to give your classes members. When I first started out testing I believed that exposing the private members for testing was acceptable. However, I quickly learned this approach was naive and a much better solution is to simply test through the public interface. Also, if a method is marked as private, but I feel it needs to be tested I likely wouldn't hesitate to simply make it public.

The reason I believed I needed to access private methods was because I was using a Code Coverage tool that told me my code was only about 80% covered. However, the answer wasn't that I needed to devise a solution to reach 100%, it was that I needed someone to tell me that 80% or higher is good enough. Should you strive for more than 80%? Sure, but don't make any compromises to get there. You will find the return on investment is generally not near as valuable.

That's the majority of my previously documented lessons learned concerning testing. I hope this entry and my previous entries can provide some guidance and help you avoid some common stumbling blocks. In part 2 of this entry I'll document some observations that I haven't previously written about.

Unit Test Guidelines

Tests live under many names. And, many tests masquerade as unit tests.

Unit testing is defined as:

... a procedure used to verify that a particular module of source code is working properly. The idea about unit tests is to write test cases for all functions and methods so that whenever a change causes a regression, it can be quickly identified and fixed. Ideally, each test case is separate from the others ... [Wikipedia]

There are existing practices that I consider to be good guidelines concerning unit testing.

• William Caputo - Do Not Cross Boundaries. The central idea is to decouple the code from the resources. An added benefit is the reduced running time of the tests. The faster the tests are, the more often they are run.
  
• Dave Astels - One Assertion Per Test. While the idea is controversial it's also very thought provoking. If you follow this rule your tests become simple, expressive, and elegant. I don't follow this rule 100% of the time; however, I do believe it's a great guideline.
  

I follow another guideline that I haven't seen documented anywhere.

Only one concrete class (with behavior) should be used per test.

The central reason for testing concrete classes individually is to promote durable tests. When several concrete classes are used in one test the test becomes brittle. Making a change to any of the coupled concrete classes can cause cascading test failures.

Both mocks and stubs can be used in place of concrete classes where necessary while testing an individual class. If you find your hierarchy too complicated to use mocks or stubs this is probably a sign that you need a simpler and less coupled hierarchy. Using Object Mother is a common alternative to refactoring to a better hierarchy. If you find yourself reaching for Object Mother, take the time to refactor instead.