Rails: How we test

Last weekend, at The Rails Edge (which was fantastic), Mike Clark (jokingly) told me that there was enough buzz around "The Jay Fields Way of Testing" that I should trademark it. I took that as a huge compliment; however, it's also not fair to the teams I've worked with. On all the project teams I am apart of, "How we test" is a collaborative decision. We don't test "The Jay Fields Way", we test our way.

So, what is "our way" as of Monday, September 3rd, 2007?

File Structure
We use RAILS_ROOT/test/unit & RAILS_ROOT/test/functional, which Rails provides by default. Under unit and functional we mirror the structure of the RAILS_ROOT/app folder.

For example, a user model that lives in

RAILS_ROOT/app/models/user.rb

would have unit tests in

RAILS_ROOT/test/unit/models/user_test.rb

and functional tests in

RAILS_ROOT/test/functional/models/user_test.rb

Tests In General
We use dust to define our tests, and we use the disallow_setup! method to ensure that setup methods aren't being added to the codebase.

Unit Tests
Unit tests are where we test classes in isolation. Within unit tests, dependencies are mocked or stubbed to ensure that a breaking test is broken because a feature of the class under test has changed. We defer testing object interactions to the functional tests. As a result, there are rarely cascading failures.

In general, there are far more unit tests than functional tests. This is a result of testing permutations, edge cases, etc within the unit test suite. If a format class contains 10 logical code paths, there should be at least 10 unit tests to verify that it works correctly; however, you may only need 1 functional test to verify that the format class works correctly with the other objects it interacts with.

Unit tests utilize a test helper specific to unit tests (RAILS_ROOT/test/unit/unit_test_helper.rb), which is the one and only require statement at the top of each unit test. In the unit_test_helper.rb we require mocha, UnitRecord and any other library that we need for unit testing. We require mocha only in unit_test_helper.rb to ensure that mocking/stubbing via mocha is only done in the unit tests. UnitRecord is a gem by Dan Manges that disables access to the database and provides the ability to easily unit test ActiveRecord::Base subclasses.

Functional Tests
Functional tests are where we verify that all the pieces of the application interact seamlessly. While stubs do exist, they are hand-coded and only stub external systems. Ideally, no stubs would exist; however, a trade-off is necessary when hitting external systems that significantly decrease your ability to quickly run the functional tests.

Functional tests often need small graphs of objects. Generally, this type of code is put in a setup method or created by fixtures. Both setup and fixtures provide a solution; however, we've found a more maintainable solution is to create a factory. Dan Manges has a great entry on our Functional Test Factory. A Factory has been very helpful; however, one thing to stress is that the Factory contains one create method per model. What's implied in that statement is that the methods are not defined on a scenario basis. If you need an Apartment that has a Renter that has a Job, you'll need to create each model within the test. Adding a method for a specific scenario (to the Factory) is the road to a completely unmaintainable Factory. We toyed with the idea of creating a fluent interface builder; however, Rails associations mostly remove the need.

For example, the following snippet can create an Apartment that has a Renter that has a job.

Factory.create_apartment(:renter => Factory.create_user(:job => Factory.create_job))

External Tests
In functional tests we stub external services; however, it's important to verify that the API of an external service hasn't changed. To perform that verification we write external services tests that interact with the external services. Since these tests are slower to run, they are generally only run on the continuous integration server or while debugging a contract change.

Why Bother?
There are several reasons for each decision we made; however, the context is the most important factor to consider. We work on large teams where the tests are run as often as possible and need to execute quickly. Those same tests also need to be as readable as possible, because it's likely that you'll be looking at tests you didn't write more often than not. While I think the above ideas are also good for small teams, I haven't personally put them to use on a small team.

Integration Tests, Selenium Tests, View Tests, aren't you missing something?
Yes, the above discussion does leave off the area of Acceptance Testing. We've yet to come up with something that works from project to project. On one project we were very successful with creating a DSL that executed as Rails integration tests locally and ran as Selenium on the build. While this might be a great solution, I haven't seen it used enough to recommend it.

View tests, in my experience, aren't necessary unless you are putting logic in your view. I prefer to keep the logic out of the view and ignore view tests entirely.

Look for updates on this entry, I'm sure I've forgotten a few things.