Wednesday, April 09, 2008

Splitting your Rails Test Suite

Before I began working primarily with Rails, I spent most of my time building C# applications. The majority of the C# projects that I was a part of had a unit test suite and a functional test suite.

The unit tests didn't interface with the filesystem, databases, or any external resource. The unit tests focused on testing one thing at a time, either isolated or with few collaborators (classes). They were designed to quickly give us feedback without using the full stack.

We also had functional tests that did test the full stack to ensure proper integration. As a result, we could run the unit tests to gain confidence that our code worked, and we could run the functional tests to ensure the code integrated correctly.

Things changed when I starting working with Rails. At first I was amazed by how quickly I could create websites, but I was disappointed by how slow writing tests became. I felt like I'd taken two steps forward and one step back. Rails had both unit and functional tests, but the definitions were different from the ones I'd grown used to in the Java and C# world. The unit tests felt more like model tests, while the functional tests felt more like controller tests. And, neither of them ran fast enough for me.

I decided to try splitting my Rails test suite based on the lessons I'd previously learned.

Speed
The original motivation for splitting my test suite was speed. I value feedback and I want it as quick as possible.

In 2006, I wrote about the idea of disconnecting the database for unit tests and the results. In 2007, Dan Manges rolled the concept into the UnitRecord gem. Then George Malamidis showed me that unit tests can run significantly faster if you don't load the Rails environment. I liked the concept of unit testing without Rails, but I knew we needed to provide a solution for unit testing ActiveRecord::Base subclasses. This is when arbs was born. The arbs gem is designed to make any ActiveRecord::Base subclass behave basically like a struct.

The result of using arbs was a unit test suite that ran in less than 1 second, including time to start and finish the process. When running tests in TextMate, the tests completed before the results window had finished drawing.

Positive Consequences
The original motivation for splitting our test suite was speed, but it resulted in a few other side benefits. A split test suite enabled us to write different tests based on what suite we were working with. For example, we use mocks and stubs in our unit test suite, but not in our functional test suite. Also, we use expectations to write unit tests and RSpec to write functional tests.

We use mocks in our unit tests to test in isolation. I prefer unit testing in isolation because it's a good way to mitigate cascading failures. However, by not allowing mocks in our functional test suite we ensure that the functional tests verify proper integration. We also use code coverage tools to ensure that everything is tested in isolation and tested while integrated. The result is robust tests that run quickly and verify integration.

Having two different suites also allows us to separate our testing concerns. Our unit tests have one assertion or one expectation per test. These fine grained unit tests focus on testing a single responsibility and generally do not break because of unrelated behavior changes. We find that the expectations unit testing framework allows us to focus on the essence of unit testing.

Our functional tests validate from a different perspective. They test at a much higher level and ensure that everything integrates as expected. Generally these types of tests need a description of why the test exists. Functional tests are generally also more resource intensive. We always strive to have one assertion per test, but due to resource requirements it's often necessary to have multiple assertions. We find that RSpec is the best tool for writing our functional tests.

Negative Consequences
Unfortunately, neither UnitRecord or arbs offer painless solutions. Splitting your test suite requires effort. Both UnitRecord and arbs rely on altering the behavior of core Rails classes. If you are unfamiliar with UnitRecord or arbs, you may see unexpected behavior when testing. Even though this is the case, I think the benefits outweigh the occasional confusion.

Having a split test suite can also cause confusion among developers. I've often worked with developers who wanted to write a new test but didn't know where the test belonged. I think "where should the test go" is the wrong question. Building comprehensive test suites requires that new functionality be tested but at the unit and functional level. Therefore, you should always write a unit test (if possible) and then ensure the logic is functionally tested. Since functional tests are generally course grained, it's often the case that the functionality will be covered by an existing test. If the functionality isn't covered by an existing test, then it makes sense to write a new functional test.

Conclusion
I prefer a split test suite because I value readable, reliable, and performant feedback. If you also value these things, you should give splitting your test suite a shot. Unfortunately, you'll be joining the minority. I believe that most Rails developers don't think it's worth the effort. Of course, at one point someone was a minority advocating for the same thing in Java, and now it's the norm.

12 comments:

  1. Anonymous11:23 AM

    Do your functional tests still correlate with your controllers?

    Are they organized by the methods in your controller?

    So for instance, if you have an Articles controller do you organize your tests for each of the methods (create, update, show, etc.)?

    If I understood correctly the distinction between a rails functional test defined as one that just tests a controller and one in the traditional sense is that you are not mocking out the models, you are creating an integration test that calls actual model methods rather than just checking the controller in isolation to see if it's calling the methods on mocked models.

    Are the functional tests more resource intensive because you are using fixtures?

    No use of rspec stories or rails integrations?

    ReplyDelete
  2. Anonymous12:02 PM

    Hi Tim,

    I do test my controllers in my functional tests; however, I also test the methods of my models that need to use the database. Any class (or module) in the application that relies on an external resource (db, web service, file system, etc) is tested functionally with nothing mocked out. Additionally any object who's responsibility is to coordinate several other objects (controllers) are functionally tested.

    I still tend to create a foo_test or foo_spec to verify all the methods foo class, but I'm not religious about it. I'd try something new, but I haven't yet seen a superior solution.

    In general I find that Rails controllers are very hard to unit test, so I try to make them as slim as possible and I don't bother unit testing them (but I do functional test them). I wish it wasn't the case, but they rely to heavily on Rails to properly unit test them. (more opinions on testing controllers)

    The functional tests are more resource intensive than the unit tests because they use the database, web services, or files.

    I do not use fixtures.

    When my last project started RSpec stories weren't quite ready, but I am planning on giving them a try on my next project.

    I don't use Rails Integration tests.

    I have used Selenium in the past, but I've yet to have a great experience with it. I would use it again if unit and functional tests weren't good enough. (e.g. I needed to test a lot of Javascript)

    ReplyDelete
  3. Anonymous12:20 PM

    Could you please post a couple of test suites as an example?

    I want to change the way i test right now, but i am not sure where to start from. There are so many references...

    Thanks a lot!!

    ReplyDelete
  4. Anonymous6:48 PM

    Hello Gaizka,
    I'll be putting together a blog post in the next few days. Keep an eye out for it.

    Cheers, Jay

    ReplyDelete
  5. Anonymous1:32 AM

    You mentioned in one of the comments that you don't use fixtures. What do you use to create the test fixture then? iirc, you're a big fan of inline setup, so the test is self-contained and easily readable. Do you do anything to reduce duplication between tests, like use an object mother? How is your test setup different when writing unit tests vs functional tests.

    Another thing you mentioned is that anything that uses external resources is functionally tested. So if you were working on a special finder, you would write a functional test for it but not a unit test? I assume no unit test because there's no viable way to unit test that kind of thing, I think.

    ReplyDelete
  6. Anonymous6:38 AM

    Hi Pat,
    I generally use a factory to create test data.

    You are exactly right, I wouldn't have a unit test for something that cannot be realistically unit tested. Of course, in that situation the first thing I do is look for a way to make it unit testable. However, in some cases, like a finder, it makes sense to just skip the unit test.

    ReplyDelete
  7. Anonymous12:49 PM

    Hi Jay, I tried to give arbs a look and I'm wondering if you've seen the issue I've run into:

    After installing the gem and adding


    require 'arbs'
    ArbsGenerator.run(File.dirname(__FILE__) + "/../db/schema.rb")


    to my test_helper, I keep running into


    NoMethodError: undefined method ‘new’ for ActiveRecord::ConnectionAdapters::MysqlAdapter:0x118faf0


    this happens during the migrations that get run from schema.rb. I can't understand why this is happening as the Mysql adapter seems to respond to new at the moment when it receives the send.

    Have you seen this?

    Thanks!

    ReplyDelete
  8. Anonymous7:32 AM

    Hi Tim,
    I'm not getting the same error, but I'm happy to try to help you out.

    John Hume has been adding a lot of behavior recently, so I released version 02.0 this morning.

    If you get the latest version and still have problem, shoot me an email and I'll help you troubleshoot it.

    jay at [mydomain] .com

    ReplyDelete
  9. Anonymous6:39 PM

    Hi Jay, I was able to get arbs going! The key was to go back and read your original post on unit testing without rails. Once I configured my unit_test_helper to look like where the magic lies, it all fell into place. I was slightly confused by the README for arbs that says

    To use arbs all you need to do is include the following 2 lines.

    require 'arbs'
    ArbsGenerator.run("[path to]/schema.rb")


    This worked for me:

    require 'rubygems'
    require 'test/unit'
    require 'dust'
    require 'active_support'
    require 'initializer'
    require 'arbs'
    require 'expectations'
    RAILS_ROOT = File.expand_path(File.dirname(__FILE__) + "/../../")
    Rails::Initializer.run(:set_load_path)
    Rails::Initializer.run(:set_autoload_paths)
    ArbsGenerator.run(RAILS_ROOT + "/db/schema.rb")

    Now everything works and I'm kind of blown away by how oddly liberating it is to run rails unit tests without the initial lag I'm accustomed to.

    Thanks for your help. Now to try this stuff out...

    ReplyDelete
  10. Anonymous1:44 PM

    Jay, how do you unit test assertions using arbs? Do you still use Validateable Assertions? I hacked arbs to include AR's validations instead of just stubbing them, so I can test validations in my arbed unit tests. This led to hacking more AR functionality like callbacks into arbs until I reached a point, where I feel like I was the losing the point of arbs.

    How do you test validations?

    ReplyDelete
  11. Anonymous1:46 PM

    I meant to say how do you unit test validations, not how do you unit test assertions...

    ReplyDelete
  12. Anonymous2:52 AM

    Tim, I use Validatable assertions instead. The major reason I do this is because Validatable provides support for testing individual validations, but a nice side effect is that I don't depend on AR.

    Cheers, Jay

    ReplyDelete

Note: Only a member of this blog may post a comment.