Splitting your Rails Test Suite

Before I began working primarily with Rails, I spent most of my time building C# applications. The majority of the C# projects that I was a part of had a unit test suite and a functional test suite.

The unit tests didn't interface with the filesystem, databases, or any external resource. The unit tests focused on testing one thing at a time, either isolated or with few collaborators (classes). They were designed to quickly give us feedback without using the full stack.

We also had functional tests that did test the full stack to ensure proper integration. As a result, we could run the unit tests to gain confidence that our code worked, and we could run the functional tests to ensure the code integrated correctly.

Things changed when I starting working with Rails. At first I was amazed by how quickly I could create websites, but I was disappointed by how slow writing tests became. I felt like I'd taken two steps forward and one step back. Rails had both unit and functional tests, but the definitions were different from the ones I'd grown used to in the Java and C# world. The unit tests felt more like model tests, while the functional tests felt more like controller tests. And, neither of them ran fast enough for me.

I decided to try splitting my Rails test suite based on the lessons I'd previously learned.

Speed
The original motivation for splitting my test suite was speed. I value feedback and I want it as quick as possible.

In 2006, I wrote about the idea of disconnecting the database for unit tests and the results. In 2007, Dan Manges rolled the concept into the UnitRecord gem. Then George Malamidis showed me that unit tests can run significantly faster if you don't load the Rails environment. I liked the concept of unit testing without Rails, but I knew we needed to provide a solution for unit testing ActiveRecord::Base subclasses. This is when arbs was born. The arbs gem is designed to make any ActiveRecord::Base subclass behave basically like a struct.

The result of using arbs was a unit test suite that ran in less than 1 second, including time to start and finish the process. When running tests in TextMate, the tests completed before the results window had finished drawing.

Positive Consequences
The original motivation for splitting our test suite was speed, but it resulted in a few other side benefits. A split test suite enabled us to write different tests based on what suite we were working with. For example, we use mocks and stubs in our unit test suite, but not in our functional test suite. Also, we use expectations to write unit tests and RSpec to write functional tests.

We use mocks in our unit tests to test in isolation. I prefer unit testing in isolation because it's a good way to mitigate cascading failures. However, by not allowing mocks in our functional test suite we ensure that the functional tests verify proper integration. We also use code coverage tools to ensure that everything is tested in isolation and tested while integrated. The result is robust tests that run quickly and verify integration.

Having two different suites also allows us to separate our testing concerns. Our unit tests have one assertion or one expectation per test. These fine grained unit tests focus on testing a single responsibility and generally do not break because of unrelated behavior changes. We find that the expectations unit testing framework allows us to focus on the essence of unit testing.

Our functional tests validate from a different perspective. They test at a much higher level and ensure that everything integrates as expected. Generally these types of tests need a description of why the test exists. Functional tests are generally also more resource intensive. We always strive to have one assertion per test, but due to resource requirements it's often necessary to have multiple assertions. We find that RSpec is the best tool for writing our functional tests.

Negative Consequences
Unfortunately, neither UnitRecord or arbs offer painless solutions. Splitting your test suite requires effort. Both UnitRecord and arbs rely on altering the behavior of core Rails classes. If you are unfamiliar with UnitRecord or arbs, you may see unexpected behavior when testing. Even though this is the case, I think the benefits outweigh the occasional confusion.

Having a split test suite can also cause confusion among developers. I've often worked with developers who wanted to write a new test but didn't know where the test belonged. I think "where should the test go" is the wrong question. Building comprehensive test suites requires that new functionality be tested but at the unit and functional level. Therefore, you should always write a unit test (if possible) and then ensure the logic is functionally tested. Since functional tests are generally course grained, it's often the case that the functionality will be covered by an existing test. If the functionality isn't covered by an existing test, then it makes sense to write a new functional test.

Conclusion
I prefer a split test suite because I value readable, reliable, and performant feedback. If you also value these things, you should give splitting your test suite a shot. Unfortunately, you'll be joining the minority. I believe that most Rails developers don't think it's worth the effort. Of course, at one point someone was a minority advocating for the same thing in Java, and now it's the norm.