Monday, September 03, 2007

Rails: How we test

Last weekend, at The Rails Edge (which was fantastic), Mike Clark (jokingly) told me that there was enough buzz around "The Jay Fields Way of Testing" that I should trademark it. I took that as a huge compliment; however, it's also not fair to the teams I've worked with. On all the project teams I am apart of, "How we test" is a collaborative decision. We don't test "The Jay Fields Way", we test our way.

So, what is "our way" as of Monday, September 3rd, 2007?

File Structure
We use RAILS_ROOT/test/unit & RAILS_ROOT/test/functional, which Rails provides by default. Under unit and functional we mirror the structure of the RAILS_ROOT/app folder.

For example, a user model that lives in
RAILS_ROOT/app/models/user.rb
would have unit tests in
RAILS_ROOT/test/unit/models/user_test.rb
and functional tests in
RAILS_ROOT/test/functional/models/user_test.rb
Tests In General
We use dust to define our tests, and we use the disallow_setup! method to ensure that setup methods aren't being added to the codebase.

Unit Tests
Unit tests are where we test classes in isolation. Within unit tests, dependencies are mocked or stubbed to ensure that a breaking test is broken because a feature of the class under test has changed. We defer testing object interactions to the functional tests. As a result, there are rarely cascading failures.

In general, there are far more unit tests than functional tests. This is a result of testing permutations, edge cases, etc within the unit test suite. If a format class contains 10 logical code paths, there should be at least 10 unit tests to verify that it works correctly; however, you may only need 1 functional test to verify that the format class works correctly with the other objects it interacts with.

Unit tests utilize a test helper specific to unit tests (RAILS_ROOT/test/unit/unit_test_helper.rb), which is the one and only require statement at the top of each unit test. In the unit_test_helper.rb we require mocha, UnitRecord and any other library that we need for unit testing. We require mocha only in unit_test_helper.rb to ensure that mocking/stubbing via mocha is only done in the unit tests. UnitRecord is a gem by Dan Manges that disables access to the database and provides the ability to easily unit test ActiveRecord::Base subclasses.

Functional Tests
Functional tests are where we verify that all the pieces of the application interact seamlessly. While stubs do exist, they are hand-coded and only stub external systems. Ideally, no stubs would exist; however, a trade-off is necessary when hitting external systems that significantly decrease your ability to quickly run the functional tests.

Functional tests often need small graphs of objects. Generally, this type of code is put in a setup method or created by fixtures. Both setup and fixtures provide a solution; however, we've found a more maintainable solution is to create a factory. Dan Manges has a great entry on our Functional Test Factory. A Factory has been very helpful; however, one thing to stress is that the Factory contains one create method per model. What's implied in that statement is that the methods are not defined on a scenario basis. If you need an Apartment that has a Renter that has a Job, you'll need to create each model within the test. Adding a method for a specific scenario (to the Factory) is the road to a completely unmaintainable Factory. We toyed with the idea of creating a fluent interface builder; however, Rails associations mostly remove the need.

For example, the following snippet can create an Apartment that has a Renter that has a job.

Factory.create_apartment(:renter => Factory.create_user(:job => Factory.create_job))

External Tests
In functional tests we stub external services; however, it's important to verify that the API of an external service hasn't changed. To perform that verification we write external services tests that interact with the external services. Since these tests are slower to run, they are generally only run on the continuous integration server or while debugging a contract change.

Why Bother?
There are several reasons for each decision we made; however, the context is the most important factor to consider. We work on large teams where the tests are run as often as possible and need to execute quickly. Those same tests also need to be as readable as possible, because it's likely that you'll be looking at tests you didn't write more often than not. While I think the above ideas are also good for small teams, I haven't personally put them to use on a small team.

Integration Tests, Selenium Tests, View Tests, aren't you missing something?
Yes, the above discussion does leave off the area of Acceptance Testing. We've yet to come up with something that works from project to project. On one project we were very successful with creating a DSL that executed as Rails integration tests locally and ran as Selenium on the build. While this might be a great solution, I haven't seen it used enough to recommend it.

View tests, in my experience, aren't necessary unless you are putting logic in your view. I prefer to keep the logic out of the view and ignore view tests entirely.

Look for updates on this entry, I'm sure I've forgotten a few things.

20 comments:

  1. Thanks for the explanation!

    I wrote a quick rake task to follow these instructions for an existing Rails project stored in Subversion.

    WARNING: Try this on a branch before running it on the trunk of your application!

    http://pastie.caboo.se/93520

    ReplyDelete
  2. Interesting stuff, thanks.

    Noticed a typo: "create_appartment" -> "create_apartment".

    ReplyDelete
  3. Hey Jay, I have to say that this is an excellent post. 100% informative and gives excellent insight into how we should consider tests as part of a project and not as a means to themselves. People have started to forget that the reason for testing is so we can get working software into prod, and when you remind us of this it can only be good. Cheers mate, see you in Tribeca sometime soon!

    ReplyDelete
  4. Anonymous2:45 PM

    Great post, thanks for the insight, that's how my tests are being structured from now on. I have a couple of questions/points:

    * You advise against creating multiple states of models via a Factory "the Factory contains one create method per model". Would you consider it bad practice to have both a create_valid_apartment and a create_invalid_apartment method? If so, what would you consider best practice for accomplishing something similar?

    * Not testing views - I have found it beneficial to test the key parts of a view i.e. is there a link to X, is the form method set to 'put' (via the hidden field) etc. as it can save on silly mistakes such as calling a helper with invalid args, using wrong helpers etc.

    ReplyDelete
  5. Anonymous2:56 PM

    I wouldn't consider it a bad practice to have a method for valid and invalid models. I suggest trying it out and see what works well for you.

    One thing to consider though, the models created in the factory should be created with default data. If your test depends on any attribute of the model, you should probably set that attribute in the test.

    So, if you need an apartment with 2 rooms, your test should probably have a line similar to the one below.

    Factory.create_apartment(:rooms => 2)

    Writing tests that depend on the default data can create maintenance issues if the default data ever needs to change. Also, specifying the attributes that you care about in the test creates more readable/understandable/maintainable tests.

    So, if you write your tests and specify the data specific to the test, you probably wont need both valid and invalid model methods. You've touched on something else though, which is: How do you get an invalid model from a create method?

    What I left off originally was that our factory is beginning to get new_[model] methods that set the defaults (and merge in the passed in parameters) and call .new instead of .create. If you use the new_[model] methods you should be able to create a model in an invalid state and test what you need.

    Cheers, Jay

    ReplyDelete
  6. Anonymous3:27 PM

    Ah yes. I only skimmed Dan's article and didn't notice that a merge was happening with supplied attributes.

    Your new_[model](attributes) suggestion solves the valid/invalid scenario and makes it much clearer in the test as to which attribute is causing the model to be invalid.

    Great stuff, many thanks.

    ReplyDelete
  7. What kind of tools are you using? Autotest needs some custom configuration to work with this kind of file structure. I guess you could use topfunky's rstakeout to run tests on save, but its not as granular/fast as everyone's favorite autotest.

    ReplyDelete
  8. Anonymous6:19 PM

    I don't use autotest. Performance is a very large focus in my test suites, thus I don't need to run only the tests for files that have changed.

    My normal workflow is: Write a breaking test and execute only that test (via Run Focused Test in TextMate). Fix that test and run the entire test file. Assuming that passes, either immediately run all tests and check in if the change is large enough, or repeat the first 2 steps until the change is large enough to check in.

    How large is "large enough?" Not very large, I try to check in every 5 to 15 minutes if I can. Since I check in often, I run all the tests very often, thus I need all my tests to run quickly. Since all my tests run quickly, I don't find that autotest gains me anything. I've had that discussion several times and I'm totally cool with people disagreeing on the topic. The reality is that I've worked on several teams where several people have starting using autotest to see if it would help them. Not a single person continued using autotest for more than a week.

    There counter argument is that the additional information doesn't hurt so why not use it? There are several arguments that state that if something isn't helping you then it is adding noise and hurting you. I'm not up to debating that fact, but that's what it feels like when I've tried using autotest in the context in which I work.

    Lastly, I've known a few teams that like autotest and have hacked it to work with that file structure. They told me it wasn't very hard.

    Cheers, Jay

    ReplyDelete
  9. Jay, I agree 100% with your workflow. Too often I've seen the results of TDD when it's done as "write a test, write some code, run focused test, watch it pass, check in".

    I don't agree 100% with keeping the unit/functional split. While I do appreciate the correct usage of "unit" and "functional" (as opposed to the apparent Rails meanings of "model" and "controller/view/helper"), it gets to be something of a bear to keep tests split up like that. I know because I've seen it in use and been part of it on a large project. I prefer having everything together as a spec, like, say, RSpec, or shoulda.

    Recently I've hacked shoulda into letting both unit and functional tests reside in one 'spec' file for a model (or controller, or what-have-you), with some contexts allowed db access and others denied it. I think it's pretty nice.

    And as an aside, I was surprised your comment form knew my name. I don't remember signing up for any Blogger stuff at all.

    ReplyDelete
  10. Anonymous10:58 AM

    Yossef,
    I can appreciate that it didn't work in your experience. Context is very important when deciding what practices to follow. I can say that what we did worked very well; however, we tried a lot of different things. Not everything worked out and when things weren't working, we adapted the process to something that was more successful. I'm sure there are plenty of people delivering quality software doing something completely different.

    "How we test" is simply that.

    ReplyDelete
  11. Anonymous12:14 PM

    With regards autotest, the only way I have found to get it to run unit and functional separately is to put a .autotest file in both unit and functional folders and run them both with the following:

    global_autotest_file = File.expand_path('~/.autotest')
    load(global_autotest_file) if File.exists?(global_autotest_file)

    class Autotest
    def tests_for_file(filename)
    Dir['**/*_test.rb']
    end
    end

    ReplyDelete
  12. Anonymous4:19 PM

    I have finally (after a few hours hunting around the zen test source) found a solution for testing ActiveUnit style directory structures with autotest.

    I have packaged it in a plugin as can be seen here: http://www.thelucid.com/articles/2007/09/05/rails-using-autotest-with-unitrecord

    ReplyDelete
  13. Thanks for clearing up your workflow Jay, its insightful to understand the process in addition to the file structure. And Jamie, thanks for putting together the autotest plugin, you saved me a ton of time!

    ReplyDelete
  14. Anonymous6:34 PM

    Jay, just out of interest, how do you go about testing before filters on models such as before_save :encrypt_password ?

    The only way I can see is as functional tests, but ideally I don't want to hit the database and therefore would also have unit tests for this.

    ReplyDelete
  15. Anonymous9:38 PM

    Jamie,

    I'm going to assume that :encrypt_password is a method that you could explicitly call. That's probably what I would do to test the behavior of the method. As far as testing that it has been wired up as a before_save, I'd probably stick to functional tests for that since it involves the framework and object collaboration.

    Cheers, Jay

    ReplyDelete
  16. Anonymous8:44 AM

    Thanks, that makes sense ...however :encrypt_password is protected since it is a before_filter.

    Maybe I will just have to make a compromise in testing private/protected methods if they are before filters (I normally avoid testing private/protected methods)

    ReplyDelete
  17. Jay, why do you feel view tests are unnecessary. Have you had a bad experience with them?

    I find that view tests are very important so naturally I'm a little on edge when you find theme useless. Could provide some more context, or do you just find all view testing useless?

    http://www.continuousthinking.com/2007/9/18/not-testing-views-how-jay-tests

    ReplyDelete
  18. Anonymous1:41 PM

    I find that the View tests that RSpec and Zentest provide are fragile and provide dubious value. Rarely do they find bugs and often they break for unimportant reasons. I just don't find them to have a good return on investment. YMMV

    ReplyDelete
  19. Anonymous11:12 AM

    Jay,

    I really like this approach -- thanks for putting it out there.

    I did run into one problem, so I thought I'd post this in case it might save others some time...

    I started out with the default test_helper, which has use_transactional_fixtures set to true. The unit tests worked fine as long as they were all in the same directory, but failed when there were test(s) in both the 'models' and the 'controllers' directories.

    With transaction fixtures on, the setup and teardown methods on active_record/fixtures.rb were getting called, and trying to access the connection. Obviously, disconnected_active_record didn't like that...

    ReplyDelete
  20. Anonymous2:39 AM

    Another approach to the fixtures/factory problem can be found @ http://github.com/grosser/valid_attributes/

    Its centered on providing valid fixtures through a simple set of valid attributes, no new factories/fixtures.

    ReplyDelete

Note: Only a member of this blog may post a comment.