Developer Testing and the Importance of Context

How is it we keep falling for the same trick? Why is it so hard to remember: there is no silver bullet.

I've spent a significant amount of my time for the past 3 years focusing on testing. I've learned several lessons.

• Setup methods are evil
• One assertion or expectation per test
• Duplicate code in tests can be a good thing
• Test names aren't always required
• Replacing mocks with stubs can improve essence and reduce high implementation specification
• Expecting literals increases readability and maintainability
• Whether to write a unit test or a functional test, and when only a unit test is necessary

Unfortunately, that list doesn't represent best practices or rules for better testing, it represents patterns for creating a readable, reliable, and performant test suite -- if and only if you work on the kind of projects I work on and you use tests the way I do.

And that's the killer. Context is still king.

I work on teams generally of size 6-16 developers. More often than not I fix tests that I've never seen before. I never read test files to understand class responsibilities. I never generate documentation based on my tests. I do my best to write perfect tests so that my application runs perfectly, but the best tests I write, I never look at again. My tests have a thankless task: guide my system design and ensure that I don't introduce regression bugs, and that's it. I run the entire test suite every minute on average.

I expect some of my readers work the way that I do, and in environments similar to mine; however, the vast majority probably don't. That means that a small minority can blindly follow my suggestions, but the majority of my readers will need to understand why I prefer those patterns and if they apply to their work environment.

Perhaps an example is appropriate. Which of these tests would you prefer to find when the build fails because of it.

test "when attribute is not nil or empty, then valid is true" do
  validation = Validatable::ValidatesPresenceOf.new stub, :name
  assert_equal true, validation.valid?(stub(:name=>"book"))
end

test "when attribute is not nil or empty, then valid is true" do
  assert_equal true, @validation.valid?(@stub)
end

It's actually a trick question. The context is too important to ignore when composing an answer. If this test lived in a project where I was the sole author and expected maintainer then the latter is probably a better solution because I would know where to find the creation of @validation. However, on a large team where it's more likely that I'll never see this test until it's broken, there's a great argument for keeping all the necessary logic within the test itself.

The same test could be written with or without a test name.

test "when attribute is not nil or empty, then valid is true" do
  validation = Validatable::ValidatesPresenceOf.new stub, :name
  assert_equal true, validation.valid?(stub(:name=>"book"))
end

expect Validatable::ValidatesPresenceOf.new(stub, :name).to.be.valid?(stub(:name=>"book"))

Again, the context is critical in deciding which approach to use on your project. The second test is one line, but it provides very little understanding of why it exists. You can resolve this issue by adding a comment or (what I would prefer) by changing the class to have a more fluent interface that explains why as well as how. However, both of these solutions make it hard to easily scan the file for an understanding of ValidatesPresenceOf or generate documentation based on the tests. Are those things important to you?

Dan North and I agree more than we disagree, but we have very different styles of testing. We also use test in different ways, but we both have the same goal in mind -- use tests to create reliable, readable, and performant software. I believe striving for reliable, readable, and performant applications and tests is a good goal to have and there are several ways to get there. Your best bet is to understand the patterns that work for me, understand the patterns that work for Dan, and understand the patterns that work for anyone else who is passionate about developer testing. You'll find that some of their approaches are in direct conflict. This isn't because one pattern is superior to another in isolation, it's because one pattern is superior to another in context.

There's also another factor worth mentioning. Innovation around testing is still happening at a rapid pace. It seems as though there's a new testing or mocking framework appearing on a weekly basis. I suspect this is probably representative of the future of testing -- testing frameworks targeted at specific contexts. As of right now you may write your unit tests and functional tests using the same framework; however, in the future you may prefer Synthesized Testing when focusing on developer tests and RSpec Story Runner for acceptance tests. It's also possible that the newest features of XUnit.net, JMock or Mockito will give you a better way to model your domain. These, and the other testing frameworks, are evolving at a rapid pace because developer testing patterns are still immature and are being adapted to the contexts in which they are used. In the future you may not use the same tool for several different types of tasks -- and that's probably a good thing.

It all comes back to context. The best advice anyone can give you is to consider yours and take the patterns that should help the most... and then adapt as your context changes.