Splitting your Rails Test Suite

Before I began working primarily with Rails, I spent most of my time building C# applications. The majority of the C# projects that I was a part of had a unit test suite and a functional test suite.

The unit tests didn't interface with the filesystem, databases, or any external resource. The unit tests focused on testing one thing at a time, either isolated or with few collaborators (classes). They were designed to quickly give us feedback without using the full stack.

We also had functional tests that did test the full stack to ensure proper integration. As a result, we could run the unit tests to gain confidence that our code worked, and we could run the functional tests to ensure the code integrated correctly.

Things changed when I starting working with Rails. At first I was amazed by how quickly I could create websites, but I was disappointed by how slow writing tests became. I felt like I'd taken two steps forward and one step back. Rails had both unit and functional tests, but the definitions were different from the ones I'd grown used to in the Java and C# world. The unit tests felt more like model tests, while the functional tests felt more like controller tests. And, neither of them ran fast enough for me.

I decided to try splitting my Rails test suite based on the lessons I'd previously learned.

Speed
The original motivation for splitting my test suite was speed. I value feedback and I want it as quick as possible.

In 2006, I wrote about the idea of disconnecting the database for unit tests and the results. In 2007, Dan Manges rolled the concept into the UnitRecord gem. Then George Malamidis showed me that unit tests can run significantly faster if you don't load the Rails environment. I liked the concept of unit testing without Rails, but I knew we needed to provide a solution for unit testing ActiveRecord::Base subclasses. This is when arbs was born. The arbs gem is designed to make any ActiveRecord::Base subclass behave basically like a struct.

The result of using arbs was a unit test suite that ran in less than 1 second, including time to start and finish the process. When running tests in TextMate, the tests completed before the results window had finished drawing.

Positive Consequences
The original motivation for splitting our test suite was speed, but it resulted in a few other side benefits. A split test suite enabled us to write different tests based on what suite we were working with. For example, we use mocks and stubs in our unit test suite, but not in our functional test suite. Also, we use expectations to write unit tests and RSpec to write functional tests.

We use mocks in our unit tests to test in isolation. I prefer unit testing in isolation because it's a good way to mitigate cascading failures. However, by not allowing mocks in our functional test suite we ensure that the functional tests verify proper integration. We also use code coverage tools to ensure that everything is tested in isolation and tested while integrated. The result is robust tests that run quickly and verify integration.

Having two different suites also allows us to separate our testing concerns. Our unit tests have one assertion or one expectation per test. These fine grained unit tests focus on testing a single responsibility and generally do not break because of unrelated behavior changes. We find that the expectations unit testing framework allows us to focus on the essence of unit testing.

Our functional tests validate from a different perspective. They test at a much higher level and ensure that everything integrates as expected. Generally these types of tests need a description of why the test exists. Functional tests are generally also more resource intensive. We always strive to have one assertion per test, but due to resource requirements it's often necessary to have multiple assertions. We find that RSpec is the best tool for writing our functional tests.

Negative Consequences
Unfortunately, neither UnitRecord or arbs offer painless solutions. Splitting your test suite requires effort. Both UnitRecord and arbs rely on altering the behavior of core Rails classes. If you are unfamiliar with UnitRecord or arbs, you may see unexpected behavior when testing. Even though this is the case, I think the benefits outweigh the occasional confusion.

Having a split test suite can also cause confusion among developers. I've often worked with developers who wanted to write a new test but didn't know where the test belonged. I think "where should the test go" is the wrong question. Building comprehensive test suites requires that new functionality be tested but at the unit and functional level. Therefore, you should always write a unit test (if possible) and then ensure the logic is functionally tested. Since functional tests are generally course grained, it's often the case that the functionality will be covered by an existing test. If the functionality isn't covered by an existing test, then it makes sense to write a new functional test.

Conclusion
I prefer a split test suite because I value readable, reliable, and performant feedback. If you also value these things, you should give splitting your test suite a shot. Unfortunately, you'll be joining the minority. I believe that most Rails developers don't think it's worth the effort. Of course, at one point someone was a minority advocating for the same thing in Java, and now it's the norm.

Extend modules instead of defining methods on a metaclass

In the entry Replace method_missing with dynamic method definitions I have the following example code.

class Decorator
  def initialize(subject)
    subject.public_methods(false).each do |meth|
      (class << self; self; end).class_eval do
        define_method meth do |*args|
          subject.send meth, *args
        end
      end
    end
  end
end

The context of the example can be summarized as, you want to delegate from the instance all the public methods defined on the constructor argument.

Ali Aghareza pointed out to me that defining methods on the metaclass of an instance isn't the nicest thing to do. The problem with it is that you've made it much harder for anyone else to change the behavior of the instance.

Here's a more simplified example. The following code creates a new Object and defines the hello_world method on the Object instance.

class Object
  def metaclass
    class << self; self; end
  end
end

obj = Object.new
obj.metaclass.class_eval do
  def hello_world
    "hello"
  end
end

obj.hello_world # => "hello"

This works fine; however, if someone wanted to change the way hello_world behaved, by defining the method on the metaclass you force them to make their change by redefining the method on the metaclass. The current solution does not allow you to extend modules and alter the behavior of the instance.

The following example demonstrates that extending a module does not change the behavior of an instance if the behavior has been defined on the metaclass.

class Object
  def metaclass
    class << self; self; end
  end
end

obj = Object.new
obj.metaclass.class_eval do
  def hello_world
    "hello"
  end
end

obj.hello_world # => "hello"

module Spanish
  def hello_world
    "hola"
  end
end

obj.extend Spanish

obj.hello_world # => "hello"

A better solution is to change the behavior of the instance by extending modules instead of defining behavior on the metaclass.

obj = Object.new

module English
  def hello_world
    "hello"
  end
end

obj.extend(English).hello_world # => "hello"

Now that the behavior is defined on an ancestor instead of the metaclass you can change the behavior by extending another module.

obj = Object.new

module English
  def hello_world
    "hello"
  end
end

obj.extend(English).hello_world # => "hello"

module Spanish
  def hello_world
    "hola"
  end
end

obj.extend(Spanish).hello_world # => "hola"

This solution works fine for our simple example, but it can also be applied to our first (much more complicated) example, even without knowing how to define the module. In the case of the Decorator, you can simply define an anonymous module and immediately extend it.

class Decorator
  def initialize(subject)
    mod = Module.new do
      subject.public_methods(false).each do |meth|
        define_method meth do |*args|
          subject.send meth, *args
        end
      end
    end
    extend mod
  end
end

Alternatives for redefining methods

Ruby's open classes allow you define and redefine behavior pretty much at will; unfortunately, almost every option comes with caveats.

The example below is a gateway class that defines a process method. For the purposes of the example, assume that we need to redefine the process method on Gateway itself and call the original process method.* Also, assume that Gateway is not our class, so we cannot easily alter the original definition of process.

class Gateway
  def process(document)
    p "gateway processed document: #{document}"
  end
end

Gateway.new.process("hello world") 
# >> "gateway processed document: hello world"

Solution 1: alias

The following example uses alias to redefine the process method.

class Gateway
  alias old_process process
  def process(document)
    p "do something else"
    old_process(document)
  end
end

Gateway.new.process("hello world") 
# >> "do something else"
# >> "gateway processed document: hello world"

The example above creates an alias (old_process) for the process method. With an alias in place you can redefine the process method to anything you want and call the old_process method using the alias. This is probably the easiest solution and the most commonly used solution.

Unfortunately, it's not without problem. First of all, if someone redefines old_process you will get unexpected behavior. Second of all, the old_process method is really nothing more than an artifact of the fact that you have no other way to refer to the original method definition. Lastly, if the code is loaded twice, an infinite loop is created that causes the always painful to see 'stack level too deep' error.

Solution 2: alias_method_chain
Like I said, solution 1 is the most popular way to redefine a method. In fact, it's so popular Rails defines the alias_method_chain method to encapsulate the pattern. From the Rails source above alias_method_chain:

Encapsulates the common pattern of:
#
# alias_method :foo_without_feature, :foo
# alias_method :foo, :foo_with_feature
#
# With this, you simply do:
#
# alias_method_chain :foo, :feature
#
# And both aliases are set up for you.
#
# Query and bang methods (foo?, foo!) keep the same punctuation:
#
# alias_method_chain :foo?, :feature
#
# is equivalent to
#
# alias_method :foo_without_feature?, :foo?
# alias_method :foo?, :foo_with_feature?
#
# so you can safely chain foo, foo?, and foo! with the same feature.

Using alias_method_chain we can define our Gateway as the example below.

class Gateway
  def process_with_logging(document)
    p "do something else"
    process_without_logging(document)
  end
  alias_method_chain :process, :logging
end

Gateway.new.process("hello world") 
# >> "do something else"
# >> "gateway processed document: hello world"

Using alias_method_chain is nice because it's something familiar to many Rails developers. Unfortunately, it also suffers from the same problems as using alias on your own.

Solution 3: Close on an unbound method
The following code uses the class method "instance_method" to assign the "process" method (as an unbound method) to a local variable. The "process_method" local variable is in scope of the closure used to define the new process method, so it can be used within the process definition. Calling an unbound method is as simple as binding it to any instance of the class that it was unbound from and then using the call method.

class Gateway
  process_method = instance_method(:process)
  define_method :process do |document|
    p "do something else"
    process_method.bind(self).call(document)
  end
end

Gateway.new.process("hello world") 
# >> "do something else"
# >> "gateway processed document: hello world"

I've always preferred this solution because it doesn't rely on artifact methods that may or may not collide with other method definitions. Also, if the code is loaded multiple times the behavior is altered multiple times, but I find that easier to diagnose than when my only clue is "stack level too deep".

Unfortunately, this solution is not without flaws. Firstly, it relies on the fact that define_method uses a closure and has access to the unbound method. Of course this also implies that you have a handle on anything else defined in the same context. As with any closure, it's possible to accidentally create a memory leak. Also, (in MRI) I'm told that define_method takes 3 times as long to execute when compared to def.

Solution 4: Extend a module that redefines the method and uses super
This solution relies on creating a module with the new behavior and extending an instance with the module. Since the module is extended from the instance it will be checked first for the method definition when "process" is called (because it's the first ancestor). Since the module is the first ancestor it can use super to execute the process method defined in Gateway (the second ancestor).

module ProcessLogging
  def process(document)
    p "do something else"
    super
  end
end

Gateway.new.extend(ProcessLogging).instance_eval("class << self; self; end").ancestors 
# => [ProcessLogging, Gateway, Object, Kernel]
Gateway.new.extend(ProcessLogging).process("hello world")
# >> "do something else"
# >> "gateway processed document: hello world"

This solution is my favorite because I can use def and super and never worry about creating any memory leaks or artifact methods.

Of course, it assumes that you get the opportunity to extend instances of the class. However, I haven't found that to be a problematic requirement.

* There are generally other options such as delegation, defining hooks, etc. Often I find these to be cleaner solutions and try that route first. But, sometimes redefining a method cannot be avoided.

Domain Specific Language Simplexity

There's one kind of simplicity that I like to call simplexity. When you take something incredibly complex and try to wrap it in something simpler, you often just shroud the complexity -- Anders Hejlsberg

At QCon London I caught the Domain Specific Language (DSL) tutorial by Martin Fowler, Neal Ford, and Rebecca Parsons. While Martin covered how and why you would want to create a DSL he discussed hiding complexity and designing solutions for specific problems. As Martin went into further detail all I could think was: Simplexity.

Anders prefers simple all the way down. Of course, hopefully we all prefer simplicity; however, the devil generally lives in the details.

Some things are complex by nature. Designing a ORM is a great example of a problem that I've yet to see a bullet proof solution for. Most ORMs are complex by necessity, but hiding that complexity in a DSL specific to your application is a good thing. I'll take simplexity over complexity pretty much any day.

I've designed several simplex Domain Specific Languages.

Domain Specific Flow Language
About a year and a half ago Tim Cochran and I designed a few objects that allowed you to define the flow of an application using code similar to the example below.

pages :customer, :shipping, :billing, :summary, :confirmation, :offer, :retailer

flow.customer_driven customer >> shipping >> billing >> summary >> confirmation
flow.retailer_driven retailer >> billing >> summary >> confirmation
flow.offer_driven offer >> shipping >> billing >> summary >> confirmation

The business led us to believe that the flow of the application was going to change often and it needed to be pliable. With that in mind Tim and I set out to create something that was simple to alter and didn't require complete understanding. We wrote out the syntax first and then made it execute. It was fairly easy for us to follow and we had a solution within a few hours. Then we presented the solution to the team.

We showed how to use the DSL and everyone loved it. I think it was actually most people's favorite part of the codebase. Then we dove into how it worked. I'm fairly sure that no one understood the underlying code. The solution was very simplex. It didn't really matter at the time because it was well tested and I don't remember it ever changing in the 6 months I was on the project.

About a year later Jake Scruggs wrote about how the current version of the Flow code is unnecessarily complex. I'm sure Jake is right. If the flows are rarely changing and the underlying code is causing any problems, it should definitely be removed.

The situation brings about an interesting question: how do you know when it's worth creating a DSL? In this example it was a requirement that it be easy to change a flow, but what is easy? I think the original implementation is easier to read and change, but much harder to fully understand and redesign. Unfortunately, there's no simple answer. Like so many things in software development, whether or not to introduce a DSL really depends on the context.

In this case, Tim and I decided to use a DSL because we expected the frequency of change to be high and there was a requirement that the time to make a change needed to be short. Given those requirements, something that is easy to read and change, but hides the underlying complexity is probably a good solution.

Expectations Delegation
I recently created a testing framework: expectations. One of the features of expectations is a solution for testing delegation. Delegation is such a commonly used pattern that I found it helpful to have an simple way to test that delegation is happening as expected.

The following code verifies that delegation does occur to the property that you expect and the result of the method you delegate to is returned from the delegating method.

expect PersonProxy.new.to.delegate(:name).to(:subject) do |proxy|
  proxy.name
end

In the example, the expectation ensures that the proxy.name method calls proxy.subject.name and the return value from proxy.subject.name is also returned by proxy.name.

The resulting expectation is very easy to read and write, but the underlying code is actually quite complex. First of all you need to stub the proxy.subject method. The return value of proxy.subject needs to be a mock that expects a name method call. If the mock receives the name method call you know that delegation happened correctly. The behavior based testing isn't overly complex, but it's not simple either, and it's the easy bit.

Ensuring that the result of proxy.subject.name is returned from proxy.subject is much more complicated. Part of the problem is that the call to proxy.name happens within the block. You could use the result of the block and compare it to what proxy.subject.name is supposed to return, but then the following code would fail.

expect PersonProxy.new.to.delegate(:name).to(:subject) do |proxy|
  proxy.name
  nil
end

You could argue that the above code should cause a failure, but if it does it creates some less than desirable results. First of all, the error message will need to stay something like "Delegation may have occurred, but the return value of the block returned something different so delegation cannot be verified". Also, comparing the result of the block for state based tests makes sense, but here we are only testing delegation so it's not intuitive that the result of the block is significant.

I solved this by specifying the return value of proxy.subject.name, dynamically defining a module that defines the name method, captures the result of the delegation and stores it for later verification, and then extending the proxy with the dynamically defined module. If that's a lot to take in, don't worry I've yet to meet anyone thought it was easy to follow.

Underlying complexity is not desirable. If I could find an easier way to test delegation, I would absolutely change my implementation. Unfortunately, no one has been able to come up with a better solution, yet.

I do think the current solution is better than having to write traditional repetitive and overly verbose code to test every delegation. The DSL for testing delegation allows you to focus on higher value problems. Whether you follow the underlying implementation or not, you must admit that being able to test delegation in a readable and reliable way is a good thing.

Sometimes, complexity is warranted when the resulting interface is simple enough that you gain productivity overall.

Conveying Intention and Removing Noise
Part of what introduces simplexity to a Domain Specific Language is the need to write what you want without introducing side effects. In the expectations example, the return block of the expectation shouldn't be used for comparison because it's not immediately obvious that it would be significant. Using the block return value for comparison is easier from an implementation perspective, but it makes the DSL less usable. This is not a good trade-off.

Noise is defined differently by everyone, but I like to think of it as the difference between what I would like to write and what I need to write.

Removing noise is also a common cause of simplexity. The following code is fairly easy to implement.

expect Process.new.to_have(:finished) do |process|
  process.finished = true
end

But, with expectations I chose to make the code below the legal syntax.

expect Process.new.to.have.finished do |process|
  process.finished = true
end

I chose the later version for a few reasons. The "to" method is the gateway method to all types of expectations. If you want a delegation expectation you write "expect Object.to.delegate...", if you want a behavior based expectation you write "expect Object.to.receive..", and if you want to use a boolean method defined on the class you can use "expect Object.to.be..." or "expect Object.to.have..." depending on what reads better. Using dots allows me to create a consistent interface for all the different expectations, and it also allows me to create expectations on any object without creating several different methods on the Object class.

I also chose to allow dot syntax because once you call "to.be" you can begin calling methods exactly as you would on the object itself. If the object is designed with fluency in mind the test can read as a dot delimited pseudo sentence. For example, the following test is an easy way to verify validation.

class Person < ActiveRecord::Base
  validates_presence_of :name
end

expect Person.new.to.have.errors.on(:name) do |person|
  person.valid?
end

To get this desired behavior I rely on the use of method_missing. Using the method_missing method almost always increases complexity by an order of magnitude. However, in this case, if I didn't use method missing I'd need a solution similar to the one below.

expect Person.new.to.be.true.when(:errors).then(:on, [:name]) do |person|
  person.valid?
end

While this version is easier to implement, it's much less friendly to use.

The level of simplexity should probably be defined by the usage of the Domain Specific Language. If the language is only used by a small subset of employees on few occasions then it might not make sense to increase simplexity. Also, if the complexity of the implementation is raised to a level that it cannot effectively be maintained, you absolutely must back off some. However, if the complexity is maintainable and the DSL is used frequently, I would do my best to reduce "noise".

Conclusion
I believe that simplexity is inevitable when designing a Domain Specific Language, and it shouldn't be something that stops you from using one. However, it's always valuable to weigh how much complexity has been introduced by the desire to create a simple user experience. If the complexity is more painful than the benefit to the consumers, you've clearly gone too far. However, if the complexity is painless, but the DSL is so unpleasant that it's usage is limited, you might need to work on designing more friendly DSL, even at the expense of simplicity.

The Language Question

My first job was working with Cold Fusion for a year. For the next two years I worked primarily with Javascript and a proprietary language. Next was a brief stretch with PHP and ASP, followed by about 3 years of .net. Finally, for the past 2 years I've been doing Ruby. I do not consider myself a Ruby developer. I prefer to just be a developer.

I don't expect that Ruby will be the last language I ever work with. In fact, I expect I'll probably be doing something new in the near future. The question is what language should I work with next? Should I go for a classic like Lisp or Smalltalk? Should I give Scala or Erlang a shot and ride the concurrency wave?

The problem is, I'm asking the wrong question. Rarely when discussing languages do the languages themselves dominate the conversation. I can't remember ever being in a conversation where someone won a language debate with things like closures, metaprogramming or static typing. Instead, the primary focus of language debates are usually around frameworks, performance, IDE support, developer availability, what OS does it run on, and any number of other factors that are only related to the language.

The question extends even beyond factors related to the language.

At Google I’ll work with C++ rather than (for example) Ruby but I do get to be part of changing the world. -- Jon Tirsen

Jon gave up a job using his language of choice to work with another language he likes far less, but for a cause he's very interested in. I have another friend who recently starting working with an investment company because he was interested in the domain.

Two years ago I started looking at Ruby because several of my colleagues were giving it a look. I preferred Rake to NAnt, and starting using it on my .net projects. Before long, someone asked me to be part of a Ruby project, because of my limited exposure to Rake.

I got introduced to Ruby by coworkers who were interested. I got involved with Ruby because I prefer build files that do not require XML programming. I stuck with Ruby because we had a steady stream of Ruby projects that needed experienced developers, I got plenty of blog content, I liked the composition of the Ruby teams I got to work with, and I liked working with clients who are comfortable with early adoption.

Notice, none of the reasons I use Ruby have anything to do with the Ruby language itself.

I'm interested in doing something new because I feel like I've been doing the same thing for about a year now. I'm also interested in traveling to other ThoughtWorks offices and getting some fresh ideas for innovation.

Again, none of my desires have anything to do with language.

I'm not giving you the tired "no silver bullet" or the hand waving "the right tool for the right job". I'm asserting that people use those phrases to justify their language choice, but you'd be better off asking what the real motivations for choosing a language are. What other factors does the language introduce that make it their choice.

It's also helpful to have this understanding when considering criticizing someone's language choice. My friends aren't using Java because they like Java, they are using it because they like IntelliJ, high performance on a few boxes, simple deployment, Hibernate, String Template, Spring, and a hundred other factors. Therefore, criticizing Java as a language doesn't really do anyone any good. Even if I convinced them that Lisp is a better language than Java, I still wouldn't have convinced them to use Lisp on their next project.