Taking a Second Look at Collective Code Ownership

It's common to hear proponents of Agile discussing the benefits of collective code ownership. The benefits can be undeniable. Sharing knowledge ensures at least one other perspective and drastically reduces Bus Risk. However, sharing comes at a cost: time. The ROI of sharing with a few people can be greatly different than the ROI of sharing with 5 or more people.

I do believe in the benefits of collective code ownership. Collective code ownership is a step towards taking an underachieving team and turning them into a good team. However, I'm becoming more and more convinced that it's not the way to take good team and make them great.

(context: I believe these ideas apply to teams larger than 3. Teams of 2-3 should likely be working with everyone on everything.)

If you pair program, you will incur a context switch every time you rotate pairs. Context switches always have a non-zero cost, and the more you work on, the larger the context switch is likely to be. Teams where everyone works on everything are very likely paying for expensive context switches on a regular basis. In fact, the words Context Switch are often used specifically to point out the cost.

Working on everything also ensures a non-trivial amount of time ramping up on whatever code you are about to start working on. Sometimes you work on something you know fairly well. Other times you're working on something you've never seen before. Working on something you've never seen before creates two choices (assuming pair-programming): go along for the ride, understanding little - or - slow your pair down significantly while they explain what's going on.

Let's say you choose to slow your pair down for the full explanation: was it worth it? If you're the only other person that knows the component, it's very likely that it was worth it. What if everyone else on the team already knows that component deeply? Well, if they all die, you can maintain the app, but I don't think that's going to be the largest issue on the team.

(the same ideas apply if you don't pair-program, except you don't have the "go along for the ride" option)

I can hear some of you right now: We rotate enough that the context switch is virtually free and because we rotate so much there's little ramp up time. You might be right. Your problem domain might be so simple that jumping on and off of parts of your system is virtually free. However, if you're domain is complex in anyway, I think you're underestimating the cost of context switches and ramp up time. Also, the devil is traditionally in the details, so you're "simple domain" probably isn't as simple as you think.

Let's assume your domain is that simple: it might be cheaper to rewrite the software than take your Bus Number from 3 to 4.

Another benefit of pair programming combined with collective code ownership is bringing up everyone's skill level to that of the most skilled team member. In my opinion, that's something you need to worry about on an underachieving team, not a good team. If you're on a good team, it's likely that you can learn just as much from any member of your team; therefore, you are not losing anything by sticking to working with just a few of them in specific areas. You really only run into an education problem if you're team has more learners than mentors - and, in that case, you're not ready to worry about going from good to great.

There's also opportunity cost of not sticking to certain areas. Focusing on a problem allows you to create better solutions. Specifically, it allows you to create a vision of what needs to be done, work towards that vision and constantly revise where necessary.

Mark Twain once wrote that his letters would be shorter if he had more time. The same is often true of software. The simplest solution is not always the most obvious. If you're jumping from problem to problem, you're more likely to create an inferior solution. You'll solve problems, but you'll be creating higher maintenance costs for the project in the long term.

Instead, I often find it very helpful to ponder a problem and create a more simple and, very often, a more concise solution. In my experience, the maintenance costs are also greatly reduced by the simplified, condensed solution.

I'd like to repeat for clarity: Collective code ownership has benefits. There's no doubt that it is better to have everyone work on everything than have everyone focused on individual parts of the codebase. However, it's worth considering the cost of complete sharing if you are trying to move from good to great.

Experience Report: Feature Toggle over Feature Branch

We often use Feature Toggle on my current team (when gradual release isn't possible). My experience so far has been: gradual release is better than Feature Toggle, and Feature Toggle is better than Feature Branch.

I found Martin's bliki entry on Feature Toggle to be a great description, but the entry doesn't touch on the primary reasons why I prefer Feature Toggle to Feature Branch.

When using Feature Branch I have to constantly rebase to avoid massive merge issues, in general. That means rebasing during development and testing. Rebasing a branch while it's complete and being tested, has often lead to subtle, merge related bugs that go unnoticed because the feature is not under active development.

Additionally, (and likely more problematic) once I merge a feature branch I am committed (no pun intended). If a bug in the new feature, that requires rolling production back to the previous release, is found after the branch has been merged I find myself rolling back the feature commit or continuing with a un-releasable trunk. Rolling back the commit is painful because it is likely a large commit. If I continue on with an un-releasable trunk and another (unrelated to the new feature) bug is found in trunk I'm in trouble: I can't fix the new bug and release.

That's bad. I either lose significant time rolling back the release (terrible), or I roll the dice (terrifying). I was burned by this situation once already this year. It is not something I'm looking to suffer again in the near future.

Feature Toggle avoids this entirely by allowing me to run with the toggle available and turn it back on if things go wrong. When I feel comfortable that everything is okay (usually, a week in prod is good enough), I clean up the unnecessary toggles.

Of course, nothing is black & white. Sometimes a Feature Toggle increases the scope of the work to an unacceptable level, and Feature Branch is the correct decision. However, I always weigh the terrible/terrifying situation when I'm choosing which direction is optimal.

It's also worth noting, I roll the current day's changes into production every night. It's possible that your experience will be vastly different if your release schedules are multi-day or multi-week.

Clojure: Flatten Keys

I recently needed to take a nested map and flatten the keys. After a bit of trial and error I came up with the following code. (note: the example expects Expectations)

(ns example
  (:use expectations))

(defn flatten-keys* [a ks m]
  (if (map? m)
    (reduce into (map (fn [[k v]] (flatten-keys* a (conj ks k) v)) (seq m)))
    (assoc a ks m)))

(defn flatten-keys [m] (flatten-keys* {} [] m))

(expect
  {[:z] 1, [:a] 9, [:b :c] Double/NaN, [:b :d] 1, [:b :e] 2, [:b :f :g] 10, [:b :f :i] 22} 
  (flatten-keys {:z 1 :a 9 :b {:c Double/NaN :d 1 :e 2 :f {:g 10 :i 22}}}))

As the test shows, the code converts

{:z 1 :a 9 :b {:c Double/NaN :d 1 :e 2 :f {:g 10 :i 22}}}

into

{[:z] 1, [:a] 9, [:b :c] Double/NaN, [:b :d] 1, [:b :e] 2, [:b :f :g] 10, [:b :f :i] 22}

Improvement suggestions welcome.

Clojure: Another Testing Framework - Expectations

Once upon a time I wrote Expectations for Ruby. I wanted a simple testing framework that allowed me to specify my test with the least amount of code.

Now that I'm spending the majority of my time in Clojure, I decided to create a version of Expectations for Clojure.

At first it started as a learning project, but I kept adding productivity enhancements. Pretty soon, it became annoying when I wasn't using Expectations. Obviously, if you write your own framework you are going to prefer to use it. However, I think the productivity enhancements might be enough for other people to use it as well.

So why would you want to use it?

Tests run automatically. Clojure hates side effects, yeah, I hear you. But, I hate wasting time and repeating code. As a result, Expectations runs all the tests on JVM shutdown. This allows you to execute a single file to run all the tests in that file, without having to specify anything additional. There's also a hook you can call if you don't want the tests to automatically run. (If you are looking for an example, there's a JUnit runner that disables running tests on shutdown)

What to test is inferred from your "expected" value. An equality test is probably the most common test written. In Expectations, an equality test looks like the following example.

(expect 3 (+ 1 2))

That's simple enough, but what if you want to match a regex against a string? The following example does exactly that, and it uses the same syntax.

(expect #"foo" "afoobar")

Other common tests are verifying an exception is thrown or checking the type of an actual value. The following snippets test those two conditions.

(expect ArithmeticException (/ 12 0))

(expect String "foo")

Testing subsets of the actual value. Sometimes you want an exact match, but there are often times when you only care about a subset of the actual value. For example, you may want to test all the elements of a map except the time and id pairs (presumably because they are dynamic). The following tests show how you can verify that some key/value pairs are in a map, a element is in a set, or an element is in a list.

;; k/v pair in map. matches subset
(expect {:foo 1} (in {:foo 1 :cat 4}))

;; key in set
(expect :foo (in (conj #{:foo :bar} :cat)))

;; val in list
(expect :foo (in (conj [:bar] :foo)))

Double/NaN is annoying. (not= Double/NaN Double/NaN) ;=> true. I get it, conceptually. In practice, I don't want my tests failing because I can't compare two maps that happen to have Double/NaN as the value for a matching key. In fact, 100% of the time I want (= Double/NaN Double/NaN) ;=> true. And, yes, I can rewrite the test and use Double/isNaN. I can. But, I don't want to. Expectations allows me to pretend (= Double/NaN Double/NaN) ;=> true. It might hurt me in the future. I'll let you know. For now, I prefer to write concise tests that behave as "expected".

Try rewriting this and using Double/isNaN (it's not fun)

(expect
  {:x 1 :a Double/NaN :b {:c Double/NaN :d 2 :e 4 :f {:g 11 :h 12}}}
  {:x 1 :a Double/NaN :b {:c Double/NaN :d 2 :e 4 :f {:g 11 :h 12}}})

Concise Java Object testing. Inevitably, I seem to end up with a few Java objects. I could write a bunch of different expect statements, but I opted for a syntax that allows me to check everything at once.

(given (java.util.ArrayList.)
       (expect
        .size 0
        .isEmpty true))

Trimmed Stacktraces. I'm sure it's helpful to look through Clojure and Java's classes at times. However, I find the vast majority of the time the problem is in my code. Expectations trims many of the common classes that are from Clojure and Java, leaving much more signal than noise. Below is the stacktrace reported when running the failure examples from the Expectations codebase.

failure in (failure_examples.clj:8) : failure.failure-examples
      raw: (expect 1 (one))
  act-msg: exception in actual: (one)
    threw: class java.lang.ArithmeticException-Divide by zero
           failure.failure_examples$two__375 (failure_examples.clj:4)
           failure.failure_examples$one__378 (failure_examples.clj:5)
           failure.failure_examples$G__381__382$fn__387 (failure_examples.clj:8)
           failure.failure_examples$G__381__382 (failure_examples.clj:8)

Every stacktrace line is from my code, where the problem lives.

Descriptive Error Messages. Expectations does it's best to give you all the important information when a failure does occur. The following failure shows what keys are missing from actual and expected as well is which values do not match.

running

   (expect
     {:z 1 :a 9          :b {:c Double/NaN :d 1 :e 2 :f {:g 10 :i 22}}}
     {:x 1 :a Double/NaN :b {:c Double/NaN :d 2 :e 4 :f {:g 11 :h 12}}})

generates

   failure in (failure_examples.clj:110) : failure.failure-examples
         raw: (expect {:z 1, :a 9, :b {:c Double/NaN, :d 1, :e 2, :f {:g 10, :i 22}}} {:x 1, :a Double/NaN, :b {:c Double/NaN, :d 2, :e 4, :f {:g 11, :h 12}}})
      result: {:z 1, :a 9, :b {:c NaN, :d 1, :e 2, :f {:g 10, :i 22}}} are not in {:x 1, :a NaN, :b {:c NaN, :d 2, :e 4, :f {:g 11, :h 12}}}
     exp-msg: :x is in actual, but not in expected
              :b {:f {:h is in actual, but not in expected
     act-msg: :z is in expected, but not in actual
              :b {:f {:i is in expected, but not in actual
     message: :b {:e expected 2 but was 4
              :b {:d expected 1 but was 2
              :b {:f {:g expected 10 but was 11
              :a expected 9 but was NaN

note: I know it's a bit hard to read, but I wanted to cover all the possible errors with one example. In practice you'll get a few messages that will tell you exactly what is wrong.

For example, the error tells you

:b {:f {:g expected 10 but was 11

With that data it's pretty easy to see the problem in

(expect
  {:z 1 :a 9          :b {:c Double/NaN :d 1 :e 2 :f {:g 10 :i 22}}}
  {:x 1 :a Double/NaN :b {:c Double/NaN :d 2 :e 4 :f {:g 11 :h 12}}})

Expectations also tells you, when comparing two lists:

if the lists are the same, but differ only in order

if the lists are the same, but one list has duplicates

if the lists are not the same, which list is larger

JUnit integration. My project uses both Java and Clojure. I like running my tests in IntelliJ and I like TeamCity running my tests as part of the build. To accomplish this using Expectations all you need to do is create a java class similar to the example below.

import expectations.junit.ExpectationsTestRunner;
import org.junit.runner.RunWith;

@RunWith(expectations.junit.ExpectationsTestRunner.class)
public class FailureTest implements ExpectationsTestRunner.TestSource{

    public String testPath() {
        return "/path/to/the/root/folder/holding/your/tests";
    }
}

The Expectations Test Runner runs your Clojure tests in the same way that the Java tests run, including the green/red status icons and clickable links when things fail.

Why wouldn't you use Expectations?

Support. I'm using it to test my production code, but if I find errors I have to go fix them. You'll be in the same situation. I'll be happy to fix any bugs you find, but I might not have the time to get to it as soon as you send me email.

If you're willing to live on the bleeding edge, feel free to give it a shot.

Clojure: Mocking

An introduction to clojure.test is easy, but it doesn't take long before you feel like you need a mocking framework. As far as I know, you have 3 options.

1. Take a look at Midje. I haven't gone down this path, but it looks like the most mature option if you're looking for a sophisticated solution.
   


2. Go simple. Let's take an example where you want to call a function that computes a value and sends a response to a gateway. Your first implementation looks like the code below. (destructuring explained)

   (defn withdraw [& {:keys [balance withdrawal account-number]}]
    (gateway/process {:balance (- balance withdrawal)
                      :withdrawal withdrawal
                      :account-number account-number}))

   No, it's not pure. That's not the point. Let's pretend that this impure function is the right design and focus on how we would test it.
   
   You can change the code a bit and pass in the gateway/process function as an argument. Once you've changed how the code works you can test it by passing identity as the function argument in your tests. The full example is below.

   (ns gateway)
   
   (defn process [m] (println m))
   
   (ns controller
    (:use clojure.test))
   
   (defn withdraw [f & {:keys [balance withdrawal account-number]}]
    (f {:balance (- balance withdrawal)
        :withdrawal withdrawal
        :account-number account-number}))
   
   (withdraw gateway/process :balance 100 :withdrawal 22 :account-number 4)
   ;; => {:balance 78, :withdrawal 22, :account-number 4}
   
   (deftest withdraw-test
    (is (= {:balance 78, :withdrawal 22, :account-number 4}
     (withdraw identity :balance 100 :withdrawal 22 :account-number 4))))
   
   (run-all-tests #"controller")

   If you run the previous example you will see the println output and the clojure.test output, verifying that our code is working as we expected. This simple solution of passing in your side effect function and using identity in your tests can often obviate any need for a mock.
   


3. Solution 2 works well, but has the limitations that only one side-effecty function can be passed in and it's result must be used as the return value.
   
   Let's extend our example and say that we want to log a message if the withdrawal would cause insufficient funds. (Our gateway/process and log/write functions will simply println since this is only an example, but in production code their behavior would differ and both would be required)

   (ns gateway)
   
   (defn process [m] (println "gateway: " m))
   
   (ns log)
   
   (defn write [m] (println "log: " m))
   
   (ns controller
     (:use clojure.test))
   
   (defn withdraw [& {:keys [balance withdrawal account-number]}]
     (let [new-balance (- balance withdrawal)]
       (if (> 0 new-balance)
         (log/write "insufficient funds")
         (gateway/process {:balance new-balance
                           :withdrawal withdrawal
                           :account-number account-number}))))
   
   (withdraw :balance 100 :withdrawal 22 :account-number 4)
   ;; => gateway:  {:balance 78, :withdrawal 22, :account-number 4}
   
   (withdraw :balance 100 :withdrawal 220 :account-number 4)
   ;; => log:  insufficient funds

   Our new withdraw implementation calls two functions that have side effects. We could pass in both functions, but that solution doesn't seem to scale very well as the number of passed functions grows. Also, passing in multiple functions tends to clutter the signature and make it hard to remember what is the valid order for the arguments. Finally, if we need withdraw to always return a map showing the balance and withdrawal amount, there would be no easy solution for verifying the string sent to log/write.
   
   Given our implementation of withdraw, writing a test that verifies that gateway/process and log/write are called correctly looks like a job for a mock. However, thanks to Clojure's binding function, it's very easy to redefine both of those functions to capture values that can later be tested.
   
   The following code rebinds both gateway/process and log/write to partial functions that capture whatever is passed to them in an atom that can easily be verified directly in the test.

   (ns gateway)
   
   (defn process [m] (println "gateway: " m))
   
   (ns log)
   
   (defn write [m] (println "log: " m))
   
   (ns controller
     (:use clojure.test))
   
   (defn withdraw [& {:keys [balance withdrawal account-number]}]
     (let [new-balance (- balance withdrawal)]
       (if (> 0 new-balance)
         (log/write "insufficient funds")
         (gateway/process {:balance new-balance
                           :withdrawal withdrawal
                           :account-number account-number}))))
   
   (deftest withdraw-test1
     (let [result (atom nil)]
       (binding [gateway/process (partial reset! result)]
         (withdraw :balance 100 :withdrawal 22 :account-number 4)
         (is (= {:balance 78, :withdrawal 22, :account-number 4} @result)))))
   
   (deftest withdraw-test2
     (let [result (atom nil)]
       (binding [log/write (partial reset! result)]
         (withdraw :balance 100 :withdrawal 220 :account-number 4)
         (is (= "insufficient funds" @result)))))
   
   (run-all-tests #"controller")

In general I use option 2 when I can get away with it, and option 3 where necessary. Option 3 adds enough additional code that I'd probably look into Midje quickly if I found myself writing a more than a few tests that way. However, I generally go out of my way to design pure functions, and I don't find myself needing either of these techniques very often.