Jay Fields' Thoughts

An introduction to clojure.test is easy, but it doesn't take long before you feel like you need a mocking framework. As far as I know, you have 3 options.

1. Take a look at Midje. I haven't gone down this path, but it looks like the most mature option if you're looking for a sophisticated solution.
   


2. Go simple. Let's take an example where you want to call a function that computes a value and sends a response to a gateway. Your first implementation looks like the code below. (destructuring explained)

   (defn withdraw [& {:keys [balance withdrawal account-number]}]
    (gateway/process {:balance (- balance withdrawal)
                      :withdrawal withdrawal
                      :account-number account-number}))

   No, it's not pure. That's not the point. Let's pretend that this impure function is the right design and focus on how we would test it.
   
   You can change the code a bit and pass in the gateway/process function as an argument. Once you've changed how the code works you can test it by passing identity as the function argument in your tests. The full example is below.

   (ns gateway)
   
   (defn process [m] (println m))
   
   (ns controller
    (:use clojure.test))
   
   (defn withdraw [f & {:keys [balance withdrawal account-number]}]
    (f {:balance (- balance withdrawal)
        :withdrawal withdrawal
        :account-number account-number}))
   
   (withdraw gateway/process :balance 100 :withdrawal 22 :account-number 4)
   ;; => {:balance 78, :withdrawal 22, :account-number 4}
   
   (deftest withdraw-test
    (is (= {:balance 78, :withdrawal 22, :account-number 4}
     (withdraw identity :balance 100 :withdrawal 22 :account-number 4))))
   
   (run-all-tests #"controller")

   If you run the previous example you will see the println output and the clojure.test output, verifying that our code is working as we expected. This simple solution of passing in your side effect function and using identity in your tests can often obviate any need for a mock.
   


3. Solution 2 works well, but has the limitations that only one side-effecty function can be passed in and it's result must be used as the return value.
   
   Let's extend our example and say that we want to log a message if the withdrawal would cause insufficient funds. (Our gateway/process and log/write functions will simply println since this is only an example, but in production code their behavior would differ and both would be required)

   (ns gateway)
   
   (defn process [m] (println "gateway: " m))
   
   (ns log)
   
   (defn write [m] (println "log: " m))
   
   (ns controller
     (:use clojure.test))
   
   (defn withdraw [& {:keys [balance withdrawal account-number]}]
     (let [new-balance (- balance withdrawal)]
       (if (> 0 new-balance)
         (log/write "insufficient funds")
         (gateway/process {:balance new-balance
                           :withdrawal withdrawal
                           :account-number account-number}))))
   
   (withdraw :balance 100 :withdrawal 22 :account-number 4)
   ;; => gateway:  {:balance 78, :withdrawal 22, :account-number 4}
   
   (withdraw :balance 100 :withdrawal 220 :account-number 4)
   ;; => log:  insufficient funds

   Our new withdraw implementation calls two functions that have side effects. We could pass in both functions, but that solution doesn't seem to scale very well as the number of passed functions grows. Also, passing in multiple functions tends to clutter the signature and make it hard to remember what is the valid order for the arguments. Finally, if we need withdraw to always return a map showing the balance and withdrawal amount, there would be no easy solution for verifying the string sent to log/write.
   
   Given our implementation of withdraw, writing a test that verifies that gateway/process and log/write are called correctly looks like a job for a mock. However, thanks to Clojure's binding function, it's very easy to redefine both of those functions to capture values that can later be tested.
   
   The following code rebinds both gateway/process and log/write to partial functions that capture whatever is passed to them in an atom that can easily be verified directly in the test.

   (ns gateway)
   
   (defn process [m] (println "gateway: " m))
   
   (ns log)
   
   (defn write [m] (println "log: " m))
   
   (ns controller
     (:use clojure.test))
   
   (defn withdraw [& {:keys [balance withdrawal account-number]}]
     (let [new-balance (- balance withdrawal)]
       (if (> 0 new-balance)
         (log/write "insufficient funds")
         (gateway/process {:balance new-balance
                           :withdrawal withdrawal
                           :account-number account-number}))))
   
   (deftest withdraw-test1
     (let [result (atom nil)]
       (binding [gateway/process (partial reset! result)]
         (withdraw :balance 100 :withdrawal 22 :account-number 4)
         (is (= {:balance 78, :withdrawal 22, :account-number 4} @result)))))
   
   (deftest withdraw-test2
     (let [result (atom nil)]
       (binding [log/write (partial reset! result)]
         (withdraw :balance 100 :withdrawal 220 :account-number 4)
         (is (= "insufficient funds" @result)))))
   
   (run-all-tests #"controller")

In general I use option 2 when I can get away with it, and option 3 where necessary. Option 3 adds enough additional code that I'd probably look into Midje quickly if I found myself writing a more than a few tests that way. However, I generally go out of my way to design pure functions, and I don't find myself needing either of these techniques very often.

Labels: clojure, mock, TDD

Clojure sets and maps are functions.

Since they are functions, you don't need functions to get values out of them. You can use the map or set as the example below shows.

(#{1 2} 1)
> 1

({:a 2 :b 3} :a)
> 2

That's nice, but it's not exactly game changing. However, when you use sets or maps with high order functions you can get a lot of power with a little code.

For example, the following code removes all of the elements of a vector if the element is also in the set.

(def banned #{"Steve" "Michael"})
(def guest-list ["Brian" "Josh" "Steve"])

(remove banned guest-list)
> ("Brian" "Josh")

I'm a big fan of using sets in the way described above, but I don't often find myself using maps in the same way. The following code works, but I rarely use maps as predicates.

(def banned {"Steve" [] "Michael" []})
(def guest-list ["Brian" "Josh" "Steve"])

(remove banned guest-list)
> ("Brian" "Josh")

However, yesterday I needed to compare two maps and get the list of ids in the second map where the quantities didn't match the quantities in the first map. I started by using filter and defining a function that checks if the quantities are not equal. The following code shows solving the problem with that approach.

; key/value pairs representing order-id and order-quantity
(def map1 {1 44 2 33})
(def map2 {1 55 2 33})

(defn not=quantities [[id qty]] (not= (map1 id) qty))
(keys (filter not=quantities map2))
> (1)

However, since you can use maps as filter functions you can also solve the problem by merging the maps with not= and filtering by the result. The following code shows an example of merging and using the result as the predicate.

; key/value pairs representing order-id and order-quantity
(def map1 {1 44 2 33})
(def map2 {1 55 2 33})

(filter (merge-with not= map1 map2) (keys map2))
> (1)

I don't often find myself using maps as predicates, but in certain cases it's exactly what I need.

Labels: clojure, functions, maps, sets

I'll admit it, the first thing I like to do when learning a new language is fire up a REPL. However, I'm usually ready for the next step after typing in a few numbers, strings and defining a function or two.

What feels like centuries ago, Mike Clark wrote an article about using unit testing to learn a new language. Mike was ahead of his time. This blog entry should help you if you want to follow Mike's advice.

Luckily, Clojure has built in support for simple testing. (I'm currently using Clojure 1.2, you can download it from clojure.org)

Before we get started, let's make sure everything is working. Save a file with the following clojure in it and run* it with clojure.

(ns clojure.test.example
  (:use clojure.test))

(run-all-tests)

If everything is okay, you should see something similar the following output.

Testing clojure.walk

Testing clojure.core

(a bunch of other namespaces tested)

Testing clojure.zip

Ran 0 tests containing 0 assertions.
0 failures, 0 errors.

If you've gotten this far, you are all set to start writing your own tests. If you are having any trouble, I suggest logging into the #clojure IRC chat room on Freenode.net

The syntax for defining tests is very simple. The following test verifies that 1 + 1 = 2. You'll want to add the test after the ns definition and before the (run-all-tests) in the file you just created.

(deftest add-1-to-1
  (is (= 2 (+ 1 1))))

Running the test should produce something similar to the following output.

Testing clojure.walk

Testing clojure.test.example

(a bunch of other namespaces tested)

Ran 1 tests containing 1 assertions.
0 failures, 0 errors.

We see all of the standard clojure namespaces; however, we see our namespace (clojure.test.example) in the results as well. The output at the bottom also tells us that 1 test with 1 assertion was executed.

The following example shows testing a custom add function. (we will add additional tests from here, without ever deleting the old tests)

(defn add [x y] (+ x y))

(deftest add-x-to-y
  (is (= 5 (add 2 3))))

If everything goes to plan, running your tests should now produce the following text towards the bottom of the output.

Ran 2 tests containing 2 assertions.
0 failures, 0 errors.

At this point you might want to pass in a few different numbers to verify that add works as expected.

(deftest add-x-to-y-a-few-times
  (is (= 5 (add 2 3)))
  (is (= 5 (add 1 4)))
  (is (= 5 (add 3 2))))

Running the tests shows us our status

Ran 3 tests containing 5 assertions.
0 failures, 0 errors.

This works perfectly fine; however, clojure.test also provides are for verifying several values.

The following example tests the same conditions using are.

(deftest add-x-to-y-a-using-are
  (are [x y] (= 5 (add x y))
       2 3
       1 4
       3 2))

And, the unsurprising results.

Ran 4 tests containing 8 assertions.

That's a simple are; however, you can do whatever you need in the form. Let's grab the value out of a map as an additional example.

(deftest grab-map-values-using-are
  (are [y z] (= y (:x z))
       2 {:x 2}
       1 {:x 1}
       3 {:x 3 :y 4}))

Leaving us with

Ran 5 tests containing 11 assertions.

The is and are macros will be all that you need for 90% of all the tests you'll ever want to write. For additional assertions and more details you can check out the clojure.test documentation.

Advanced Topics (very unnecessary to get started)

I get annoyed with noise in my test results. Our results have been very noisy due to the namespace reporting. The run-all-tests function takes a regular expression (documented here). We can change our test running call to include a regular expression, as the following example shows.

(run-all-tests #"clojure.test.example")

Once we switch to providing a regular expression the results should be limited to the following output.

Testing clojure.test.example

Ran 5 tests containing 11 assertions.
0 failures, 0 errors.

This approach works fine for our current sample file; however, it seems like a better solution would be to stop reporting namespaces that do not contain any tests. The following snippet changes the report multimethod to ignore namespaces that don't contain any tests.

(defmethod report :begin-test-ns [m]
    (with-test-out
      (when (some #(:test (meta %)) (vals (ns-interns (:ns m))))
        (println "\nTesting" (ns-name (:ns m))))))

If you're just getting started, don't worry you don't need to understand what's going on in that snippet. I've copied the original report method and made it conditional by adding the code in bold. As a result, the namespace is only printed if it contains any tests.

Now that our results are clean, let's talk about ways of getting those results.

Adding calls to the run-all-tests function isn't a big deal when working with one namespace; however, you'll need to get clever when you want to run a suite of tests. I've been told that leiningen and Maven have tasks that allow you to run all the tests. You might want to start there. I don't currently use either one, and I'm lazy. I don't want to set up either, especially since all I want to do is run all my tests.

It turns out it's very easy to add a shutdown hook in Java. So, as a simple solution, I run all my tests from the Java shutdown hook.

(.addShutdownHook
 (Runtime/getRuntime)
 (proxy [Thread] []
   (run []
 (run-all-tests))))

In general, I create a test_helper.clj with the following code.

(ns test-helper
  (:use clojure.test))

(defmethod report :begin-test-ns [m]
    (with-test-out
      (if (some #(:test (meta %)) (vals (ns-interns (:ns m))))
        (println "\nTesting" (ns-name (:ns m))))))

(.addShutdownHook
 (Runtime/getRuntime)
 (proxy [Thread] []
   (run []
 (run-all-tests))))

Once you've created a test_helper.clj you can use test-helper (just like you used clojure.test) (example below) and your tests will automatically be run on exit, and only namespaces with tests will be included in the output.

It's worth noting that some clojure.contrib namespaces seem to include tests, so in practice I end up using a regular expression that ignores all namespaces beginning with "clojure"** when running all tests. With all of those ideas combined, I find I can execute all my tests or only the tests in the current namespace very easily.

Below you can find all the code from this entry.

clojure.test.example.clj

(ns clojure.test.example
  (:use clojure.test test-helper))

(deftest add-1-to-1
  (is (= 2 (+ 1 1))))

(defn add [x y] (+ x y))

(deftest add-x-to-y
  (is (= 5 (add 2 3))))

(deftest add-x-to-y-a-few-times
  (is (= 5 (add 2 3)))
  (is (= 5 (add 1 4)))
  (is (= 5 (add 3 2))))

(deftest add-x-to-y-a-using-are
  (are [x y] (= 5 (add x y))
       2 3
       1 4
       3 2))

(deftest grab-map-values-using-are
  (are [y z] (= y (:x z))
       2 {:x 2}
       1 {:x 1}
       3 {:x 3 :y 4}))

test_helper.clj

(ns test-helper
  (:use clojure.test))

(defmethod report :begin-test-ns [m]
    (with-test-out
      (if (some #(:test (meta %)) (vals (ns-interns (:ns m))))
        (println "\nTesting" (ns-name (:ns m))))))

(.addShutdownHook
 (Runtime/getRuntime)
 (proxy [Thread] []
   (run []
 (run-all-tests))))

* Running a clojure file should be as easy as: java -cp /path/to/clojure.jar clojure.main -i file.to.run.clj

** (run-all-tests #"[^(clojure)].*") ; careful though, now your clojure.test.example tests will be ignored. Don't let that confuse you.

Labels: clojure, testing, unit testing

In The Joy of Clojure (TJoC) destructuring is described as a mini-language within Clojure. It's not essential to learn this mini-language; however, as the authors of TJoC point out, destructuring facilitates concise, elegant code.

What is destructuring?

Clojure supports abstract structural binding, often called destructuring, in let binding lists, fn parameter lists, and any macro that expands into a let or fn. -- http://clojure.org/special_forms

The simplest example of destructuring is assigning the values of a vector.

user=> (def point [5 7])
#'user/point

user=> (let [[x y] point]
         (println "x:" x "y:" y))
x: 5 y: 7

note: I'm using let for my examples of destructuring; however, in practice I tend to use destructuring in function parameter lists at least as often, if not more often.

I'll admit that I can't remember ever using destructuring like the first example, but it's a good starting point. A more realistic example is splitting a vector into a head and a tail. When defining a function with an arglist** you use an ampersand. The same is true in destructuring.

user=> (def indexes [1 2 3])
#'user/indexes

user=> (let [[x & more] indexes]
         (println "x:" x "more:" more))
x: 1 more: (2 3)

It's also worth noting that you can bind the entire vector to a local using the :as directive.

user=> (def indexes [1 2 3])
#'user/indexes

user=> (let [[x & more :as full-list] indexes]
         (println "x:" x "more:" more "full list:" full-list))
x: 1 more: (2 3) full list: [1 2 3]

Vector examples are the easiest; however, in practice I find myself using destructuring with maps far more often.

Simple destructuring on a map is as easy as choosing a local name and providing the key.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{the-x :x the-y :y} point]
         (println "x:" the-x "y:" the-y))
x: 5 y: 7

As the example shows, the values of :x and :y are bound to locals with the names the-x and the-y. In practice we would never prepend "the-" to our local names; however, using different names provides a bit of clarity for our first example. In production code you would be much more likely to want locals with the same name as the key. This works perfectly well, as the next example shows.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{x :x y :y} point]
         (println "x:" x "y:" y))
x: 5 y: 7

While this works perfectly well, creating locals with the same name as the keys becomes tedious and annoying (especially when your keys are longer than one letter). Clojure anticipates this frustration and provides :keys directive that allows you to specify keys that you would like as locals with the same name.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{:keys [x y]} point]
         (println "x:" x "y:" y))
x: 5 y: 7

There are a few directives that work while destructuring maps. The above example shows the use of :keys. In practice I end up using :keys the most; however, I've also used the :as directive while working with maps.

The following example illustrates the use of an :as directive to bind a local with the entire map.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{:keys [x y] :as the-point} point]
         (println "x:" x "y:" y "point:" the-point))
x: 5 y: 7 point: {:x 5, :y 7}

We've now seen the :as directive used for both vectors and maps. In both cases the local is always assigned to the entire expression that is being destructured.

For completeness I'll document the :or directive; however, I must admit that I've never used it in practice. The :or directive is used to assign default values when the map being destructured doesn't contain a specified key.

user=> (def point {:y 7})
#'user/point
 
user=> (let [{:keys [x y] :or {x 0 y 0}} point]
         (println "x:" x "y:" y))
x: 0 y: 7

Lastly, it's also worth noting that you can destructure nested maps, vectors and a combination of both.

The following example destructures a nested map

user=> (def book {:name "SICP" :details {:pages 657 :isbn-10 "0262011530"}})
#'user/book

user=> (let [{name :name {pages :pages isbn-10 :isbn-10} :details} book]
         (println "name:" name "pages:" pages "isbn-10:" isbn-10))
name: SICP pages: 657 isbn-10: 0262011530

As you would expect, you can also use directives while destructuring nested maps.

user=> (def book {:name "SICP" :details {:pages 657 :isbn-10 "0262011530"}})
#'user/book
user=> 
user=> (let [{name :name {:keys [pages isbn-10]} :details} book]
         (println "name:" name "pages:" pages "isbn-10:" isbn-10))
name: SICP pages: 657 isbn-10: 0262011530

Destructuring nested vectors is also very straight-forward, as the following example illustrates

user=> (def numbers [[1 2][3 4]])
#'user/numbers

user=> (let [[[a b][c d]] numbers]
  (println "a:" a "b:" b "c:" c "d:" d))
a: 1 b: 2 c: 3 d: 4

Since binding forms can be nested within one another arbitrarily, you can pull apart just about anything -- http://clojure.org/special_forms

The following example destructures a map and a vector at the same time.

user=> (def golfer {:name "Jim" :scores [3 5 4 5]})
#'user/golfer

user=> (let [{name :name [hole1 hole2] :scores} golfer] 
         (println "name:" name "hole1:" hole1 "hole2:" hole2))
name: Jim hole1: 3 hole2: 5

The same example can be rewritten using a function definition to show the simplicity of using destructuring in parameter lists.

user=> (defn print-status [{name :name [hole1 hole2] :scores}] 
  (println "name:" name "hole1:" hole1 "hole2:" hole2))
#'user/print-status

user=> (print-status {:name "Jim" :scores [3 5 4 5]})
name: Jim hole1: 3 hole2: 5

There are other (less used) directives and deeper explanations available on http://clojure.org/special_forms and in The Joy of Clojure. I recommend both.

**(defn do-something [x y & more] ... )

Labels: clojure, destructuring

Before Clojure, I had never used a Functional Programming language. My experience has primarily been in C#, Ruby, & Java. So, learning how to use Clojure effectively has been a fun and eye-opening experience.

I've noticed a few things:

• The Clojure language has a lot of built in support for maps and hashes (literal syntax, destructuring, etc)
• Rich Hickey is quoted as saying "It is better to have 100 functions operate on one data abstraction than 10 functions on 10 data structures." Clojure reflects this opinion and has loads of functions that work with Sequences.
• I strongly dislike working with people who think Map<String, Map<String, String>> is generally a good idea in Java. However, all the support for maps and vectors in Clojure makes working with them easy and efficient.
• Once you embrace functions, maps, vectors, etc, composing functions together to transform data becomes effortless.

Like I said, I'm still learning Clojure. Recently I refactored some code in a way that I thought was worth documenting for other people who are also learning Clojure.

Let's start with a simple example. Let's assume we want to track the score in the British Open (Golf). The data structure we will be working with will be a map of maps, keyed by country and then player.

{:England {"Paul Casey" -1}}

The above example shows that Paul Casey plays for England and currently has the score of -1. We need a function that allows you to update a players score. Let's also assume that scores for individual holes come in the following map form.

{:player "Paul Casey" :country :England :score -1}

Given the above code, you would expect the following test to pass.

(def current-score {:England {"Paul Casey" -1}})

(deftest update-score-test
  (is (= 
    {:England {"Paul Casey" -2}} 
    (update-score current-score {:player "Paul Casey" :country :England :score -1}))))

(Let's ignore the fact that I'm def'ing current-score. It's only for simplicity's sake.)

There are a lot of ways to make that test pass. It took me a few tries to come up with something I liked. Below is the path I took.

Step one, get the test passing using the functions I know.

(defn update-score [current update]
  (let [player (:player update)
        country (:country update)
        hole-score (:score update)]
    (merge current {country {player (+ ((current country) player) hole-score)}})))

The above example code is a victory when you are learning Clojure. The tests pass, you've used a few features of the language. It's a great first pass. However, I'm sure the Clojure veterans are drooling over the refactoring possibilities.

Step two, learn the update-in function. Like I previously mentioned, Clojure has a lot of functions for working with sequences (maps, vectors, lists, etc). It's not always obvious which function you should be using. However, I've had loads of success learning from the mailing list, and getting immediate answers in the #clojure chat room on freenode. One way or another, I eventually found the update-in function, which is designed up update a nested key.

(defn update-score [current update]
  (let [player (:player update)
        country (:country update)
        hole-score (:score update)]
    (update-in current [country player] (fn [_] (+ ((current country) player) hole-score)))))

The switch to update-in is relatively easy and cleans up a bit of code, but now I've introduced an anonymous function. Anonymous functions are great, but if I can get away with not creating them, I always take that route.

However, before we eliminate the anonymous function, we're actually better off making sure we understand the new function (update-in). While the current version of the code works, it's much cleaner if you use the current value that is passed into the anonymous function instead of looking up the value yourself.

(defn update-score [current update]
  (update-in current [(:country update) (:player update)] (fn [overall-score] (+ overall-score (:score update)))))

Since we no longer use country and player in multiple places, it seems like removing the let statement is a logical choice. As a result, the amount of code necessary to solve this problem seems to get smaller and smaller.

However, we can actually take this a step farther and use destructuring to provide an even cleaner implementation.

(defn update-score [current {:keys [country player score]}]
  (update-in current [country player] (fn [overall-score] (+ overall-score score))))

If you aren't familiar with destructuring the syntax might be a bit confusing. There's a decent write up with destructuring examples available on clojure.org. I would also recommend writing a few simple functions and try out the examples for yourself.

There's one more step to take, but for me it was a big one. The documentation for update-in states:

Usage: (update-in m [k & ks] f & args)

'Updates' a value in a nested associative structure, where ks is a sequence of keys and f is a function that will take the old value and any supplied args and return the new value, and returns a new nested structure.

What I currently have works fine, but the additional args that update-in takes allows me to provide a more concise solution that I greatly prefer.

(defn update-score [current {:keys [country player score]}]
  (update-in current [country player] + score))

This final version doesn't require any anonymous functions. Instead it relies on on update-in to apply the current value and any additional args to the function that you passed in. The final version is so concise you begin to wonder if you really need to define an update-score function.

The final snippet is an example of why I'm very intrigued by Clojure these days. The ability to compose functions and mix them in with the powerful functions of the language results in very concise code.

I consider what I would need to write to accomplish the same thing in Java or Ruby, and I have to wonder if the Functional Programming supporters might just be on to something.

Labels: clojure, composing, functions, refactoring