Clojure: Using Sets and Maps as Functions

Clojure sets and maps are functions.

Since they are functions, you don't need functions to get values out of them. You can use the map or set as the example below shows.

(#{1 2} 1)
> 1

({:a 2 :b 3} :a)
> 2

That's nice, but it's not exactly game changing. However, when you use sets or maps with high order functions you can get a lot of power with a little code.

For example, the following code removes all of the elements of a vector if the element is also in the set.

(def banned #{"Steve" "Michael"})
(def guest-list ["Brian" "Josh" "Steve"])

(remove banned guest-list)
> ("Brian" "Josh")

I'm a big fan of using sets in the way described above, but I don't often find myself using maps in the same way. The following code works, but I rarely use maps as predicates.

(def banned {"Steve" [] "Michael" []})
(def guest-list ["Brian" "Josh" "Steve"])

(remove banned guest-list)
> ("Brian" "Josh")

However, yesterday I needed to compare two maps and get the list of ids in the second map where the quantities didn't match the quantities in the first map. I started by using filter and defining a function that checks if the quantities are not equal. The following code shows solving the problem with that approach.

; key/value pairs representing order-id and order-quantity
(def map1 {1 44 2 33})
(def map2 {1 55 2 33})

(defn not=quantities [[id qty]] (not= (map1 id) qty))
(keys (filter not=quantities map2))
> (1)

However, since you can use maps as filter functions you can also solve the problem by merging the maps with not= and filtering by the result. The following code shows an example of merging and using the result as the predicate.

; key/value pairs representing order-id and order-quantity
(def map1 {1 44 2 33})
(def map2 {1 55 2 33})

(filter (merge-with not= map1 map2) (keys map2))
> (1)

I don't often find myself using maps as predicates, but in certain cases it's exactly what I need.

clojure.test Introduction

I'll admit it, the first thing I like to do when learning a new language is fire up a REPL. However, I'm usually ready for the next step after typing in a few numbers, strings and defining a function or two.

What feels like centuries ago, Mike Clark wrote an article about using unit testing to learn a new language. Mike was ahead of his time. This blog entry should help you if you want to follow Mike's advice.

Luckily, Clojure has built in support for simple testing. (I'm currently using Clojure 1.2, you can download it from clojure.org)

Before we get started, let's make sure everything is working. Save a file with the following clojure in it and run* it with clojure.

(ns clojure.test.example
  (:use clojure.test))

(run-all-tests)

If everything is okay, you should see something similar the following output.

Testing clojure.walk

Testing clojure.core

(a bunch of other namespaces tested)

Testing clojure.zip

Ran 0 tests containing 0 assertions.
0 failures, 0 errors.

If you've gotten this far, you are all set to start writing your own tests. If you are having any trouble, I suggest logging into the #clojure IRC chat room on Freenode.net

The syntax for defining tests is very simple. The following test verifies that 1 + 1 = 2. You'll want to add the test after the ns definition and before the (run-all-tests) in the file you just created.

(deftest add-1-to-1
  (is (= 2 (+ 1 1))))

Running the test should produce something similar to the following output.

Testing clojure.walk

Testing clojure.test.example

(a bunch of other namespaces tested)

Ran 1 tests containing 1 assertions.
0 failures, 0 errors.

We see all of the standard clojure namespaces; however, we see our namespace (clojure.test.example) in the results as well. The output at the bottom also tells us that 1 test with 1 assertion was executed.

The following example shows testing a custom add function. (we will add additional tests from here, without ever deleting the old tests)

(defn add [x y] (+ x y))

(deftest add-x-to-y
  (is (= 5 (add 2 3))))

If everything goes to plan, running your tests should now produce the following text towards the bottom of the output.

Ran 2 tests containing 2 assertions.
0 failures, 0 errors.

At this point you might want to pass in a few different numbers to verify that add works as expected.

(deftest add-x-to-y-a-few-times
  (is (= 5 (add 2 3)))
  (is (= 5 (add 1 4)))
  (is (= 5 (add 3 2))))

Running the tests shows us our status

Ran 3 tests containing 5 assertions.
0 failures, 0 errors.

This works perfectly fine; however, clojure.test also provides are for verifying several values.

The following example tests the same conditions using are.

(deftest add-x-to-y-a-using-are
  (are [x y] (= 5 (add x y))
       2 3
       1 4
       3 2))

And, the unsurprising results.

Ran 4 tests containing 8 assertions.

That's a simple are; however, you can do whatever you need in the form. Let's grab the value out of a map as an additional example.

(deftest grab-map-values-using-are
  (are [y z] (= y (:x z))
       2 {:x 2}
       1 {:x 1}
       3 {:x 3 :y 4}))

Leaving us with

Ran 5 tests containing 11 assertions.

The is and are macros will be all that you need for 90% of all the tests you'll ever want to write. For additional assertions and more details you can check out the clojure.test documentation.

Advanced Topics (very unnecessary to get started)

I get annoyed with noise in my test results. Our results have been very noisy due to the namespace reporting. The run-all-tests function takes a regular expression (documented here). We can change our test running call to include a regular expression, as the following example shows.

(run-all-tests #"clojure.test.example")

Once we switch to providing a regular expression the results should be limited to the following output.

Testing clojure.test.example

Ran 5 tests containing 11 assertions.
0 failures, 0 errors.

This approach works fine for our current sample file; however, it seems like a better solution would be to stop reporting namespaces that do not contain any tests. The following snippet changes the report multimethod to ignore namespaces that don't contain any tests.

(defmethod report :begin-test-ns [m]
    (with-test-out
      (when (some #(:test (meta %)) (vals (ns-interns (:ns m))))
        (println "\nTesting" (ns-name (:ns m))))))

If you're just getting started, don't worry you don't need to understand what's going on in that snippet. I've copied the original report method and made it conditional by adding the code in bold. As a result, the namespace is only printed if it contains any tests.

Now that our results are clean, let's talk about ways of getting those results.

Adding calls to the run-all-tests function isn't a big deal when working with one namespace; however, you'll need to get clever when you want to run a suite of tests. I've been told that leiningen and Maven have tasks that allow you to run all the tests. You might want to start there. I don't currently use either one, and I'm lazy. I don't want to set up either, especially since all I want to do is run all my tests.

It turns out it's very easy to add a shutdown hook in Java. So, as a simple solution, I run all my tests from the Java shutdown hook.

(.addShutdownHook
 (Runtime/getRuntime)
 (proxy [Thread] []
   (run []
 (run-all-tests))))

In general, I create a test_helper.clj with the following code.

(ns test-helper
  (:use clojure.test))

(defmethod report :begin-test-ns [m]
    (with-test-out
      (if (some #(:test (meta %)) (vals (ns-interns (:ns m))))
        (println "\nTesting" (ns-name (:ns m))))))

(.addShutdownHook
 (Runtime/getRuntime)
 (proxy [Thread] []
   (run []
 (run-all-tests))))

Once you've created a test_helper.clj you can use test-helper (just like you used clojure.test) (example below) and your tests will automatically be run on exit, and only namespaces with tests will be included in the output.

It's worth noting that some clojure.contrib namespaces seem to include tests, so in practice I end up using a regular expression that ignores all namespaces beginning with "clojure"** when running all tests. With all of those ideas combined, I find I can execute all my tests or only the tests in the current namespace very easily.

Below you can find all the code from this entry.

clojure.test.example.clj

(ns clojure.test.example
  (:use clojure.test test-helper))

(deftest add-1-to-1
  (is (= 2 (+ 1 1))))

(defn add [x y] (+ x y))

(deftest add-x-to-y
  (is (= 5 (add 2 3))))

(deftest add-x-to-y-a-few-times
  (is (= 5 (add 2 3)))
  (is (= 5 (add 1 4)))
  (is (= 5 (add 3 2))))

(deftest add-x-to-y-a-using-are
  (are [x y] (= 5 (add x y))
       2 3
       1 4
       3 2))

(deftest grab-map-values-using-are
  (are [y z] (= y (:x z))
       2 {:x 2}
       1 {:x 1}
       3 {:x 3 :y 4}))

test_helper.clj

(ns test-helper
  (:use clojure.test))

(defmethod report :begin-test-ns [m]
    (with-test-out
      (if (some #(:test (meta %)) (vals (ns-interns (:ns m))))
        (println "\nTesting" (ns-name (:ns m))))))

(.addShutdownHook
 (Runtime/getRuntime)
 (proxy [Thread] []
   (run []
 (run-all-tests))))

* Running a clojure file should be as easy as: java -cp /path/to/clojure.jar clojure.main -i file.to.run.clj

** (run-all-tests #"[^(clojure)].*") ; careful though, now your clojure.test.example tests will be ignored. Don't let that confuse you.

Clojure: Destructuring

In The Joy of Clojure (TJoC) destructuring is described as a mini-language within Clojure. It's not essential to learn this mini-language; however, as the authors of TJoC point out, destructuring facilitates concise, elegant code.

What is destructuring?

Clojure supports abstract structural binding, often called destructuring, in let binding lists, fn parameter lists, and any macro that expands into a let or fn. -- http://clojure.org/special_forms

The simplest example of destructuring is assigning the values of a vector.

user=> (def point [5 7])
#'user/point

user=> (let [[x y] point]
         (println "x:" x "y:" y))
x: 5 y: 7

note: I'm using let for my examples of destructuring; however, in practice I tend to use destructuring in function parameter lists at least as often, if not more often.

I'll admit that I can't remember ever using destructuring like the first example, but it's a good starting point. A more realistic example is splitting a vector into a head and a tail. When defining a function with an arglist** you use an ampersand. The same is true in destructuring.

user=> (def indexes [1 2 3])
#'user/indexes

user=> (let [[x & more] indexes]
         (println "x:" x "more:" more))
x: 1 more: (2 3)

It's also worth noting that you can bind the entire vector to a local using the :as directive.

user=> (def indexes [1 2 3])
#'user/indexes

user=> (let [[x & more :as full-list] indexes]
         (println "x:" x "more:" more "full list:" full-list))
x: 1 more: (2 3) full list: [1 2 3]

Vector examples are the easiest; however, in practice I find myself using destructuring with maps far more often.

Simple destructuring on a map is as easy as choosing a local name and providing the key.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{the-x :x the-y :y} point]
         (println "x:" the-x "y:" the-y))
x: 5 y: 7

As the example shows, the values of :x and :y are bound to locals with the names the-x and the-y. In practice we would never prepend "the-" to our local names; however, using different names provides a bit of clarity for our first example. In production code you would be much more likely to want locals with the same name as the key. This works perfectly well, as the next example shows.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{x :x y :y} point]
         (println "x:" x "y:" y))
x: 5 y: 7

While this works perfectly well, creating locals with the same name as the keys becomes tedious and annoying (especially when your keys are longer than one letter). Clojure anticipates this frustration and provides :keys directive that allows you to specify keys that you would like as locals with the same name.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{:keys [x y]} point]
         (println "x:" x "y:" y))
x: 5 y: 7

There are a few directives that work while destructuring maps. The above example shows the use of :keys. In practice I end up using :keys the most; however, I've also used the :as directive while working with maps.

The following example illustrates the use of an :as directive to bind a local with the entire map.

user=> (def point {:x 5 :y 7})
#'user/point

user=> (let [{:keys [x y] :as the-point} point]
         (println "x:" x "y:" y "point:" the-point))
x: 5 y: 7 point: {:x 5, :y 7}

We've now seen the :as directive used for both vectors and maps. In both cases the local is always assigned to the entire expression that is being destructured.

For completeness I'll document the :or directive; however, I must admit that I've never used it in practice. The :or directive is used to assign default values when the map being destructured doesn't contain a specified key.

user=> (def point {:y 7})
#'user/point
 
user=> (let [{:keys [x y] :or {x 0 y 0}} point]
         (println "x:" x "y:" y))
x: 0 y: 7

Lastly, it's also worth noting that you can destructure nested maps, vectors and a combination of both.

The following example destructures a nested map

user=> (def book {:name "SICP" :details {:pages 657 :isbn-10 "0262011530"}})
#'user/book

user=> (let [{name :name {pages :pages isbn-10 :isbn-10} :details} book]
         (println "name:" name "pages:" pages "isbn-10:" isbn-10))
name: SICP pages: 657 isbn-10: 0262011530

As you would expect, you can also use directives while destructuring nested maps.

user=> (def book {:name "SICP" :details {:pages 657 :isbn-10 "0262011530"}})
#'user/book
user=> 
user=> (let [{name :name {:keys [pages isbn-10]} :details} book]
         (println "name:" name "pages:" pages "isbn-10:" isbn-10))
name: SICP pages: 657 isbn-10: 0262011530

Destructuring nested vectors is also very straight-forward, as the following example illustrates

user=> (def numbers [[1 2][3 4]])
#'user/numbers

user=> (let [[[a b][c d]] numbers]
  (println "a:" a "b:" b "c:" c "d:" d))
a: 1 b: 2 c: 3 d: 4

Since binding forms can be nested within one another arbitrarily, you can pull apart just about anything -- http://clojure.org/special_forms

The following example destructures a map and a vector at the same time.

user=> (def golfer {:name "Jim" :scores [3 5 4 5]})
#'user/golfer

user=> (let [{name :name [hole1 hole2] :scores} golfer] 
         (println "name:" name "hole1:" hole1 "hole2:" hole2))
name: Jim hole1: 3 hole2: 5

The same example can be rewritten using a function definition to show the simplicity of using destructuring in parameter lists.

user=> (defn print-status [{name :name [hole1 hole2] :scores}] 
  (println "name:" name "hole1:" hole1 "hole2:" hole2))
#'user/print-status

user=> (print-status {:name "Jim" :scores [3 5 4 5]})
name: Jim hole1: 3 hole2: 5

There are other (less used) directives and deeper explanations available on http://clojure.org/special_forms and in The Joy of Clojure. I recommend both.

**(defn do-something [x y & more] ... )

Clojure: Composing Functions

Before Clojure, I had never used a Functional Programming language. My experience has primarily been in C#, Ruby, & Java. So, learning how to use Clojure effectively has been a fun and eye-opening experience.

I've noticed a few things:

• The Clojure language has a lot of built in support for maps and hashes (literal syntax, destructuring, etc)
• Rich Hickey is quoted as saying "It is better to have 100 functions operate on one data abstraction than 10 functions on 10 data structures." Clojure reflects this opinion and has loads of functions that work with Sequences.
• I strongly dislike working with people who think Map<String, Map<String, String>> is generally a good idea in Java. However, all the support for maps and vectors in Clojure makes working with them easy and efficient.
• Once you embrace functions, maps, vectors, etc, composing functions together to transform data becomes effortless.

Like I said, I'm still learning Clojure. Recently I refactored some code in a way that I thought was worth documenting for other people who are also learning Clojure.

Let's start with a simple example. Let's assume we want to track the score in the British Open (Golf). The data structure we will be working with will be a map of maps, keyed by country and then player.

{:England {"Paul Casey" -1}}

The above example shows that Paul Casey plays for England and currently has the score of -1. We need a function that allows you to update a players score. Let's also assume that scores for individual holes come in the following map form.

{:player "Paul Casey" :country :England :score -1}

Given the above code, you would expect the following test to pass.

(def current-score {:England {"Paul Casey" -1}})

(deftest update-score-test
  (is (= 
    {:England {"Paul Casey" -2}} 
    (update-score current-score {:player "Paul Casey" :country :England :score -1}))))

(Let's ignore the fact that I'm def'ing current-score. It's only for simplicity's sake.)

There are a lot of ways to make that test pass. It took me a few tries to come up with something I liked. Below is the path I took.

Step one, get the test passing using the functions I know.

(defn update-score [current update]
  (let [player (:player update)
        country (:country update)
        hole-score (:score update)]
    (merge current {country {player (+ ((current country) player) hole-score)}})))

The above example code is a victory when you are learning Clojure. The tests pass, you've used a few features of the language. It's a great first pass. However, I'm sure the Clojure veterans are drooling over the refactoring possibilities.

Step two, learn the update-in function. Like I previously mentioned, Clojure has a lot of functions for working with sequences (maps, vectors, lists, etc). It's not always obvious which function you should be using. However, I've had loads of success learning from the mailing list, and getting immediate answers in the #clojure chat room on freenode. One way or another, I eventually found the update-in function, which is designed up update a nested key.

(defn update-score [current update]
  (let [player (:player update)
        country (:country update)
        hole-score (:score update)]
    (update-in current [country player] (fn [_] (+ ((current country) player) hole-score)))))

The switch to update-in is relatively easy and cleans up a bit of code, but now I've introduced an anonymous function. Anonymous functions are great, but if I can get away with not creating them, I always take that route.

However, before we eliminate the anonymous function, we're actually better off making sure we understand the new function (update-in). While the current version of the code works, it's much cleaner if you use the current value that is passed into the anonymous function instead of looking up the value yourself.

(defn update-score [current update]
  (update-in current [(:country update) (:player update)] (fn [overall-score] (+ overall-score (:score update)))))

Since we no longer use country and player in multiple places, it seems like removing the let statement is a logical choice. As a result, the amount of code necessary to solve this problem seems to get smaller and smaller.

However, we can actually take this a step farther and use destructuring to provide an even cleaner implementation.

(defn update-score [current {:keys [country player score]}]
  (update-in current [country player] (fn [overall-score] (+ overall-score score))))

If you aren't familiar with destructuring the syntax might be a bit confusing. I created a blog entry for destructuring that should help if you aren't familiar with it. There's also a good write up with destructuring examples available on clojure.org. I would also recommend writing a few simple functions and try out the examples for yourself.

There's one more step to take, but for me it was a big one. The documentation for update-in states:

Usage: (update-in m [k & ks] f & args)

'Updates' a value in a nested associative structure, where ks is a sequence of keys and f is a function that will take the old value and any supplied args and return the new value, and returns a new nested structure.

What I currently have works fine, but the additional args that update-in takes allows me to provide a more concise solution that I greatly prefer.

(defn update-score [current {:keys [country player score]}]
  (update-in current [country player] + score))

This final version doesn't require any anonymous functions. Instead it relies on on update-in to apply the current value and any additional args to the function that you passed in. The final version is so concise you begin to wonder if you really need to define an update-score function.

The final snippet is an example of why I'm very intrigued by Clojure these days. The ability to compose functions and mix them in with the powerful functions of the language results in very concise code.

I consider what I would need to write to accomplish the same thing in Java or Ruby, and I have to wonder if the Functional Programming supporters might just be on to something.

High Level Testing with a High Level Language

In the early days of my project we made the decision to high-level test our Java application with Clojure*. One and a half years later, we're still following that path. It seemed worthwhile to document the progress so far.

My current preferred style of testing is rigorous unit testing and less than a dozen high level tests.

This style of testing doesn't catch everything; however, context is king. In my context, we constantly have to balance the number of bugs against the speed at which we deliver. We could test more, but it would slow down delivery. Since we don't want to drastically impact delivery, we try to get the most we can out of the tests that we do write.

A few more notes on context. Our high level tests are written by programmers for programmers. The application is fairly large and complex. We use Java, Clojure, Ruby, C#, & others. We take advantage of open-source frameworks as well as vendor and in-house frameworks. The result is a user-facing application used exclusively in-house. It's not a service or a 'for-sale' product.

The Bad
Context: The team knows Java fairly well and writes the majority of the domain code in Java. The team didn't have any previous experience with Clojure.
Result: Happiness with using a high level language was often impacted by no knowledge of that high level language.

Context: The vast majority of the code is written in Java and is able to be manipulated with IntelliJ.
Result: Some members of the team felt that using an additional language hampered their ability to use automated refactoring tools, and thus the rewards were not worth the cost. Other members believe the powerful language features provide benefits that out-weigh the costs.

Context: The tests need to run on developer boxes and build machines. One and a half years ago, there was no easy way to run Clojure tests in JUnit.
Result: The team hacked together something custom. It didn't take long to write, but the integration with IntelliJ and JUnit is not nearly as nice as using pure Java.

The Interesting
Context: The team has plenty of experience with Object Oriented (OO) based, C-style languages. The team didn't have any previous experience with Functional Programming (FP). Tests are procedural. However, the team was much more experienced writing procedural code in OO than FP.
Result: The paradigm shift impacted the speed at which the team was able to learn Clojure, but the team was able to peek into the world of FP without writing an entire application using an FP language.

The Good
Context: The team needed to build several utilities that allowed them to high level test the application.
Result: It was trivial to create a REPL that allowed us to communicate at a high level with our application. We practically got a command line interface to our application for free.

Context: The team was curious if Clojure would be the right language for solving other problems on the project. The team knew that with less than a dozen high level tests, rewriting the tests in another language would not be a large time investment.
Result: The team was able to determine that Clojure is a viable language choice for several tasks without having to take the leap on a mission critical component whose size may be variable.

Context: High level tests often require a significant amount of setup code. Clojure provides language features (duck-typing, macros, etc) that allow you to reduce the noise often generated by setup code.
Result: The tests are easier to read and maintain (assuming you understand what and how things are being hidden) and the amount of utility code required to run high level tests was significantly reduced.

*We still use Java to unit test our Java, for now.