Tuesday, July 20, 2010

Clojure: Composing Functions

Before Clojure, I had never used a Functional Programming language. My experience has primarily been in C#, Ruby, & Java. So, learning how to use Clojure effectively has been a fun and eye-opening experience.

I've noticed a few things:
  • The Clojure language has a lot of built in support for maps and hashes (literal syntax, destructuring, etc)
  • Rich Hickey is quoted as saying "It is better to have 100 functions operate on one data abstraction than 10 functions on 10 data structures." Clojure reflects this opinion and has loads of functions that work with Sequences.
  • I strongly dislike working with people who think Map<String, Map<String, String>> is generally a good idea in Java. However, all the support for maps and vectors in Clojure makes working with them easy and efficient.
  • Once you embrace functions, maps, vectors, etc, composing functions together to transform data becomes effortless.
Like I said, I'm still learning Clojure. Recently I refactored some code in a way that I thought was worth documenting for other people who are also learning Clojure.

Let's start with a simple example. Let's assume we want to track the score in the British Open (Golf). The data structure we will be working with will be a map of maps, keyed by country and then player.
{:England {"Paul Casey" -1}}
The above example shows that Paul Casey plays for England and currently has the score of -1. We need a function that allows you to update a players score. Let's also assume that scores for individual holes come in the following map form.
{:player "Paul Casey" :country :England :score -1}
Given the above code, you would expect the following test to pass.
(def current-score {:England {"Paul Casey" -1}})

(deftest update-score-test
(is (=
{:England {"Paul Casey" -2}}
(update-score current-score {:player "Paul Casey" :country :England :score -1}))))
(Let's ignore the fact that I'm def'ing current-score. It's only for simplicity's sake.)

There are a lot of ways to make that test pass. It took me a few tries to come up with something I liked. Below is the path I took.

Step one, get the test passing using the functions I know.
(defn update-score [current update]
(let [player (:player update)
country (:country update)
hole-score (:score update)]
(merge current {country {player (+ ((current country) player) hole-score)}})))
The above example code is a victory when you are learning Clojure. The tests pass, you've used a few features of the language. It's a great first pass. However, I'm sure the Clojure veterans are drooling over the refactoring possibilities.

Step two, learn the update-in function. Like I previously mentioned, Clojure has a lot of functions for working with sequences (maps, vectors, lists, etc). It's not always obvious which function you should be using. However, I've had loads of success learning from the mailing list, and getting immediate answers in the #clojure chat room on freenode. One way or another, I eventually found the update-in function, which is designed up update a nested key.
(defn update-score [current update]
(let [player (:player update)
country (:country update)
hole-score (:score update)]
(update-in current [country player] (fn [_] (+ ((current country) player) hole-score)))))
The switch to update-in is relatively easy and cleans up a bit of code, but now I've introduced an anonymous function. Anonymous functions are great, but if I can get away with not creating them, I always take that route.

However, before we eliminate the anonymous function, we're actually better off making sure we understand the new function (update-in). While the current version of the code works, it's much cleaner if you use the current value that is passed into the anonymous function instead of looking up the value yourself.
(defn update-score [current update]
(update-in current [(:country update) (:player update)] (fn [overall-score] (+ overall-score (:score update)))))
Since we no longer use country and player in multiple places, it seems like removing the let statement is a logical choice. As a result, the amount of code necessary to solve this problem seems to get smaller and smaller.

However, we can actually take this a step farther and use destructuring to provide an even cleaner implementation.
(defn update-score [current {:keys [country player score]}]
(update-in current [country player] (fn [overall-score] (+ overall-score score))))
If you aren't familiar with destructuring the syntax might be a bit confusing. I created a blog entry for destructuring that should help if you aren't familiar with it. There's also a good write up with destructuring examples available on clojure.org. I would also recommend writing a few simple functions and try out the examples for yourself.

There's one more step to take, but for me it was a big one. The documentation for update-in states:
Usage: (update-in m [k & ks] f & args)

'Updates' a value in a nested associative structure, where ks is a sequence of keys and f is a function that will take the old value and any supplied args and return the new value, and returns a new nested structure.
What I currently have works fine, but the additional args that update-in takes allows me to provide a more concise solution that I greatly prefer.
(defn update-score [current {:keys [country player score]}]
(update-in current [country player] + score))
This final version doesn't require any anonymous functions. Instead it relies on on update-in to apply the current value and any additional args to the function that you passed in. The final version is so concise you begin to wonder if you really need to define an update-score function.

The final snippet is an example of why I'm very intrigued by Clojure these days. The ability to compose functions and mix them in with the powerful functions of the language results in very concise code.

I consider what I would need to write to accomplish the same thing in Java or Ruby, and I have to wonder if the Functional Programming supporters might just be on to something.

Wednesday, July 07, 2010

High Level Testing with a High Level Language

In the early days of my project we made the decision to high-level test our Java application with Clojure*. One and a half years later, we're still following that path. It seemed worthwhile to document the progress so far.

My current preferred style of testing is rigorous unit testing and less than a dozen high level tests.

This style of testing doesn't catch everything; however, context is king. In my context, we constantly have to balance the number of bugs against the speed at which we deliver. We could test more, but it would slow down delivery. Since we don't want to drastically impact delivery, we try to get the most we can out of the tests that we do write.

A few more notes on context. Our high level tests are written by programmers for programmers. The application is fairly large and complex. We use Java, Clojure, Ruby, C#, & others. We take advantage of open-source frameworks as well as vendor and in-house frameworks. The result is a user-facing application used exclusively in-house. It's not a service or a 'for-sale' product.

The Bad
Context: The team knows Java fairly well and writes the majority of the domain code in Java. The team didn't have any previous experience with Clojure.
Result: Happiness with using a high level language was often impacted by no knowledge of that high level language.

Context: The vast majority of the code is written in Java and is able to be manipulated with IntelliJ.
Result: Some members of the team felt that using an additional language hampered their ability to use automated refactoring tools, and thus the rewards were not worth the cost. Other members believe the powerful language features provide benefits that out-weigh the costs.

Context: The tests need to run on developer boxes and build machines. One and a half years ago, there was no easy way to run Clojure tests in JUnit.
Result: The team hacked together something custom. It didn't take long to write, but the integration with IntelliJ and JUnit is not nearly as nice as using pure Java.

The Interesting
Context: The team has plenty of experience with Object Oriented (OO) based, C-style languages. The team didn't have any previous experience with Functional Programming (FP). Tests are procedural. However, the team was much more experienced writing procedural code in OO than FP.
Result: The paradigm shift impacted the speed at which the team was able to learn Clojure, but the team was able to peek into the world of FP without writing an entire application using an FP language.

The Good
Context: The team needed to build several utilities that allowed them to high level test the application.
Result: It was trivial to create a REPL that allowed us to communicate at a high level with our application. We practically got a command line interface to our application for free.

Context: The team was curious if Clojure would be the right language for solving other problems on the project. The team knew that with less than a dozen high level tests, rewriting the tests in another language would not be a large time investment.
Result: The team was able to determine that Clojure is a viable language choice for several tasks without having to take the leap on a mission critical component whose size may be variable.

Context: High level tests often require a significant amount of setup code. Clojure provides language features (duck-typing, macros, etc) that allow you to reduce the noise often generated by setup code.
Result: The tests are easier to read and maintain (assuming you understand what and how things are being hidden) and the amount of utility code required to run high level tests was significantly reduced.

*We still use Java to unit test our Java, for now.

Tuesday, July 06, 2010

Sacrificing some IDE capabilities

When discussing adopting Clojure or JRuby there are often heated discussions concerning the pros and cons of working with a language that has less IDE support. This blog entry will focus on the common concerns I hear.

I've never met anyone who has mastered Ruby or Clojure and had the opinion that they are more productive in Java. I think that's the best "proof" that the pros outweigh the cons.*

Other than that, things get complicated. The problem is, I understand where reluctant adopters are coming from. But, I just don't see any way to convince them to make the leap.

Many Java programmers do a bit of Ruby or Clojure and make a determination. I think their concerns are inflated by 3 factors: api/library fear, shallow understanding of powerful language features, and inability to optimize problem resolution.

A common concern is that learning a new language requires learning new libraries. Even though Java's libraries are available in Clojure or JRuby, both languages still have their own apis/libs to learn. In Java, you often use auto-completion. In Ruby and Clojure some auto-completion is available; however, it's often suspect. Few people use a Java REPL, so they don't understand how helpful/powerful a REPL is. In Clojure and Ruby, the REPL is often the most effective way to learn new apis/tools. Since most Java developers haven't experienced the value of the REPL, they exaggerate the overhead of learning apis/libs.

Writing idiomatic Java using Ruby or Clojure will produce painful and likely unmaintainable code. Writing idiomatic Ruby and Clojure often require understanding the powerful aspects of a language (metaprogramming, method_missing, macros, duck-typing). Understanding those aspects of a language can greatly change the design of a system. The result of mastering language features is often concise and significantly more maintainable code. However, if you aren't able to see the concise/maintainable version, you will picture a design that may likely be hard to manipulate without IDE support. Enter unnecessary concern and doubt.

Lastly, when things go wrong you need to know how to fix them. Fixing issues is often a mix of tweaking the language and mixing in a few tools. In Java it's often Ignore the noise in the stacktrace, Click a few links and open a few files, Drop in some break points or print lines or change a few tests. Getting to the source of the problem efficiently requires an IDE. In Clojure and Ruby I have those tools (since I do my Clojure and Ruby in IntelliJ), but I also have the REPLs. I know what noise to ignore. Even without those tools (when I used TextMate for Ruby), I know what patterns generally represent what issues. I can see the signal in the noise, and I have an additional tool (the REPL) to help me determine what is signal and what is noise.

However, if you don't know what is noise and what is signal, and (once again) you don't understand the value of a REPL then you exaggerate the cost of problem resolution.

Cosmin Stejerean recently mentioned "To me good IDE support for Clojure is very much like using training wheels on a bicycle."

That's almost true. I'll freely admit that I'd love some free refactoring. It would make me more productive. However, I'm already more productive so it's icing on the cake, not a reason to stop me from using the language.

Unfortunately, those training wheels are very much a deal-breaker to conservative adopters.

* I have recently heard of a few projects starting as Python and switching to Java, but I haven't ever done any Python, so I wont speculate as to why that happened.

Monday, April 12, 2010

Clojure: Converting a Custom Collection to a Sequence

I've been doing a bit of clojure lately and I've often found myself looking to convert a custom collection to a sequence. Unfortunately, several custom collections from libraries that we use don't implement Iterable; therefore, I can't simply use the seq function to create a seq.

In Java it's common to simply use a for loop and a get method to iterate through custom collections that don't implement Iterable.
for (int i=0; i<fooCollection.size(); i++) {
Foo foo = fooCollection.get(i);
}
There are patterns for doing something similar in Clojure; however, I much prefer to work with Sequences and functions that operate on Sequences. Shane Harvie and I added the following snippet to our code to quickly convert a custom collection to a seq.
(defmacro obj->seq [obj count-method]
`(loop [result# []
i# 0]
(if (< i# (. ~obj ~count-method))
(recur (conj result# (.get ~obj i#)) (inc i#))
result#)))
Our first version only worked with a custom collection that had a .size() method; however, the second collection we ran into used a .length() method, so we created the above macro that allows you to specify the (size|length|count|whatever) method.

The obj->seq function can be used similar to the following example.
(let [positions (obj->seq position-collection size)]
; do stuff with positions
)
I'm fairly new to Clojure, so please feel free to provide recommendations and feedback.

Thursday, March 11, 2010

Pairing isn't the Solution

In 2007 & 2008 I wrote several blog entires on Pair Programming (tagged with pair programming). Pair programming solved a lot of problems for me: knowledge transfer, mentoring, code review, etc. It also solved another problem at the same time, even though I wasn't aware of it. Pairing helps reduce the number of cooks in the kitchen.

These days I'm working on a project with some really talented people. The pace at which we deliver features is far faster than any project I worked on before I joined DRW. However, I've seen two effects of having so many talented developers on the same team: wiki coding and spork sharing. (both metaphors coined by my tech lead, badams)

Wiki coding occurs when the churn on a class or component is so great that whoever commits last ends up deciding what it should do. Wiki coding leads to inconsistent design and lack of convention. Spork sharing occurs when a fork is designed and a spoon is also needed. Instead of creating a spoon, you want to share the handle, so you create a spork instead. Now, you have no spoon or fork, you have a spork, and sporks suck if you really want a spoon or a fork. Both problems seem to stem from differing vision for the classes or components.

It appears that the more talented programmers you put on a team, the more fractured the vision for the project becomes. Of course, complexity (as well as many other factors) can also increase the fracture.

You won't catch me advocating for 'hard-working average programmers'. What I do believe is: you should stock your team with only rockstars, between 2 and 4 of them. I've worked on teams that only had 3 people. I've worked on teams with about 16. I was a consultant, I've worked on a lot of teams - big and small. My experience was that the smaller teams were much more effective, every time.

Reflecting on the past few years, I came up with the following reasons for keeping your teams small:
  • New Technology: Small teams can more easily come to a decision to try a new technology. Also, if a small team selects a new technology, the entire team is likely to learn the new technology. However, an individual on a larger team may find a way to never learn the new technology and never contribute to that area of the code. Worse, a larger team may shy away from trying new technology because they cannot come to a consensus. New technology can offer productivity boosts, and missing those boosts can hurt a teams efficiency.
  • Smaller Problems: Smaller teams generally implies solving smaller problems. Most problems can be broken down into smaller, less complex problems. Once the problem is broken down, it should be easier for the small teams to craft elegant solutions that can integrate to solve the larger business need.
  • Improved Maintainability: Smaller teams generate less code, which allows all team members to have depth and breadth concerning a solution. Having depth and breadth allows you to easily share vision and fix broken windows.
  • Ownership: Ownership isn't bad. Single points of failure are bad, but having a small team that feels ownership over an application will generally lead to higher quality and more emotional attachment to it's success.
  • Context Switching: Smaller codebases maintained by small teams solving a smaller problem will context switch less, because there's nothing else to context switch to.
  • Responsibility: The Bystander Effect applies to code (e.g. production bugs) as well.
  • Unified Vision: In a large team it's easy to have an opposing vision and never be proven wrong. In a small team it's likely that you will agree on a vision for the project (process, technology, etc) as an entire team.
  • Adding: Adding one more person to a 2 person team will likely result in a new productive team member relatively quickly. Smaller team, smaller problem, smaller codebase = easier team to join.
Sometimes I wonder if consultancies sell large teams, and then use pairing to make the large teams more effective. It's definitely my experience that large teams pairing are much more effective than large teams that aren't pairing. However, I wonder if anyone ever stopped to ask: would a smaller team have been sufficient (or, better)?

I still believe pairing is an answer to many problems. However, the best solution to making a 8 person team more effective isn't pairing. I believe that a superior solution is finding a way to solve the problem with a smaller, more effective team (or teams).