Clojure: Truthy and Falsey

Truthy and Falsey are important concepts in Clojure. The best description I've seen is in The Joy of Clojure, go buy it.

Depending on your language background you may have very different ideas on what evaluates to true and what evaluates to false. In Clojure, the answer to this is very straightforward.

Both nil and false are treated as "false" and everything else is treated as true.

Let's hit the REPL.

user=> (if true "yes" "no")
"yes"
user=> (if false "yes" "no")
"no"

Okay, that much should be obvious. However, it's important to note that nil is also considered false, which is why we say:

In Clojure nil and false are considered "false" and therefore we say they are both "falsey".

Another quick trip to the REPL to verify that nil is falsey.

user=> (if nil "yes" "no")  
"no"

As expected, the else is evaluated.

There are other values that are considered false in other languages, this does not apply to clojure.

All non-falsey values are considered "truthy" and evaluate as such.

Here are some common examples of truthy values in Clojure that are falsey in other languages.

user=> (if -1 "still true" "false")
"still true"
user=> (if 0 "still true" "false") 
"still true"
user=> (if [] "still true" "false")
"still true"
user=> (if (list) "still true" "false")
"still true"

As it says in The Joy of Clojure: Every[thing] is "true" all the time, unless it is nil or false.

The Impact of Accidental Complexity on Estimates

At speakerconf Aruba, J.B. Rainsberger recently gave an enlightening talk which included discussion on the impact of accidental complexity to estimates.

In his talk, he pointed out that estimates are composed of accidental and essential complexity. It's often easy to estimate the essential complexity of a problem; however, estimating accidental complexity is an entirely different matter. This point really hit home for me. On my last project I often strongly disagreed with certain members of the team on estimates. I was often higher than the average, and much higher than the lowest estimates. J.B.'s talk really openend my eyes to a few points.

• Not all developers consider both accidental and essential complexity while estimating
• Features that touch accidentally complex portions of the system are harder to estimate as the accidental complexity can often be hard to completely understand. (If you completely understood it, you'd probably remove the accidental complexity)
• Introducing technical debt increases accidental complexity, and as a side-effect invalidates previous estimates and increases the likelihood that future estimates will differ in size drastically.

The first two points had always been intuitive to me; however, it was good to hear the ideas clearly articulated from someone else's point of view. The last point was the one that really interested me.

I'm lucky enough to work for a company where I don't need to explain the benefits of addressing technical debt. As developers we sometimes consciously introduce technical debt when it will allow us a short-term benefit, and we address that technical debt when appropriate. I've always considered technical debt to be part of the developer's world, and I only mention it to the business when completing a feature takes longer than expected due to tech debt. I never considered the impact of technical debt on my ability to produce accurate estimates for the business.

J.B.'s talk made me ask myself a few questions about decisions I'd been making. Most importantly: If the business knew that I was taking a short-cut to deliver sooner, but it was going to damage estimates, would that be acceptable? I think that's an important question to consider if your client values estimates, and it's something I'll continue to ask myself in the future.

The President and the CEO of an Application

The CEO is a visionary, often leaving day-to-day operations to the President -- wikipedia

I recently switched teams. As part of the transition process, I started taking on less and less of the responsibility for things that were long-term goals. I was still very interested in helping the team succeed, but not at the risk of leaving them with something that would be harder to maintain.

So, I found myself sitting with the users and looking for the lowest-risk, highest-impact features that I could work on for my final days. This seemed like the responsible thing to do for the long-term health of the project, but an interesting side-effect was the unexpected impact to the short-term success of the project.

At first I started grabbing the story-cards that fit this criteria, and I worked my way from the highest prioritized on down. As you can imagine, I moved down the list quickly - many of the cards were sufficiently complicated that working on them would have created maintenance risk. The users were pleased with the quick additions of highly prioritized features; however, as I moved down the list their happiness became very erratic. Some features were still very appreciated; while others were met largely with ambivalence. It became obvious that I needed to adapt the plan.

I stopped working in priority order. In fact, I stopped working from the story cards entirely. I sat down next to the users twice a day, for extended periods of time, listened to anything they had to say, and I watched the way the application was used. I started making tweaks to the application based on their complaints and any wasted effort that I witnessed. I looked for "every time I do A, then I do B". Sometimes the interaction was complicated and I couldn't automate it away. However, there were non-trivial amounts of tasks that I entirely eliminated.

The users were great about providing feedback. There was some feedback that I had to dismiss, because it wasn't something I could fix safely (in the short-term); however, a large amount of their complaints took 5-20 minutes to address. Those issues were nagging and annoying, but not (individually) more important than the long term goals of the project. But, as I began to quickly address the issues the application became more polished. It went from a good application, to an application that was surprisingly nice to work with. One of the most demanding users I've ever worked for sent me a few different emails letting me know how great the app was and how much he appreciated my effort. I was shocked.

Delivering all of these little, helpful features so greatly impacted the happiness and effectiveness of the users that I had to take a step back and see if this was something I could reproduce on future projects. The first thing that came to mind was the relationship between a CEO and a President. As the quote above says, The CEO is a visionary, often leaving day-to-day operations to the President. This idea seemed similar to what occurred in my final days. I was adding features that made day-to-day activities more efficient, and at the same time a pair was working on the features that solved our long term goals. The application was moving in the right direction for the long-term, while still experiencing short bursts of improvement in the short-term as well.

It wasn't unfiltered awesomeness. I delivered a few things that addressed an immediate problem, but not in a way that satisfied the users' needs. I wasn't working off story cards or deep conversations. I was often speculating on what I thought the application needed most. Sometimes I was wrong. However, each failure occurred quickly, and led to delivering something that the users did need. And, this cycle was complete within a few hours. (e.g. [in an hour] deliver a feature that isn't quite right, get immediate feedback, spend an hour fixing it, deliver what's really needed)

You could argue that I wasted that first hour, but spending 30 minutes with 2 users to get it right the first time would have cost the team more in the end. But, we're quickly approaching dangerous territory. Developers may like the idea of no requirements and delivering speculative features all-day, but I expect businesses would (rightly-so) object. I think a key variable in my scenario was the 2 hour failure-success cycle. If we tried this approach, it took 2 weeks, and it could have been avoided by a 30 minute discussion I think everyone would consider it a failure (again, rightly-so).

This led me to the observation that developers are more likely to correctly prioritize features that take less than a day to deliver and users are more likely to correctly prioritize features based on long-term goals. I've definitely seen this occur in two ways in the past.

• I often see users prioritize based solely on the value of the feature, even when an estimate is present.
• I have personally thought prioritizing some small features over a larger feature was the right choice, only to see that the impact of the large feature was significantly greater than I understood

This is also intuitive to me; developers are more likely to correctly estimate a smaller change and users are more likely to correctly estimate the impact of a larger change.

This experience has definitely left me wondering if teams would benefit from:

• developers prioritizing and keeping a separate backlog of smaller feature requests
• users prioritizing and keeping a separate backlog of larger feature requests
• ensuring there is always at least one person working on each backlog.

I was very pleased with the results of using this style, and I'm hoping to try it again in the future.

Clojure: Web Socket Introduction

Web Sockets may sound intimidating, but if you use Clojure and jQuery it's actually quite simple.

The first thing you'll want to do is create a new project. I use lein for simplicity's sake.

jfields$ lein new ws-intro
Created new project in: /Users/jfields/src/ws-intro
Look over project.clj and start coding in ws_intro/core.clj

Next we'll need to get Clojure, Webbit (a WebSocket capable server), and clojure.data.json. My project.clj file looks like the following example.

(defproject ws-intro "1.0.0-SNAPSHOT"
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.webbitserver/webbit "0.4.3"]
                 [org.clojure/data.json "0.1.2"]])

Once the project.clj file has been updated, a quick command-line run of "lein deps" will get all the jars you need. A quick ls of the lib dir should look something like this:

-rw-r--r--  1 jfields  domain.users  3390414 Feb 20 15:25 clojure-1.3.0.jar
-rw-r--r--  1 jfields  domain.users     5177 Feb 20 15:25 data.json-0.1.2.jar
-rw-r--r--  1 jfields  domain.users   801614 Feb 20 15:25 netty-3.2.7.Final.jar
-rw-r--r--  1 jfields  domain.users   154735 Feb 20 15:25 webbit-0.4.3.jar

Next, I converted the Hello World example available in the Webbit README to Clojure (ws-intro/core.clj).

(ns ws-intro.core
  (:require [clojure.data.json :as json]
            [clojure.string :as s])
  (:import [org.webbitserver WebServer WebServers WebSocketHandler]
           [org.webbitserver.handler StaticFileHandler]))

(defn -main []
  (doto (WebServers/createWebServer 8080)
    (.add "/websocket"
          (proxy [WebSocketHandler] []
            (onOpen [c] (println "opened" c))
            (onClose [c] (println "closed" c))
            (onMessage [c j] (println c j))))

    (.add (StaticFileHandler. "."))
    (.start)))

You'll notice that we defined a "-main" function. This is a convention that we can take advantage of by adding a :main option to our project.clj.

(defproject ws-intro "1.0.0-SNAPSHOT"
  :main ws-intro.core
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.webbitserver/webbit "0.4.3"]
                 [org.clojure/data.json "0.1.2"]])

With the addition of the :main option, we can start listening to our web socket by running "lein run" from the command-line.

At this point we have a webserver running on port 8080 and a websocket available at localhost:8080/websocket; however, we have no way to access our new websocket.

Again, the examples already on the internet give us 90% of what we need. We're going to use jquery-websocket, and the example at the bottom of the page is just about exactly what we need. The following code is a slightly modified version that should suit our purposes.

<html>
<body>
<h1>WebSocket Demo</h1>
<input id="message" type="text"/>
<section id="content"></section>
<script src="http://www.google.com/jsapi"></script>
<script>google.load("jquery", "1.3")</script>
<script src="http://jquery-json.googlecode.com/files/jquery.json-2.2.min.js">
</script>
<script src="http://jquery-websocket.googlecode.com/files/jquery.websocket-0.0.1.js">
</script>
<script>
  var ws = $.websocket("ws://127.0.0.1:8080/websocket", {
    events: { 
      upcased: function(e) { $("#content").html(e.message); }}});

  $('#message').change(function(){
    ws.send('message', {type: "downcase", message: $("#message").val()});});
</script>
</body>
</html>

Our client isn't much, but it does connect to a web socket and it sends a message to the web socket when the text in the input is changed.

Assuming you still have the server running you should be able to load up your page once you save your index.html in the ws-intro project directory (the path specified to our StaticFileHandler).

Is your page not loading? =(
What URL did you use? I've been told http://localhost:8080/ doesn't work as well as http://127.0.0.1:8080/
What browser did you use? Everything works for me in Chrome (version 17.0.963.56)

If your page loads, your server must be up and running. You should see something similar to the following line in your server console.

opened #<NettyWebSocketConnection webbit.netty.NettyWebSocketConnection@8c5488>

You might as well type something into the input and tab out (or whatever you prefer to do that fires the "change" event). I typed in "hello" and got the following output in my server console.

#<NettyWebSocketConnection webbit.netty.NettyWebSocketConnection@8c5488> 
  {"type":"message","data":{"type":"downcase","message":"hello"}}

Okay, everything is working. Let's add a little behavior to our server. Upon receiving a message, our server is going to take the text, upcase it, and send it back to the client.

Here's an updated version of core.clj.

(ns ws-intro.core
  (:require [clojure.data.json :as json]
            [clojure.string :as s])
  (:import [org.webbitserver WebServer WebServers WebSocketHandler]
           [org.webbitserver.handler StaticFileHandler]))

(defn on-message [connection json-message]
  (let [message (-> json-message json/read-json (get-in [:data :message]))]
    (.send connection (json/json-str
                       {:type "upcased" :message (s/upper-case message) }))))

(defn -main []
  (doto (WebServers/createWebServer 8080)
    (.add "/websocket"
          (proxy [WebSocketHandler] []
            (onOpen [c] (println "opened" c))
            (onClose [c] (println "closed" c))
            (onMessage [c j] (on-message c j))))

    (.add (StaticFileHandler. "."))
    (.start)))

The new version of core.clj takes advantage of clojure.data.json support. The net result of this is that we can work with Clojure maps and basically ignore json throughout our application.

After making the above changes to core.clj we can restart our server (ctrl+c & 'lein run' again), refresh our webpage, enter some text, tab out of the input, and then we should see our text on the webpage as upper-case.

And, we're done. We have working client-server interaction. We're ready to put this into production. It's that easy.

You might have noticed a few things that make the magic happen. On the server we sent a map that has :type "upcased". This type corresponds to the events that are defined in our client. The jquery-websocket takes care of routing our new message to the function associated with upcased. Extending on this idea, you can send messages from the server with different types and handle each one on the ui as a different event.

That's it. The app should be working, and you should have everything you need to begin expanding the capabilities of the application. If you run into any trouble, the documentation for webbit and jquery-websocket should get you through.

Clojure: &env and &form

Inside the body of defmacro you can call &env and &form to get a bit of interesting information that may or may not be helpful.

Here's a few examples that demonstrate how &env and &form can be used.
(note: I'm using Clojure 1.2)

&env
By default &env is nil.

user=> (defmacro show-env [] (println &env))
#'user/show-env
user=> (show-env)
nil

However, if any bindings exist, &env gives you the names of the bindings as the keys of a map.

user=> (let [band "zeppelin" city "london"] (show-env))
{city #<LocalBinding clojure.lang.Compiler$LocalBinding@78ff9053>, band #<LocalBinding clojure.lang.Compiler$LocalBinding@525c7734>}

Okay, now we're getting somewhere. What's a Compiler$LocalBinding? I'm not exactly sure, and I've never bothered to look into it. I've been told that the 'values' from &env may change in the future, so I wouldn't rely on them anyway. Since I can't rely on them, I haven't found the need to look into what they are.

Back to the keys. They sure look like symbols.

user=> (defmacro show-env [] (println (keys &env)) (println (map class (keys &env))))
#'user/show-env
user=> (let [band "zeppelin" city "london"] (show-env))                              
(city band)
(clojure.lang.Symbol clojure.lang.Symbol)

As the example shows, they are definitely symbols. However, these symbols don't have a namespace.

user=> (defmacro show-env [] (println (keys &env)) (println (map namespace (keys &env))))
#'user/show-env
user=> (let [band "zeppelin" city "london"] (show-env))                                  
(city band)
(nil nil)

Since the symbols don't have a namespace there didn't seem to be much fun I could do with them; however, you can use the symbols in your macro to print the values, as the following example shows.

user=> (defmacro show-env [] (println (keys &env)) `(println ~@(keys &env)))                      
#'user/show-env
user=> (let [band "zeppelin" city "london"] (show-env))                     
(city band)
london zeppelin

Printing the values of bindings can be a helpful trick while you are debugging.
&form
&form can be used to get the original macro invocation.

user=> (defmacro show-form [] (println &form))                               
#'user/show-form
user=> (show-form)
(show-form)

Okay, not very interesting so far. It gets a bit more interesting when your macro takes a few arguments.

user=> (defmacro show-form [a b] (println &form))       
#'user/show-form
user=> (show-form 50 100)                        
(show-form 50 100)
user=> (show-form a 100) 
(show-form a 100)

So, you can get the arguments. Notice you can grab both 50 and 100.

user=> (defmacro show-form [a b] (println (next &form)))
#'user/show-form
user=> (show-form 50 100)                               
(50 100)
user=> (defmacro show-form [a b] (println (map class (next &form))))
#'user/show-form
user=> (show-form 50 100)                                           
(java.lang.Integer java.lang.Integer)

Interesting. So I have a few integers I can work with, if I wish. What about 'show-form'?

user=> (defmacro show-form [a b] (println (map class &form)))       
#'user/show-form
user=> (show-form 50 100)                                    
(clojure.lang.Symbol java.lang.Integer java.lang.Integer)

'show-form' is a symbol, as expected. Which brings us back to a previous example, shown again below.

user=> (defmacro show-form [a b] (println (map class &form)))
#'user/show-form
user=> (show-form a 100)                                     
(clojure.lang.Symbol clojure.lang.Symbol java.lang.Integer)

Okay, 'a' is also a symbol, unsurprising, but perhaps it's interesting since 'a' doesn't exist anywhere except in our invocation. You can probably do some interesting things here, like allow people to specify enum values and append the enum yourself.

user=> (ns user (:import [java.util.concurrent TimeUnit]))                                                     
java.util.concurrent.TimeUnit
user=> (defmacro time-units [& l] (->> (next &form) (map (partial str "TimeUnit/")) (map symbol) (cons 'list)))
#'user/time-units
user=> (time-units SECONDS MILLISECONDS)                                                                       
(#< SECONDS> #< MILLISECONDS>)

Would you want to use &form instead of just using the arguments (stored in l)? Probably not. This isn't an exercise in what you should do, but it does demonstrate what you could do.

So &form must be returning a list, right?

user=> (defmacro show-form [a b] (println (map class &form)) (println (class &form))) 
#'user/show-form
user=> (show-form 50 100)                                                            
(clojure.lang.Symbol java.lang.Integer java.lang.Integer)
clojure.lang.PersistentList

A list. Correct.

And, I'm in a macro, so I can do anything I want with this list. Maybe I just want to print the arguments. Easy enough.

user=> (defmacro show-form [a b] `(println ~@(next &form)))
#'user/show-form
user=> (show-form 50 100)                                  
50 100

Of course, there are a million things I could do. You get the idea.

One other interesting thing to note is that &form has metadata.

user=> (show-form 50 100)                                 
{:line 132}

Perhaps you don't care about line numbers, but they definitely can come in handy when you are writing a testing framework.

I use &form in expectations and I believe LazyTest uses &env. I guess you never know what you're going to need...