REPL Driven Development

When I describe my current workflow I use the TLA RDD, which is short for REPL Driven Development. I've been using REPL Driven Development for all of my production work for awhile now, and I find it to be the most effective workflow I've ever used. RDD differs greatly from any workflow I've used in the past, and (despite my belief that it's superior) I've often had trouble concisely describing what makes the workflow so productive. This entry is an attempt to describe what I consider RDD to be, and to demonstrate why I find it the most effective way to work.

RDD Cycle

First, I'd like to address the TLA RDD. I use the term RDD because I'm relying on the REPL to drive my development. More specifically, when I'm developing, I create an s-expression that I believe will solve my problem at hand. Once I'm satisfied with my s-expression, I send that s-expression to the REPL for immediate evaluation. The result of sending an s-expression can either be a value that I manually inspect, or it can be a change to a running application. Either way, I'll look at the result, determine if the problem is solved, and repeat the process of crafting an s-expression, sending it to the REPL, and evaluating the result.

If that isn't clear, hopefully the video below demonstrates what I'm talking about.



If you're unfamiliar with RDD, the previous video might leave you wondering: What's so impressive about RDD? To answer that question, I think it's worth making explicit what the video is: an example of a running application that needs to change, a change taking place, and verification that the application runs as desired. The video demonstrates change and verification; what makes RDD so effective to me is what's missing: (a) restarting the application, (b) running something other than the application to verify behavior, and (c) moving out of the source to execute arbitrary code. Eliminating those 3 steps allows me to focus on what's important, writing and running code that will be executed in production.

Feedback

I've found that, while writing software, getting feedback is the single largest time thief. Specifically, there are two types of feedback that I want to get as quickly as possible: (1) Is my application doing what I believe it is? (2) What does this arbitrary code return when executed? I believe the above video demonstrates how RDD can significantly reduce the time needed to answer both of those questions.

In my career I've spent significant time writing applications in C#, Ruby, & Java. While working in C# and Java, if I wanted to make and verify (in the application) any non-trivial change to an application, I would need to stop the application, rebuild/recompile, & restart the application. I found the slowness of this feedback loop to be unacceptable, and wholeheartedly embraced tools such as NUnit and JUnit.

I've never been as enamored with TDD as some of my peers; regardless, I absolutely endorsed it. The Design aspect of TDD was never that enticing to me, but tests did allow me to get feedback at a significantly superior pace. Tests also provide another benefit while working with C# & Java: They're the poorest man's REPL. Need to execute some arbitrary code? Write a test, that you know you're going to immediately delete, and execute away. Of course, tests have other pros and cons. At this moment I'm limiting my discussion around tests to the context of rapid feedback, but I'll address TDD & RDD later in this entry.

Ruby provided a more effective workflow (technically, Rails provided a more effective workflow). Rails applications I worked on were similar to my RDD experience: I was able to make changes to a running application, refresh a webpage and see the result of the new behavior. Ruby also provided a REPL, but I always ran the REPL external to my editor (I knew of no other option). This workflow was the closest, in terms of efficiency, that I've ever felt to what I have with RDD; however, there are some minor differences that do add up to an inferior experience: (a) having to switch out of a source file to execute arbitrary code is an unnecessary nuisance and (b) refreshing a webpage destroys any client side state that you've built up. I have no idea if Ruby now has editor & repl integration, if it does, then it's likely on par with the experience I have now.

Semantics

• It's important to distinguish between two meanings of "REPL" - one is a window that you type forms into for immediate evaluation; the other is the process that sits behind it and which you can interact with from not only REPL windows but also from editor windows, debugger windows, the program's user interface, etc.
• It's important to distinguish between REPL-based development and REPL-driven development:
  • REPL-based development doesn't impose an order on what you do. It can be used with TDD or without TDD. It can be used with top-down, bottom-up, outside-in and inside-out approaches, and mixtures of them.
  • REPL-driven development seems to be about "noodling in the REPL window" and later moving things across to editor buffers (and so source files) as and when you are happy with things. I think it's fair to say that this is REPL-based development using a series of mini-spikes. I think people are using this with a bottom-up approach, but I suspect it can be used with other approaches too.

-- Simon Katz

I like Simon's description, but I don't believe that we need to break things down to two different TLAs. Quite simply, (sadly) I don't think enough people are developing in this way, and the additional specification causes a bit of confusion among people who aren't familiar with RDD. However, Simon's description is so spot on I felt the need to describe why I'm choosing to ignore his classifications.

RDD & TDD

RDD and TDD are not in direct conflict with each other. As Simon notes above, you can do TDD backed by a REPL. Many popular testing frameworks have editor specific libraries that provide immediate feedback through REPL interaction.

When working on a feature, the short term goal is to have it working in the application as fast as possible. Arbitrary execution, live changes, and only writing what you need are 3 things that can help you complete that short term goal as fast as possible. The video above is the best example I have of how you go from a feature request to software that does what you want in the smallest amount of time. In the video, I only leave the buffer to verify that the application works as intended. If the short term goal was the only goal, RDD without writing tests would likely be the solution. However, we all know that that are many other goals in software. Good design is obviously important. If you think tests give you better design, then you should probably mix both TDD & RDD. Preventing regression is also important, and that can be accomplished by writing tests after you have a working feature that you're satisfied with. Regression tests are great for giving confidence that a feature works as intended and will continue to in the future.

REPL Driven Development doesn't need to replace your current workflow, it can also be used to extend your existing TDD workflow.

Coding: Increase Your Reading and Writing Speed

A teammate of mine recently expressed a desire for a shortcut for something we type often. I started looking into our shortcut options and came to a common determination: We can do this, but the number of 2 key shortcuts available to us is finite, so we better use them wisely.

I wrote the following unix to give me a rough idea of what we type frequently.

find . -name "*.clj" | xargs cat | tr -s '[:space:]:#()[]{}\"' '\n' | sort | uniq -c | sort -n

note: If you're not writing clojure you'll want to look for something other than .clj files, and you might also want to tweak what you replace with a new line.

The above unix gave me an ordered list of the most typed 'words' across all of my codebases. At this point I had some science for setting up some shortcuts.

Writing

You'll want to look into whatever editor/ide you use and see if you can find key shortcuts and snippet expansion. My editor is emacs; I assigned some key-chords and some yasnippets. If you're not using emacs you should have something similar in whatever you are using.

While I wanted to define some shortcuts, I also didn't want to create so many that I was constantly wasting time looking up what I'd created. Based on that desire I created:

• 2 shortcuts (key-chords) for two of the most duplicated words. The shortcuts are concise by design, but that makes them a bit harder to remember. You can probably get started with more than 2, but I didn't see much harm in starting there.
• a dozen snippets for the next most used words. These snippets are descriptive enough to easily remember, thus I felt comfortable defining several of them. e.g. pps expands to (println (pr-str )).

Having shortcuts and snippets will obviously make me more productive, and the unix helped me figure out which words were the most important to optimize for.

Reading

Most editors/ides also give you a summary view for common code patterns. For example, IntelliJ displays lambdas when the actual code is actually an anonymous class. Emacs gives you font-lock for turning patterns into individual characters. Armed with my list of the most common words in my codebase, I created font-locks for the 11 most duplicated.

Sometimes a picture is worth a thousand words. Below is a function definition without any font locks applied.


The following image is what the same function looks like with custom font locks applied. It might be a bit jarring at first, but in the long run it should add up to many small victories such as quickly identifying patterns in code and less line breaks.

Results

I've been working with these settings for a little over a week now. I haven't needed to look up anything I defined, and I get a little burst of satisfaction when I read or write something faster than I'd been able to in the past. I'd definitely recommend doing something similar with your codebase and ide/editor.

Emacs Lisp: Find Java Sources

Confession: I really hope someone can tell me I'm doing this wrong. I can't believe there isn't an easier way.

I work with Clojure, in Emacs, almost every day. Navigating the source is usually fairly easy. If I want to navigate to a function definition, all I need to press is M-., and if I want to navigate back, M-, does the trick. This works for Clojure that I've written, as well as Clojure that lives in the libraries that I reference. That's fine the vast majority of the time, but occasionally I need to navigate to the Java source of some library I'm using. This is where I can't believe that no one else has solved this problem.* If I'm in a clj file, my cursor is on a Java class, I don't know of any way to easily navigate to the class definition.

Context: I use fig for dependency management (at work). I have my projects set to put sources in ./the-project/lib/sources; therefore, the following solution assumes the sources are in that directory. If you use Lein (for deps), I'm sure the sources are on your local drive somewhere. All you need to do is change the lisp below to point to your source dir.

Additionally, I use Lein, so I have a project.clj at the root of my project; however, there's nothing special or required about "project.clj". You could just as easily put a this.is.a.project.root file in the root of your project, and search for that file.

The following code will search the your source jars for a Java class, and open the first match that it finds (or no match, if no match is found)

disclaimer, I'm still an emacs lisp beginner.

The previous code is pretty straightforward. Line 3 uses expand region to mark whatever Java class my cursor is currently on or near. You could type the word instead if you like, this page should help you understand how.
Line 4 and 6 find and verify where the project root lives.
Line 8 (and 9) greps for a string (the Java class) in a directory (my source dir).
Line 10 switches to the grep results.
Line 11 sleeps, waiting for the grep results to return.
Line 12 searches forward, looking for the first match.
Line 13 opens the jar of the first match.
Line 14 assigns the current point to a var named 'current-point'.
Line 15 searches forward to the end of the jar name.
Line 16 grabs the name of the jar and switches to that buffer.
Line 17 searches the jar's dired buffer for the name of the class.
Line 18 opens the first class name match.
(Line 19 lets you know if your project root could not be determined)

That's it, you can now easily find java source (with the occasional conflicting name annoyance). It's not pretty, but it gets the job done.

* I've been told that a few Java modes are good, if I can easily use those to navigate from my Clojure to Java, please leave a link to a manual I can dig into. I assume there are etags solutions, but it's not clear what the best way to go is. I'm sure there's an easy solution for navigating Java from Clojure, I'm just having a hard time finding it.

Elisp: Duplicate Line

After switching to emacs I quickly noticed that I was missing a simple keystroke for duplicating a line. Yes, I know it's as easy as C-a, C-k, C-y, Enter, C-y. Still, I wanted a keystroke. So, I coded up the following snippet. It's similar to what you'd find elsewhere on the net, but it also moves the cursor to where I've gotten used to it ending up.

Elisp: Automated Switching Between Clojure Test and Source Files

The majority of the work that I do in emacs is Clojure programming. The Clojure navigation support (M-.) is usually all I need, but one thing that I find myself doing manually fairly often is jumping between tests and source. After suffering manual navigation for a few days, I finally automated the task using the Elisp from this blog entry.

All of my Clojure projects use both Leiningen and expectations; therefore, my directory structures always look similar to what you see below.

source - /Users/jfields/src/project-name/src/clojure/
tests - /Users/jfields/src/project-name/test/clojure/expectations/

Since my projects follow this convention, I'm able to make several assumptions about where the expectations and where the source will actually live. If you don't use expectations, or you follow a slightly different directory structure, you'll want to hack this a bit to follow your conventions.

If you're in a source file, find (and open) the expectations.


If you're in a test file, find (and open) the source.

Elisp: Grep in Clojure Project

Grep'ing within my current project is something I do frequently. It's not much to type, but I do it often enough that I was looking for a keystroke. Most of my projects are Clojure and use Leiningen, thus I'm able to make some pretty safe assumptions. The following snippets allow you to easily grep within your project.

note: both use expand-region to grab the clojure that the cursor is on or immediately after, and grep for whatever was selected.

grep recursively starting at the directory where the project.clj file lives

grep recursively, but allow the user to select the root directory (defaulting to the location of the project.clj file). I often use this one for selecting only the src or test directories of my project.

emacs lisp: removing a lamba hook

disclaimer: I know almost nothing about emacs lisp, so please forgive any mistakes or incorrect assumptions.

I've been using Clojure for over 4 years at this point, but it's generally been on projects that are mostly Java with a few small components in Clojure. Given that context my teammates preferred that we stick with IntelliJ; however, things have recently changed and the emacs journey has begun.

Taking emacs from 'factory settings' to 'impressive' for me was as easy as getting started with emacs-live.

Emacs Live can be summarized as:

a nice structured approach to organising your Emacs config

modular in that functionality is organised by discrete packs

Indeed, Emacs Live does structure your config nicely; however, in it's own words: Emacs Live is also an opinionated set of defaults. I like all of the defaults... except one: rainbow-delimiters

I get that people like rainbow-delimeters, and I would never try to convince you not to use them - they're just not for me, and I needed to find out how to get rid of them.

Emacs Live adds rainbow-delimiters in two different ways. The following code shows how Emacs Live uses add-hook to enable rainbow delimiters for scheme, emacs-lisp, & lisp.

(dolist (x '(scheme emacs-lisp lisp))
  (add-hook 
    (intern (concat (symbol-name x) "-mode-hook"))
    'rainbow-delimiters-mode))

The code required to remove rainbow-delimiters from scheme, emacs-lisp, & lisp is very straightforward, and can be found below.

(dolist (x '(scheme emacs-lisp lisp))
  (remove-hook 
    (intern (concat (symbol-name x) "-mode-hook"))
    'rainbow-delimiters-mode))

As you can see, replacing add-hook with remove-hook will remove the hook that Emacs Live added for me. Since Emacs Live loads my personal settings last, my remove should successfully work every time. It's a best practice that you create hooks that can be run in any order - and this change is obviously order specific; however, I can't think of a way to follow the hook best practice without hacking the emacs-live checkout. Therefore, it seems like this solution is the most pragmatic.

The next snippet of code is how Emacs Live adds rainbow delimiters (& a few other things) to clojure-mode.

(add-hook 'clojure-mode-hook
          (lambda ()
            (enable-paredit-mode)
            (rainbow-delimiters-mode)
            (add-to-list 'ac-sources 'ac-source-yasnippet)
            (setq buffer-save-without-query t)))

The previous hook was easy to remove, since I knew exactly what function I needed to remove. This hook is more of a problem, since it's anonymous. Additionally, this function is defined within Emacs Live code, so simply changing it to a named function isn't an option (since I don't want to modify the emacs-live checkout).

Finding a solution wasn't very hard. The first thing I did was try to get a list of the functions that will be fired by the hook. This is as simple as printing, as the following code shows.

(print clojure-mode-hook)

I put that simple print statement in my .emacs-live.el, restarted emacs, went to the *Messages* buffer, and found the following line printed out.

((lambda nil 
         (enable-paredit-mode)
         (rainbow-delimiters-mode)
         (add-to-list (quote ac-sources) (quote ac-source-yasnippet)) 
         (setq buffer-save-without-query t)))

As you can see, clojure-mode-hook has a list of the functions that have been added via add-hook. With this information, I added the following code, which first removes the existing hook and then adds a new anonymous function with everything previously specified - sans rainbow-delimiters.

(remove-hook 'clojure-mode-hook (first clojure-mode-hook))

(add-hook 'clojure-mode-hook
          (lambda ()
            (enable-paredit-mode)
            (add-to-list 'ac-sources 'ac-source-yasnippet)
            (setq buffer-save-without-query t)))

The above snippet removes my unwanted lambda hook and adds a new hook with everything I do want.

Is this fragile? You bet. If things change in Emacs Live, I'll need to mirror those changes in my .emacs-live.el - which is why it's recommended that you don't rely on ordering when adding and removing hooks. However, given the situation, this seems like a pragmatic solution.

Feedback definitely welcome.