Wednesday, May 15, 2013

Emacs Lisp: Font Lock for Clojure's Partial

I love using partial, but I dislike the length of the function name. There's a simple solution, define another function with a shorter name that simply calls (or is) partial. This is exactly what I did in the jry library.

I liked the use of % due to partial feeling similar to creating a function using #(), and % having a special meaning inside #(). I thought they tied well together. Unfortunately, there's an obvious problem, things would be very broken if you tried to use the '%' function in an anonymous function defined with #(). Somewhere along the way this issue caused me to stop using jry/%.

Using partial is great: it's part of the standard lib, and I don't need to explain it to anyone who joins my team or any future maintainers of the code I write. Still, I want something shorter, and I've always had a background thread looking for another shorter-than-partial solution. While recently contributing to emacs-live I found the solution I was looking for: clojure-mode font lock.

The following code can now be found in my emacs configuration.

This solution feels like the best of both worlds. My code still uses the function from the standard library, my colleagues still see a function they already know, and 'partial' only takes up one character space in my buffer. The image below is what you'll see if you put the above emacs-lisp in your config.

Tuesday, May 14, 2013

Clojure: Testing The Creation Of A Partial Function

I recently refactored some code that takes longs from two different sources to compute one value. The code originally stored the longs and called a function when all of the data arrived. The refactored version partials the data while it's incomplete and executes the partial'd function when all of the data is available. Below is a contrived example of what I'm taking about.

Let's pretend we need a function that will allow us to check whether or not another drink would make us legally drunk in New York City.

The code below stores the current bac and uses the value when legally-drunk? is called.

The following (passing) tests demonstrate that everything works as expected.

This code works without issue, but can also be refactored to store a partial'd function instead of the bac value. Why you would want to do such a thing is outside of the scope of this post, so we'll just assume this is a good refactoring. The code below no longer stores the bac value, and instead stores the pure-legally-drunk? function partial'd with the bac value.

Two of the three of the tests don't change; however, the test that was verifying the state is now broken.

note: The test output has been trimmed and reformatted to avoid horizontal scrolling.

In the output you can see that the test is failing as you'd expect, due to the change in what we're storing. What's broken is obvious, but there's not an obvious solution. Assuming you still want this state based test, how do you verify that you've partial'd the right function with the right value?

The solution is simple, but a bit tricky. As long as you don't find the redef too magical, the following solution allows you to easily verify the function that's being partial'd as well as the arguments.

Those tests all pass, and should provide security that the legally-drunk? and update-bac functions are sufficiently tested. The pure-legally-drunk? function still needs to be tested, but that should be easy since it's a pure function.

Would you want this kind of test? I think that becomes a matter of context and personal preference. Given the various paths through the code the following tests should provide complete coverage.

The above tests make no assumptions about the implementation - they actually pass whether you :use the 'original namespace or the 'refactored namespace. Conversely, the following tests verify each function in isolation and a few of them are very much tied to the implementation.

Both sets of tests would give me confidence that the code works as expected, so choosing which tests to use would become a matter of maintenance cost. I don't think there's anything special about these examples; I think they offer the traditional trade-offs between higher and lower level tests. A specific trade-off that stands out to me is identifying defect localization versus having to update the test when you update the code.

As I mentioned previously, the high-level-expectations work for both the 'original and the 'refactored namespaces. Being able to change the implementation without having to change the test is obviously an advantage of the high level tests. However, when things go wrong, the lower level tests provide better feedback for targeting the issue.

The following code is exactly the same as the code in refactored.clj, except it has a 1 character typo. (it's not necessary to spot the typo, the test output below will show you want it is)

The high level tests give us the following feedback.
failure in (high_level_expectations.clj:14) : expectations.high-level-expectations
(expect
 true
 (with-redefs
  [state (atom {})]
  (update-bac 0.01)
  (legally-drunk? 0.07)))

           expected: true 
                was: false
There's not much in that failure report to point us in the right direction. The unit-level-expectations provide significantly more information, and the details that should make it immediately obvious where the typo is.
failure in (unit_level_expectations.clj:8) : expectations.unit-level-expectations
(expect
 {:legally-drunk?* [pure-legally-drunk? 0.04]}
 (with-redefs [state (atom {}) partial vector] (update-bac 0.04)))

           expected: {:legally-drunk?* [# 0.04]} 
                was: {:legally-drunk?** [# 0.04]}
 
           :legally-drunk?** with val [# 0.04] 
                             is in actual, but not in expected
           :legally-drunk?* with val [# 0.04] 
                            is in expected, but not in actual
The above output points us directly to the extra asterisk in update-bac that caused the failure.

Still, I couldn't honestly tell you which of the above tests that I prefer. This specific example provides a situation where I think you could convincingly argue for either set of tests. However, as the code evolved I would likely choose one path or the other based on:
  • how much 'setup' is required for always using high-level tests?
  • how hard is it to guarantee integration using primarily unit-level tests?
In our examples the high level tests require redef'ing one bit of state. If that grew to a few pieces of state and/or a large increase in the complexity of the state, then I may be forced to move towards more unit-level tests. A rule of thumb I use: If a significant amount of the code within a test is setting up the test context, there's probably a smaller function and a set of associated tests waiting to be extracted.

By definition, the unit-level tests don't test the integration of the various functions. When I'm using unit-level tests, I'll often test the various code paths at the unit level and then have a happy-path high-level test that verifies integration of the various functions. My desire to have more high-level tests increases as the integration complexity increases, and at some point it makes sense to simply convert all of the tests to high-level tests.

If you constantly re-evaluate which tests will be more appropriate and switch when necessary, you'll definitely come out ahead in the long run.

Tuesday, May 07, 2013

Recovering Lost Post Data

I recently typed out a long, thoughtful response in a textarea. I clicked submit, like I've done millions of times, and I got the dreaded "session expired" error message. This happens very, very rarely, but it's devastating when it does. Creating long & thoughtful responses isn't something that comes naturally for me. I crossed my fingers and clicked back. No luck, web 2.0 dynamically created text boxes ensured Chrome had no chance to preserve my editing state.

My first reaction was: I guess I'm not responding after all. Then it occurred to me, DevTools must have my data somewhere, right? Lucky for me, the answer was yes.

There might be easier ways, this is what worked for me:
  • open DevTools
  • go to the "Network" tab.
  • look for the row with the method POST.
    • If you don't see a POST row, try refreshing the page. With any luck you'll get a repost confirmation dialog, giving you some hope that your data is still around. (You'll want to allow the data repost)
  • click on the POST row, and scroll down till you see "Form Data". If you've gotten this far, hopefully you'll find your data in clear text and able to be copied.
The examples from this post are from following the instructions above and logging in to twitter.com. If you've ever lost post data in the past, you may want to give these directions a dry-run now.

Thursday, May 02, 2013

Emacs Lisp: Toggle Between a Clojure String and Keyword

When I was doing a fair bit of Ruby I often used the TextMate's shortcut (Ctrl+:) to convert a Ruby String to a Symbol or a Ruby Symbol to a String. It's something I've periodically missed while doing Clojure, and yesterday I found myself in the middle of a refactoring that was going to force the conversion of 5+ Clojure Keywords to Strings.

The following emacs lisp is my solution for toggling between Clojure Strings and Keywords. The standard disclaimers apply - it works on my machine, and I've never claimed to know emacs lisp well.

A quick video of the behavior:

Tuesday, April 30, 2013

Year Five

The average lifespan for a software engineering job is 4 years. Okay, I've never actually seen proof (or contradiction), but that's the general feeling in the groups I associate with. Perhaps that's selection bias - my employer has generally changed on year 3 or 4. Perhaps this is the exception and not the rule, in that case feel free to simply read this as an experience report. However, I do think it's somewhat common for developers to leave around year 3 or 4. This entry contains speculation on why they leave, and offers one idea on what employers can do to break that cycle.

My 4 year employment cycle generally looks like this
  • Year One: "I'm in over my head. My semi-bluff was in-fact a bluff. They're going to fire me any day."
  • Year Two: "It's nice to feel like a productive team member"
  • Year Three: "This is fun, and I'm not bad at it. It's satisfying to pass on knowledge to teammates."
  • Year Four: "This feels repetitive, that grass over there sure looks greener"
I expect that I, like many programmers, probably undervalue my contribution in the early days and overvalue my contribution in the latter days.

In Year Three and Four at DRW I spent some time thinking about how I felt, and observing the behavior of some colleagues that were also on year three and four. A few things stood out to me.
  • A company you don't work at always seems to have infinite possibilities; however, after a few years with an employer, it's extremely clear what your options are. More importantly, it's very clear what limitations will likely always be there.
  • A company you don't work at contains no code you're responsible for. Conversely, any company you've been with for 4 years probably has plenty of code you're not proud of. If you're responsible for that code, it's a constant reminder of your previous limitations. If you're not responsible for it, your co-workers aren't likely to let you forget about it anytime soon.
  • There's always someone willing to pay you more than you're worth. After several years with a company it's likely that they're going to pay you what you're worth, but not what some other company thinks you're worth. I'm surprised that more companies don't pay (the employees they want to keep) what their "flawless market value" would be. In other words, what would you pay them if they interviewed, you determined what they knew, you determined what value they would bring, and you were completely ignorant of their flaws? That's what your competition is likely doing. That's what you're fighting against if you want to keep them around.
  • A new job often offers a new challenge. Once you feel like you've given that challenge your best shot, what remains? If you did a great job, it's likely that you'll have plenty of other options. However, if you've done a good job, you may be stuck in a spot where there aren't as many open doors and challenges to choose from - not nearly as many as a position at another company will appear to offer.
I was recently in Punta Cana for wedding, and I was on the beach - working on my laptop. My wife asked: don't you want some time off? My response was short and immediate: no. Later that evening my wife and I discussed my work situation. I observed that I'm in Year Five at DRW and I'm happy, happier than ever, strange - given my previous experiences. She asked if I thought that I was working too much, and if I thought that I would burn out. I remarked: I'd rather have a job that I love, that I don't like to be away from, than a job where I feel like I need a week or two off.

I hear you, nice work if you can get it. I don't have a general recipe for getting there, but I know how I got there.

Back in 2009 I interviewed at DRW. At the time I was working for ThoughtWorks, and my client was Forward. I considered the founder of Forward to be a friend and someone I would gladly work for. I decided it was time to leave ThoughtWorks (after 3.5 years), and I was sure that Forward would be my future home. I remarked to my DRW recruiter "H" (who also happened to be a friend from my ThoughtWorks days) that one of the best things about Forward was knowing that I liked and trusted the man who ran Forward. H said nothing, but made a brilliant move.

In my interview I was grilled, killed even, and then things turned. I met with a guy who asked me a few questions and then told me about the company: the vision, the people, and where I could fit in. He was smart, easy to talk to, and someone I related to. We discussed things casually, it didn't feel like a company pitch in any way at all, it felt like small-talk - something I was very grateful for after the beating I'd taken previously in of the day. After everything concluded I hit the bar with my friends, including H. At that point they revealed to me that the guy I'd met was the partner at the firm that was (among other things) responsible for the firm's technology. I'd also met the CTO, and various other people responsible for technology in the firm. H had shown me that DRW, just like Forward, had what I like to call Awesome All the Way Up.**

Awesome All the Way Up has served me very well at DRW. To this day I remain in fairly common contact with the CTO and several of DRW's partners. About 6 months ago I asked 3 favors. First of all, I asked for enough money to pay someone's salary for 6 months. I identified a project that I wanted to undertake, and I needed help to complete it. Then things got unconventional, I asked if I could create a contract-to-hire situation. Even more unconventional, I pursued a friend and previous colleague who lived in Austin, Texas. DRW rarely uses contractors, and has no other remote employees that I'm aware of. An appropriate amount of questions were asked, but in the end my request was granted.

The experiment is on-going, but I'm very happy with our progress so far. That's all well-and-good, but the support of DRW is the important aspect of the story. I'm confident that their support of my unconventional requests was a major factor in ensuring my happiness in Year Five. We recently hired John Hume, thus declaring success at some level already. However, if things had gone poorly, both parties could have gone their separate ways with little lost and lessons learned. More importantly to me, DRW would have continued to give me confidence that they were willing to take chances to provide me with opportunities and ensure my continued happiness at the firm.

There's a similar discussion around DRW allowing me to use Clojure as my primary development language. I'll spare you the long version. tl; dr: They gave me a reasonable amount of space to try something new, and supported me appropriately as we found more and more success.

Not all of my experiments are green-lighted, and I've also had unsuccessful outcomes. DRW has done a good job of not setting me up to fail; my ideas that have a low probability of succeeding are fleshed out and appropriately shot down. All experiments have risk measures put in place, limited downside, and are reassessed constantly. It's great to have support when things are going well, and it's essential to have support when things don't go as planned.

For me, that's been the secret for keeping me around more than 4 years: An appropriate amount of trust and a willingness to experiment.

A foreign thought also recently came to mind. For the first time in my life I can say that I see myself happy and successful at my current employer in 10 years. This is a question I've asked many people since it occurred to me. To date, +AdeOshineye (http://www.oshineye.com/) is the only person who's responded affirmatively. The results aren't surprising to me, but I do wonder why more employees and employers aren't looking for ways to extend relationships.

Perhaps the secret for keeping me around isn't more broadly applicable; however, simply asking what will keep an individual around is probably the more important message in this entry. It's good to know what will make someone happy now, but it seems like it's equally important to know what will make them happy in the long term. I suspect the answers will be at least a little, if not very different.

The way things currently stand, I'm looking forward to writing about Year Six.

** DRW became my home in the end; however, Forward continues to do well. I suspect Awesome All the Way Up would have ensured happy and gainful employment at either destination. I remain in regular contact with my friends at Forward.