So, You Dropped Out of College

I was recently chatting to some friends about all the apprenticeship experiments going on and (name withheld to protect the innocent) lamented:

So, what happens when they quit that first job (or worse, get laid-off) and their only skill is writing a Rails app?

I know what happens. It happened to me.

Okay, I didn't actually participate in an apprenticeship program that taught me Rails. However, I did drop out of college after gaining some experience programming, get a job building web applications, and I was laid off about a year later. Getting another job wasn't so hard, but it was hard to find a job that paid as well. It turns out, there's not a huge demand for people who don't have formal education or much experience, go figure.

While I was looking for a new job I noticed that there were significantly more jobs for people with certain skills. I'm sure it differs by location and time - in Jacksonville FL, around 2000, 75% of the jobs openings were looking for people with Microsoft .net skills. Sadly, I did not have .net skills. That was when I learned that I needed to have not just the skills necessary to do my job today, but also the skills necessary to land a new job tomorrow. Since I wasn't already in that situation, I had to settle for a job making .5 of my previous salary; however, my new job did allow me to learn .net.

I firmly believe: if you have less than 4 years of experience, you should always know what technology is in the highest demand for your area, and you should be proficient enough that you can do well in an interview. Perhaps your day to day job already teaches you the most highly sought after skills, fantastic, you should probably learn the 2nd most sought after skill. Don't wait until you're already looking for a job to try to learn a new technology. The truth is, learning something new will almost certainly help you with your current job as well, so everyone wins.

note: searching hipster-programming-jobs.com to find out what's in demand is a selection bias failure. Use monster.com & dice.com. The goal is to appeal to as many companies as possible, not just the cool ones. You can focus on your dream job once you've got enough experience to ensure that your career doesn't suffer a failure to launch.

Whatever technology you choose will certainly have specific books that you should read; however, there are other books that will also help make up for the experience you (sadly, we) missed by not getting a Computer Science degree.

Refactoring was the first book I ever read that really opened my eyes to better ways to program. It's a pretty easy book to read, even for a beginner, and I can't recommend it enough. It's likely that you find that the most in-demand technology is Java or C#, and Refactoring will complement learning either of those languages. However, if you prefer, there's a Refactoring: Ruby Edition as well.

Once you've read refactoring, it's good to dig into Patterns of Enterprise Application Architecture (PofEAA). PofEAA contains patterns that are found all over the enterprise world (including ActiveRecord). You should read this book not because it is a bible, but because it shows you various solutions to commonly encountered problems. It gives you ideas about how other people are solving problems, and provides reference data that you can refer to if you encounter any of those issues in the future.

Those two books will demonstrate to potential employers that you're interested in building successful business applications; however, you'll also want to read a few books that diversify your experience and open your eyes to what can be done by using less mainstream techniques.

The wizard book and the dragon book sound like interesting fictional books, but are actually building blocks for your programming career. Both books are widely regarded as classics that any serious programmer should read. That said, they are not easy to digest. I'm not going to claim it will be fun, but you should push your way through both of these books - and then do it again a year later. These books can help shape your career in a positive way, especially the wizard book, perhaps more than any other text available.

Once you've pushed through both of those books, there are a few follow up options. If you preferred the wizard book, I would recommend following up with The Joy of Clojure or Learn You a Haskell For Great Good. You're unlikely to get a job using either technology, but both will broaden your horizons and also demonstrate to potential employers that you are deeply interested in increasing your programming capabilities. If you preferred the dragon book, I would move on to Domain Specific Languages. Domain Specific Languages is the definitive guide to DSL programming, and provides you with many examples of how to put DSLs to practical use within your applications.

Once you've made it through all (or most) of those books, you shouldn't have any problem finding a job that is both satisfying and also pays well.

Domain Specific Languages don't follow the Principle of Least Surprise

Ola Bini gets it right, as usual, in Evil Hook Methods?, but I think you can actually take the idea a bit further.

DataMapper allows you to gain it's methods simply by an include statement.

class Category
  include DataMapper::Resource
  # ...
end

Ola points out that the include method should not add class methods. At least, that's not what it was designed to do. Include should (by way of append_features) add the constants, methods, and module variables of this module to the class or module that called include.

The problem for me is: Should DataMapper add it's methods to your class when you use Ruby methods as they were originally intended, or when you use DataMapper's Domain Specific Language (DSL).

If DataMapper is a framework and should be used traditionally then you should add it's methods in the following way.

class Category
  include DataMapper::Resource
  extend DataMapper::Resource::ClassMethods
  # ...
end

However, you can't blame DataMapper for following the pattern that's been around since long before DataMapper. At this point I would consider the trick to definitely be an idiom even if it is an anti-pattern. The reality is that include has been stolen by those that prefer the simplest possible Domain Specific Language for adding their behavior.

Martin Fowler describes how a framework can have a traditional API as well as a thin veneer that allows you to use the framework in a more fluent way.

Unfortunately, in the Ruby world we've designed our veneer in a way that doesn't allow for traditional usage.

The other day I noticed something that I thought was equally interesting in the Java world. I was working on a test that used JMock and IntelliJ formatted my code as shown below.

    1 class PublisherTest extends TestCase {
    2     Mockery mockery = new Mockery();
    3 
    4     public void testNamesAreAnnoying() {
    5         final Subscriber subscriber = context.mock(Subscriber.class);
    6 
    7         mockery.checking(new Expectations() {
    8             {
    9                 one (subscriber).receive(message);
   10             }
   11         });
   12         
   13         // ...
   14     }
   15 }

Unimpressed by lines 8 and 10, I changed the code to look like the following snippet.

class PublisherTest extends TestCase {
    Mockery mockery = new Mockery();

    public void testNamesAreAnnoying() {
        final Subscriber subscriber = context.mock(Subscriber.class);

        mockery.checking(new Expectations() {{
            one (subscriber).receive(message);
        }});
        
        // ...
    }
}

Mike Ward said I shouldn't do that because the IntelliJ formatting properly shows an initializer for an anonymous class. Which is absolutely correct, but I don't want an anonymous class with an initializer, I want to use JMock's DSL for defining expectations. And, while the second version might not highlight how those expectations are set, that's not what I care about.

When I write the code I want to create expectations in the easiest way possible, and when I read the code I want the fact that they are expectations to be obvious. I don't think removing lines 8 and 10 reduces readability, in fact it may improve it. Truthfully, I don't care what tricks JMock uses to define it's DSL (okay, within reason), I only care that the result is the most readable option possible.

Back to DataMapper, I believe there's a superior option that allows them to have both a clean DSL and a traditional API. The following code would allow you to add methods as Ola desires (traditionally) and it would allow you to get everything with one method invocation for those that prefer DSL syntax.

class Object
  def data_mapper_resource
    include DataMapper::Resource
    extend DataMapper::Resource::ClassMethods
  end
end

class Category
  include DataMapper::Resource
  extend DataMapper::Resource::ClassMethods
  # ...
end

class Entry
  data_mapper_resource
  # ...
end

The obvious drawback is if everyone starts adding methods to Object we may start to see method collision madness. Of course, if the method names are given decent names it shouldn't be an issue. It's not likely that someone else is going to want to define a method named data_mapper_resource.

Don't worry. For those of you who prefer complexity "just in case", I have a solution for you also.

module DataMapper; end
module DataMapper::Resource
  def self.instance_behaviors
    DataMapper::Resource::InstanceMethods
  end
  
  def self.class_behaviors
    DataMapper::Resource::ClassMethods
  end
end
module DataMapper::Resource::InstanceMethods
  def instance_method
    "instance method"
  end
end
module DataMapper::Resource::ClassMethods
  def class_method
    "class method"
  end
end

class Object
  def become(mod)
      include mod.instance_behaviors
      extend mod.class_behaviors
  end
end

class Category
  include DataMapper::Resource::InstanceMethods
  extend DataMapper::Resource::ClassMethods
  # ...
end

class Entry
  become DataMapper::Resource
end

Entry.class_method            # => "class method"
Entry.new.instance_method     # => "instance method"
Category.class_method         # => "class method"
Category.new.instance_method  # => "instance method"

Domain Specific Language Simplexity

There's one kind of simplicity that I like to call simplexity. When you take something incredibly complex and try to wrap it in something simpler, you often just shroud the complexity -- Anders Hejlsberg

At QCon London I caught the Domain Specific Language (DSL) tutorial by Martin Fowler, Neal Ford, and Rebecca Parsons. While Martin covered how and why you would want to create a DSL he discussed hiding complexity and designing solutions for specific problems. As Martin went into further detail all I could think was: Simplexity.

Anders prefers simple all the way down. Of course, hopefully we all prefer simplicity; however, the devil generally lives in the details.

Some things are complex by nature. Designing a ORM is a great example of a problem that I've yet to see a bullet proof solution for. Most ORMs are complex by necessity, but hiding that complexity in a DSL specific to your application is a good thing. I'll take simplexity over complexity pretty much any day.

I've designed several simplex Domain Specific Languages.

Domain Specific Flow Language
About a year and a half ago Tim Cochran and I designed a few objects that allowed you to define the flow of an application using code similar to the example below.

pages :customer, :shipping, :billing, :summary, :confirmation, :offer, :retailer

flow.customer_driven customer >> shipping >> billing >> summary >> confirmation
flow.retailer_driven retailer >> billing >> summary >> confirmation
flow.offer_driven offer >> shipping >> billing >> summary >> confirmation

The business led us to believe that the flow of the application was going to change often and it needed to be pliable. With that in mind Tim and I set out to create something that was simple to alter and didn't require complete understanding. We wrote out the syntax first and then made it execute. It was fairly easy for us to follow and we had a solution within a few hours. Then we presented the solution to the team.

We showed how to use the DSL and everyone loved it. I think it was actually most people's favorite part of the codebase. Then we dove into how it worked. I'm fairly sure that no one understood the underlying code. The solution was very simplex. It didn't really matter at the time because it was well tested and I don't remember it ever changing in the 6 months I was on the project.

About a year later Jake Scruggs wrote about how the current version of the Flow code is unnecessarily complex. I'm sure Jake is right. If the flows are rarely changing and the underlying code is causing any problems, it should definitely be removed.

The situation brings about an interesting question: how do you know when it's worth creating a DSL? In this example it was a requirement that it be easy to change a flow, but what is easy? I think the original implementation is easier to read and change, but much harder to fully understand and redesign. Unfortunately, there's no simple answer. Like so many things in software development, whether or not to introduce a DSL really depends on the context.

In this case, Tim and I decided to use a DSL because we expected the frequency of change to be high and there was a requirement that the time to make a change needed to be short. Given those requirements, something that is easy to read and change, but hides the underlying complexity is probably a good solution.

Expectations Delegation
I recently created a testing framework: expectations. One of the features of expectations is a solution for testing delegation. Delegation is such a commonly used pattern that I found it helpful to have an simple way to test that delegation is happening as expected.

The following code verifies that delegation does occur to the property that you expect and the result of the method you delegate to is returned from the delegating method.

expect PersonProxy.new.to.delegate(:name).to(:subject) do |proxy|
  proxy.name
end

In the example, the expectation ensures that the proxy.name method calls proxy.subject.name and the return value from proxy.subject.name is also returned by proxy.name.

The resulting expectation is very easy to read and write, but the underlying code is actually quite complex. First of all you need to stub the proxy.subject method. The return value of proxy.subject needs to be a mock that expects a name method call. If the mock receives the name method call you know that delegation happened correctly. The behavior based testing isn't overly complex, but it's not simple either, and it's the easy bit.

Ensuring that the result of proxy.subject.name is returned from proxy.subject is much more complicated. Part of the problem is that the call to proxy.name happens within the block. You could use the result of the block and compare it to what proxy.subject.name is supposed to return, but then the following code would fail.

expect PersonProxy.new.to.delegate(:name).to(:subject) do |proxy|
  proxy.name
  nil
end

You could argue that the above code should cause a failure, but if it does it creates some less than desirable results. First of all, the error message will need to stay something like "Delegation may have occurred, but the return value of the block returned something different so delegation cannot be verified". Also, comparing the result of the block for state based tests makes sense, but here we are only testing delegation so it's not intuitive that the result of the block is significant.

I solved this by specifying the return value of proxy.subject.name, dynamically defining a module that defines the name method, captures the result of the delegation and stores it for later verification, and then extending the proxy with the dynamically defined module. If that's a lot to take in, don't worry I've yet to meet anyone thought it was easy to follow.

Underlying complexity is not desirable. If I could find an easier way to test delegation, I would absolutely change my implementation. Unfortunately, no one has been able to come up with a better solution, yet.

I do think the current solution is better than having to write traditional repetitive and overly verbose code to test every delegation. The DSL for testing delegation allows you to focus on higher value problems. Whether you follow the underlying implementation or not, you must admit that being able to test delegation in a readable and reliable way is a good thing.

Sometimes, complexity is warranted when the resulting interface is simple enough that you gain productivity overall.

Conveying Intention and Removing Noise
Part of what introduces simplexity to a Domain Specific Language is the need to write what you want without introducing side effects. In the expectations example, the return block of the expectation shouldn't be used for comparison because it's not immediately obvious that it would be significant. Using the block return value for comparison is easier from an implementation perspective, but it makes the DSL less usable. This is not a good trade-off.

Noise is defined differently by everyone, but I like to think of it as the difference between what I would like to write and what I need to write.

Removing noise is also a common cause of simplexity. The following code is fairly easy to implement.

expect Process.new.to_have(:finished) do |process|
  process.finished = true
end

But, with expectations I chose to make the code below the legal syntax.

expect Process.new.to.have.finished do |process|
  process.finished = true
end

I chose the later version for a few reasons. The "to" method is the gateway method to all types of expectations. If you want a delegation expectation you write "expect Object.to.delegate...", if you want a behavior based expectation you write "expect Object.to.receive..", and if you want to use a boolean method defined on the class you can use "expect Object.to.be..." or "expect Object.to.have..." depending on what reads better. Using dots allows me to create a consistent interface for all the different expectations, and it also allows me to create expectations on any object without creating several different methods on the Object class.

I also chose to allow dot syntax because once you call "to.be" you can begin calling methods exactly as you would on the object itself. If the object is designed with fluency in mind the test can read as a dot delimited pseudo sentence. For example, the following test is an easy way to verify validation.

class Person < ActiveRecord::Base
  validates_presence_of :name
end

expect Person.new.to.have.errors.on(:name) do |person|
  person.valid?
end

To get this desired behavior I rely on the use of method_missing. Using the method_missing method almost always increases complexity by an order of magnitude. However, in this case, if I didn't use method missing I'd need a solution similar to the one below.

expect Person.new.to.be.true.when(:errors).then(:on, [:name]) do |person|
  person.valid?
end

While this version is easier to implement, it's much less friendly to use.

The level of simplexity should probably be defined by the usage of the Domain Specific Language. If the language is only used by a small subset of employees on few occasions then it might not make sense to increase simplexity. Also, if the complexity of the implementation is raised to a level that it cannot effectively be maintained, you absolutely must back off some. However, if the complexity is maintainable and the DSL is used frequently, I would do my best to reduce "noise".

Conclusion
I believe that simplexity is inevitable when designing a Domain Specific Language, and it shouldn't be something that stops you from using one. However, it's always valuable to weigh how much complexity has been introduced by the desire to create a simple user experience. If the complexity is more painful than the benefit to the consumers, you've clearly gone too far. However, if the complexity is painless, but the DSL is so unpleasant that it's usage is limited, you might need to work on designing more friendly DSL, even at the expense of simplicity.

Implementing an internal DSL in Ruby

On Christmas Day 2007 I released the 0.0.1 version of expectations. Expectations is the result of several months of designing what I wanted the Domain Specific Language (DSL) to look like, and a few days of making it executable. To design the DSL, I created what I was looking for without worrying about implementation. This entry is going to focus on what it took to take the DSL from looking good to actually executing.

The following code is an example of what I decided I wanted the DSL to look like (for a state based test, expectations also has support for behavior based testing).

Expectations do
  expect :expected do
    :expected
  end
end

When making a DSL executable I generally begin by building up objects with the values expressed in the DSL. For example, the above code creates a test suite containing 1 expectation where the expected value is :expected and the actual value can be determined by evaluating the block. While designing expectations I tested it using expectations, but that's probably more complex than is necessary for this example, so I'll simply print values instead.

The first step is ensuring that an Expectations::Suite is created when Expectations is called with a block. The following code demonstrates that an Expectations::Suite is created successfully. The code executes since the block given to Expectations is never executed.

require 'singleton'
module Expectations; end

class Expectations::Suite
  include Singleton
end

def Expectations(&block)
  Expectations::Suite.instance
end

Expectations do
  expect :expected do
    :expected
  end
end

Expectations::Suite.instance # => #<Expectations::Suite:0x1d4fc>

note: Expectations::Suite is a singleton so that it can be appended to multiple times and later used for execution without being stored within another object.

The next step is to verify the number of expectations the suite stores for later execution. The following code verifies that one object is stored in the expectations array.

require 'singleton'
module Expectations; end

class Expectations::Suite
  include Singleton
  attr_accessor :expectations
  
  def initialize
    self.expectations = []
  end
  
  def expect(expected, &actual)
    expectations << :expectation
  end
  
end

def Expectations(&block)
  Expectations::Suite.instance.instance_eval(&block)
end

Expectations do
  expect :expected do
    :expected
  end
end

Expectations::Suite.instance # => #<Expectations::Suite:0x1c82c @expectations=[:expectation]>
Expectations::Suite.instance.expectations.size # => 1

The final step is to create an object for each expectation and verify the values of both the expected and the actual.

require 'singleton'
module Expectations; end

class Expectations::Suite
  include Singleton
  attr_accessor :expectations
  
  def initialize
    self.expectations = []
  end
  
  def expect(expected, &actual)
    expectations << Expectations::Expectation.new(expected, actual)
  end
  
end

class Expectations::Expectation
  attr_accessor :expected, :actual
  
  def initialize(expected, actual)
    self.expected, self.actual = expected, actual.call
  end
end

def Expectations(&block)
  Expectations::Suite.instance.instance_eval(&block)
end

Expectations do
  expect :expected do
    :expected
  end
end

Expectations::Suite.instance # => #<Expectations::Suite:0x1b184 @expectations=[#<Expectations::Expectation:0x1b0e4 @expected=:expected, @actual=:expected>]>
Expectations::Suite.instance.expectations.size # => 1
Expectations::Suite.instance.expectations.first.expected # => :expected
Expectations::Suite.instance.expectations.first.actual # => :expected

note: expectations actually defers determining the value of actual until the suite is executed, but that's outside the scope of this example.

The bottom two lines of the last example show that the Expectations::Expectation instance is being populated successfully.

That's it, at this point the DSL can successfully be parsed. Of course, making the expectations run to verify that the values match still needs to be done, but that's no different than working with any plain old ruby code. The DSL can be evaluated and the framework can work with the objects that have been created.

As the example demonstrates, designing an internal DSL in Ruby is very easy largely due to open classes, class methods, and the ability to eval. The above code for evaluating the DSL is less than 30 lines of code.

Designing a Domain Specific Language

I've worked on and with various Domain Specific Languages (DSLs), but I've found designing them to be the same process no matter what type of DSL is being created.

In March of 2005 I started one of the more technically interesting projects of my life. ThoughtWorks had been hired to design a system that would allow the director of finance for the client bank to edit, test, and deploy the business rules of the system without any assistance from IT. It was a bold step, but the return on investment was realized within weeks, not months (or worse, years).

One of the major challenges of designing this type of system is determining what the language should look like. Being new to the idea, we started out by telling the business what was possible. That was a mistake. The early versions of the language were easy for IT to work with, but were clunky for the business. The result was clearly suboptimal. As we continued evolving the language for new features we took a different approach, we asked the director what he wanted the language to look like. The more he shaped the language the easier it was for him to take ownership of the domain logic.

We figured out that we didn't need to write the domain logic in our language, we needed to make the domain language executable. The resulting project was a huge success, which led to me coming up with the term Business Natural Languages and presenting the idea at several conferences*.

Almost 3 years later, I decided I needed a new framework for unit testing. The primary motivation for designing a new framework was my desire for a more syntactically elegant way of defining tests. This time I knew I wanted to design what the language looked like and then worry about making it work at a later time. In fact, when writing expectations, I had a small suite of example tests before I had any way to execute them.

None of the languages I've helped design are flawless. But, I've seen significantly more success with the languages that were designed by a subject matter expert concerned with the language, instead of a language designer who was concerned with how to make the language execute.

* If you'd like to catch the Business Natural Language presentation you can see it at QCon London 2008 or rupy.eu 2008.

ThoughtWorks DSL Podcast (part 2)

Part 2 of the DSL podcast Martin Fowler, Rebecca Parsons, Neal Ford and I participated in is now available. The podcast is once again being released as part of the ThoughtWorks IT Matters series and is available at ThoughtWorks: What we say.

ThoughtWorks DSL Podcast

I recently participated in a podcast with Martin Fowler, Rebecca Parsons and Neal Ford on the topic of Domain Specific Languages. The podcast is being released as part of the ThoughtWorks IT Matters series and is available at ThoughtWorks: What we say.

Feedback welcome.

DSL interview

At RubyConf 2006, Zak Tamsen and I were interviewed by Obie Fernandez. In the months leading up to the conference I worked with both Obie and Zak on different projects where a DSL was used by the customer to specify the business rules of an application. Both projects were among the more interesting of my career. The interview discusses how we approached the problem and the successes that followed.

Check out the interview on InfoQ: Jay Fields and Zak Tamsen on Domain Specific Languages

Ruby: Unit testing delegations

When working with Rails I tend to use a lot of delegations. Since I'm a big fan of the Law of Demeter I often find myself delegating an_ad_instance.lister_name to the name method of the lister attribute.

Class Ad < ActiveRecord::Base
  extend Forwardable
  def_delegator :lister, name, lister_name
  def lister
    ...
  end
end

Often, I end up writing tests for something even as simple as delegation, since I try to TDD all the code I write.

class AdTest < Test::Unit::TestCase
  def test_lister_name_is_delegated_to_listers_name
    ad = Ad.new(:lister => Lister.new(:name => 'lister_name'))
    assert_equal 'lister_name', ad.lister_name
  end
end

After a few of these tests, the duplication begins to tell me that I can do something simpler with some metaprogramming. So, I stole the idea from Validatable and created some delegation custom assertions.

class AdTest < Test::Unit::TestCase
  Ad.delegates do
    lister_name.to(:lister).via(:name)
    lister_address.to(:lister).via(:address)
    ...
  end
end

The above delegation testing DSL allows me to test multiple validations without the need for many (very similar) tests.

The implementation to get this working is fairly straightforward: I create a subclass of the class under test, add a constructor to the subclass that takes the objec tthat's going to be delegated to, and set the value of the attribute on the delegate object. I also add an attr_accessor to the subclass, since all I care about is the delegation (for this test). It doesn't matter to me that I'm changing the implementation of the lister method because I'm testing the delegation, not the lister method.

class DelegateAssertion
  attr_accessor :desired_method, :delegate_object, :original_method
  
  def initialize(desired_method)
    self.desired_method = desired_method
  end
  
  def to(delegate_object) 
    self.delegate_object = delegate_object
    self
  end
  
  def via(original_method)
    self.original_method = original_method
  end
end

class DelegateCollector
  def self.gather(block) 
    collector = new
    collector.instance_eval(&block)
    collector.delegate_assertions
  end
  
  attr_accessor :delegate_assertions
  
  def delegate_assertions
    @delegate_assertions ||= [] 
  end
  
  def method_missing(sym, *args)
    assertion = DelegateAssertion.new(sym)
    delegate_assertions << assertion
    assertion
  end
end

class Class
  def delegates(&block) 
    test_class = eval "self", block.binding
    assertions = DelegateCollector.gather(block)
    assertions.each do |assertion|
      klass = Class.new(self)
      klass.class_eval do
        attr_accessor assertion.delegate_object
        define_method :initialize do |delegate_object|
          self.send :"#{assertion.delegate_object}=", delegate_object
        end
      end
      test_class.class_eval do 
        define_method "test_#{assertion.desired_method}_is_delegated_to_#{assertion.delegate_object}_via_#{assertion.original_method}" do
          klass_instance = klass.new(stub(assertion.original_method => :original_value))
          begin
            assert_equal :original_value, klass_instance.send(assertion.desired_method)
          rescue Exception => ex
            add_failure "Delegating #{assertion.desired_method } to #{assertion.delegate_object} via #{assertion.original_method} doesn't work"
          end
        end
      end
    end
  end
end

The resulting tests make it hard to justify leaving of tests, even if you are 'only doing simple delegation'.

note: In the example I use an ActiveRecord object, but since I'm adding this behavior to Class any class can take advantage of these custom validations.