Domain Specific Language Simplexity

There's one kind of simplicity that I like to call simplexity. When you take something incredibly complex and try to wrap it in something simpler, you often just shroud the complexity -- Anders Hejlsberg

At QCon London I caught the Domain Specific Language (DSL) tutorial by Martin Fowler, Neal Ford, and Rebecca Parsons. While Martin covered how and why you would want to create a DSL he discussed hiding complexity and designing solutions for specific problems. As Martin went into further detail all I could think was: Simplexity.

Anders prefers simple all the way down. Of course, hopefully we all prefer simplicity; however, the devil generally lives in the details.

Some things are complex by nature. Designing a ORM is a great example of a problem that I've yet to see a bullet proof solution for. Most ORMs are complex by necessity, but hiding that complexity in a DSL specific to your application is a good thing. I'll take simplexity over complexity pretty much any day.

I've designed several simplex Domain Specific Languages.

Domain Specific Flow Language
About a year and a half ago Tim Cochran and I designed a few objects that allowed you to define the flow of an application using code similar to the example below.

pages :customer, :shipping, :billing, :summary, :confirmation, :offer, :retailer

flow.customer_driven customer >> shipping >> billing >> summary >> confirmation
flow.retailer_driven retailer >> billing >> summary >> confirmation
flow.offer_driven offer >> shipping >> billing >> summary >> confirmation

The business led us to believe that the flow of the application was going to change often and it needed to be pliable. With that in mind Tim and I set out to create something that was simple to alter and didn't require complete understanding. We wrote out the syntax first and then made it execute. It was fairly easy for us to follow and we had a solution within a few hours. Then we presented the solution to the team.

We showed how to use the DSL and everyone loved it. I think it was actually most people's favorite part of the codebase. Then we dove into how it worked. I'm fairly sure that no one understood the underlying code. The solution was very simplex. It didn't really matter at the time because it was well tested and I don't remember it ever changing in the 6 months I was on the project.

About a year later Jake Scruggs wrote about how the current version of the Flow code is unnecessarily complex. I'm sure Jake is right. If the flows are rarely changing and the underlying code is causing any problems, it should definitely be removed.

The situation brings about an interesting question: how do you know when it's worth creating a DSL? In this example it was a requirement that it be easy to change a flow, but what is easy? I think the original implementation is easier to read and change, but much harder to fully understand and redesign. Unfortunately, there's no simple answer. Like so many things in software development, whether or not to introduce a DSL really depends on the context.

In this case, Tim and I decided to use a DSL because we expected the frequency of change to be high and there was a requirement that the time to make a change needed to be short. Given those requirements, something that is easy to read and change, but hides the underlying complexity is probably a good solution.

Expectations Delegation
I recently created a testing framework: expectations. One of the features of expectations is a solution for testing delegation. Delegation is such a commonly used pattern that I found it helpful to have an simple way to test that delegation is happening as expected.

The following code verifies that delegation does occur to the property that you expect and the result of the method you delegate to is returned from the delegating method.

expect PersonProxy.new.to.delegate(:name).to(:subject) do |proxy|
  proxy.name
end

In the example, the expectation ensures that the proxy.name method calls proxy.subject.name and the return value from proxy.subject.name is also returned by proxy.name.

The resulting expectation is very easy to read and write, but the underlying code is actually quite complex. First of all you need to stub the proxy.subject method. The return value of proxy.subject needs to be a mock that expects a name method call. If the mock receives the name method call you know that delegation happened correctly. The behavior based testing isn't overly complex, but it's not simple either, and it's the easy bit.

Ensuring that the result of proxy.subject.name is returned from proxy.subject is much more complicated. Part of the problem is that the call to proxy.name happens within the block. You could use the result of the block and compare it to what proxy.subject.name is supposed to return, but then the following code would fail.

expect PersonProxy.new.to.delegate(:name).to(:subject) do |proxy|
  proxy.name
  nil
end

You could argue that the above code should cause a failure, but if it does it creates some less than desirable results. First of all, the error message will need to stay something like "Delegation may have occurred, but the return value of the block returned something different so delegation cannot be verified". Also, comparing the result of the block for state based tests makes sense, but here we are only testing delegation so it's not intuitive that the result of the block is significant.

I solved this by specifying the return value of proxy.subject.name, dynamically defining a module that defines the name method, captures the result of the delegation and stores it for later verification, and then extending the proxy with the dynamically defined module. If that's a lot to take in, don't worry I've yet to meet anyone thought it was easy to follow.

Underlying complexity is not desirable. If I could find an easier way to test delegation, I would absolutely change my implementation. Unfortunately, no one has been able to come up with a better solution, yet.

I do think the current solution is better than having to write traditional repetitive and overly verbose code to test every delegation. The DSL for testing delegation allows you to focus on higher value problems. Whether you follow the underlying implementation or not, you must admit that being able to test delegation in a readable and reliable way is a good thing.

Sometimes, complexity is warranted when the resulting interface is simple enough that you gain productivity overall.

Conveying Intention and Removing Noise
Part of what introduces simplexity to a Domain Specific Language is the need to write what you want without introducing side effects. In the expectations example, the return block of the expectation shouldn't be used for comparison because it's not immediately obvious that it would be significant. Using the block return value for comparison is easier from an implementation perspective, but it makes the DSL less usable. This is not a good trade-off.

Noise is defined differently by everyone, but I like to think of it as the difference between what I would like to write and what I need to write.

Removing noise is also a common cause of simplexity. The following code is fairly easy to implement.

expect Process.new.to_have(:finished) do |process|
  process.finished = true
end

But, with expectations I chose to make the code below the legal syntax.

expect Process.new.to.have.finished do |process|
  process.finished = true
end

I chose the later version for a few reasons. The "to" method is the gateway method to all types of expectations. If you want a delegation expectation you write "expect Object.to.delegate...", if you want a behavior based expectation you write "expect Object.to.receive..", and if you want to use a boolean method defined on the class you can use "expect Object.to.be..." or "expect Object.to.have..." depending on what reads better. Using dots allows me to create a consistent interface for all the different expectations, and it also allows me to create expectations on any object without creating several different methods on the Object class.

I also chose to allow dot syntax because once you call "to.be" you can begin calling methods exactly as you would on the object itself. If the object is designed with fluency in mind the test can read as a dot delimited pseudo sentence. For example, the following test is an easy way to verify validation.

class Person < ActiveRecord::Base
  validates_presence_of :name
end

expect Person.new.to.have.errors.on(:name) do |person|
  person.valid?
end

To get this desired behavior I rely on the use of method_missing. Using the method_missing method almost always increases complexity by an order of magnitude. However, in this case, if I didn't use method missing I'd need a solution similar to the one below.

expect Person.new.to.be.true.when(:errors).then(:on, [:name]) do |person|
  person.valid?
end

While this version is easier to implement, it's much less friendly to use.

The level of simplexity should probably be defined by the usage of the Domain Specific Language. If the language is only used by a small subset of employees on few occasions then it might not make sense to increase simplexity. Also, if the complexity of the implementation is raised to a level that it cannot effectively be maintained, you absolutely must back off some. However, if the complexity is maintainable and the DSL is used frequently, I would do my best to reduce "noise".

Conclusion
I believe that simplexity is inevitable when designing a Domain Specific Language, and it shouldn't be something that stops you from using one. However, it's always valuable to weigh how much complexity has been introduced by the desire to create a simple user experience. If the complexity is more painful than the benefit to the consumers, you've clearly gone too far. However, if the complexity is painless, but the DSL is so unpleasant that it's usage is limited, you might need to work on designing more friendly DSL, even at the expense of simplicity.

The Language Question

My first job was working with Cold Fusion for a year. For the next two years I worked primarily with Javascript and a proprietary language. Next was a brief stretch with PHP and ASP, followed by about 3 years of .net. Finally, for the past 2 years I've been doing Ruby. I do not consider myself a Ruby developer. I prefer to just be a developer.

I don't expect that Ruby will be the last language I ever work with. In fact, I expect I'll probably be doing something new in the near future. The question is what language should I work with next? Should I go for a classic like Lisp or Smalltalk? Should I give Scala or Erlang a shot and ride the concurrency wave?

The problem is, I'm asking the wrong question. Rarely when discussing languages do the languages themselves dominate the conversation. I can't remember ever being in a conversation where someone won a language debate with things like closures, metaprogramming or static typing. Instead, the primary focus of language debates are usually around frameworks, performance, IDE support, developer availability, what OS does it run on, and any number of other factors that are only related to the language.

The question extends even beyond factors related to the language.

At Google I’ll work with C++ rather than (for example) Ruby but I do get to be part of changing the world. -- Jon Tirsen

Jon gave up a job using his language of choice to work with another language he likes far less, but for a cause he's very interested in. I have another friend who recently starting working with an investment company because he was interested in the domain.

Two years ago I started looking at Ruby because several of my colleagues were giving it a look. I preferred Rake to NAnt, and starting using it on my .net projects. Before long, someone asked me to be part of a Ruby project, because of my limited exposure to Rake.

I got introduced to Ruby by coworkers who were interested. I got involved with Ruby because I prefer build files that do not require XML programming. I stuck with Ruby because we had a steady stream of Ruby projects that needed experienced developers, I got plenty of blog content, I liked the composition of the Ruby teams I got to work with, and I liked working with clients who are comfortable with early adoption.

Notice, none of the reasons I use Ruby have anything to do with the Ruby language itself.

I'm interested in doing something new because I feel like I've been doing the same thing for about a year now. I'm also interested in traveling to other ThoughtWorks offices and getting some fresh ideas for innovation.

Again, none of my desires have anything to do with language.

I'm not giving you the tired "no silver bullet" or the hand waving "the right tool for the right job". I'm asserting that people use those phrases to justify their language choice, but you'd be better off asking what the real motivations for choosing a language are. What other factors does the language introduce that make it their choice.

It's also helpful to have this understanding when considering criticizing someone's language choice. My friends aren't using Java because they like Java, they are using it because they like IntelliJ, high performance on a few boxes, simple deployment, Hibernate, String Template, Spring, and a hundred other factors. Therefore, criticizing Java as a language doesn't really do anyone any good. Even if I convinced them that Lisp is a better language than Java, I still wouldn't have convinced them to use Lisp on their next project.

Example Dilemma

Creating examples for blog entries, presentations, articles, and books is very, very hard. You need an example that is easy enough that hopefully all, but at least most can understand it quickly. However, any example that trivial almost always has to make several conscious omissions.

For example, rarely do examples take into account error handling, performance, security, etc. Almost no example will include code that addresses cross cutting concerns. Nor should they, in my opinion. When the topic is error handling, the examples should include error handling. But, when the topic is how to design a DSL, it's much better to concentrate on communicating only that.

Cross cutting concerns aren't the only things ignored in the name of concise examples. Proper domain modeling, encapsulation, and many other best practices are also often ignored. I believe this is because many of those choices depend so heavily on context it would be impossible to devise an example that fits all possible scenarios. Instead of building the proper context, it's better to create an example that focuses on the problem and simply admit that you'll need to mold the example to your domain.

Of course, I'm not the first to write about these things. The following excerpt is from Martin Fowler, in chapter 1 of Refactoring.

With an introductory example, however, I run into a big problem. If I pick a large program, describing it and how it is refactored is too complicated for any reader to work through. (I tried and even a slightly complicated example runs to more than a hundred pages.) However, if I pick a program that is small enough to be comprehensible, refactoring does not look like it is worthwhile.

Thus I'm in the classic bind of anyone who wants to describe techniques that are useful for real-world programs. Frankly it is not worth the effort to do the refactoring that I'm going to show you on a small program like the one I'm going to use. But if the code I'm showing you is part of a larger system, then the refactoring soon becomes important. So I have to ask you to look at this and imagine it in the context of a much larger system.

Michael Feathers also has a section in the Preface of Working Effectively with Legacy Code where he describes his approach to coming up with examples.

One thing that you will notice as you read this book is that it is not a book about pretty code. The examples that I use in the book are fabricated because I work under nondisclosure agreements with clients. But in many of the examples, I've tried to preserve the spirit of code that I've seen in the field. I won't say that the examples are always representative. There certainly are oases of great code out there, but, frankly, there are also pieces of code that are far worse than anything I can use as an example in this book. Aside from client confidentiality, I simply couldn't put code like that in this book without boring you to tears and burying important points in a morass of detail. As a result, many of the examples are relatively brief. If you look at one of them and think "No, he doesn't understand—my methods are much larger than that and much worse," please look at the advice that I am giving at face value and see if it applies, even if the example seems simpler.

I see this most often with blog entries. Of course, this is because example code for blog entires isn't (usually) reviewed before it's published. Generally, a blog entry is published and a few comments begin to appear with a "better" way to solve the problem. Of course, the "better" way doesn't address the topic of the blog entry, so they aren't very relevant. The author (or another comment) responds with the traditional "You missed the point" and neither side really gains anything.

I used to believe that it wasn't my problem if the reader missed the point. I was wrong. Martin and Michael's books are proof, this isn't a new problem. The entire reason I write is to provide ideas to my readers. I do believe it's my responsibility as an author to come up with the best possible example, which includes accounting for ways that someone might miss the point. If they miss the point, I've failed.

Of course, feedback comes in many forms. The last comment I got started with "I know this is off topic...". When a conversation is framed that way, the author is much less likely to become defensive of their example, and much more likely to happily address the off topic comment. I've often gotten similar comments related to examples that start with "I understand why the example addresses the problem you're writing about, but why not solve the problem using...". The people who leave comments this way seem to understand the Example Dilemma, and always seem to get better responses from authors.

Most people are curious and excited when they have what they believe to be a superior solution. This explains the constant flow of "better" examples and off topic questions. I think this is a good thing, otherwise I would turn comments off on my blog. But, I think understanding the Example Dilemma can turn an adversarial exchange into a beneficial one.

I don't believe the Example Dilemma is going anywhere, but I do believe we can all benefit from being more conscious of it.

Ruby: inject

I love inject. To be more specific, I love Enumerable#inject. I find it easy to read and easy to use. It's powerful and it lets me be more concise. Enumerable#inject is a good thing.

Of course, I didn't always love it. When I was new to Ruby I didn't understand what it was, so I found it hard to follow. However, finding it hard to understand didn't make me run from it, instead I wanted to know what all the hype was about. Enumerable#inject is an often used method by many great Rubyists, and I wanted to know what I was missing.

So, to learn about Enumerable#inject I did what I always do, I used it every possible way I could think of.

Example 1: Summing numbers
Summing numbers is the most common example for using inject. You have an array of numbers and you want the sum of those numbers.

[1, 2, 3, 4].inject(0) { |result, element| result + element } # => 10

If the example isn't straightforward, don't worry, we're going to break it down. The inject method takes an argument and a block. The block will be executed once for each element contained in the object that inject was called on ([1,2,3,4] in our example). The argument passed to inject will be yielded as the first argument to the block, the first time it's executed. The second argument yielded to the block will be the first element of the object that we called inject on.

So, the block will be executed 4 times, once for every element of our array ([1,2,3,4]). The first time the block executes the result argument will have a value of 0 (the value we passed as an argument to inject) and the element argument will have a value of 1 (the first element in our array).

You can do anything you want within the block, but the return value of the block is very important. The return value of the block will be yielded as the result argument the next time the block is executed.

In our example we add the result, 0, to the element, 1. Therefore, the return value of the block will be 0 + 1, or 1. This will result in 1 being yielded as the result argument the second time the block is executed.

The second time the block is executed the result of the previous block execution, 1, will be yielded as the result, and the second element of the array will be yielded as the element. Again the result, 1, and the element, 2 will be added together, resulting in the return value of the block being 3.

The third time the block is executed the result of the second block execution, 3, is yielded as the result argument and the third element of the array, 3, will be yielded as the element argument. Again, the result and the element will be added, and the return value of the block for the third execution will be 6.

The fourth time will be the final time the block is executed since there are only 4 elements in our array. The result value will be 6, the result from the third execution of the block, and the element will be 4, the fourth element of the array. The block will execute, adding four plus six, and the return value of the block will be 10. On the final execution of the block the return value is used as the return value of the inject method; therefore, as the example shows, the result of executing the code above is 10.

That's the very long version of how inject works, but you could actually shortcut one of the block executions by not passing an argument to inject.

[1, 2, 3, 4].inject { |result, element| result + element } # => 10

As the example shows, the argument to inject is actually optional. If a default value is not passed in as an argument the first time the block executes the first argument (result from our example) will be set to the first element of the enumerable (1 from our example) and the second argument (element from our example) will be set to the second element of the enumerable (2 from our example).

In this case the block will only need to be executed 3 times, since the first execution will yield both the first and the second element. The first time the block executes it will add the result, 1, to the element, 2, and return a value of 3. The second time the block executes the result will be 3 and the element will also be 3. All additional steps will be the same, and the result will be 10 once again.

Summing numbers with inject is a simple example of taking an array of numbers and building a resulting sum one element at a time.

Example 2: Building a Hash
Sometimes you'll have data in one format, but you really want it in another. For example, you may have an array that contains keys and values as pairs, but it's really just an array of arrays. In that case, inject is a nice solution for quickly converting your array of arrays into a hash.

hash = [[:first_name, 'Shane'], [:last_name, 'Harvie']].inject({}) do |result, element|
  result[element.first] = element.last
  result
end

hash # => {:first_name=>"Shane", :last_name=>"Harvie"}

As the example shows, I start with an empty hash (the argument to inject) and I iterate through each element in the array adding the key and value one at a time to the result. Also, since the result of the block is the next yielded result, I need to add to the hash, but explicitly return the result on the following line.

Ola Bini and rubikitch both pointed out that you can also create a hash from an array with the following code.

Hash[*[[:first_name, 'Shane'], [:last_name, 'Harvie']].flatten] # => {:first_name=>"Shane", :last_name=>"Harvie"}

Of course, I can do other things in inject also, such as converting the keys to be strings and changing the names to be lowercase.

hash = [[:first_name, 'Shane'], [:last_name, 'Harvie']].inject({}) do |result, element|
  result[element.first.to_s] = element.last.downcase
  result
end

hash # => {"first_name"=>"shane", "last_name"=>"harvie"}

This is a central value for inject, it allows me to easily convert an enumerable into an object that is useful for the problem I'm trying to solve.

Example 3: Building an Array
Enumerable gives you many methods you need for manipulating arrays. For example, if want all the integers of an array, that are even, as strings, you can do so chaining various methods from Enumerable.

[1, 2, 3, 4, 5, 6].select { |element| element % 2 == 0 }.collect { |element| element.to_s } # => ["2", "4", "6"]

Chaining methods of Enumerable is a solution that's very comfortable for many developers, but as the chain gets longer I prefer to use inject. The inject method allows me to handle everything I need without having to chain multiple independent methods.

The code below achieves the same thing in one method, and is just as readable, to me.

array = [1, 2, 3, 4, 5, 6].inject([]) do |result, element|
  result << element.to_s if element % 2 == 0
  result
end

array # => ["2", "4", "6"]

Of course, that example is a bit contrived; however, a realistic example is when you have an object with two different properties and you want to build an array of one, conditionally based on the other. A more concrete example is an array of test result objects that know if they've failed or succeeded and they have a failure message if they've failed. For reporting, you want all the failure messages.

You can get this with the built in methods of Enumerable.

TestResult = Struct.new(:status, :message)
results = [
  TestResult.new(:failed, "1 expected but was 2"),
  TestResult.new(:sucess),
  TestResult.new(:failed, "10 expected but was 20")
]

messages = results.select { |test_result| test_result.status == :failed }.collect { |test_result| test_result.message }
messages # => ["1 expected but was 2", "10 expected but was 20"]

But, it's not obvious what you are doing until you read the entire line. You could build the array the same way using inject and if you are comfortable with inject it reads slightly cleaner.

TestResult = Struct.new(:status, :message)
results = [
  TestResult.new(:failed, "1 expected but was 2"),
  TestResult.new(:sucess),
  TestResult.new(:failed, "10 expected but was 20")
]

messages = results.inject([]) do |messages, test_result|
  messages << test_result.message if test_result.status == :failed
  messages
end
messages # => ["1 expected but was 2", "10 expected but was 20"]

I prefer to build what I want using inject instead of chaining methods of Enumerable and effectively building multiple objects on the way to what I need.

Example 4: Building a Hash (again)
Building from the Test Result example you might want to group all results by their status. The inject method lets you easily do this by starting with an empty hash and defaulting each key value to an empty array, which is then appended to with each element that has the same status.

TestResult = Struct.new(:status, :message)
results = [
  TestResult.new(:failed, "1 expected but was 2"),
  TestResult.new(:sucess),
  TestResult.new(:failed, "10 expected but was 20")
]

grouped_results = results.inject({}) do |grouped, test_result|
  grouped[test_result.status] = [] if grouped[test_result.status].nil?
  grouped[test_result.status] << test_result
  grouped
end

grouped_results
# >> {:failed => [
# >>    #<struct TestResult status=:failed, message="1 expected but was 2">, 
# >>    #<struct TestResult status=:failed, message="10 expected but was 20">],
# >>  :sucess => [ #<struct TestResult status=:sucess, message=nil> ]
# >> }

You might be sensing a theme here.

Example 5: Building a unknown result
Usually you know what kind of result you are looking for when you use inject, but it's also possible to use inject to build an unknown object.

Consider the following Recorder class that saves any messages you send it.

class Recorder
  instance_methods.each do |meth|
    undef_method meth unless meth =~ /^(__|inspect|to_str)/
  end

  def method_missing(sym, *args)
    messages << [sym, args]
    self
  end

  def messages
    @messages ||= []
  end
end

Recorder.new.will.record.anything.you.want
# >> #<Recorder:0x28ed8 @messages=[[:will, []], [:record, []], [:anything, []], [:you, []], [:want, []]]>

By simply defining the play_for method and using inject you can replay each message on the argument and get back anything depending on how the argument responds to the recorded methods.

class Recorder
  def play_for(obj)
    messages.inject(obj) do |result, message|
      result.send message.first, *message.last
    end
  end
end

recorder = Recorder.new
recorder.methods.sort
recorder.play_for(String)

# >> ["<", "<=", "<=>", "==", "===", "=~", ">", ">=", "__id__", ...]

Conclusion
How do I know when I want to use inject? I like to use inject anytime I am building an object a piece at a time. In the case of summing, creating a hash, or an array I'm building a result by applying changes based on the elements of the enumerable. After I'm done applying changes for each element, I have the finished object I'm looking for. The same is true of the Recorder example, I'm sending the methods one at a time until I return the result of sending all the methods.

So next time you need to build an object based on elements of an enumerable, consider using inject.