Rails: Autoloading Gems in Vendor

It's no secret that I'm a big fan of RubyGems. Due to this bias, I tend to package up all my reusable code as RubyGems. Then, when I'm developing a new Rails application I gem unpack [gem] each of the gems I'm using in the vendor directory. This works out well, but I am stuck maintaining the require [path/to/gem] statements that are necessary for loading the libraries.

To resolve this maintenance pain I put the following code in environment.rb.

Dir[File.dirname(__FILE__) + "/../vendor/*"].each do |path|
  gem_name = File.basename(path.gsub(/-\d+.\d+.\d+$/, ''))
  gem_path = path + "/lib/" + gem_name + ".rb"
  require gem_path if File.exists? gem_path
end

The above script relies on a few RubyGems conventions: 3 digit versioning and a file in lib that requires all relevant ruby files for the library. I'm comfortable with these assumptions; however, not all gems follow them. For example, postgres-pr uses a ruby file named postres.rb to load the library. For situations like these, I still need to do an explicit require.

BNL: DRY code, DAMP BNL - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

DRY code, DAMP Business Natural Languages

Don't Repeat Yourself (DRY) is a best practice in software development. Specifying behavior once and only once is the essence of the DRY rule. When you follow this rule the maintainability of the software is increases. The rule helps ensure that if you are required to make a change you can be sure that you will only need to change the software in one location. The benefits of this rule should be quite obvious; however, does it also apply to Business Natural Languages?

Based on my experience I believe that the DRY rule does not apply to Business Natural Languages. A major reason for using a Business Natural Language is to separate the business logic from the complexities of the under-lying system. When using a Business Natural Language, business users who are the most familiar with the domain can maintain the business logic. To a business user, a Business Natural Language should be no different than a group of phrases that describe the rules for running the business correctly.

A well-designed Business Natural Language will appear as Descriptive And Meaningful Phrases (DAMP).

In 'The Solution' I used the following business logic to calculate Jackie Johnson's bonus:

employee Jackie Johnson
compensate $3000 for each deal closed in the past 30 days
compensate $800 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 5% of gross profits if gross profits are greater than $2,000,000

By creating shorter phrases you could rewrite the logic to be more concise:

employee Jackie Johnson
compensate $3000 each past 30
compensate $800 each closed 365
compensate 5% of profits if profits greater $1,000,000
compensate 5% of profits if profits greater $2,000,000

The new logic can be considered more DRY; however, much of the readability has been lost at the expense of removing the some of the duplication. You should not trade readability for conciseness when designing Business Natural Languages.

As businesses change their software requires change. If software can be understood and altered by subject matter experts the business can be more responsive to change. Business Natural Languages facilitate empowering subject matter experts by using a meaningful syntax to express business logic. Creating meaningful syntax often requires defining verbose phrases; however, they increase the overall readability of the phrase. Therefore, the verbose phrases carry maintainability value for the subject matter experts. If the verbose phrases are removed the logic becomes less maintainable.

It is also important to note that employees often change roles within an organization. Because staffing does change, you should not design a Business Natural Language that sacrifices readability for conciseness simply because the current subject matter expert is able to comprehend the concise version. Instead, you should design the Business Natural Language to be easily understood by any subject matter expert placed in that role.

Some programmers object to verbose phrases because they are not necessary for program execution. This is a fairly common thought because programmers are still thinking in terms of 'what will make the system work' instead of trying to solve a more valuable question: How can I empower a subject matter expert to make the system work? When you begin to think in these terms you see that verbose phrases are not strictly required; however, it's highly important to the subject matter expert who will need to continually understand and update the system.

The main goal of a DAMP Business Natural Language is empowering subject matter experts by providing the most maintainable syntax available. A Business Natural Language clearly has room for improvement in the syntax if it requires training to understand. An ideal Business Natural Language contains phrases that are descriptive and meaningful enough that they require no explanation at all.

Rails: Loading mock data

The last two Ruby/Rails projects that I've been involved with had both data that the application created and data that was expected to be there. In production, the data that was expected to be there was data created by an external batch process that ran monthly.

This all works very well; however, it does present a challenge when developing locally: How do we load data that mocks the expected external data? Both projects approached the problem differently, and both approaches provided pros and cons.

Approach 1: Conditionally load the mock data using migrations. On the first project we used PostgreSQL for local development and Oracle in production. Since we used different databases we were able to write our migrations to only load mock data when the database being loaded was a PostgreSQL database. A pro to this approach is that running all the migrations completely sets up the database, local or production. A con to this approach is that more migrations are necessary and some of the migrations lose maintainability based on conditional noise.

Approach 2: Create a rake task to load data that your application does not own. PragDave inspired this approach with his blog entry on how to use Migrations Outside Rails. Using ActiveRecord::Schema.define it's possible to easily create the mock tables.

ActiveRecord::Schema.define do
  create_table books, :force => true do |t|
    t.column :title, :integer
    t.column :pages, :string
    t.column :classification, :integer
  end
end

Following table creation, you can load the mock tables by creating models and saving.

class Book < ActiveRecord::Base
end

Book.create(:title=>'RubyFoo', :pages=>300, :classification=>'Technical')

A pro of this implementation is the clear separation between the mock data and the data necessary for running your application. This also helps reduce the risk of loading invalid data in production. A con is that an extra step is required when first setting up or resetting a database. This con can be partially mitigated by creating a task that runs both the migrations and loads the mock data; however, this introduces more code to the codebase and requires developers to remember one more rake task.

BNL: The Solution - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

The Solution
A Business Natural Language is used to specify application behavior. The language should be comprised of descriptive and maintainable phrases. The structure of the language should be simple, yet verbose. Imagine each line of the specification as one complete sentence. Because a Business Natural Language is natural language it is important to limit the scope of the problem being solved. An application's requirements can often be split to categories that specify similar functionality. Each category can be a candidate for a simple Business Specific Language. Lucky for us, our current application has a very limited scope: calculating compensation. Therefore, our application will only require one Business Natural Language.

We've already seen the requirements:

employee John Jones
compensate $2500 for each deal closed in the past 30 days
compensate $500 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 3% of gross profits if gross profits are greater than $2,000,000
compensate 1% of gross profits if gross profits are greater than $3,000,000

A primary driver for using a Business Natural Language is the ability to execute the requirements as code. Lets change the existing application to work with a Business Natural Language.

To start with we can take the requirements and store them in a file named jjones.bnl. Now that we have our Business Natural Language file in our application we'll need to alter the existing code to read it. The first step is changing process_payroll.rb to search for all Business Natural Language files, create a vocabulary, create a parse tree, and report the results.

File: process_payroll.rb

Dir[File.dirname(__FILE__) + "/*.bnl"].each do |bnl_file|
  vocabulary = CompensationVocabulary.new(File.basename(bnl_file, '.bnl'))
  compensation = CompensationParser.parse(File.read(bnl_file), vocabulary)
  puts "#{compensation.name} compensation: #{compensation.amount}"
end

Creating a vocabulary is nothing more than defining the phrases that the Business Natural Language should understand. Each phrase returns values that are appended together to create valid ruby syntax. The vocabulary also calls out to the SalesInfo class to return sales data for each employee.

File: compensation_vocabulary.rb

class CompensationVocabulary
  extend Vocabulary
  
  def initialize(data_for)
    @data_for = data_for
  end
  
  phrase "active deal that closed more than 365 days ago!" do 
    SalesInfo.send(@data_for).year_old_deals.to_s
  end
  
  phrase "are greater than" do
    " > "
  end
  
  phrase "deal closed in the past 30 days!" do 
    SalesInfo.send(@data_for).deals_this_month.to_s
  end
  
  phrase "for each"  do 
    "*" 
  end
  
  phrase "gross profits" do 
    SalesInfo.send(@data_for).gross_profit.to_s 
  end
  
  phrase "if" do 
    " if " 
  end
  
  phrase "of" do 
    "*" 
  end
  
end

The CompensationVocabulary class does extend Vocabulary, which is how the phrase class method is added to CompensationVocabulary.

File: vocabulary.rb

module Vocabulary

  def phrase(name, &block)
    define_method :"_#{name.to_s.gsub(" ","_")}", block
  end

end

After the vocabulary is defined the CompensationParser class processes the BNL file.

File: compensation_parser.rb

class CompensationParser

  class << self
    def parse(script, vocabulary)
      root = Root.new(vocabulary)
      script.split(/\n/).each { |line| root.process(preprocess(line)) }
      root
    end
    
    def preprocess(line)
      line.delete!('$,')
      line.gsub!(/(\d+)%/, '\1percent')
      line.gsub!(/\s/, '._')
      "_#{line.downcase}!"
    end
  end

end

The CompensationParser class is responsible for parsing the script and converting the BNL syntax into valid ruby. The parse class method of CompensationParser creates a new instance of Root, splits the script, sends the preprocessed line to the process method of the instance of Root, and then returns the instance of Root. The CompensationParser preprocess method removes the superfluous characters, replaces special characters with alphabetic representations, converts the line into a chain of methods being called on the result of the previous method call, and appends underscores and an exclamation point to avoid method collisions and signal the end of a phrase.

For example, when preprocess is called on:
"compensate 5% of gross profits if gross profits are greater than $1,000,000"
it becomes:
"_compensate._5percent._of._gross._profits._if._gross._profits._are._greater._than._1000000!"

The Root class is responsible for processing each line of the Business Natural Language.

File: root.rb

class Root
  extend Vocabulary

  def initialize(vocabulary)
    @compensations = []
    @vocabulary = vocabulary
  end
  
  def name
    @employee.name
  end
  
  def amount
    @compensations.collect do |compensation| 
      compensation.amount 
    end.inject { |x, y| x + y }
  end
  
  def process(line)
    instance_eval(line)
  end
  
  phrase :employee do
    @employee = Employee.new
  end
  
  phrase :compensate do
    @compensations << Compensation.new(@vocabulary)
    @compensations.last
  end

end

Root is responsible for storing reference to the child objects of the parse tree. The references are maintained via instance variables that are initialized when the methods that correspond to phrases of the language are processed. For example, the 'employee' phrase creates and stores an instance of the Employee class. Root processes each line of the Business Natural Language by passing the line to the instance_eval method.

When the first line is sent to instance_eval, 'employee' returns an instance of the Employee class. The Employee class instance stores each subsequent message in an array and always returns self.

File: employee.rb

class Employee

  def initialize
    @name_parts = []
  end
  
  def method_missing(sym,*args)
    @name_parts << sym.to_s.delete('_!')
    self
  end
  
  def name
    @name_parts.collect { |part| part.to_s.capitalize }.join(' ')
  end

end

When the second and each following line are sent to instance_eval a Compensation instance is created. The primary task of the instance of Compensation is to build phrases and reduce them to ruby code when possible. Each message sent to compensation from the instance_eval is handled by method_missing. The method_missing method attempts to reduced the phrase instance variable each time method_missing is executed. A phrase can be reduced if the entire phrase is a number or if the vocabulary contains the phrase. If a phrase can be reduced the result is appended to the compensation_logic instance variable and the phrase is reset to empty. If a phrase cannot be reduced it is simply stored awaiting the next call to method_missing. If the phrase contains the end of phrase delimiter (the exclamation point) and it cannot be reduced an exception is thrown.

File: compensation.rb

class Compensation

  def initialize(vocabulary)
    @phrase, @compensation_logic = '', ''
    @vocabulary = vocabulary
  end
  
  def method_missing(sym, *args)
    @phrase = reduce(@phrase + sym.to_s)
    if @phrase.any? && sym.to_s =~ /!$/
      raise NoMethodError.new("#{@phrase} not found")
    end
    self
  end
  
  def reduce(phrase)
    case
      when phrase =~ /^_\d+[(percent)|!]*$/
        append(extract_number(phrase))
      when @vocabulary.respond_to?(phrase)
        append(@vocabulary.send(phrase))
      else phrase
    end
  end
  
  def append(piece)
    @compensation_logic += piece
    ""
  end
  
  def extract_number(string)
    string.gsub(/(\d+)percent$/, '0.0\1').delete('_!')
  end
  
  def amount
    instance_eval(@compensation_logic) || 0
  end

end

After making all of the above changes a quick run of payroll_process.rb produces the following output:

Jackie Johnson compensation: 256800.0
John Jones compensation: 88500.0

We still have a lot of work to do, but we've proven our concept. Our application behavior is entirely dependent on the compensation specifications that can be altered by our subject matter experts.

In the upcoming chapters we will discuss moving our application to the web, putting the specifications in a database instead of using flat files, providing syntax checking and contextual grammar recommendations, and many other concepts that will take us through developing a realistic Business Natural Language application.

Ruby Regular Expression Replace

Ruby provides Regular Expression replacement via the gsub method of String. For example, if I would like to replace a word I simply need the regex to match the word and the replacement text.

irb(main):006:0> "replacing in regex".gsub /\sin\s/, ' is easy in '
=> "replacing is easy in regex"

The gsub method also lets you use the regex match in the replacement.

irb(main):007:0> "replacing in regex".gsub /\sin\s/, ' is easy\0'
=> "replacing is easy in regex"

If you want part of the regex, but not the whole thing you will need to use a capturing group (parenthesis).

irb(main):008:0> "replacing in regex".gsub /\si(n)\s/, ' is easy i\1 '
=> "replacing is easy in regex"

When using a capturing group that matches multiple times in a single line you can still use \1 to include the match in the result.

irb(main):015:0> "1%, 10%, 100%".gsub /(\d+)%/, '0.0\1'
=> "0.01, 0.010, 0.0100"

Another important detail to note is that I'm using single quotes in my replacement string. Using double quotes neither works with \0 or \1 since "\0" #=> "\000".

irb(main):009:0> "replacing in regex".gsub /\sin\s/, " is easy\0"
=> "replacing is easy\000regex"
irb(main):010:0> "replacing in regex".gsub /\si(n)\s/, " is easy i\1 "
=> "replacing is easy i\001 regex"

Rake: Tasks with parameters

I've written several Rake tasks in the past that could have been improved by accepting parameters, but I never knew how to make it work. It's not that I thought it was impossible to pass parameters to Rake tasks, I just never bothered to figure out how. Lucky for me, a teammate recently took the time to figure it out. Today I'm changing an existing rakefile to take advantage of passing in parameters.

The file now:

require 'writer'
require 'html_writer'
require 'doc_writer'

directory "built_docs"
directory "built_html"

task :doc => [:built_docs, :clean_doc] do
  Dir['chapters/*.txt'].each do |chapter|
    output = "built_docs/#{File.basename(chapter,'txt')}doc"
    File.open(output, 'w') { |file| DocWriter.new(file).write(chapter) }
    puts "generated #{output}"
    system "chmod 444 #{output}"
  end
end

task :html => [:built_html, :clean_html] do
  Dir['chapters/*.txt'].each do |chapter|
    output = "built_html/#{File.basename(chapter,'txt')}html"
    File.open(output, 'w') { |file| HtmlWriter.new(file).write(chapter) }
    puts "generated #{output}"
    system "chmod 444 #{output}"
  end
end

task :clean_doc do
  Dir['built_docs/*'].each do |output|
    rm output
  end
end

task :clean_html do
  Dir['built_html/*'].each do |output|
    rm output
  end
end

Clearly, this file is not near as DRY as it could be.

The first step is to create a new task that will generate output and accept the format as an argument.

task :default do
  unless ENV.include?("output") && (ENV['output']=='doc' || ENV['output']=='html')
    raise "usage: rake output= # valid formats are [html] or [doc]" 
  end
  format = ENV['output']
  puts format
  rm_rf format
  mkdir format
end

The above code verifies that a valid format is specified. It also removes the need for both the directory and clean tasks. The only work left to do is combine the output generation tasks.

task :default do
  unless ENV.include?("output") && (ENV['output']=='doc' || ENV['output']=='html')
    raise "usage: rake output= # valid formats are [html] or [doc]" 
  end
  format = ENV['output']
  puts format
  rm_rf format
  mkdir format
  Dir['chapters/*.txt'].each do |chapter|
    output = "#{format}/#{File.basename(chapter,'txt')}#{format}"
    writer = (format=='doc' ? DocWriter : HtmlWriter)
    File.open(output, 'w') { |file| writer.new(file).write(chapter) }
    puts "generated #{output}"
    system "chmod 444 #{output}"
  end
end

The resulting task is cleaner and easier to maintain.

Running the new task from the command line: focus:~ jay$ rake output=html

BNL: The Problem - Update

The concept behind this 'chapter' is basically the same as the last time I put it on my blog; however, the example is largely new. As always, feedback welcome.

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

The Problem
To demonstrate the value in implementing a Business Natural Language we will start with an existing ruby application and then introduce a simple Business Natural Language. This approach should provide a good example of how to include a Business Natural Language in an application.

For the sample application imagine you've been contracted to replace a payroll calculation system for a consulting company. The consulting company chose to custom develop the software for calculating payroll because of the varying compensation packages. They have given you profiles of 2 employees and the business rules required to generate compensation.

Profiles:
John Jones
compensate $2500 for each deal closed in the past 30 days
compensate $500 for each active deal that closed more than 365 days ago
compensate 5 percent of gross profits if gross profits are greater than $1,000,000
compensate 3 percent of gross profits if gross profits are greater than $2,000,000
compensate 1 percent of gross profits if gross profits are greater than $3,000,000

Jackie Johnson
compensate $3000 for each deal closed in the past 30 days
compensate $800 for each active deal that closed more than 365 days ago
compensate 5 percent of gross profits if gross profits are greater than $1,000,000
compensate 5 percent of gross profits if gross profits are greater than $2,000,000

The application need only be run monthly; therefore, manual execution of the application is acceptable. The existing application is executed at the command line and the results are displayed to the screen. The payroll department records the application results in another system that is outside of the scope of your work. A preprocess is run to put monthly sales information in a known location in an agreed upon format, this is also outside the scope of your work. The only input files we will need for our sample application are jjones.yml and jjohnson.yml.

File: jjones.yml

employee: jjones
deals_this_month: 7
year_old_deals: 2
gross_profit: 1400000

File: jjohnson.yml

employee: jjohnson
deals_this_month: 12
year_old_deals: 1
gross_profit: 2200000

The SalesInfo class currently reads the sales data and exposes it via class method calls that are handled by method_missing.

File: sales_info.rb

require 'yaml'

class SalesInfo

  @employee_sales_info = Dir['*.yml'].collect do |filename|
    YAML::load(File.open(filename))
  end
  
  def self.method_missing(sym, *args)
    hash = @employee_sales_info.find { |hash| hash.value?(sym.to_s) }
    super if hash.nil?
    EmployeeInfo.new(hash)
  end

  class EmployeeInfo
    attr_reader :employee, :deals_this_month, :year_old_deals, :gross_profit
    def initialize(hash)
      hash.each_pair do |key, value|
        instance_variable_set :"@#{key}", value
      end
    end
  end

end

The ruby file that executes the current payroll process is very straightforward:

File: process_payroll.rb

require 'jjones'
require 'jjohnson'

puts "John Jones compensation: #{JJones.calculate}"
puts "Jackie Johnson compensation: #{JJohnson.calculate}"

Each employee ruby file is also fairly straightforward. They contain the logic to calculate compensation, expressed as ruby.

File: jjones.rb

require 'sales_info'

class JJones
  def self.calculate
    total = SalesInfo.jjones.deals_this_month * 2500
    total += SalesInfo.jjones.year_old_deals * 500
    profit = SalesInfo.jjones.gross_profit
    total += profit * 0.05 if profit > 1000000
    total += profit * 0.03 if profit > 2000000
    total += profit * 0.01 if profit > 3000000
    total
  end
end

File: jjohnson.rb

require 'sales_info'

class JJohnson
  def self.calculate
    total = SalesInfo.jjohnson.deals_this_month * 3000
    total += SalesInfo.jjohnson.year_old_deals * 800
    profit = SalesInfo.jjohnson.gross_profit
    total += profit * 0.05 if profit > 1000000
    total += profit * 0.05 if profit > 2000000
    total
  end
end

The existing application produces the following output.

John Jones compensation: 88500.0
Jackie Johnson compensation: 256800.0

The code produces accurate results; however, the code has limited ability to easily be extended for new employees. Additionally, employees are required to re-negotiate their compensation structure every year. Following a compensation change a programmer must be involved to change the employees logic. The current process is clearly sub-optimal

BNL: Introduction - Update

I'm wrapping up another project these days and I'm revisiting my previous Business Natural Language information. The project provided many more lessons learned that I'm eager to add to the existing material. The first 'chapter' remains much the same; however, I made a few small changes. I'm also hoping to generate some more feedback this time around.

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

Introduction
Why use a Business Natural Language?

Since the introduction of computers to the general workforce businesses have searched for a solution that will enable subject matter experts to specify the business logic of an application. This solution is highly sought after since it will allow the application to be changed without the assistance of a programmer. Programmers are still required to create the application; however, the application is written in a way that empowers the subject matter experts to maintain the business logic. Enabling the subject matter expert greatly increases efficiency of maintaining an application as the needs of the business change.

Using a Domain Specific Language (DSL) is the most recent solution to this problem. A Domain Specific Language can be defined as a language created to describe a specific set of tasks. A Domain Specific Language is often used to define a specification or business rule without using a general purpose programming language such as Ruby or C#. Domain Specific Languages have been around under various names for many years. In fact, it's rare in today's business world to find a subject matter expert who isn't fluent in the Domain Specific Language in which they define their spreadsheet macros.

Unfortunately, there is a problem with the term Domain Specific Language. Programmers commonly know YACC as a DSL. Spreadsheet macros are widely used within the business community. Recently Rake and Ruby on Rails have gained popularity as DSLs. Some even consider Cobol to be a DSL. Thus the problem, you can make a case for almost anything to be considered a DSL.

Is that a really a problem? Yes, it turns out it is a problem. When we are unable to be specific about what type of Domain Specific Language we are using we are forced to eliminate types of DSLs that are not relevant to the current conversation. This elimination process often slows discussions down. Or worse, people dismiss the conversations because they don't believe that DSLs can work in the way described. This, unfortunately, is fairly common because of DSLs of the past that have failed.

Because of the previous reason I think it's clear that there is a need to classify the different types of Domain Specific Languages. I believe using specific classifications will allow conversations to become more productive. The primary focus of this book is enabling subject matter experts to specify the business logic in a natural language. For several months I used the term Business Domain Specific Language to describe this type of work; unfortunately, the business prefix was generally lost in conversation and the same issues emerged. Because the prefix was not effective at solving the problem I chose to coin the term Business Natural Language (BNL).

A Business Natural Language is a Domain Specific Language; however, not all Domain Specific Languages are Business Natural Languages. Business Natural Languages use natural language to represent business logic. Business Natural Languages are expressed as descriptive and maintainable phrases.
For example, a marketing executive for an airline could specify point award descriptions as:
award 1 point for each mile flown where the total flight length is greater than 500 miles
award 500 points for each flight where the total flight length is less than or equal to 500 miles

A Business Natural Language is a specific type of Domain Specific Language that will not contain unnecessary commas, semi-colons or any other constraint imposed by a general-purpose programming language. Additionally, a general-purpose language should not influence the structure of a Business Natural Language phrase. Any domain expert, with no explanation required, can read a well-written Business Natural Language as if it were simply a phrase specifying logic.

The previous airline example appears to be a specification written by a business analyst. The specification could be used to describe business logic to be implemented in a general-purpose language. However, when using a Business Natural Language the above example is a specification, but also much more. The above example is executable code, which will be used to determine point allocation after each flight flown. The above example is also documentation of the business rules contained in the point allocation application. And, the above example can be used to formulate a test case to verify the system works as expected.

Business Natural Languages allow you to specify, in one location, exactly how your application should work. By reducing duplication the maintainability of the system is greatly increased. When using a Business Natural Language the lines are not blurred between the specification and code, there are no lines. The specification is the code.

When should you use a Business Natural Language?

Business Natural Languages are ideal for systems where the behavior often changes. Generally the value of these types of systems can be directly linked to their ability to be quickly changed. In most traditional applications the business rules are specified in a general-purpose language. Changes to this style of application generally require a programmer to understand and alter the existing code base. However, a superior approach allows the behavior of the system to be specified by a subject matter expert. A Business Natural Language is an obvious choice for this scenario because it removes the programmer from the equation, often greatly reducing the time required to alter the application's behavior.

Frequently changing systems are not the only applications that can benefit from Business Natural Languages. Systems that contain a large number of similar business logic are also ideal candidates. For example, imagine a payroll system that is used to compensate your sales force. In this payroll system each employee has specific variables that determine their compensation for each pay period. Their pay could be altered by variables such as the amount of business sold, profit margin per transaction, years with the company, etc. Each employee could have their own negotiated parameters for determining their pay:

employee John Doe
compensate 500 dollars for each deal closed in the past 30 days
compensate 100 dollars for each active deal that closed more than 365 days ago
compensate 5 percent of gross profits if gross profits are greater than 1,000,000 dollars
compensate 3 percent of gross profits if gross profits are greater than 2,000,000 dollars
compensate 1 percent of gross profits if gross profits are greater than 3,000,000 dollars

When a system utilizes a Business Natural Language it becomes easy to add a new employee into the system and ensure that their compensation will directly match exactly what their offer letter details.

When else is a Business Natural Language appropriate? Business Natural Languages are ideal for any application that contains logic specific to the business. The more business specific logic you can abstract out of an application the less programmer involvement you will need to alter the business logic. Ideally all business specific logic can be contained in one or several Business Natural Languages within your application. By enabling the subject matter experts to maintain the business rules of the application the teams can work together to quickly develop a robust application.

I was recently involved in developing an application that was estimated to take 6-8 months using a traditional development approach. Because of time to market issues this was not a viable plan. The alternative approach, using a Business Natural Language, allowed the general application to be written by the developers while the subject matter experts wrote the business logic. The application, including business logic, was complete in 4 months. During the 4 months the programmers and subject matter experts worked in parallel constantly evolving the application. While the application was being developed the subject matter experts found they needed more flexibility in their Business Natural Language. Because the teams were working in parallel the programmers were able to immediately expand the boundaries of the Business Natural Language to meet the needs of the subject matter experts.

These types of stories can vary in value due to the fact that we will never know how long it would have taken to complete the task in a traditional manner. However, what is measurable is the level of effort that the application requires to change the business logic. For this application the logic of the business can change on a monthly basis. If the rules were written in a language that required a programmer to make the change, each month a change request and prioritization session would be required. The customer for this application stated that this process generally took 2 to 3 days. Following the change, regression testing was necessary to verify that the programmer correctly implemented the change. Compared to what we actually delivered, this process is greatly inefficient. Because we chose to use a Business Natural Language, a change could be made to the business logic directly by the subject matter expert. We also provided a test environment to run the proposed business logic. This allowed the subject matter expert to make a change, execute, and verify without any programmer support. The result, in his words was "A change that used to take 3-4 days now takes about 10 minutes."

Traditional applications waste precious time in various ways. While developing software, much time is spent educating programmers on every minute detail of the business. With this knowledge, programmers are busy developing code in a general-purpose language that mirrors the specifications given by subject matter experts. Following completion of the programming task there are often issues discovered. This cycle repeats until the program executes producing correct results. This common development loop is a time consuming process. Systems that are able to introduce Business Natural Languages avoid this loop entirely by empowering the subject matter experts to create, test, and approve the business logic with little or no assistance.

Clearly, the decision to use a Business Natural Language depends greatly on the ability of the business to provide at least one subject matter expert. At first glance it appears that a Business Natural Language creates more work for subject matter experts; however, it is important to remember that traditional application development requires subject matter experts to write specifications for the application being developed. Business Natural Languages also require the subject matter experts to create specifications, but by following some simple conventions the specifications are used within the application as actual business logic. Furthermore, anyone who has been involved in conversations where subject matter experts explain business logic to programmers has seen how challenging it can be to bring everyone's understanding to the same level. Also, even following the conversation the programmer still only has as much context as the subject matter expert was able to express. This can lead to costly assumptions because the programmer does not posses the full context that a subject matter expert has. In these scenarios a Business Natural Language can reduce the amount of overall work for a subject matter expert.

Using a Business Natural Language can also greatly simplify deployment of new business logic. For example, in the last application I delivered the business logic was stored in a database. The application provided a UI that allowed the subject matter experts to make changes to the business logic. When a subject matter expert was content with their changes they would save their changes back to the database. In this application all changes are required to pass through a workflow before becoming active. The entire workflow process is also done through the application's user interface. Submitted changes require approval from another employee before being included in the application. Once the logic is approved it may be immediately activated through the user interface. Each time a process executes it will execute all activated logic. In this scenario, the workflow is the deployment.

Program execution when using a Business Natural Language

A common concern when using a Business Natural Language is that the logic is executed in an expected way. For example, if you work for a casino and you want to find high rollers you could use these phrases to segment your player population:

include players who bet less than 100 dollars a hand in the unimportant category
include players who bet between 100 and 150 dollars a hand in the valuable category
include remaining players in the high roller category

If the last phrase executes first, all of the casino's guests will be considered high rollers. Obviously, this is a serious problem. To mitigate this issue each phrase should be executable in isolation without dependency on any other phrase. The last phrase could be rewritten as such to conform to this rule:

include players who bet more than 150 dollars a hand in the high roller category.

When statements can be executed in any order the dependency problem is solved; however, this does require that each statement contain all information required to execute the business logic. For this reason it is acceptable and even expected that duplication will exist in several if not all of your descriptive and maintainable phrases.

Along the same lines, it should be clear that you wouldn't chain or link multiple statements together. Also, a Business Natural Language would not contain variables or looping constructs. Again, each Business Natural Language phrase should be valid entirely in isolation.

Because of these limitations it should be clear that a Business Natural Language is not a silver bullet. For example, some companies use rules engines to manage their business logic. While a Business Natural Language could replace the simpler implementations, I do not expect that you would ever attempt to replace a complex implementation with a BNL. The value of a Business Natural Language can be derived almost directly from the narrowness of the scope of the problem being solved.

Tracability is fairly simple due to the the constraints associated with Business Natural Languages. As the amount of logic increases the complexity does increase; however, since each phrase can execute in isolation it is fairly easy to follow behavior as the phrases execute individually. Also, at the heart of it a Business Natural Language is really a facade to an API. Therefore, if necessary, a programmer can step through each line of the Business Natural Language while as it executes.

Ruby Project Tree

I was recently asked: How would you structure a (non-Rails) Ruby project?

Here's a list of the things I generally do when creating a new project:

• Create lib and test folders to house the code and tests respectively.
• Create a rakefile.rb with a task for running all the tests:

  task :default => :test
  
  task :test do
    require File.dirname(__FILE__) + '/test/all_tests.rb'  
  end

• Create a test file that runs all tests (all_tests.rb):

  Dir['**/*_test.rb'].each { |test_case| require test_case }

• Create a test_helper file that handles common require calls:

  require 'test/unit'
  require 'mocha'
  require 'stubba'
  require File.dirname(__FILE__) + '/../lib/[project name]'

• Create a file with the same name as my project in the lib folder. This file contains all the requires necessary to run the application/use the library:

  [example: sqldsl.rb]
  
  lib = File.dirname(__FILE__)
  
  require lib + '/object.rb'
  require lib + '/symbol.rb'
  require lib + '/array.rb'
  require lib + '/string.rb'
  require lib + '/numeric.rb'
  require lib + '/time.rb'
  require lib + '/sql_statement.rb'
  require lib + '/where_builder.rb'
  require lib + '/select.rb'
  require lib + '/insert.rb'
  require lib + '/update.rb'
  require lib + '/delete.rb'

I find this structure makes my life easier.

Of course, if you want this automated, you could always just use tree_surgeon.rb

require 'fileutils'

project_name = ARGV[0]
lib = project_name + "/lib"
test = project_name + "/test"
rakefile = project_name + '/rakefile.rb'
all_tests = test + "/all_tests.rb"
main = project_name + "/lib/" + project_name + ".rb"
test_helper = test + "/test_helper.rb"

FileUtils.rm_rf project_name

puts "creating: " + project_name
Dir.mkdir project_name

puts "creating: " + lib
Dir.mkdir lib

puts "creating: " + test
Dir.mkdir test

puts "creating: " + rakefile
File.open(rakefile, 'w') do |file|
  file << <<-eos
task :default => :test

task :test do
  require File.dirname(__FILE__) + '/test/all_tests.rb'
end
eos
end

puts "creating: " + all_tests
File.open(all_tests, 'w') do |file|
  file << "Dir['**/*_test.rb'].each { |test_case| require test_case }"
end

puts "creating: " + main
FileUtils.touch main

puts "creating: " + test_helper
File.open(test_helper, 'w') do |file|
  file << <<-eos
require 'test/unit'
require File.dirname(__FILE__) + '/../lib/#{project_name}'
eos
end

Also, if you are creating a RubyGem it is recommended that you create README, TODO, and CHANGES files. Additionally, if your gem uses executables it is suggested that you store the executables in a bin folder and create a install.rb file that calls your executables.

Do you have other suggestions?

RubyGems: shared Ruby code

From the RubyGems website:

RubyGems is the premier ruby packaging system. It provides:

• A standard format for distributing Ruby programs and libraries.
• An easy to use tool for managing the installation of gem packages.
• A gem server utility for serving gems from any machine where RubyGems is installed.

RubyGems is currently the mostly widely accepted way of distributing Ruby code. However, there are at least two other options for sharing code: svn:externals, rails plugins

Subversion Externals can be used to share code between multiple projects. Subversion Externals allow common code to be housed in a single repository. Any repositories that wish to share the code may simply create an external definition and link to the source. After a change is committed to the source repository, each check out that has the source repository as an external will receive the change when an svn update is issued. This provides consumers the ability to receive changes in real time. A notable consequence of this option is the risk of unexpected breakages when the external code changes. This can generally be mitigated by specifying a revision; however, this removes the ability to receive any future changes.

Rails Plugins are an additional way to distribute code that is useful for Rails. Rails Plugins are generally designed for shared code that is specific to Ruby on Rails backed applications. The most common complaint I hear is that versioning plugins can be problematic.

In fact, versioning seems to be a common problem with Ruby code distribution.

I'm not sure why RubyGems isn't more widely adopted as a distribution solution. Rake gives you a task to easily create RubyGems. RubyGems comes with a Gem Server that can be used to host gems internally. RubyGems also allows consumers to install gems and optionally specify a version.

Currently, RubyGems works well with Rails applications. To ensure that your application runs against expected code it is recommended that you unpack your gems in the vendor directory of rails.

focus:~ jay$ gem unpack sqldsl

The above code will create a sqldsl-1.0.0 directory in the directory in which the command is executed. After unpacking a gem you will need to add a line in environment.rb that requires the main file of the library. For example, to add the SQL DSL library to my Rails application I would add the following line to my environment.rb.

require "#{File.expand_path(RAILS_ROOT)}/vendor/sqldsl-1.0.0/lib/sqldsl.rb"

After adding the above line of code, the SQL DSL library is available anywhere in my Rails application.

Unpacking Gems in my vendor directory also ensures that my application only runs against code that I control and deploy. This mitigates the risk of my application breaking when gems are updated on my production box.

Give RubyGems a try.