Monday, October 30, 2006

Rails: Autoloading Gems in Vendor

It's no secret that I'm a big fan of RubyGems. Due to this bias, I tend to package up all my reusable code as RubyGems. Then, when I'm developing a new Rails application I gem unpack [gem] each of the gems I'm using in the vendor directory. This works out well, but I am stuck maintaining the require [path/to/gem] statements that are necessary for loading the libraries.

To resolve this maintenance pain I put the following code in environment.rb.
Dir[File.dirname(__FILE__) + "/../vendor/*"].each do |path|
gem_name = File.basename(path.gsub(/-\d+.\d+.\d+$/, ''))
gem_path = path + "/lib/" + gem_name + ".rb"
require gem_path if File.exists? gem_path
end
The above script relies on a few RubyGems conventions: 3 digit versioning and a file in lib that requires all relevant ruby files for the library. I'm comfortable with these assumptions; however, not all gems follow them. For example, postgres-pr uses a ruby file named postres.rb to load the library. For situations like these, I still need to do an explicit require.

Saturday, October 21, 2006

BNL: DRY code, DAMP BNL - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

DRY code, DAMP Business Natural Languages

Don't Repeat Yourself (DRY) is a best practice in software development. Specifying behavior once and only once is the essence of the DRY rule. When you follow this rule the maintainability of the software is increases. The rule helps ensure that if you are required to make a change you can be sure that you will only need to change the software in one location. The benefits of this rule should be quite obvious; however, does it also apply to Business Natural Languages?

Based on my experience I believe that the DRY rule does not apply to Business Natural Languages. A major reason for using a Business Natural Language is to separate the business logic from the complexities of the under-lying system. When using a Business Natural Language, business users who are the most familiar with the domain can maintain the business logic. To a business user, a Business Natural Language should be no different than a group of phrases that describe the rules for running the business correctly.

A well-designed Business Natural Language will appear as Descriptive And Meaningful Phrases (DAMP).

In 'The Solution' I used the following business logic to calculate Jackie Johnson's bonus:

employee Jackie Johnson
compensate $3000 for each deal closed in the past 30 days
compensate $800 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 5% of gross profits if gross profits are greater than $2,000,000

By creating shorter phrases you could rewrite the logic to be more concise:

employee Jackie Johnson
compensate $3000 each past 30
compensate $800 each closed 365
compensate 5% of profits if profits greater $1,000,000
compensate 5% of profits if profits greater $2,000,000

The new logic can be considered more DRY; however, much of the readability has been lost at the expense of removing the some of the duplication. You should not trade readability for conciseness when designing Business Natural Languages.

As businesses change their software requires change. If software can be understood and altered by subject matter experts the business can be more responsive to change. Business Natural Languages facilitate empowering subject matter experts by using a meaningful syntax to express business logic. Creating meaningful syntax often requires defining verbose phrases; however, they increase the overall readability of the phrase. Therefore, the verbose phrases carry maintainability value for the subject matter experts. If the verbose phrases are removed the logic becomes less maintainable.

It is also important to note that employees often change roles within an organization. Because staffing does change, you should not design a Business Natural Language that sacrifices readability for conciseness simply because the current subject matter expert is able to comprehend the concise version. Instead, you should design the Business Natural Language to be easily understood by any subject matter expert placed in that role.

Some programmers object to verbose phrases because they are not necessary for program execution. This is a fairly common thought because programmers are still thinking in terms of 'what will make the system work' instead of trying to solve a more valuable question: How can I empower a subject matter expert to make the system work? When you begin to think in these terms you see that verbose phrases are not strictly required; however, it's highly important to the subject matter expert who will need to continually understand and update the system.

The main goal of a DAMP Business Natural Language is empowering subject matter experts by providing the most maintainable syntax available. A Business Natural Language clearly has room for improvement in the syntax if it requires training to understand. An ideal Business Natural Language contains phrases that are descriptive and meaningful enough that they require no explanation at all.

Rails: Loading mock data

The last two Ruby/Rails projects that I've been involved with had both data that the application created and data that was expected to be there. In production, the data that was expected to be there was data created by an external batch process that ran monthly.

This all works very well; however, it does present a challenge when developing locally: How do we load data that mocks the expected external data? Both projects approached the problem differently, and both approaches provided pros and cons.

Approach 1: Conditionally load the mock data using migrations. On the first project we used PostgreSQL for local development and Oracle in production. Since we used different databases we were able to write our migrations to only load mock data when the database being loaded was a PostgreSQL database. A pro to this approach is that running all the migrations completely sets up the database, local or production. A con to this approach is that more migrations are necessary and some of the migrations lose maintainability based on conditional noise.

Approach 2: Create a rake task to load data that your application does not own. PragDave inspired this approach with his blog entry on how to use Migrations Outside Rails. Using ActiveRecord::Schema.define it's possible to easily create the mock tables.
ActiveRecord::Schema.define do
create_table books, :force => true do |t|
t.column :title, :integer
t.column :pages, :string
t.column :classification, :integer
end
end
Following table creation, you can load the mock tables by creating models and saving.
class Book < ActiveRecord::Base
end

Book.create(:title=>'RubyFoo', :pages=>300, :classification=>'Technical')
A pro of this implementation is the clear separation between the mock data and the data necessary for running your application. This also helps reduce the risk of loading invalid data in production. A con is that an extra step is required when first setting up or resetting a database. This con can be partially mitigated by creating a task that runs both the migrations and loads the mock data; however, this introduces more code to the codebase and requires developers to remember one more rake task.

Saturday, October 14, 2006

BNL: The Solution - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

The Solution
A Business Natural Language is used to specify application behavior. The language should be comprised of descriptive and maintainable phrases. The structure of the language should be simple, yet verbose. Imagine each line of the specification as one complete sentence. Because a Business Natural Language is natural language it is important to limit the scope of the problem being solved. An application's requirements can often be split to categories that specify similar functionality. Each category can be a candidate for a simple Business Specific Language. Lucky for us, our current application has a very limited scope: calculating compensation. Therefore, our application will only require one Business Natural Language.

We've already seen the requirements:

employee John Jones
compensate $2500 for each deal closed in the past 30 days
compensate $500 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 3% of gross profits if gross profits are greater than $2,000,000
compensate 1% of gross profits if gross profits are greater than $3,000,000

A primary driver for using a Business Natural Language is the ability to execute the requirements as code. Lets change the existing application to work with a Business Natural Language.

To start with we can take the requirements and store them in a file named jjones.bnl. Now that we have our Business Natural Language file in our application we'll need to alter the existing code to read it. The first step is changing process_payroll.rb to search for all Business Natural Language files, create a vocabulary, create a parse tree, and report the results.

File: process_payroll.rb
Dir[File.dirname(__FILE__) + "/*.bnl"].each do |bnl_file|
vocabulary = CompensationVocabulary.new(File.basename(bnl_file, '.bnl'))
compensation = CompensationParser.parse(File.read(bnl_file), vocabulary)
puts "#{compensation.name} compensation: #{compensation.amount}"
end
Creating a vocabulary is nothing more than defining the phrases that the Business Natural Language should understand. Each phrase returns values that are appended together to create valid ruby syntax. The vocabulary also calls out to the SalesInfo class to return sales data for each employee.

File: compensation_vocabulary.rb
class CompensationVocabulary
extend Vocabulary

def initialize(data_for)
@data_for = data_for
end

phrase "active deal that closed more than 365 days ago!" do
SalesInfo.send(@data_for).year_old_deals.to_s
end

phrase "are greater than" do
" > "
end

phrase "deal closed in the past 30 days!" do
SalesInfo.send(@data_for).deals_this_month.to_s
end

phrase "for each" do
"*"
end

phrase "gross profits" do
SalesInfo.send(@data_for).gross_profit.to_s
end

phrase "if" do
" if "
end

phrase "of" do
"*"
end

end
The CompensationVocabulary class does extend Vocabulary, which is how the phrase class method is added to CompensationVocabulary.

File: vocabulary.rb
module Vocabulary

def phrase(name, &block)
define_method :"_#{name.to_s.gsub(" ","_")}", block
end

end
After the vocabulary is defined the CompensationParser class processes the BNL file.

File: compensation_parser.rb
class CompensationParser

class << self
def parse(script, vocabulary)
root = Root.new(vocabulary)
script.split(/\n/).each { |line| root.process(preprocess(line)) }
root
end

def preprocess(line)
line.delete!('$,')
line.gsub!(/(\d+)%/, '\1percent')
line.gsub!(/\s/, '._')
"_#{line.downcase}!"
end
end

end
The CompensationParser class is responsible for parsing the script and converting the BNL syntax into valid ruby. The parse class method of CompensationParser creates a new instance of Root, splits the script, sends the preprocessed line to the process method of the instance of Root, and then returns the instance of Root. The CompensationParser preprocess method removes the superfluous characters, replaces special characters with alphabetic representations, converts the line into a chain of methods being called on the result of the previous method call, and appends underscores and an exclamation point to avoid method collisions and signal the end of a phrase.

For example, when preprocess is called on:
"compensate 5% of gross profits if gross profits are greater than $1,000,000"
it becomes:
"_compensate._5percent._of._gross._profits._if._gross._profits._are._greater._than._1000000!"

The Root class is responsible for processing each line of the Business Natural Language.

File: root.rb
class Root
extend Vocabulary

def initialize(vocabulary)
@compensations = []
@vocabulary = vocabulary
end

def name
@employee.name
end

def amount
@compensations.collect do |compensation|
compensation.amount
end.inject { |x, y| x + y }
end

def process(line)
instance_eval(line)
end

phrase :employee do
@employee = Employee.new
end

phrase :compensate do
@compensations << Compensation.new(@vocabulary)
@compensations.last
end

end
Root is responsible for storing reference to the child objects of the parse tree. The references are maintained via instance variables that are initialized when the methods that correspond to phrases of the language are processed. For example, the 'employee' phrase creates and stores an instance of the Employee class. Root processes each line of the Business Natural Language by passing the line to the instance_eval method.

When the first line is sent to instance_eval, 'employee' returns an instance of the Employee class. The Employee class instance stores each subsequent message in an array and always returns self.

File: employee.rb
class Employee

def initialize
@name_parts = []
end

def method_missing(sym,*args)
@name_parts << sym.to_s.delete('_!')
self
end

def name
@name_parts.collect { |part| part.to_s.capitalize }.join(' ')
end

end
When the second and each following line are sent to instance_eval a Compensation instance is created. The primary task of the instance of Compensation is to build phrases and reduce them to ruby code when possible. Each message sent to compensation from the instance_eval is handled by method_missing. The method_missing method attempts to reduced the phrase instance variable each time method_missing is executed. A phrase can be reduced if the entire phrase is a number or if the vocabulary contains the phrase. If a phrase can be reduced the result is appended to the compensation_logic instance variable and the phrase is reset to empty. If a phrase cannot be reduced it is simply stored awaiting the next call to method_missing. If the phrase contains the end of phrase delimiter (the exclamation point) and it cannot be reduced an exception is thrown.

File: compensation.rb
class Compensation

def initialize(vocabulary)
@phrase, @compensation_logic = '', ''
@vocabulary = vocabulary
end

def method_missing(sym, *args)
@phrase = reduce(@phrase + sym.to_s)
if @phrase.any? && sym.to_s =~ /!$/
raise NoMethodError.new("#{@phrase} not found")
end
self
end

def reduce(phrase)
case
when phrase =~ /^_\d+[(percent)|!]*$/
append(extract_number(phrase))
when @vocabulary.respond_to?(phrase)
append(@vocabulary.send(phrase))
else phrase
end
end

def append(piece)
@compensation_logic += piece
""
end

def extract_number(string)
string.gsub(/(\d+)percent$/, '0.0\1').delete('_!')
end

def amount
instance_eval(@compensation_logic) || 0
end

end
After making all of the above changes a quick run of payroll_process.rb produces the following output:

Jackie Johnson compensation: 256800.0
John Jones compensation: 88500.0

We still have a lot of work to do, but we've proven our concept. Our application behavior is entirely dependent on the compensation specifications that can be altered by our subject matter experts.

In the upcoming chapters we will discuss moving our application to the web, putting the specifications in a database instead of using flat files, providing syntax checking and contextual grammar recommendations, and many other concepts that will take us through developing a realistic Business Natural Language application.

Ruby Regular Expression Replace

Ruby provides Regular Expression replacement via the gsub method of String. For example, if I would like to replace a word I simply need the regex to match the word and the replacement text.
irb(main):006:0> "replacing in regex".gsub /\sin\s/, ' is easy in '
=> "replacing is easy in regex"
The gsub method also lets you use the regex match in the replacement.
irb(main):007:0> "replacing in regex".gsub /\sin\s/, ' is easy\0'
=> "replacing is easy in regex"
If you want part of the regex, but not the whole thing you will need to use a capturing group (parenthesis).
irb(main):008:0> "replacing in regex".gsub /\si(n)\s/, ' is easy i\1 '
=> "replacing is easy in regex"
When using a capturing group that matches multiple times in a single line you can still use \1 to include the match in the result.
irb(main):015:0> "1%, 10%, 100%".gsub /(\d+)%/, '0.0\1'
=> "0.01, 0.010, 0.0100"
Another important detail to note is that I'm using single quotes in my replacement string. Using double quotes neither works with \0 or \1 since "\0" #=> "\000".
irb(main):009:0> "replacing in regex".gsub /\sin\s/, " is easy\0"
=> "replacing is easy\000regex"
irb(main):010:0> "replacing in regex".gsub /\si(n)\s/, " is easy i\1 "
=> "replacing is easy i\001 regex"