Wednesday, December 27, 2006

Rails: Unit Testing ActiveRecord Validations

There are at least 2 easy ways to unit test the validations of an ActiveRecord::Base subclass.

The most straight-forward way is to create a model and use the valid? method to determine if it is valid or not. For an example I'm going to reuse the PhoneNumber class I defined in a previous entry on ActiveRecord Unit Testing. The following code shows the three tests I've written to ensure that I've correctly defined a validates_presence_of for digits.
class PhoneNumberTest < Test::Unit::TestCase  Column = ActiveRecord::ConnectionAdapters::Column    test "digits are required to be valid" do    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])    number = PhoneNumber.new(:digits => "1234567890")    assert_equal true, number.valid?  end    test "invalid with no digits " do    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])    number = PhoneNumber.new()    assert_equal false, number.valid?  end    test "errors include digits on invalid digits" do    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])    number = PhoneNumber.new()    number.valid?    assert_equal "can't be blank", number.errors.on('digits')  endend
After adding the validates_presence_of call to the PhoneNumber class, all the above tests pass.
class PhoneNumber < ActiveRecord::Base  validates_presence_of :digitsend
This all works fine, but I'm not sure it's necessary to actually cause a validation to occur just to test that one has been defined.

An alternative test implementation could mock the validates_presence_of call and load the file. If the validates_presence_of is defined correctly the class methods will be invoked.
class PhoneNumberTest < Test::Unit::TestCase  Column = ActiveRecord::ConnectionAdapters::Column  test "validates_presence_of is defined for digits" do    PhoneNumber.expects(:validates_presence_of).with(:digits)    load "#{RAILS_ROOT}/app/models/phone_number.rb"  endend
This method of testing doesn't require actually creating a PhoneNumber instance, instead it tests that the PhoneNumber validations are defined correctly.

The load call is a bit messy; however, we can clean that up by adding another method to ActiveRecord::Base.
class << ActiveRecord::Base  def standard_path    File.expand_path("#{RAILS_ROOT}/app/models/#{self.name.underscore}.rb")  endend
Following the above code change the test can be rewritten to the code shown below.
class PhoneNumberTest < Test::Unit::TestCase  Column = ActiveRecord::ConnectionAdapters::Column  test "validates_presence_of is defined for digits" do    PhoneNumber.expects(:validates_presence_of).with(:digits)    load PhoneNumber.standard_path  endend
Which implementation is better? Both implementations provide pros and cons. I suggest trying both out to determine which works best for you (or your team).

Monday, December 25, 2006

Rails: ActiveRecord Unit Testing

There's been a fair amount of information posted recently on the topic of unit testing ActiveRecord::Base subclasses. Of all the information, I think the most valuable was James' observation that a better data model would result in better tests with less dependencies.

That said, sometimes it would be nice to truly unit test an ActiveRecord::Base subclass. When I say unit test, I mean no dependency on the database. For example, I may store a phone number in the database as a 10 digit string, but I may want to expose that phone number with formatting. I may also want to expose the area code, exchange, and station as methods of the PhoneNumber class. To test this behavior I wrote the following tests that do hit the database.
class PhoneNumberTest < Test::Unit::TestCase  test "to_formatted_s returns US format" do    number = PhoneNumber.new(:digits => "1234567890")    assert_equal "(123) 456-7890", number.to_formatted_s  end    test "area code returns first 3 numbers" do    number = PhoneNumber.new(:digits => "1234567890")    assert_equal "123", number.area_code  endend
If this syntax looks bizarre at all, you might want to read about how these tests take advantage of the test class method.

The PhoneNumber class has the following implementation.
class PhoneNumber < ActiveRecord::Base    def to_formatted_s    '(' + area_code + ') ' + exchange + '-' + station  end      def area_code    digits[0..2]  end    def exchange    digits[3..5]  end  def station    digits[6..9]  end  end
As the implementation shows, the digits are stored in the database. Splitting the digits up or formatting them is handled by methods on the model.

Now that we have a few tests we'll add the code that disallows database access from unit tests and rerun the tests. As expected the tests fail with the following error.
1) Error:test_area_code_returns_first_3_numbers(PhoneNumberTest):ArgumentError: You cannot access the database from a unit test
Looking at the stack trace provided by the error you can track the database access to the columns method of ActiveRecord::Base.

If you want to unit test a model the columns method is a good one to stub. I tried a few different options and the best one I found was stubbing the columns method with a method that returns an array of ActiveRecord::ConnectionAdapters::Column instances. Since I only needed the digits attribute for these tests, it was the only column I needed to return in the array. The following tests test my PhoneNumber class without requiring a trip to the database.
class PhoneNumberTest < Test::Unit::TestCase  Column = ActiveRecord::ConnectionAdapters::Column  test "to_formatted_s returns US format" do    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])    number = PhoneNumber.new(:digits => "1234567890")    assert_equal "(123) 456-7890", number.to_formatted_s  end    test "area code returns first 3 numbers" do    PhoneNumber.stubs(:columns).returns([Column.new("digits", nil, "string", false)])    number = PhoneNumber.new(:digits => "1234567890")    assert_equal "123", number.area_code  endend

Rails: Migrations with a large team part I

I've been working on a big (in Rails terms) team for a few months now and we've come to some conclusions concerning Migrations. I'll start with the bad and in part II I'll talk about the decisions we made and why.

Migrations are great, but they do come at a cost. When working with a large team (my current team size is 14 and growing) migration conflicts happen. This can be mitigated with communication, but migrations can definitely be a bottleneck. Also, the process of creating a migration can be painful on a large team. Before creating a migration you should always update from source control to ensure you get all the migrations checked in by your teammates. Then, the best case scenario is when you can create a migration that doesn't change anything and immediately check it in. Checking in a new migration immediately helps ensure you don't block other teammates from creating migrations; however, it's not always as simple as adding a new table. Migrations that alter the database structure often break several tests. Obviously, you can't check those migrations in until you fix all the breaking tests, which can take time. During this time, database changes are blocked for the entire team.

It can also be troublesome to find specific changes to the database within multiple migration files. Finding which migration adds a specific column can take a fair amount of time when you are working with over 50 migration files. Naming conventions can mitigate this issue some; however, naming conventions generally require that only one action occur per file. For example, the 023_create_customers_table.rb file can only create the customer table and cannot alter the purchases table to add the customer_id column. This type of naming convention helps on searching for specific changes to the database; however, it also results in a large number of migration files.

Sunday, December 24, 2006

Ruby: Alias method alternative

Originally blogged by Martin @ split-s. I'm reposting because I don't see this method used very often.

Using Ruby's alias it is possible to reopen a class, override a method and still call the original.
class ClockRadio  def on!    @on = true  end    def on?    @on  endendclass ClockRadio  alias :old_on! :on!    def on!    old_on!    @display_time = true  end    def display_time?    @display_time  endend
While this works, it can cause unexpected results and leave around artifacts. In isolation this doesn't look risky; however, in a large codebase someone could easily define old_on! or use it as their alias name also. The other, much smaller, issue is that old_on! will be left as a method on ClockRadio when you actually have no desire to expose this method outside of calling it from on!.

An alternative is to capture the on! method as an unbound method, bind it to the current instance, and call it explicitly.
class ClockRadio  on = self.instance_method(:on!)    define_method(:on!) do    on.bind(self).call    @display_time = true  end    def display_time?    @display_time  endend
The above version ensures that the correct version of on! will be called from the on! implementation defined in the reopened version ClockRadio. This version also lets the reference to the old_on! fall out of scope after the class is defined; therefore, there are no additional methods left around as side effects.

Below are the tests that prove the concept.
class AliasMethodAlternativeTest < Test::Unit::TestCase  def test_aliased_method_returns_true_for_on    radio = ClockRadio.new    radio.on!    assert_equal true, radio.on?  end    def test_aliased_method_returns_true_for_display_time    radio = ClockRadio.new    radio.on!    assert_equal true, radio.display_time?  end  end

Saturday, December 23, 2006

Ruby: Multiline strings - here doc or quotes

Ruby supports multiline strings by providing two types of here doc syntax. The first syntax uses and additional dash, but allows you to indent the "end of here doc" delimiter ('eos' in the example).
  <<-eos    Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor     incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud     exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute     irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla     pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia    deserunt mollit anim id est laborum.  eos
Another here doc syntax doesn't require you to use the dash, but it does require that the "end of here doc" delimiter is in column 1 (or there are no spaces that precede it).
  <<eos    Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor     incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud     exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute     irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla     pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia    deserunt mollit anim id est laborum.eos
Both options support the following syntax when passed as a parameter.
  Content.new(:value => <<eos)    Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor     incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud     exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute     irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla     pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officiaeos  Content.new(:value => <<eos    Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor     incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud     exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute     irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla     pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officiaeos)
As an alternative, you could also pass the parameter using quotes.
  Content.new(:value => "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor     incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud     exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute     irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla     pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia")
The solution with quotes requires less lines, so I tend to prefer it. Also, the only benefit I can think of for the here doc syntax is that it allows quotes without having to escape them.

Friday, December 22, 2006

Ruby: Constant values

I generally use constants in Ruby for the following two situations: Markers or Constant values.

Markers are used as a standard for comparison.
module CreditCardTypes  Visa = 0  Mastercard = 1end
Markers are initialized to a value; however, that value is unimportant as long as it is unique. Markers are generally used in an application within conditional statements.
case card.typewhen CreditCardTypes::Visa then VisaLuhnValidatorwhen CreditCardTypes::Mastercard then MastercardLuhnValidatorend
Note: you can also use symbols as markers, but I prefer constants. This preference is based on the fact that if I mistype CreditCardTypes::Vissa it will fail fast; however, if I mistype :credit_card_type_vissa, I will get a possibly hard to find bug.

Constant values are global values that should never change during the life of your application.
module MathValues  PI = 3.14end
Constant values can be used throughout applications to ensure that the same value is consistently used.
circumference = circle.diameter * MathValues::PI
Based on these usages, I'm a bit concerned about some behavior I recently found.
irb(main):019:0> module MathVariablesirb(main):020:1>   PI = 3.14irb(main):021:1> end=> 3.14irb(main):022:0> module MathVariablesirb(main):023:1>   PI = 3.14159265irb(main):024:1> end(irb):23: warning: already initialized constant PI=> 3.14159265
Warning? This means anyone can redefine my constants at any time? Did I do something wrong? Does anyone else think this is dangerous?

Thursday, December 21, 2006

Rake: db:migration:conflict

This task originally written by Stephen Chu.

If you are using migrations on a large team, migration conflicts can and do occur. To help avoid conflicts you can add the following task to your codebase and run it after updating and before checking in (hopefully in your commit task).
namespace :db do  namespace :migration do    desc "After downloading migrations from server, check to see if we have conflicting migration numbers."    task :conflict do      existing_migration_numbers = []      Dir.entries("#{RAILS_ROOT}/db/migrate").each do |entry|        migration_number = entry.scan(/^\d+/).first        next if migration_number.nil?        if existing_migration_numbers.include? migration_number          raise ArgumentError, "Migration #{migration_number} is already checked in on the server. Verify your migration numbers."        end        existing_migration_numbers << migration_number      end        end  endend

Ruby: rake commit

update: Paul Gross created a GitHub project where these tasks live and can be improved. Check it out at http://github.com/pgr0ss/rake_commit_tasks/tree/master

I wrote in the past about rake pc, the task that I run before checking in. The rake pc task is good, but it stops one step short: actually checking in.

The first implementation of this task used a task with parameters; however, the final implementation uses STDIN.gets and a data file.

The rake commit task also allows you specify a pair if you are working on multiple pairing stations.

desc "Run to check in"task :commit => :pc do  commit_pair = retrieve_saved_data "pair"  commit_message = retrieve_saved_data "message"  command = %[svn ci -m "#{commit_pair.chomp} - #{commit_message.chomp}"]  puts command  puts %x[#{command}]enddef retrieve_saved_data attribute  data_path = File.expand_path(File.dirname(__FILE__) + "/#{attribute}.data")  touch #{data_path} unless File.exist? data_path  saved_data = File.read(data_path)    puts "last #{attribute}: " + saved_data unless saved_data.chomp.empty?  print "#{attribute}: "  input = STDIN.gets  while (saved_data.chomp.empty? && (input.chomp.empty?))    print "#{attribute}: "    input = STDIN.gets  end  if input.chomp.any?    File.open(data_path, "w") { |file| file << input }  else    puts "using: " + saved_data.chomp  end  input.chomp.any? ? input : saved_dataend

Sunday, December 17, 2006

BNL: New location

Instead of continuing to post BNL updates to my blog I've put all the current content on http://bnl.jayfields.com.

I'll post an occasional blog entry here when significant updates occur.

Rails: Plugins, Why?

I've been wondering for awhile if the Plugin system is something worth taking out of Rails. It's not that I have anything against Rails plugins, but I cant find any way in which they are superior to RubyGems.

Two benefits I have heard are: The code can be packaged with the application and Rails auto-loads plugins. While I agree that these two things are important, it's possible to achieve the first by using gem unpack and it's also possible to auto-load unpacked gems.

Update: I hadn't even seen this post about Rails autoloading gems, by Chad, until today. It seems that more than a few people would like their gems to play nicely with Rails.

Another benefit I've heard is that plugins are easier to create. I've never created a plugin, but Dr Nic has made it quite easy to create gems using newgem.

While I cant find any features that plugins provide and gems do not, RubyGems do provide features that are not available in plugins such as versioning and dependencies.

So, a question and a request:

What benefits do plugins provide?

If there are no benefits to plugins, create gems for your reusable Ruby code instead of plugins.

Ruby: instance and class methods from a module

Context: You have a presenter class that needs to validate attributes. You like the way ActiveRecord allows you to define validations using class methods. You also want to use a valid? method on your presenter instances to determine if they are valid or not.

Step one: You define the validation methods in a module.
module Presenters::Validations  def validates_presence_of(*args)    ..  end  def validates_format_of(*args)    ..  endend
Then, you extend the module to add the class methods to your presenter.
class AccountInformationPresenter  extend Presenters::Validations  validates_presence_of :username, :password  ..
This implementation also requires that you define valid? in each presenter class.
  def valid?    self.class.validations.collect do |validation|      unless validation.valid?(self)        self.errors.add(validation.on, validation.message)       end    end.all?  endend
When building the second presenter it should be clear that the valid? should be abstracted. This abstraction could result in another module. The new module could be included thus providing valid? as an instance method.

Step two: Another common approach is to define the class methods in a ClassMethods module inside the Presenters::Validations module.
module Presenters::Validations  module ClassMethods    def validates_presence_of(*args)      ..    end    def validates_format_of(*args)      ..    end  end  def self.included(base)    base.extend(ClassMethods)  end  def valid?    self.class.validations.collect do |validation|      unless validation.valid?(self)        self.errors.add(validation.on, validation.message)       end    end.all?  endend
This approach, while common in Rails, can generate some dislike. A counter argument is that include is designed to add instance methods and using self.included is clever, but provides unexpected behavior. I've found that people who dislike the self.included trick prefer to explicitly use both include and extend.
class AccountInformationPresenter  include Presenter::Validations  extend Presenter::Validations::ClassMethods  ..end
Which approach is better?
I tend to prefer explicitness; however, if something is used often enough it can turn from an anti-pattern to an idiom. I'm not sure if that is the case or not here, but I think it might be.

Saturday, December 16, 2006

Ruby: Use class methods to reduce duplication

Context: I had two method definitions in my class that were similar to the code below.
class TaxRemoteFacade  def state_tax    ...    @remote.close  end  def federal_tax    ...    @remote.close  endend
I've only shown two, but we actually needed the @remote.close code at the end of several methods.

Solution: To reduce the duplication you could introduce a mixin with a method to remove the duplication.
module RemoteFacade  def remote_call(method_name, &block)    class_eval do      define_method name do        instance_eval &block        @remote.close      end    end  endend
The TaxRemoteFacade can now be defined as the code below.
class TaxRemoteFacade  extend RemoteFacade  remote_call do :state_tax do    ...  end  remote_call do :federal_tax do    ...  endend

Thursday, December 14, 2006

Rails: Running migrations in another environment

From the command line it's possible to run migrations in any environment. For example, using the code below it is possible to run migrations in the test environment.
rake db:migrate RAILS_ENV=test
Because the above code works it would appear that you could create the following rake task to automate running the migrations in the test environment.
task :test_migrate do  ENV["RAILS_ENV"] = "test"  Rake::Task["db:migrate"].invokeend
Unfortunately, that didn't work, but the following change does work.
task :test_migrate do  ActiveRecord::Base.establish_connection "test"  Rake::Task["db:migrate"].invokeend

Wednesday, December 13, 2006

Rake: --dry-run, --trace, and -T

When using rake it can often be useful to know the order in which tasks are being executed. To list the execution order you can issue the following command.
rake --dry-run
The --dry-run option works well; however, it does not report tasks that are explicitly invoked.
Rake::Task["test:units"].invoke
The above code can be used to explicitly invoke a task (test:units in the example). Rails uses this in various database and testing tasks. However, it's still possible to see which tasks are being executed. The following code will show each task as it's executed.
rake --trace
The --trace option provides a lot of information, but if you have grep available it can be trimmed down easily.
rake --trace | grep Execute
Also, when running rake with the -T option you can also pass another parameter to list only the commands that contain that parameter. For example, the following code returns all the tasks that contain 'db' in their description.
rake -T db

Tuesday, December 05, 2006

BNL: Extracting Sales Person

Versioning on an individual sales person
Before we add version control to our application we need something to version. Logically it seems you would like to have a document history showing compensation agreements for each sales person. To accommodate this requirement, we can pull the sales person from the scripts and save them off separately. The first step is creating migrations that create and alter the tables to match our new intentions.

File: 003_create_sales_people.rb
class CreateSalesPeople < ActiveRecord::Migration  def self.up    create_table :sales_people do |t|      t.column :name, :string    end  end  def self.down    drop_table :sales_people  endend

class AddSalesPersonToCompensationScript < ActiveRecord::Migration  def self.up    add_column :compensation_scripts, :sales_person_id, :integer    add_column :compensation_scripts, :created_at, :datetime  end  def self.down    remove_column :compensation_scripts, :sales_person_id    remove_column :compensation_scripts, :created_at  endend

Following the above migrations the existing codebase will need to be updated to reflect the new database structure. The "new" page has been updated to include a textbox for entering the sales person's name.

File: new.rhtml
<% form_for :script, @script, :url => { :action => "create" } do |form| %>Name<br><%= form.text_field :name %><br><br>Compensation Rules<br><%= form.text_area :logic %><br><%= submit_tag %><% end %>

Introducing the name textbox requires us to create the new SalesPersonCompensationScript class. This is because a CompensationScript still contains the logic, but the SalesPerson class maintains the name captured in the "name" textbox. The SalesPersonCompensationScript is a simple class that is used only to store the data that will be used by the controller to save a SalesPerson and CompensationScript.

File: sales_person_compensation_script.rb
class SalesPersonCompensationScript  attr_accessor :name, :logic    def initialize(hash=nil)    unless hash.nil?      self.name = hash[:name]      self.logic = hash[:logic]    end  endend

The PayrollController has been updated to use the SalesPersonCompensationScript class on the "new" view and on the create action. The create action is where the SalesPersonCompensationScript is used to create a CompensationScript and a new SalesPerson if necessary.

File: payroll_controller.rb
class PayrollController < ApplicationController    def index    @sales_people = SalesPerson.find :all  end    def new    @script = SalesPersonCompensationScript.new  end    def create    person_script = SalesPersonCompensationScript.new(params[:script])    script = CompensationScript.create(:logic => person_script.logic)    sales_person = SalesPerson.find_by_name(person_script.name) ||                    SalesPerson.create(:name => person_script.name)    sales_person.compensation_scripts << script    redirect_to :action => :index  end    def view    @person = SalesPerson.find(params[:person_id])  end    def execute    @person = SalesPerson.find(params[:person_id])    vocabulary = InlineVocabulary.new(@person.short_name)    logic = @person.active_compensation_script.logic    @compensation = CompensationParser.parse(logic, vocabulary, InlineContext.new)  end    def execute_all    @compensations = {}    SalesPerson.find(:all).each do |person|      vocabulary = SqlVocabulary.new(person.short_name)      logic = person.active_compensation_script.logic      compensation = CompensationParser.parse(logic, vocabulary, SqlContext.new)      @compensations[person.name.to_sym] = compensation.amount    end  end  end

Every action within the PayrollController has been changed to reflect the new domain model that includes a SalesPerson as a concept. The index action now relies on an array of SalesPerson instances instead of an array of CompensationScript instances. The index.rhtml has also been updated to reflect this change.

File: index.rhtml
Employees:<ul><% @sales_people.each do |person| %> <li>  <%= person.name %> -   <%= link_to 'view', :action => :view, :person_id => person.id %> |  <%= link_to 'execute', :action => :execute, :person_id => person.id %> </li><% end %></ul><%= link_to 'Execute All', :action => :execute_all %> | <%= link_to 'Create new compensation script', :action => :new %>

Navigating to the view page now requires a sales person instance to be available.

File: view.rhtml
Compensation rules for <%= @person.name %><pre><%= @person.active_compensation_script.logic %></pre><%= link_to 'back', :action => :index %>

The execute.rhtml was updated also, but only changed slightly to accommodate the introduction of the sales person concept.

File: execute.rhtml
<%= @person.name %> compensation: <%= number_to_currency @compensation.amount %><br><%= link_to 'back', :action => :index %>

The execute action in PayrollController relies on the short name of a sales person, and the view.rhtml page uses the active_compensation_script method of the SalesPerson instance.

File: sales_person.rb
class SalesPerson < ActiveRecord::Base  has_many :compensation_scripts    def active_compensation_script    compensation_scripts.sort.first  end    def short_name    self.name.gsub(/^([A-Z])[a-z]+\s/, '\1').downcase  endend

The execute_all.rhtml also contains minor updates that reflect the introduction of SalesPerson

File: execute_all.rhtml
<% @compensations.each_pair do |name, amount| %> <%= name %> compensation: <%= number_to_currency amount %><br><% end %><%= link_to 'back', :action => :index %>

All these changes also mean the CompensationScript and Root classes have been simplified by removing the name of the employee.

File: compensation_script.rb
class CompensationScript < ActiveRecord::Base  def <=>(other)    other.created_at <=> self.created_at  endend

File: root.rb
class Root  extend Vocabulary  def initialize(vocabulary, eval_context)    @compensations = []    @vocabulary, @eval_context = vocabulary, eval_context  end    def amount    @compensations.collect do |compensation|       compensation.amount     end.inject { |x, y| x + y }  end    def process(line)    instance_eval(line)  end    phrase :compensate do    @compensations << Compensation.new(@vocabulary, @eval_context)    @compensations.last  endend

Lastly, the Employee class can be completely removed since Root no longer parses an employee's name.

Saturday, December 02, 2006

xUnit: Absence of Mocks and Stubs

In September of 2004 I met James Newkirk at the Microsoft Patterns and Practice Summit in Reston Virginia. If you don't know James, among other accomplishments he was one of the creators of NUnit. At the summit I stopped him to express my appreciation for NUnit. A conversation ensued in which he caught me off guard with the question: What is missing from NUnit?

Well, it's been over two years, but I finally have an answer: Mocks and Stubs

As long as I've needed them, Mocks have been available as additional libraries. When I was using NUnit I used NMock and when using Test::Unit I use Mocha. But, why are these libraries not simply included in xUnit frameworks?

In 2004 I was also introduced to Behavioral Testing as a formal concept. I had been using mocks for a bit; however, it was wile reading Mocks Aren't Stubs by Martin Fowler when the difference between state and behavioral testing became clear. Martin concludes the write up with the following question and answer:
So which style is the best?

I find this a difficult question to answer with confidence.
It's been more than two years since Mocks Aren't Stubs was originally published and both behavioral and state based testing still exist. I'm skeptical that either will ever be declared better for all cases.

I can't imagine having to choose either behavioral or state based testing exclusively. Both have their pros and cons and, I believe, should be used accordingly. So why then do xUnit implementations continue to ignore support for behavioral testing?

Given that the tests we write as developers have matured, I believe it's fair to expect our xUnit frameworks to evolve with our needs.

Tuesday, November 28, 2006

Ruby: Protected class methods

Given the following class subjects_xml and subjects_html are public, not protected.
class Blog  def self.available_subjects(format)    case format    when :xml then subjects_xml    when :html then subjects_html    end  end  protected  def self.subjects_xml    [xml implementation]  end  def self.subjects_html    [html implementation]  endend
The def self. code can be thought of as defining a class method of Blog, but it is actually defining a method on self, where self is the virtual class of Blog. The protected visibility modifier is being used to specify the visibility of the methods of Blog. Since the subjects_xml and subjects_html methods are being defined on self (the virtual class of Blog) the protected visibility is not applied.

It is possible to define protected class methods in various ways.
class Blog  class << self    def available_subjects(format)      case format        when :xml then subjects_xml        when :html then subjects_html      end    end    protected    def subjects_xml      [xml implementation]    end    def subjects_html      [html implementation]    end  endendclass Blog  def self.available_subjects(format)    case format    when :xml then subjects_xml    when :html then subjects_html    end  end  def self.subjects_xml    [xml implementation]  end  def self.subjects_html    [html implementation]  end    class<<self;self;end.send :protected, :subjects_xml, :subjects_htmlend

Tuesday, November 21, 2006

RubyGems: Absolute Paths

In a previous post about RubyGems I mentioned that I prefer to unpack gems locally to remove dependency issues. While this does provide a code sharing solution it also relies on the developers of the RubyGems to provide compatible solutions. Unfortunately, absolute paths seems to be something many RubyGem creators are overlooking.

I'm not picking on postgres-pr, but I will use it as an example. Below the current implementation of postgres.rb is shown.
begin  require 'postgres.so'rescue LoadError  require 'postgres-pr/postgres-compat'end
The above implementation works when installed; however, when unpacked and uninstalled it raises the following error.
MissingSourceFile: no such file to load Â— postgres-pr/postgres-compat
To fix this issue you can manually change all the require statements to use absolute paths or you can add the directory where the gem is unpacked to the load path. I'm not a big fan of making changes to an unpacked gem since I will lose my changes when a new gem version comes out; therefore, adding another directory to the load path seems to be the lesser of two evils. Adding a directory to the load path in Ruby is easy enough.
$:.unshift [path] But, I really don't think this should be necessary. Instead RubyGem developers should make the effort to use absolute paths. require File.dirname(__FILE__) + [relative path] Ruby: Testing Rake Tasks If you develop in Ruby and you practice Test Driven Development you may consider the question: How do I test Rake tasks? Let's start with a task that drops each table from a database:  desc "Wipe the database" task :wipe => :environment do connection = ActiveRecord::Base.connection connection.tables.each do |table| begin connection.execute("drop table #{table}") puts "* dropped table '#{table}'" rescue end end end It may be possible to test this task in it's current form; however, rewriting the task can create a familiar and easily testable class. The key is encapsulating the behavior of the task in a class.  desc "Wipe the database" task :wipe => :environment do DatabaseFacade.clean end class DatabaseFacade def self.clean connection = ActiveRecord::Base.connection connection.tables.each do |table| begin connection.execute("drop table #{table}") puts "* dropped table '#{table}'" rescue end end end end After moving the behavior into a class method it's possible to test the class the same as any other class. class DatabaseFacadeTest < Test::Unit::TestCase def test_clean_drops_all_tables ActiveRecord::Base.expects(:connection).returns(connection=mock) connection.expects(:tables).returns(['table1','table2']) connection.expects(:execute).with("drop table table1") connection.expects(:execute).with("drop table table2") DatabaseFacade.clean endend Monday, November 20, 2006 BNL: Common Pieces Business Natural Languages: Introduction Business Natural Languages: The Problem Business Natural Languages: The Solution Business Natural Languages: DRY code, DAMP BNL Business Natural Languages: Moving to the web and multiple contexts Business Natural Languages: Limitations Business Natural Languages: Common Pieces What is common between applications using Business Natural Languages? Applications using Business Natural Languages often have common functionality. The most often seen functionality includes workflow for approving proposed business logic, history containing previous versions of business logic, an environment that allows you to test proposed business logic, and an editing environment that includes syntax and contextual validation. All of these features have been fairly simple to implement in the systems I've worked on. In general the business logic is stored in a database as a string. The same table also contains a status field where you can store the current state of the business logic. For example, logic could be marked as 'proposed' if it has not yet been approved. Keeping a history is also as easy as inserting a new row into the table for each newly proposed string of logic. When the new logic is approved, the old logic is kept with a status of expired. In general, once logic has expired, it is not brought back to an active status. If you wish to revert, you should clone an expired version and activate it within the workflow. The most complicated common piece to implement is the test environment. This environment often allows you to write ad-hoc logic and immediately execute to view results. Implementation of the test environment varies by project; however, in general it means executing the logic in a slightly different context. For example, you can still use the same code to process the business logic; however, you may persist the results to a table where only test data is persisted. Another option would allow you to execute the logic and see the results rendered in the UI instead of persisting to the database. Each project will have it's own implementation that makes sense, but in general they will all require some type of test environment. While creating an editing environment that includes syntax and contextual validation sounds like a large task, it's actually nothing more than evaluating the same BNL script in another context. As previously shown a BNL can be executed to create a parse tree that is used for execution. For syntax and contextual validation the same BNL can be executed to create a parse tree of syntax validation objects. Once the syntax validation parse tree is created it can be used to provide feedback to the user. The most common question to this approach is how to keep the syntax validation classes in sync with the primary context classes. The answer to this question will vary with project and language; however, using Ruby it's possible to build the syntax checking classes using metaprogramming and using the primary context classes as templates. For example the RootSyntaxValidation class could define it's methods by grabbing all of the methods of Root that begin with an underscore. class RootSyntaxValidation Root.instance_methods.grep(/^_[^_]/).each do |phrase| define_method phrase.to_sym do ... end end def method_missing(sym, *args) SyntaxError.new(sym) endend Now that we've touched on identifying the common pieces the upcoming chapters will provide examples of how to implement these ideas in our sample application. Friday, November 17, 2006 BNL: Limitations Business Natural Languages: Introduction Business Natural Languages: The Problem Business Natural Languages: The Solution Business Natural Languages: DRY code, DAMP BNL Business Natural Languages: Moving to the web and multiple contexts Business Natural Languages: Limitations Business Natural Languages: Common Pieces Limitations of Business Natural Languages Business logic can be highly complex and unrelated. A Business Natural Language becomes more complicated as it expands the scope of problems it addresses. As the complication increases the value of the language decreases. Therefore, when designing a Business Natural Language it should address only related business logic. One application can contain several Business Natural Languages. By limiting the scope of each Business Natural Language it will be easier to understand and use. Every application should not include a Business Natural Language. In fact, businesses should not invest in creating a Business Natural Language unless the application contains logic specific to the business. An example of an application that, despite being a complex application, likely doesn't warrant the use of a Business Natural Language is a content management system (CMS). Creating a CMS is no easy task; however, the business simply wants an application that allows them to easily display static information. There is no reason to design a Business Natural Language when an application, such as a CMS, does not contain any business logic. Business Natural Languages are designed to empower people who contain specific knowledge. By empowering these individuals the business can run more efficiently. However, Business Natural Languages should not be used as an excuse to delegate work. For example, even though you could empower a subject matter expert to specify the database connection string, you shouldn't do it. Maintaining a database connection string is the responsibility of the IT staff. Creating a Business Natural Language is unnecessary in this scenario since the IT staff should be perfectly comfortable with XML, YAML, or some other form of configuration file. It should be clear that you could use a Business Natural Language within a configuration file. This makes sense because configuration files are simply files that contain logic expressed in a Domain Specific Language, and Business Natural Languages, XML, YAML, etc are all DSLs. However, before investing the effort to develop a Business Natural Language you should ensure that someone would benefit from the investment. While it may be technically interesting to use a Business Natural Language to configure an Inversion of Control container, it is likely a waste of time. When determining whether or not to make the investment it is important to consider the following questions: Who will be responsible for writing the business logic? What are the chances that this logic will change? If the answer to the first question is "a subject matter expert whose role does not require technical depth" and the answer to the second question is "it's possible" then you likely have a good candidate for a Business Natural Language. When thinking about the first question it is more important to consider the subject matter expert's role than a specific person's skill level. If you only think in terms of the person currently in that role you risk creating an un-maintainable system if the technically savvy subject matter expert leaves their current role. However, if you design a Business Natural Language that can be understood by any subject matter expert in that role, then your systems maintainability is greatly increased. The second question and answer should be fairly straightforward. It makes sense to abstract the logic in a way that allows a subject matter expert to maintain it if the logic has the possibility to change. Another situation where it could be unclear whether to use a Business Natural Language or not is when a subject matter expert creates a specification that is not specific to the business. For example, a subject matter expert may create a requirement for the system such as 'the message entry textbox on the invitation page should not allow more than 500 characters'. While this requirement does come from a subject matter expert it is not a requirement imposed by the business. These types of requirements do not often warrant the investment of creating a Business Natural Language. A major reason why a Business Natural Language loses value here is because the business probably does not have specific language around field validation. So, from their perspective it's almost the same level of effort to read either of these two DSLs: 'the message entry textbox on the invitation page should not allow more than 500 characters' page: invitation; field: message entry; limit: 500 That being said, if some of your validations were tied to business logic then it does make sense to use a Business Natural Language. Examples of this could be to only accept 9-digit social security numbers, Florida driver's licenses must always start with the first letter of the driver's last name, or if you claim a dependent you must specify their social security number also. Clearly, if some of your validation logic belongs in a Business Natural Language then it makes sense to express it all in one or more Business Natural Languages. If you were to maintain logic in two places, such as an XML file and a Business Natural language logic script, usability clearly suffers. On a similar note, if your Business Natural Language begins to explode with all the types of validations you are supporting, you should consider grouping similar validations in separate Business Natural Languages. As previously stated, you should look to separate a Business Natural Language when it will increase readability and ease of use. One major advantage of creating a Business Natural Language is empowering subject matter experts to specify the behavior of the system in the language of their business. Unfortunately, language rarely transcends multiple corporations. Therefore, it is unlikely that a generic set of Business Natural Languages could be created to address the needs of a business vertical. (I.e. insurance, leasing, etc) Monday, November 13, 2006 Rails: Get the schema_info version number from Rake Recently a coworker asked if there was a simple way to get the current version out of the schema_info table. Of course, one simple answer is to simply look in the database. However, if you want to quickly pull the version from a command line, this rake task should do the trick*: namespace :db do desc "Returns the current schema version" task :version => :environment do puts "Current version: " + ActiveRecord::Migrator.current_version.to_s endend * 2nd version submitted by Doug as a comment. Rails: Textmate Footnotes If you are developing Rails applications using TextMate you should not be without TextMate Footnotes. This plugin saves you time when moving between your browser and Textmate. It adds clickable links to the Rails backtrace when an error occurs, and clickable "footnotes" to the bottom of each page (in development mode only). These links open the correct file in Textmate and move your cursor to the precise line of interest. In addition, you will have session, params and cookies information available to you in development mode on all rhtml rendered pages. It's simple to add to your project: go to the command line and navigate to vendor/plugins and issue the following svn command. svn propset svn:externals "footnotes http://macromates.com/svn/Bundles/trunk/Bundles/Rails.tmbundle/Support/plugins/footnotes" . Sunday, November 12, 2006 Rails: Generate SQL from Migrations Unfortunately, my new client will not allow migrations to be run in production. This isn't a surprise since they have a DB group that maintains their databases. And, the DB group requires application developers to submit SQL scripts when database modifications are necessary. At first this may sound like a pain, but it's actually quite easy to capture the SQL that migrations generate. There may be easier ways; however, it seemed simple enough to temporarily change the behavior of the execute method. require 'find'class SqlWriter def self.write(path, string) File.open(path, 'w') { |file| file << string } end endclass MigrationSql class << self def execute connection = ActiveRecord::Base.connection old_method = connection.class.instance_method(:execute) define_execute(connection) { |*args| SqlWriter.write(@sql_write_path, args.first) } root = RAILS_ROOT + "/db/migrate" output_dir = root + "/../migration_sql" Dir.mkdir output_dir unless File.exists? output_dir Find.find(root) do |path| unless path == root require path file = File.basename(path, ".rb") write_sql(connection, output_dir, file, :up) write_sql(connection, output_dir, file, :down) end end define_execute(connection) { |*args| old_method.bind(self).call(*args) } end def write_sql(connection, output_dir, file, direction) connection.instance_variable_set :@sql_write_path, output_dir + "/" + file + "_#{direction}.sql" file.gsub(/^\d\d\d_/,'').camelize.constantize.send direction end def define_execute(connection, &block) connection.class.send :define_method, :execute, &block end endend Now that we have the behavior necessary we can wrap the call to the execute method of the MigrationSql class in a rake task. namespace "db" do namespace "generate" do desc "generate sql for migrations" task "migration_sql" => :environment do MigrationSql.execute end endend Friday, November 03, 2006 BNL: Moving to the web and multiple contexts - Update Business Natural Languages: Introduction Business Natural Languages: The Problem Business Natural Languages: The Solution Business Natural Languages: DRY code, DAMP BNL Business Natural Languages: Moving to the web and multiple contexts Business Natural Languages: Limitations Business Natural Languages: Common Pieces In 'The Solution' we proved our concept by replacing the existing system with one that allows the business logic to be written by the subject matter experts. However, the solution we provided has a few limitations our client would like us to overcome: • Our application can only be run from the command line, they would like a web interface. • Our application only executes compensation calculations for the entire group; they would like to execute individual compensation calculations. • The sales data that was previously stored in yaml files should now be read from their database. • Our application does not scale well; they would like the burden of calculation to be transferred to their database server when running for every employee. Now that we are required to connect to the database and move to the web, Ruby on Rails is the obvious solution. This book assumes familiarity with creating a new Ruby on Rails applications. If you need more info on Ruby on Rails applications I suggest 'Agile Web Development with Rails' by Dave Thomas and DHH. For those coding along, the next steps are: 1. create a Rails application named payroll 2. Set up a dev database and alter database.yml to point to your new database. 3. Add the following code to environment.rb: Inflector.inflections do |inflect| inflect.uncountable %w( employee_info )end We also need to create the sales data that we would expect to be available when our application executes. Creating the sales tables and data is accomplished with the CreateSalesData Migration class. class EmployeeInfo < ActiveRecord::Base; endclass CreateSalesData < ActiveRecord::Migration def self.up create_table :employee_info, :force=>true do |table| table.column :employee, :string table.column :deals_this_month, :integer table.column :year_old_deals, :integer table.column :gross_profit, :integer end EmployeeInfo.create(:employee=>'jjones', :deals_this_month=>7, :year_old_deals=>2, :gross_profit=>1400000) EmployeeInfo.create(:employee=>'jjohnson', :deals_this_month=>12, :year_old_deals=>1, :gross_profit=>2200000) end def self.down drop_table :employee_info endend After creating the sales data you can create the following SalesInfo class that represents the facade to the sales data. Note: The SalesInfo class still returns EmployeeInfo instances, and EmployeeInfo has now become an ActiveRecord::Base derived class. class SalesInfo def self.method_missing(sym, *args) result = EmployeeInfo.find_by_employee sym.to_s super if result.nil? result endendclass EmployeeInfo < ActiveRecord::Baseend With the sample data taken care of, the first thing necessary to accommodate the new requirements is create a table to store the compensation logic. class CreateCompensationScripts < ActiveRecord::Migration def self.up create_table :compensation_scripts do |table| table.column :logic, :string, :limit=>1000 end end def self.down drop_table :compensation_scripts endend We are also going to need a few views, a model, and a controller to allow the business users to input new compensation logic. File: new.rhtml <% form_for :script, @script, :url => { :action => "create" } do |form| %>Compensation Script<br><%= form.text_area :logic %><br><%= submit_tag %><% end %> File: index.rhtml Employees:<ul><% @scripts.each do |script| %> <li> <%= script.employee_name %> - <%= link_to 'view', :action => :view, :id => script.id %> | <%= link_to 'execute', :action => :execute, :id => script.id %> </li><% end %></ul><%= link_to 'Execute All', :action=>:execute_all %> | <%= link_to 'Create new compensation script', :action=>:new %> File: view.rhtml <pre><%= @script.logic %><pre><%= link_to 'back', :action => :index %> File: compensation_script.rb class CompensationScript < ActiveRecord::Base def employee_name logic.split(/\n/).first.gsub(/^employee\s/, '').chomp end def employee_short_name name_parts = employee_name.split (name_parts.first[0,1] + name_parts.last).downcase endend File: payroll_controller.rb class PayrollController < ApplicationController def index @scripts = CompensationScript.find :all end def new @script = CompensationScript.new end def create CompensationScript.create(params[:script]) redirect_to :action => :index end def view @script = CompensationScript.find(params[:id]) end def execute script = CompensationScript.find(params[:id]) vocabulary = InlineVocabulary.new(script.employee_short_name) @compensation = CompensationParser.parse(script.logic, vocabulary, InlineContext.new) end def execute_all @compensations = [] CompensationScript.find(:all).each do |script| vocabulary = SqlVocabulary.new(script.employee_short_name) @compensations << CompensationParser.parse(script.logic, vocabulary, SqlContext.new) end end end As you can see there is logic for executing also included; however, for the time being let's focus on adding our compensation logic to the application. To store the compensation logic navigate to http://localhost:3000/payroll and click the 'Create new compensation script' link, then enter the following logic for John Johnson. employee John Jones compensate$2500 for each deal closed in the past 30 days
compensate $500 for each active deal that closed more than 365 days ago compensate 5% of gross profits if gross profits are greater than$1,000,000
compensate 3% of gross profits if gross profits are greater than $2,000,000 compensate 1% of gross profits if gross profits are greater than$3,000,000

Repeat the previous steps and enter in the logic for Jackie Johnson.

employee Jackie Johnson
compensate $3000 for each deal closed in the past 30 days compensate$800 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000 compensate 5% of gross profits if gross profits are greater than$2,000,000

After the logic is saved to the database you should be able to click the 'view' link from the index page to verify that the logic has been entered correctly.

Now, back to the previously mentioned execute logic. The index page has an 'execute' link that links to a page that doesn't yet exist. The execute.rhtml file is a simple page that shows the employee's name and their calculated bonus:

File: execute.rhtml
<%= @compensation.name %> compensation: <%= number_to_currency @compensation.amount %><br><%= link_to 'back', :action => :index %>
A brief look in the PayrollController class reveals that execute method relies on the InlineVocabulary, CompensationParser, and InlineContext classes to create a parse tree from the logic stored in a CompensationScript. The InlineVocabulary has been previously shown as the CompensationVocabulary class. It has been renamed to clearly express intent and allow the SqlVocabulary class to be defined.

File: inline_vocabulary.rb
class InlineVocabulary  extend Vocabulary    def initialize(data_for)    @data_for = data_for  end    def logic_initializer    ''  end    phrase "active deal that closed more than 365 days ago!" do     SalesInfo.send(@data_for).year_old_deals.to_s  end    phrase "are greater than" do    " > "  end    phrase "deal closed in the past 30 days!" do     SalesInfo.send(@data_for).deals_this_month.to_s  end    phrase "for each"  do     "*"   end    phrase "gross profits" do     SalesInfo.send(@data_for).gross_profit.to_s   end    phrase "if" do     " if "   end    phrase "of" do     "*"   end  end
The Vocabulary module, which InlineVocabulary extends, remains unchanged; however, I will show it again for completeness.

File: vocabulary.rb
module Vocabulary  def phrase(name, &block)    define_method :"_#{name.to_s.gsub(" ","_")}", block  endend
The InlineContext class introduces another layer of abstraction. This is necessary to allow the compensation to be evaluated in various contexts. When the execute method is invoked the InlineVocabulary will create valid Ruby code, thus allowing the InlineContext class to simply call instance_eval to evaluate and return.

File: inline_context.rb
class InlineContext    def evaluate(logic)    instance_eval(logic)  end  end
Processing the Business Natural Language is basically the same as the previous command line application; therefore, the CompensationParser class remains largely the same as the previously shown example. The only change to the class is that the parse method now accepts a context and passes this context to the instance of the Root object that is created.

File: compensation_parser.rb
class CompensationParser  class << self        def parse(script, vocabulary, context)      root = Root.new(vocabulary, context)      script.split(/\n/).each { |line| root.process(preprocess(line)) }      root    end        def preprocess(line)      line.chomp!      line.delete!('$,') line.gsub!(/(\d+)%/, '\1percent') line.gsub!(/\s/, '._') "_#{line.downcase}!" end endend Similar to the CompensationParser class, the Root class also remains largely the same notwithstanding taking a context instance as a constructor argument and passing it to each newly created instance of the Compensation class. File: root.rb class Root extend Vocabulary def initialize(vocabulary, eval_context) @compensations = [] @vocabulary, @eval_context = vocabulary, eval_context end def name @employee.name end def amount @compensations.collect do |compensation| compensation.amount end.inject { |x, y| x + y } end def process(line) instance_eval(line) end phrase :employee do @employee = Employee.new end phrase :compensate do @compensations << Compensation.new(@vocabulary, @eval_context) @compensations.last endend The Root class also uses the unchanged Employee class. File: employee.rb class Employee def initialize @name_parts = [] end def method_missing(sym,*args) @name_parts << sym.to_s.delete('_!') self end def name @name_parts.collect { |part| part.to_s.capitalize }.join(' ') endend The Compensation class also basically mirrors it's previously show version. A minimal change now initializes the compensation_logic instance variable to the value returned from the logic_initializer method of the vocabulary construtor argument. The logic_initializer method of the InlineVocabulary is an empty string; however, we will see later how the SqlVocabulary class uses this to create valid SQL statements. Another notable change to the Compensation class allows for the logic to be executed in various ways. This subtle change can be found in the amount method of the Compensation class. By delegating the evaluation to another class we can simply instance_eval, which InlineContext does, or pass the burden of evaluation on to another process, which will be seen within the SqlContext class. Take a minute to contemplate this and the previous paragraph if you don't yet appreciate the value of various contexts. Various contexts allow you to evaluate the same code and produce different results based on your current needs. This can be used to delegate responsibility to an external application (e.g. Database server), external process (e.g. a process written in another language), or simply evaluate the code within your application in another way (e.g. Syntax checking). When using Domain Specific Languages, context is king or at least royalty. Getting back on track, if you've been typing along you should now be able to execute the logic you entered for both John Jones and Jackie Johnson. At this point we have fulfilled the first three of the new requirements. A quick check reveals that both totals report the same numbers previously generated. John Jones compensation:$88,500.00
Jackie Johnson compensation: $256,800.00 To accommodate the last requirement we are going to create a new page that displays the return values from the database, use the previously shown execute_all method of the PayrollContorller class, create the SqlVocabulary class, and create the SqlContext class which will be used to evaluate the SQL statements generated by the SqlVocabulary class. The execute_all.rhtml file is responsible for displaying the results of executing the compensation logic for all employees. File: execute_all.rhtml <% @compensations.each do |compensation| %><%= compensation.name %> compensation: <%= number_to_currency compensation.amount %><br><% end %><%= link_to 'back', :action => :index %> The execute_all method of the PayrollController is similar to the execute method except it iterates through each CompensationScript and appends the results in an instance variable array. Another difference is the use of the SqlVocabulary and SqlContext classes. The SqlVocabulary class behaves exactly the same as the InlineVocabulary class, but returns fragments of SQL statements instead of Ruby code. The SqlVocabulary class also defines the logic_initializer method to return 'select ', the necessary prefix for the SQL statements we will generate. File: sql_vocabulary.rb class SqlVocabulary extend Vocabulary def initialize(data_for) @data_for = data_for end def logic_initializer 'select ' end def select_column(column) " (select #{column} from employee_info where employee = '#{@data_for}') " end phrase "active deal that closed more than 365 days ago!" do select_column("year_old_deals") end phrase "are greater than" do " > " end phrase "deal closed in the past 30 days!" do select_column("deals_this_month") end phrase "for each" do "*" end phrase "gross profits" do select_column("gross_profit") end phrase "if" do " where " end phrase "of" do "*" end end The implementation of the SqlContext class uses the execute method of ActiveRecord::Base.connection. This anticlimactic implementation allows us to easily delegate evaluation to the database. I've used this pattern very successfully on two different projects that required calculation and modification of hundreds of millions of records within a few hours. File: sql_context.rb class SqlContext def evaluate(logic) result = ActiveRecord::Base.connection.select_value logic result.to_i unless result.nil? end end If you are still typing along you should now be able to navigate from the index page using the 'Execute All' link to a page that shows the following results. John Jones compensation:$88,500.00
Jackie Johnson compensation: $256,800.00 This doesn't appear very impressive since we already knew how to calculate the results. However, the previously mentioned point of interest is that instead of calculating the results using ruby, the results are calculated from generated sql statements. For example, below is the SQL generated to calculate Jackie Johnson's bonus. select 3000* (select deals_this_month from employee_info where employee = 'jjohnson') select 800* (select year_old_deals from employee_info where employee = 'jjohnson') select 0.05* (select gross_profit from employee_info where employee = 'jjohnson') where (select gross_profit from employee_info where employee = 'jjohnson') > 1000000select 0.05* (select gross_profit from employee_info where employee = 'jjohnson') where (select gross_profit from employee_info where employee = 'jjohnson') > 2000000 An important point to note is the that when Jackie Johnson's bonus terms change both the results for execute and execute_all will be updated by the single change that the subject matter expert will make. Obviously, system maintainability has been increased. At this point we've satisfied the final requirement for applicaiton. The next chapter will focus on some of the limitations of our current implementation and Business Natural Languages in general. Story Card Format A friend of mine recently wrote: Realizing that TW has NDA's to honor, is there anyway you could paraphrase a few User Stories for me so I can see good examples of what they should have? I realize they need only be 2-3 sentences, but since I've no experience with them at all (other than use cases) I don't know what the ideal story should be like. As in, what level of detail are they, verbage wise. Does the customer actually write them on the cards themselves while the developer is nearby to answer any questions? Below is my response, that I thought might be useful to those adopting Agile. At ThoughtWorks we generally use a form such as: As a script author I would like to save the script language to the database so that it can be executed and altered in the future. The underlined words are the 'template'. I've briefly written in the past about why I'm not a huge fan of the template; however, we have had success with it. At the end of the day a story is a placeholder for a conversation; therefore, use whatever template you can create that will encourage the conversation. Monday, October 30, 2006 Rails: Autoloading Gems in Vendor It's no secret that I'm a big fan of RubyGems. Due to this bias, I tend to package up all my reusable code as RubyGems. Then, when I'm developing a new Rails application I gem unpack [gem] each of the gems I'm using in the vendor directory. This works out well, but I am stuck maintaining the require [path/to/gem] statements that are necessary for loading the libraries. To resolve this maintenance pain I put the following code in environment.rb. Dir[File.dirname(__FILE__) + "/../vendor/*"].each do |path| gem_name = File.basename(path.gsub(/-\d+.\d+.\d+$/, ''))  gem_path = path + "/lib/" + gem_name + ".rb"  require gem_path if File.exists? gem_pathend
The above script relies on a few RubyGems conventions: 3 digit versioning and a file in lib that requires all relevant ruby files for the library. I'm comfortable with these assumptions; however, not all gems follow them. For example, postgres-pr uses a ruby file named postres.rb to load the library. For situations like these, I still need to do an explicit require.

Saturday, October 21, 2006

BNL: DRY code, DAMP BNL - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

DRY code, DAMP Business Natural Languages

Don't Repeat Yourself (DRY) is a best practice in software development. Specifying behavior once and only once is the essence of the DRY rule. When you follow this rule the maintainability of the software is increases. The rule helps ensure that if you are required to make a change you can be sure that you will only need to change the software in one location. The benefits of this rule should be quite obvious; however, does it also apply to Business Natural Languages?

Based on my experience I believe that the DRY rule does not apply to Business Natural Languages. A major reason for using a Business Natural Language is to separate the business logic from the complexities of the under-lying system. When using a Business Natural Language, business users who are the most familiar with the domain can maintain the business logic. To a business user, a Business Natural Language should be no different than a group of phrases that describe the rules for running the business correctly.

A well-designed Business Natural Language will appear as Descriptive And Meaningful Phrases (DAMP).

In 'The Solution' I used the following business logic to calculate Jackie Johnson's bonus:

employee Jackie Johnson
compensate $3000 for each deal closed in the past 30 days compensate$800 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000 compensate 5% of gross profits if gross profits are greater than$2,000,000

By creating shorter phrases you could rewrite the logic to be more concise:

employee Jackie Johnson
compensate $3000 each past 30 compensate$800 each closed 365
compensate 5% of profits if profits greater $1,000,000 compensate 5% of profits if profits greater$2,000,000

The new logic can be considered more DRY; however, much of the readability has been lost at the expense of removing the some of the duplication. You should not trade readability for conciseness when designing Business Natural Languages.

As businesses change their software requires change. If software can be understood and altered by subject matter experts the business can be more responsive to change. Business Natural Languages facilitate empowering subject matter experts by using a meaningful syntax to express business logic. Creating meaningful syntax often requires defining verbose phrases; however, they increase the overall readability of the phrase. Therefore, the verbose phrases carry maintainability value for the subject matter experts. If the verbose phrases are removed the logic becomes less maintainable.

It is also important to note that employees often change roles within an organization. Because staffing does change, you should not design a Business Natural Language that sacrifices readability for conciseness simply because the current subject matter expert is able to comprehend the concise version. Instead, you should design the Business Natural Language to be easily understood by any subject matter expert placed in that role.

Some programmers object to verbose phrases because they are not necessary for program execution. This is a fairly common thought because programmers are still thinking in terms of 'what will make the system work' instead of trying to solve a more valuable question: How can I empower a subject matter expert to make the system work? When you begin to think in these terms you see that verbose phrases are not strictly required; however, it's highly important to the subject matter expert who will need to continually understand and update the system.

The main goal of a DAMP Business Natural Language is empowering subject matter experts by providing the most maintainable syntax available. A Business Natural Language clearly has room for improvement in the syntax if it requires training to understand. An ideal Business Natural Language contains phrases that are descriptive and meaningful enough that they require no explanation at all.

The last two Ruby/Rails projects that I've been involved with had both data that the application created and data that was expected to be there. In production, the data that was expected to be there was data created by an external batch process that ran monthly.

This all works very well; however, it does present a challenge when developing locally: How do we load data that mocks the expected external data? Both projects approached the problem differently, and both approaches provided pros and cons.

Approach 1: Conditionally load the mock data using migrations. On the first project we used PostgreSQL for local development and Oracle in production. Since we used different databases we were able to write our migrations to only load mock data when the database being loaded was a PostgreSQL database. A pro to this approach is that running all the migrations completely sets up the database, local or production. A con to this approach is that more migrations are necessary and some of the migrations lose maintainability based on conditional noise.

Approach 2: Create a rake task to load data that your application does not own. PragDave inspired this approach with his blog entry on how to use Migrations Outside Rails. Using ActiveRecord::Schema.define it's possible to easily create the mock tables.
ActiveRecord::Schema.define do  create_table books, :force => true do |t|    t.column :title, :integer    t.column :pages, :string    t.column :classification, :integer  endend
Following table creation, you can load the mock tables by creating models and saving.
class Book < ActiveRecord::BaseendBook.create(:title=>'RubyFoo', :pages=>300, :classification=>'Technical')
A pro of this implementation is the clear separation between the mock data and the data necessary for running your application. This also helps reduce the risk of loading invalid data in production. A con is that an extra step is required when first setting up or resetting a database. This con can be partially mitigated by creating a task that runs both the migrations and loads the mock data; however, this introduces more code to the codebase and requires developers to remember one more rake task.

Saturday, October 14, 2006

BNL: The Solution - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Langauges: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

The Solution
A Business Natural Language is used to specify application behavior. The language should be comprised of descriptive and maintainable phrases. The structure of the language should be simple, yet verbose. Imagine each line of the specification as one complete sentence. Because a Business Natural Language is natural language it is important to limit the scope of the problem being solved. An application's requirements can often be split to categories that specify similar functionality. Each category can be a candidate for a simple Business Specific Language. Lucky for us, our current application has a very limited scope: calculating compensation. Therefore, our application will only require one Business Natural Language.

We've already seen the requirements:

employee John Jones
compensate $2500 for each deal closed in the past 30 days compensate$500 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000 compensate 3% of gross profits if gross profits are greater than$2,000,000
compensate 1% of gross profits if gross profits are greater than $3,000,000 A primary driver for using a Business Natural Language is the ability to execute the requirements as code. Lets change the existing application to work with a Business Natural Language. To start with we can take the requirements and store them in a file named jjones.bnl. Now that we have our Business Natural Language file in our application we'll need to alter the existing code to read it. The first step is changing process_payroll.rb to search for all Business Natural Language files, create a vocabulary, create a parse tree, and report the results. File: process_payroll.rb Dir[File.dirname(__FILE__) + "/*.bnl"].each do |bnl_file| vocabulary = CompensationVocabulary.new(File.basename(bnl_file, '.bnl')) compensation = CompensationParser.parse(File.read(bnl_file), vocabulary) puts "#{compensation.name} compensation: #{compensation.amount}"end Creating a vocabulary is nothing more than defining the phrases that the Business Natural Language should understand. Each phrase returns values that are appended together to create valid ruby syntax. The vocabulary also calls out to the SalesInfo class to return sales data for each employee. File: compensation_vocabulary.rb class CompensationVocabulary extend Vocabulary def initialize(data_for) @data_for = data_for end phrase "active deal that closed more than 365 days ago!" do SalesInfo.send(@data_for).year_old_deals.to_s end phrase "are greater than" do " > " end phrase "deal closed in the past 30 days!" do SalesInfo.send(@data_for).deals_this_month.to_s end phrase "for each" do "*" end phrase "gross profits" do SalesInfo.send(@data_for).gross_profit.to_s end phrase "if" do " if " end phrase "of" do "*" end end The CompensationVocabulary class does extend Vocabulary, which is how the phrase class method is added to CompensationVocabulary. File: vocabulary.rb module Vocabulary def phrase(name, &block) define_method :"_#{name.to_s.gsub(" ","_")}", block endend After the vocabulary is defined the CompensationParser class processes the BNL file. File: compensation_parser.rb class CompensationParser class << self def parse(script, vocabulary) root = Root.new(vocabulary) script.split(/\n/).each { |line| root.process(preprocess(line)) } root end def preprocess(line) line.delete!('$,')      line.gsub!(/(\d+)%/, '\1percent')      line.gsub!(/\s/, '._')      "_#{line.downcase}!"    end  endend
The CompensationParser class is responsible for parsing the script and converting the BNL syntax into valid ruby. The parse class method of CompensationParser creates a new instance of Root, splits the script, sends the preprocessed line to the process method of the instance of Root, and then returns the instance of Root. The CompensationParser preprocess method removes the superfluous characters, replaces special characters with alphabetic representations, converts the line into a chain of methods being called on the result of the previous method call, and appends underscores and an exclamation point to avoid method collisions and signal the end of a phrase.

For example, when preprocess is called on:
"compensate 5% of gross profits if gross profits are greater than $1,000,000" it becomes: "_compensate._5percent._of._gross._profits._if._gross._profits._are._greater._than._1000000!" The Root class is responsible for processing each line of the Business Natural Language. File: root.rb class Root extend Vocabulary def initialize(vocabulary) @compensations = [] @vocabulary = vocabulary end def name @employee.name end def amount @compensations.collect do |compensation| compensation.amount end.inject { |x, y| x + y } end def process(line) instance_eval(line) end phrase :employee do @employee = Employee.new end phrase :compensate do @compensations << Compensation.new(@vocabulary) @compensations.last endend Root is responsible for storing reference to the child objects of the parse tree. The references are maintained via instance variables that are initialized when the methods that correspond to phrases of the language are processed. For example, the 'employee' phrase creates and stores an instance of the Employee class. Root processes each line of the Business Natural Language by passing the line to the instance_eval method. When the first line is sent to instance_eval, 'employee' returns an instance of the Employee class. The Employee class instance stores each subsequent message in an array and always returns self. File: employee.rb class Employee def initialize @name_parts = [] end def method_missing(sym,*args) @name_parts << sym.to_s.delete('_!') self end def name @name_parts.collect { |part| part.to_s.capitalize }.join(' ') endend When the second and each following line are sent to instance_eval a Compensation instance is created. The primary task of the instance of Compensation is to build phrases and reduce them to ruby code when possible. Each message sent to compensation from the instance_eval is handled by method_missing. The method_missing method attempts to reduced the phrase instance variable each time method_missing is executed. A phrase can be reduced if the entire phrase is a number or if the vocabulary contains the phrase. If a phrase can be reduced the result is appended to the compensation_logic instance variable and the phrase is reset to empty. If a phrase cannot be reduced it is simply stored awaiting the next call to method_missing. If the phrase contains the end of phrase delimiter (the exclamation point) and it cannot be reduced an exception is thrown. File: compensation.rb class Compensation def initialize(vocabulary) @phrase, @compensation_logic = '', '' @vocabulary = vocabulary end def method_missing(sym, *args) @phrase = reduce(@phrase + sym.to_s) if @phrase.any? && sym.to_s =~ /!$/      raise NoMethodError.new("#{@phrase} not found")    end    self  end    def reduce(phrase)    case      when phrase =~ /^_\d+[(percent)|!]*$/ append(extract_number(phrase)) when @vocabulary.respond_to?(phrase) append(@vocabulary.send(phrase)) else phrase end end def append(piece) @compensation_logic += piece "" end def extract_number(string) string.gsub(/(\d+)percent$/, '0.0\1').delete('_!')  end    def amount    instance_eval(@compensation_logic) || 0  endend
After making all of the above changes a quick run of payroll_process.rb produces the following output:

Jackie Johnson compensation: 256800.0
John Jones compensation: 88500.0

We still have a lot of work to do, but we've proven our concept. Our application behavior is entirely dependent on the compensation specifications that can be altered by our subject matter experts.

In the upcoming chapters we will discuss moving our application to the web, putting the specifications in a database instead of using flat files, providing syntax checking and contextual grammar recommendations, and many other concepts that will take us through developing a realistic Business Natural Language application.

Ruby Regular Expression Replace

Ruby provides Regular Expression replacement via the gsub method of String. For example, if I would like to replace a word I simply need the regex to match the word and the replacement text.
irb(main):006:0> "replacing in regex".gsub /\sin\s/, ' is easy in '=> "replacing is easy in regex"
The gsub method also lets you use the regex match in the replacement.
irb(main):007:0> "replacing in regex".gsub /\sin\s/, ' is easy\0'=> "replacing is easy in regex"
If you want part of the regex, but not the whole thing you will need to use a capturing group (parenthesis).
irb(main):008:0> "replacing in regex".gsub /\si(n)\s/, ' is easy i\1 '=> "replacing is easy in regex"
When using a capturing group that matches multiple times in a single line you can still use \1 to include the match in the result.
irb(main):015:0> "1%, 10%, 100%".gsub /(\d+)%/, '0.0\1'=> "0.01, 0.010, 0.0100"
Another important detail to note is that I'm using single quotes in my replacement string. Using double quotes neither works with \0 or \1 since "\0" #=> "\000".
irb(main):009:0> "replacing in regex".gsub /\sin\s/, " is easy\0"=> "replacing is easy\000regex"irb(main):010:0> "replacing in regex".gsub /\si(n)\s/, " is easy i\1 "=> "replacing is easy i\001 regex"

Saturday, October 07, 2006

Rake: Tasks with parameters

I've written several Rake tasks in the past that could have been improved by accepting parameters, but I never knew how to make it work. It's not that I thought it was impossible to pass parameters to Rake tasks, I just never bothered to figure out how. Lucky for me, a teammate recently took the time to figure it out. Today I'm changing an existing rakefile to take advantage of passing in parameters.

The file now:
require 'writer'require 'html_writer'require 'doc_writer'directory "built_docs"directory "built_html"task :doc => [:built_docs, :clean_doc] do  Dir['chapters/*.txt'].each do |chapter|    output = "built_docs/#{File.basename(chapter,'txt')}doc"    File.open(output, 'w') { |file| DocWriter.new(file).write(chapter) }    puts "generated #{output}"    system "chmod 444 #{output}"  endendtask :html => [:built_html, :clean_html] do  Dir['chapters/*.txt'].each do |chapter|    output = "built_html/#{File.basename(chapter,'txt')}html"    File.open(output, 'w') { |file| HtmlWriter.new(file).write(chapter) }    puts "generated #{output}"    system "chmod 444 #{output}"  endendtask :clean_doc do  Dir['built_docs/*'].each do |output|    rm output  endendtask :clean_html do  Dir['built_html/*'].each do |output|    rm output  endend
Clearly, this file is not near as DRY as it could be.

The first step is to create a new task that will generate output and accept the format as an argument.
task :default do  unless ENV.include?("output") && (ENV['output']=='doc' || ENV['output']=='html')    raise "usage: rake output= # valid formats are [html] or [doc]"   end  format = ENV['output']  puts format  rm_rf format  mkdir formatend
The above code verifies that a valid format is specified. It also removes the need for both the directory and clean tasks. The only work left to do is combine the output generation tasks.
task :default do  unless ENV.include?("output") && (ENV['output']=='doc' || ENV['output']=='html')    raise "usage: rake output= # valid formats are [html] or [doc]"   end  format = ENV['output']  puts format  rm_rf format  mkdir format  Dir['chapters/*.txt'].each do |chapter|    output = "#{format}/#{File.basename(chapter,'txt')}#{format}"    writer = (format=='doc' ? DocWriter : HtmlWriter)    File.open(output, 'w') { |file| writer.new(file).write(chapter) }    puts "generated #{output}"    system "chmod 444 #{output}"  endend
The resulting task is cleaner and easier to maintain.

Running the new task from the command line: focus:~ jay\$ rake output=html