Friday, November 03, 2006

BNL: Moving to the web and multiple contexts - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Languages: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

In 'The Solution' we proved our concept by replacing the existing system with one that allows the business logic to be written by the subject matter experts. However, the solution we provided has a few limitations our client would like us to overcome:
  • Our application can only be run from the command line, they would like a web interface.
  • Our application only executes compensation calculations for the entire group; they would like to execute individual compensation calculations.
  • The sales data that was previously stored in yaml files should now be read from their database.
  • Our application does not scale well; they would like the burden of calculation to be transferred to their database server when running for every employee.
Now that we are required to connect to the database and move to the web, Ruby on Rails is the obvious solution. This book assumes familiarity with creating a new Ruby on Rails applications. If you need more info on Ruby on Rails applications I suggest 'Agile Web Development with Rails' by Dave Thomas and DHH.

For those coding along, the next steps are:
1. create a Rails application named payroll
2. Set up a dev database and alter database.yml to point to your new database.
3. Add the following code to environment.rb:
Inflector.inflections do |inflect|
inflect.uncountable %w( employee_info )
end
We also need to create the sales data that we would expect to be available when our application executes. Creating the sales tables and data is accomplished with the CreateSalesData Migration class.
class EmployeeInfo < ActiveRecord::Base; end

class CreateSalesData < ActiveRecord::Migration
def self.up
create_table :employee_info, :force=>true do |table|
table.column :employee, :string
table.column :deals_this_month, :integer
table.column :year_old_deals, :integer
table.column :gross_profit, :integer
end

EmployeeInfo.create(:employee=>'jjones',
:deals_this_month=>7,
:year_old_deals=>2,
:gross_profit=>1400000)
EmployeeInfo.create(:employee=>'jjohnson',
:deals_this_month=>12,
:year_old_deals=>1,
:gross_profit=>2200000)
end

def self.down
drop_table :employee_info
end
end
After creating the sales data you can create the following SalesInfo class that represents the facade to the sales data. Note: The SalesInfo class still returns EmployeeInfo instances, and EmployeeInfo has now become an ActiveRecord::Base derived class.
class SalesInfo

def self.method_missing(sym, *args)
result = EmployeeInfo.find_by_employee sym.to_s
super if result.nil?
result
end

end

class EmployeeInfo < ActiveRecord::Base
end
With the sample data taken care of, the first thing necessary to accommodate the new requirements is create a table to store the compensation logic.
class CreateCompensationScripts < ActiveRecord::Migration
def self.up
create_table :compensation_scripts do |table|
table.column :logic, :string, :limit=>1000
end
end

def self.down
drop_table :compensation_scripts
end
end
We are also going to need a few views, a model, and a controller to allow the business users to input new compensation logic.

File: new.rhtml
<% form_for :script, @script, :url => { :action => "create" } do |form| %>
Compensation Script<br>
<%= form.text_area :logic %><br>
<%= submit_tag %>
<% end %>
File: index.rhtml
Employees:
<ul>
<% @scripts.each do |script| %>
<li>
<%= script.employee_name %> -
<%= link_to 'view', :action => :view, :id => script.id %> |
<%= link_to 'execute', :action => :execute, :id => script.id %>
</li>
<% end %>
</ul>
<%= link_to 'Execute All', :action=>:execute_all %> |
<%= link_to 'Create new compensation script', :action=>:new %>
File: view.rhtml
<pre><%= @script.logic %><pre>
<%= link_to 'back', :action => :index %>
File: compensation_script.rb
class CompensationScript < ActiveRecord::Base

def employee_name
logic.split(/\n/).first.gsub(/^employee\s/, '').chomp
end

def employee_short_name
name_parts = employee_name.split
(name_parts.first[0,1] + name_parts.last).downcase
end
end
File: payroll_controller.rb
class PayrollController < ApplicationController

def index
@scripts = CompensationScript.find :all
end

def new
@script = CompensationScript.new
end

def create
CompensationScript.create(params[:script])
redirect_to :action => :index
end

def view
@script = CompensationScript.find(params[:id])
end

def execute
script = CompensationScript.find(params[:id])
vocabulary = InlineVocabulary.new(script.employee_short_name)
@compensation = CompensationParser.parse(script.logic, vocabulary, InlineContext.new)
end

def execute_all
@compensations = []
CompensationScript.find(:all).each do |script|
vocabulary = SqlVocabulary.new(script.employee_short_name)
@compensations << CompensationParser.parse(script.logic, vocabulary, SqlContext.new)
end
end

end
As you can see there is logic for executing also included; however, for the time being let's focus on adding our compensation logic to the application. To store the compensation logic navigate to http://localhost:3000/payroll and click the 'Create new compensation script' link, then enter the following logic for John Johnson.

employee John Jones
compensate $2500 for each deal closed in the past 30 days
compensate $500 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 3% of gross profits if gross profits are greater than $2,000,000
compensate 1% of gross profits if gross profits are greater than $3,000,000

Repeat the previous steps and enter in the logic for Jackie Johnson.

employee Jackie Johnson
compensate $3000 for each deal closed in the past 30 days
compensate $800 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 5% of gross profits if gross profits are greater than $2,000,000

After the logic is saved to the database you should be able to click the 'view' link from the index page to verify that the logic has been entered correctly.

Now, back to the previously mentioned execute logic. The index page has an 'execute' link that links to a page that doesn't yet exist. The execute.rhtml file is a simple page that shows the employee's name and their calculated bonus:

File: execute.rhtml
<%= @compensation.name %> compensation: <%= number_to_currency @compensation.amount %><br>
<%= link_to 'back', :action => :index %>
A brief look in the PayrollController class reveals that execute method relies on the InlineVocabulary, CompensationParser, and InlineContext classes to create a parse tree from the logic stored in a CompensationScript. The InlineVocabulary has been previously shown as the CompensationVocabulary class. It has been renamed to clearly express intent and allow the SqlVocabulary class to be defined.

File: inline_vocabulary.rb
class InlineVocabulary
extend Vocabulary

def initialize(data_for)
@data_for = data_for
end

def logic_initializer
''
end

phrase "active deal that closed more than 365 days ago!" do
SalesInfo.send(@data_for).year_old_deals.to_s
end

phrase "are greater than" do
" > "
end

phrase "deal closed in the past 30 days!" do
SalesInfo.send(@data_for).deals_this_month.to_s
end

phrase "for each" do
"*"
end

phrase "gross profits" do
SalesInfo.send(@data_for).gross_profit.to_s
end

phrase "if" do
" if "
end

phrase "of" do
"*"
end

end
The Vocabulary module, which InlineVocabulary extends, remains unchanged; however, I will show it again for completeness.

File: vocabulary.rb
module Vocabulary

def phrase(name, &block)
define_method :"_#{name.to_s.gsub(" ","_")}", block
end

end
The InlineContext class introduces another layer of abstraction. This is necessary to allow the compensation to be evaluated in various contexts. When the execute method is invoked the InlineVocabulary will create valid Ruby code, thus allowing the InlineContext class to simply call instance_eval to evaluate and return.

File: inline_context.rb
class InlineContext

def evaluate(logic)
instance_eval(logic)
end

end
Processing the Business Natural Language is basically the same as the previous command line application; therefore, the CompensationParser class remains largely the same as the previously shown example. The only change to the class is that the parse method now accepts a context and passes this context to the instance of the Root object that is created.

File: compensation_parser.rb
class CompensationParser

class << self

def parse(script, vocabulary, context)
root = Root.new(vocabulary, context)
script.split(/\n/).each { |line| root.process(preprocess(line)) }
root
end

def preprocess(line)
line.chomp!
line.delete!('$,')
line.gsub!(/(\d+)%/, '\1percent')
line.gsub!(/\s/, '._')
"_#{line.downcase}!"
end

end

end
Similar to the CompensationParser class, the Root class also remains largely the same notwithstanding taking a context instance as a constructor argument and passing it to each newly created instance of the Compensation class.

File: root.rb
class Root
extend Vocabulary

def initialize(vocabulary, eval_context)
@compensations = []
@vocabulary, @eval_context = vocabulary, eval_context
end

def name
@employee.name
end

def amount
@compensations.collect do |compensation|
compensation.amount
end.inject { |x, y| x + y }
end

def process(line)
instance_eval(line)
end

phrase :employee do
@employee = Employee.new
end

phrase :compensate do
@compensations << Compensation.new(@vocabulary, @eval_context)
@compensations.last
end

end
The Root class also uses the unchanged Employee class.

File: employee.rb
class Employee

def initialize
@name_parts = []
end

def method_missing(sym,*args)
@name_parts << sym.to_s.delete('_!')
self
end

def name
@name_parts.collect { |part| part.to_s.capitalize }.join(' ')
end

end
The Compensation class also basically mirrors it's previously show version. A minimal change now initializes the compensation_logic instance variable to the value returned from the logic_initializer method of the vocabulary construtor argument. The logic_initializer method of the InlineVocabulary is an empty string; however, we will see later how the SqlVocabulary class uses this to create valid SQL statements. Another notable change to the Compensation class allows for the logic to be executed in various ways. This subtle change can be found in the amount method of the Compensation class. By delegating the evaluation to another class we can simply instance_eval, which InlineContext does, or pass the burden of evaluation on to another process, which will be seen within the SqlContext class.

Take a minute to contemplate this and the previous paragraph if you don't yet appreciate the value of various contexts. Various contexts allow you to evaluate the same code and produce different results based on your current needs. This can be used to delegate responsibility to an external application (e.g. Database server), external process (e.g. a process written in another language), or simply evaluate the code within your application in another way (e.g. Syntax checking). When using Domain Specific Languages, context is king or at least royalty.

Getting back on track, if you've been typing along you should now be able to execute the logic you entered for both John Jones and Jackie Johnson. At this point we have fulfilled the first three of the new requirements. A quick check reveals that both totals report the same numbers previously generated.

John Jones compensation: $88,500.00
Jackie Johnson compensation: $256,800.00

To accommodate the last requirement we are going to create a new page that displays the return values from the database, use the previously shown execute_all method of the PayrollContorller class, create the SqlVocabulary class, and create the SqlContext class which will be used to evaluate the SQL statements generated by the SqlVocabulary class.

The execute_all.rhtml file is responsible for displaying the results of executing the compensation logic for all employees.

File: execute_all.rhtml
<% @compensations.each do |compensation| %>
<%= compensation.name %> compensation: <%= number_to_currency compensation.amount %><br>
<% end %>
<%= link_to 'back', :action => :index %>
The execute_all method of the PayrollController is similar to the execute method except it iterates through each CompensationScript and appends the results in an instance variable array. Another difference is the use of the SqlVocabulary and SqlContext classes. The SqlVocabulary class behaves exactly the same as the InlineVocabulary class, but returns fragments of SQL statements instead of Ruby code. The SqlVocabulary class also defines the logic_initializer method to return 'select ', the necessary prefix for the SQL statements we will generate.

File: sql_vocabulary.rb
class SqlVocabulary
extend Vocabulary

def initialize(data_for)
@data_for = data_for
end

def logic_initializer
'select '
end

def select_column(column)
" (select #{column} from employee_info where employee = '#{@data_for}') "
end

phrase "active deal that closed more than 365 days ago!" do
select_column("year_old_deals")
end

phrase "are greater than" do
" > "
end

phrase "deal closed in the past 30 days!" do
select_column("deals_this_month")
end

phrase "for each" do
"*"
end

phrase "gross profits" do
select_column("gross_profit")
end

phrase "if" do
" where "
end

phrase "of" do
"*"
end

end
The implementation of the SqlContext class uses the execute method of ActiveRecord::Base.connection. This anticlimactic implementation allows us to easily delegate evaluation to the database. I've used this pattern very successfully on two different projects that required calculation and modification of hundreds of millions of records within a few hours.

File: sql_context.rb
class SqlContext

def evaluate(logic)
result = ActiveRecord::Base.connection.select_value logic
result.to_i unless result.nil?
end

end
If you are still typing along you should now be able to navigate from the index page using the 'Execute All' link to a page that shows the following results.

John Jones compensation: $88,500.00
Jackie Johnson compensation: $256,800.00

This doesn't appear very impressive since we already knew how to calculate the results. However, the previously mentioned point of interest is that instead of calculating the results using ruby, the results are calculated from generated sql statements. For example, below is the SQL generated to calculate Jackie Johnson's bonus.
select 3000* (select deals_this_month from employee_info where employee = 'jjohnson') 
select 800* (select year_old_deals from employee_info where employee = 'jjohnson')
select 0.05* (select gross_profit from employee_info where employee = 'jjohnson') where (select gross_profit from employee_info where employee = 'jjohnson') > 1000000
select 0.05* (select gross_profit from employee_info where employee = 'jjohnson') where (select gross_profit from employee_info where employee = 'jjohnson') > 2000000
An important point to note is the that when Jackie Johnson's bonus terms change both the results for execute and execute_all will be updated by the single change that the subject matter expert will make. Obviously, system maintainability has been increased.

At this point we've satisfied the final requirement for applicaiton. The next chapter will focus on some of the limitations of our current implementation and Business Natural Languages in general.

6 comments:

  1. Anonymous9:58 PM

    Jay, thanks for publishing this series. It has given me a greater appreciation for the power of Ruby.

    ReplyDelete
  2. Anonymous10:55 AM

    Ken,
    Thanks for the kind words. I'm still trying to decide if it's worth putting a book out...

    ReplyDelete
  3. Anonymous8:13 PM

    I'm all for the book ... how many copies do I need to commit to buying to move it forward?

    ReplyDelete
  4. Anonymous9:02 PM

    Seriously, I have been working on a similar problem for some time and this is the solution I've been seeking. My problem deals with international taxation and I've been looking for a way to allow the tax experts to codify the rules. If I can help in anyway, please contact me at twshelton at hrtoolbox dot com.

    ReplyDelete
  5. Anonymous11:04 AM

    Thomas,
    Thanks, I'm glad the posts have provided you with some ideas. I'm actively working on more content; however, as of right now I only have about 30 pages, well short of a book. If you want to help you can post questions or suggestions.

    Thanks, Jay

    ReplyDelete
  6. Anonymous10:06 AM

    I'll certainly continue to follow along and make suggestions/comments.

    ReplyDelete

Note: Only a member of this blog may post a comment.