BNL: Moving to the web and multiple contexts - Update

Business Natural Languages: Introduction
Business Natural Languages: The Problem
Business Natural Languages: The Solution
Business Natural Languages: DRY code, DAMP BNL
Business Natural Languages: Moving to the web and multiple contexts
Business Natural Languages: Limitations
Business Natural Languages: Common Pieces

In 'The Solution' we proved our concept by replacing the existing system with one that allows the business logic to be written by the subject matter experts. However, the solution we provided has a few limitations our client would like us to overcome:

• Our application can only be run from the command line, they would like a web interface.
• Our application only executes compensation calculations for the entire group; they would like to execute individual compensation calculations.
• The sales data that was previously stored in yaml files should now be read from their database.
• Our application does not scale well; they would like the burden of calculation to be transferred to their database server when running for every employee.

Now that we are required to connect to the database and move to the web, Ruby on Rails is the obvious solution. This book assumes familiarity with creating a new Ruby on Rails applications. If you need more info on Ruby on Rails applications I suggest 'Agile Web Development with Rails' by Dave Thomas and DHH.

For those coding along, the next steps are:
1. create a Rails application named payroll
2. Set up a dev database and alter database.yml to point to your new database.
3. Add the following code to environment.rb:

Inflector.inflections do |inflect|
  inflect.uncountable %w( employee_info )
end

We also need to create the sales data that we would expect to be available when our application executes. Creating the sales tables and data is accomplished with the CreateSalesData Migration class.

class EmployeeInfo < ActiveRecord::Base; end

class CreateSalesData < ActiveRecord::Migration
  def self.up
    create_table :employee_info, :force=>true do |table|
      table.column :employee, :string
      table.column :deals_this_month, :integer
      table.column :year_old_deals, :integer
      table.column :gross_profit, :integer
    end
    
    EmployeeInfo.create(:employee=>'jjones', 
                        :deals_this_month=>7, 
                        :year_old_deals=>2, 
                        :gross_profit=>1400000)
    EmployeeInfo.create(:employee=>'jjohnson', 
                        :deals_this_month=>12, 
                        :year_old_deals=>1, 
                        :gross_profit=>2200000)
  end

  def self.down
    drop_table :employee_info
  end
end

After creating the sales data you can create the following SalesInfo class that represents the facade to the sales data. Note: The SalesInfo class still returns EmployeeInfo instances, and EmployeeInfo has now become an ActiveRecord::Base derived class.

class SalesInfo

  def self.method_missing(sym, *args)
    result = EmployeeInfo.find_by_employee sym.to_s
    super if result.nil?
    result
  end

end

class EmployeeInfo < ActiveRecord::Base
end

With the sample data taken care of, the first thing necessary to accommodate the new requirements is create a table to store the compensation logic.

class CreateCompensationScripts < ActiveRecord::Migration
  def self.up
    create_table :compensation_scripts do |table|
      table.column :logic, :string, :limit=>1000
    end
  end

  def self.down
    drop_table :compensation_scripts
  end
end

We are also going to need a few views, a model, and a controller to allow the business users to input new compensation logic.

File: new.rhtml

<% form_for :script, @script, :url => { :action => "create" } do |form| %>
Compensation Script<br>
<%= form.text_area :logic %><br>
<%= submit_tag %>
<% end %>

File: index.rhtml

Employees:
<ul>
<% @scripts.each do |script| %>
 <li>
  <%= script.employee_name %> - 
  <%= link_to 'view', :action => :view, :id => script.id %> |
  <%= link_to 'execute', :action => :execute, :id => script.id %>
 </li>
<% end %>
</ul>
<%= link_to 'Execute All', :action=>:execute_all %> | 
<%= link_to 'Create new compensation script', :action=>:new %>

File: view.rhtml

<pre><%= @script.logic %><pre>
<%= link_to 'back', :action => :index %>

File: compensation_script.rb

class CompensationScript < ActiveRecord::Base

  def employee_name
    logic.split(/\n/).first.gsub(/^employee\s/, '').chomp
  end
  
  def employee_short_name
    name_parts = employee_name.split
    (name_parts.first[0,1] + name_parts.last).downcase
  end
end

File: payroll_controller.rb

class PayrollController < ApplicationController
  
  def index
    @scripts = CompensationScript.find :all
  end
  
  def new
    @script = CompensationScript.new
  end
  
  def create
    CompensationScript.create(params[:script])
    redirect_to :action => :index
  end
  
  def view
    @script = CompensationScript.find(params[:id])
  end
  
  def execute
    script = CompensationScript.find(params[:id])
    vocabulary = InlineVocabulary.new(script.employee_short_name)
    @compensation = CompensationParser.parse(script.logic, vocabulary, InlineContext.new)
  end
  
  def execute_all
    @compensations = []
    CompensationScript.find(:all).each do |script|
      vocabulary = SqlVocabulary.new(script.employee_short_name)
      @compensations << CompensationParser.parse(script.logic, vocabulary, SqlContext.new)
    end
  end
  
end

As you can see there is logic for executing also included; however, for the time being let's focus on adding our compensation logic to the application. To store the compensation logic navigate to http://localhost:3000/payroll and click the 'Create new compensation script' link, then enter the following logic for John Johnson.

employee John Jones
compensate $2500 for each deal closed in the past 30 days
compensate $500 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 3% of gross profits if gross profits are greater than $2,000,000
compensate 1% of gross profits if gross profits are greater than $3,000,000

Repeat the previous steps and enter in the logic for Jackie Johnson.

employee Jackie Johnson
compensate $3000 for each deal closed in the past 30 days
compensate $800 for each active deal that closed more than 365 days ago
compensate 5% of gross profits if gross profits are greater than $1,000,000
compensate 5% of gross profits if gross profits are greater than $2,000,000

After the logic is saved to the database you should be able to click the 'view' link from the index page to verify that the logic has been entered correctly.

Now, back to the previously mentioned execute logic. The index page has an 'execute' link that links to a page that doesn't yet exist. The execute.rhtml file is a simple page that shows the employee's name and their calculated bonus:

File: execute.rhtml

<%= @compensation.name %> compensation: <%= number_to_currency @compensation.amount %><br>
<%= link_to 'back', :action => :index %>

A brief look in the PayrollController class reveals that execute method relies on the InlineVocabulary, CompensationParser, and InlineContext classes to create a parse tree from the logic stored in a CompensationScript. The InlineVocabulary has been previously shown as the CompensationVocabulary class. It has been renamed to clearly express intent and allow the SqlVocabulary class to be defined.

File: inline_vocabulary.rb

class InlineVocabulary
  extend Vocabulary
  
  def initialize(data_for)
    @data_for = data_for
  end
  
  def logic_initializer
    ''
  end
  
  phrase "active deal that closed more than 365 days ago!" do 
    SalesInfo.send(@data_for).year_old_deals.to_s
  end
  
  phrase "are greater than" do
    " > "
  end
  
  phrase "deal closed in the past 30 days!" do 
    SalesInfo.send(@data_for).deals_this_month.to_s
  end
  
  phrase "for each"  do 
    "*" 
  end
  
  phrase "gross profits" do 
    SalesInfo.send(@data_for).gross_profit.to_s 
  end
  
  phrase "if" do 
    " if " 
  end
  
  phrase "of" do 
    "*" 
  end
  
end

The Vocabulary module, which InlineVocabulary extends, remains unchanged; however, I will show it again for completeness.

File: vocabulary.rb

module Vocabulary

  def phrase(name, &block)
    define_method :"_#{name.to_s.gsub(" ","_")}", block
  end

end

The InlineContext class introduces another layer of abstraction. This is necessary to allow the compensation to be evaluated in various contexts. When the execute method is invoked the InlineVocabulary will create valid Ruby code, thus allowing the InlineContext class to simply call instance_eval to evaluate and return.

File: inline_context.rb

class InlineContext
  
  def evaluate(logic)
    instance_eval(logic)
  end
  
end

Processing the Business Natural Language is basically the same as the previous command line application; therefore, the CompensationParser class remains largely the same as the previously shown example. The only change to the class is that the parse method now accepts a context and passes this context to the instance of the Root object that is created.

File: compensation_parser.rb

class CompensationParser

  class << self
    
    def parse(script, vocabulary, context)
      root = Root.new(vocabulary, context)
      script.split(/\n/).each { |line| root.process(preprocess(line)) }
      root
    end
    
    def preprocess(line)
      line.chomp!
      line.delete!('$,')
      line.gsub!(/(\d+)%/, '\1percent')
      line.gsub!(/\s/, '._')
      "_#{line.downcase}!"
    end

  end

end

Similar to the CompensationParser class, the Root class also remains largely the same notwithstanding taking a context instance as a constructor argument and passing it to each newly created instance of the Compensation class.

File: root.rb

class Root
  extend Vocabulary

  def initialize(vocabulary, eval_context)
    @compensations = []
    @vocabulary, @eval_context = vocabulary, eval_context
  end
  
  def name
    @employee.name
  end
  
  def amount
    @compensations.collect do |compensation| 
      compensation.amount 
    end.inject { |x, y| x + y }
  end
  
  def process(line)
    instance_eval(line)
  end
  
  phrase :employee do
    @employee = Employee.new
  end
  
  phrase :compensate do
    @compensations << Compensation.new(@vocabulary, @eval_context)
    @compensations.last
  end

end

The Root class also uses the unchanged Employee class.

File: employee.rb

class Employee

  def initialize
    @name_parts = []
  end
  
  def method_missing(sym,*args)
    @name_parts << sym.to_s.delete('_!')
    self
  end
  
  def name
    @name_parts.collect { |part| part.to_s.capitalize }.join(' ')
  end

end

The Compensation class also basically mirrors it's previously show version. A minimal change now initializes the compensation_logic instance variable to the value returned from the logic_initializer method of the vocabulary construtor argument. The logic_initializer method of the InlineVocabulary is an empty string; however, we will see later how the SqlVocabulary class uses this to create valid SQL statements. Another notable change to the Compensation class allows for the logic to be executed in various ways. This subtle change can be found in the amount method of the Compensation class. By delegating the evaluation to another class we can simply instance_eval, which InlineContext does, or pass the burden of evaluation on to another process, which will be seen within the SqlContext class.

Take a minute to contemplate this and the previous paragraph if you don't yet appreciate the value of various contexts. Various contexts allow you to evaluate the same code and produce different results based on your current needs. This can be used to delegate responsibility to an external application (e.g. Database server), external process (e.g. a process written in another language), or simply evaluate the code within your application in another way (e.g. Syntax checking). When using Domain Specific Languages, context is king or at least royalty.

Getting back on track, if you've been typing along you should now be able to execute the logic you entered for both John Jones and Jackie Johnson. At this point we have fulfilled the first three of the new requirements. A quick check reveals that both totals report the same numbers previously generated.

John Jones compensation: $88,500.00
Jackie Johnson compensation: $256,800.00

To accommodate the last requirement we are going to create a new page that displays the return values from the database, use the previously shown execute_all method of the PayrollContorller class, create the SqlVocabulary class, and create the SqlContext class which will be used to evaluate the SQL statements generated by the SqlVocabulary class.

The execute_all.rhtml file is responsible for displaying the results of executing the compensation logic for all employees.

File: execute_all.rhtml

<% @compensations.each do |compensation| %>
<%= compensation.name %> compensation: <%= number_to_currency compensation.amount %><br>
<% end %>
<%= link_to 'back', :action => :index %>

The execute_all method of the PayrollController is similar to the execute method except it iterates through each CompensationScript and appends the results in an instance variable array. Another difference is the use of the SqlVocabulary and SqlContext classes. The SqlVocabulary class behaves exactly the same as the InlineVocabulary class, but returns fragments of SQL statements instead of Ruby code. The SqlVocabulary class also defines the logic_initializer method to return 'select ', the necessary prefix for the SQL statements we will generate.

File: sql_vocabulary.rb

class SqlVocabulary
  extend Vocabulary
  
  def initialize(data_for)
    @data_for = data_for
  end
  
  def logic_initializer
    'select '
  end
  
  def select_column(column)
    " (select #{column} from employee_info where employee = '#{@data_for}') "
  end
  
  phrase "active deal that closed more than 365 days ago!" do
    select_column("year_old_deals")
  end
  
  phrase "are greater than" do
    " > "
  end
  
  phrase "deal closed in the past 30 days!" do 
    select_column("deals_this_month")
  end
  
  phrase "for each"  do 
    "*" 
  end
  
  phrase "gross profits" do
    select_column("gross_profit")
  end
  
  phrase "if" do 
    " where " 
  end
  
  phrase "of" do 
    "*" 
  end
  
end

The implementation of the SqlContext class uses the execute method of ActiveRecord::Base.connection. This anticlimactic implementation allows us to easily delegate evaluation to the database. I've used this pattern very successfully on two different projects that required calculation and modification of hundreds of millions of records within a few hours.

File: sql_context.rb

class SqlContext
  
  def evaluate(logic)
    result = ActiveRecord::Base.connection.select_value logic
    result.to_i unless result.nil?
  end
  
end

If you are still typing along you should now be able to navigate from the index page using the 'Execute All' link to a page that shows the following results.

John Jones compensation: $88,500.00
Jackie Johnson compensation: $256,800.00

This doesn't appear very impressive since we already knew how to calculate the results. However, the previously mentioned point of interest is that instead of calculating the results using ruby, the results are calculated from generated sql statements. For example, below is the SQL generated to calculate Jackie Johnson's bonus.

select 3000* (select deals_this_month from employee_info where employee = 'jjohnson') 
select 800* (select year_old_deals from employee_info where employee = 'jjohnson') 
select 0.05* (select gross_profit from employee_info where employee = 'jjohnson')  where  (select gross_profit from employee_info where employee = 'jjohnson')  > 1000000
select 0.05* (select gross_profit from employee_info where employee = 'jjohnson')  where  (select gross_profit from employee_info where employee = 'jjohnson')  > 2000000

An important point to note is the that when Jackie Johnson's bonus terms change both the results for execute and execute_all will be updated by the single change that the subject matter expert will make. Obviously, system maintainability has been increased.

At this point we've satisfied the final requirement for applicaiton. The next chapter will focus on some of the limitations of our current implementation and Business Natural Languages in general.