Friday, December 09, 2005

Functional Testing using a DSL and NUnit

Selecting a functional test framework very much depends on the aspects of your project. Currently, I'm staffed on a project with the following characteristics.
  • All the development to date has been back-end. (No UI)
  • The functional tests need to be maintained by Business Analysts
  • The application is written in C#
  • The business analysts need to be able to run the tests locally
  • The tests need to run as part of the build
  • The wiki we are using cannot be used as the input for the tests
  • We already had developer maintained functional tests written for NUnit
After identifying these and a few other points we narrowed our choices to NFit or rolling our own. Before we made the decision we spiked the level of effort it would take to roll our own.

Our spike consisted of defining a testing DSL and creating a parser to parse the DSL files and convert them into NUnit tests. The entire process looked like this:
  1. A business analyst creates a test file that contains multiple tests written in our DSL.
  2. The parser parses the DSL into NUnit tests written in C#.
  3. The C# is compiled to a testing DLL.
  4. The NUnit GUI is used to execute the tests contained in the testing DLL.
The DSL was defined based on the existing NUnit tests. For example the NUnit test:
[Test]

public void TestSomething()
{
Transformer transformer = new Transformer();
transformer.LegacyValue = "ASDF";
Assert.AreEqual("42",transformer.ConvertedValue);
}
Could be expressed in our DSL as:
test
legacy_value = "ASDF"
expect "42"
The parser was written in Ruby because we wanted a lightweight solution that allowed for easy string parsing.

The ruby parser was run against a directory and created one large TestFixture class that contained each test specified in the DSL files. This step was added to the build before the compilation step. The the BAFunctionalTest class was compiled into a DLL and NUnit executed the tests in this DLL as part of the build.

Obviously, this DLL could be used with the NUnit GUI to allow the BAs to run their tests locally.

I'm sure we could have done things better, such as creating an internal Ruby DSL instead of an external one. We didn't actively make this decision, and had we chosen this solution I'm sure we would have moved in that direction. It was a fun spike and hopefully I'll actually get to try it out on my next project.

In the end, the decision was made to go with NFit because the team wasn't very familiar with Ruby and they had used Fit frameworks of a different flavor in the past. Looking back, I think that was the biggest mistake we made on the project.

2 comments:

  1. Anonymous12:37 AM

    Jay, I like the DSL idea and also would like to try it out. I do want to point out that we might be bitching NFit too much than we should. NFit does suck at several places, like finding the assembly to be tested. But once it is solved, NFit is nothing more than a pretty interface to whatever DSL you would to use underneath.
    And I think, the problem we have, is not because of we choose Fit over home-grown DSL, but because of the complexity and comprehensiveness that we expect NFit tests to achieve. Dare I say sometime it is too much, and hopefully we are moving away from letting NFit (or Functional tests for BA) to drag us too much.
    Because NFit claims it can work out of Excel, I think we mistakenly give our BAs high expectation of its ease. Once we found things aren't as easy as passing in a excel file, we got frustrated. But I think same frustration would also occur if BAs find out they have to make a big DSL input file out of their excel and undo lots of excel auto-formatting.

    ReplyDelete
  2. Anonymous12:56 AM

    A very good approach. I would really like to try using DSL and then generate the NUnit test Fixtures but i am lost in how to write the parser that does all the job? Can you suggest some guidelines or some sample source code to start with..
    Thanks
    Sharon.

    ReplyDelete

Note: Only a member of this blog may post a comment.