Functional Testing using a DSL and NUnit

Selecting a functional test framework very much depends on the aspects of your project. Currently, I'm staffed on a project with the following characteristics.

• All the development to date has been back-end. (No UI)
• The functional tests need to be maintained by Business Analysts
• The application is written in C#
• The business analysts need to be able to run the tests locally
• The tests need to run as part of the build
• The wiki we are using cannot be used as the input for the tests
• We already had developer maintained functional tests written for NUnit
  

After identifying these and a few other points we narrowed our choices to NFit or rolling our own. Before we made the decision we spiked the level of effort it would take to roll our own.

Our spike consisted of defining a testing DSL and creating a parser to parse the DSL files and convert them into NUnit tests. The entire process looked like this:

1. A business analyst creates a test file that contains multiple tests written in our DSL.
2. The parser parses the DSL into NUnit tests written in C#.
3. The C# is compiled to a testing DLL.
4. The NUnit GUI is used to execute the tests contained in the testing DLL.

The DSL was defined based on the existing NUnit tests. For example the NUnit test:

[Test]

public void TestSomething()
{
 Transformer transformer = new Transformer();
 transformer.LegacyValue = "ASDF";
 Assert.AreEqual("42",transformer.ConvertedValue);
}

Could be expressed in our DSL as:

test
legacy_value = "ASDF"
expect "42"

The parser was written in Ruby because we wanted a lightweight solution that allowed for easy string parsing.

The ruby parser was run against a directory and created one large TestFixture class that contained each test specified in the DSL files. This step was added to the build before the compilation step. The the BAFunctionalTest class was compiled into a DLL and NUnit executed the tests in this DLL as part of the build.

Obviously, this DLL could be used with the NUnit GUI to allow the BAs to run their tests locally.

I'm sure we could have done things better, such as creating an internal Ruby DSL instead of an external one. We didn't actively make this decision, and had we chosen this solution I'm sure we would have moved in that direction. It was a fun spike and hopefully I'll actually get to try it out on my next project.

In the end, the decision was made to go with NFit because the team wasn't very familiar with Ruby and they had used Fit frameworks of a different flavor in the past. Looking back, I think that was the biggest mistake we made on the project.