Improving Brittle Functional Tests

Functional tests often rely on an external resource existing in a well known state. A test could simply validate that a class returns valid data from the database. For end-to-end testing these tests are very valuable. Unfortunately, these end-to-end tests are also brittle if done incorrectly. Brittle tests often lead to builds breaking unexpectedly. These are the worst types of broken builds. The developer who checked in knows that their change did not cause the build to break. Because their change was not the cause of the broken build they feel unmotivated to fix the build. Of course, they have to fix the build, but are unhappy with the level of effort needed to track down the cause of the error. This pain is not quickly forgotten and often leads to one or both of two outcomes.

• Broken builds become common and are ignored. As long as the build occasionally passes everyone writes off the broken builds as a data problem. Much of the value of the build box is lost at this point.
  
• Developers check in less. Often a pair will work an entire day without checking in any changes. The pair understands that fixing the build will be a painful and time-consuming process and for efficiency they choose to endure this task only once a day. This leads to a larger number of code conflicts and general integration issues. Productivity will be lost.

Fortunately, there are alternatives to assuming a resource is in a well known state.

Create a database that is only used during the build. Ensure this database is in a well known state by dropping objects, adding objects, and loading data with each build. If you are using SQL Server, osql can be used to execute SQL scripts that perform these tasks.

If the database is a shared resource you may not have the ability to drop, add, and delete any time you like. In this scenario, everyone requiring access must agree on a time where the data can be reset to a well known state. The reset can be performed by an additional build that is created to reset the database and immediately run all functional tests that depend on the well known state. This option is less favorable because it requires that the tests be run on a timed interval instead of after each change. However, this negative can be mitigated by creating an In Memory Database that mimics the behavior of the real database. The test suite that runs following each change can include tests that act as end-to-end tests but use the In Memory Database.

Durable functional tests are absolutely necessary in a well running agile environment.