My current team has 2 QA roles. The QA developers are responsible for creating a acceptance suite that ensures that each story is completed without breaking previously QA'd stories. The project I'm working in is a web application, so using Selenium as the acceptance test suite was a fairly easy choice.
4 months later, the Selenium suite is painfully large. The Selenium suite is run with each CI build; however, it is not run (for performance) reasons by developers before they check in. The result: the Selenium build is broken more often than not.
Fixing the Selenium build can be time consuming for the QA developers, which takes away from the time that they have to spend doing exploratory testing. This is acceptable assuming that the Selenium tests are catching bugs. Unfortunately, we estimate that when the Selenium tests are broken only 10% of the time it is because a bug has been introduced, and the other 90% of the time the Selenium tests require updating because the functionality of the application has changed.
At the same time the build is getting longer and longer (because more tests equals more browser open and closes); therefore, the feedback loop for the Selenium tests is losing even more value.
The writing on the wall prompted my memory of stories of other projects that eventually throw out their acceptance suite because maintaining it is a full time job and it isn't providing enough value.
Based on the fact that we have a long running acceptance suite and a functional test suite that catches 90% of bugs introduced, I concluded that we should look for a more beneficial approach to acceptance testing. After a bit of discussion with a few coworkers we came up with the following idea: create a DSL for acceptance tests. The QA team can create acceptance tests using the DSL. The acceptance tests can be evaluated in one context to run as Rails Functional or Integration tests. This will allow the developers to run the entire acceptance suite before checking in without creating the overhead of running selenium. If the acceptance tests break when run as Rails Functional tests the QA team should work with the developers to update the acceptance tests. The same acceptance tests can be evaluated in another context to run as Selenium tests. This allows the CI build to run the same tests through the browser. We do believe that running through the browser is valuable and this solution allows us to continue to test through the browser.
I've begun creating this solution for my current project and hope to extract something general for other projects moving forward. Look for updates in the future.
I have ran into the same problems before (big selenium tests).
ReplyDeleteTips:
-- Create shorter tests.
-- Create selenium tests that just test the application’s features (not the data or things that constantly change), and then Selenium won’t error out (or not as much).
-- Create the Selenium tests in (Selenium IDE) and re-run a couple of times (also in different environments).
-- If you have to check data or features that constantly change, stop (Selenium IDE) and manually check.
Have you tried Immediate Test Failure Notification?
ReplyDeleteI also recommend using a non-UI functional testing framework such as Fit in addition to Selenium/Watir. Fit functional tests tend to be much less brittle than UI tests.
We've implemented an acceptance DSL, tests look like this (I'm changing the domain to protect the innocent)
ReplyDeletegiven.client("Stacy")
.hasNoCashWhatsoever();
when.client("Stacy")
.accessesTheirAccount();
then.client("Stacy")
.willBeDirectedToTheNearestSoupKitchen();
The 'given', 'when', and 'then' are instance variables with types 'PreConditions', 'Events', and 'PostConditions' respectively.
Any of the conditions or events can access the application in any way, many access it through a browser:
class ClientEvents {
void accessesTheirAccount()
{
onlineBankingHomePage.login(clientId, "password");
}
}
class OnlineBankingHomePage {
void login(String username, String password)
{
browser.type("username", username);
browser.type("password",
password);
browser.submit("login");
}
}
The 'browser' instance is a wrapper around a selenium RC driver (DefaultSelenium).
Another implementation of browser could wrap JWebUnit, etc.
The way we have been using selenium is that, in the firsts stage of development we only use Unit/functional and Integration tests.
ReplyDeleteOnly after most of the main core activity is complete, and alpha is about to start with the UI relatively stable, we start adding selenium tests. this is not really TDD, but it gives us more flexibility and doesn't makes the cost of changes too heavy for the developers.
Jay.. You should know already - That's what Twist is for.. :) Seriously, we built Twist exactly for these reasons, and I remember those projects where the big Selenium suites were thrown away. You need abstraction and the appropriate tool support to manage making the modular at the right level. Selenium, or any other driver, by itself can lead to a path of pain at scale. A DSL is the right choice.
ReplyDelete