Wednesday, June 07, 2006

Some Test Driven Development observations

Testing code is hard. There is some material available, but most of the examples are trivial. When you actually sit down to do strictly Test Driven Development (TDD) the complications appear quickly.

The first tripping point for me was cascading failures. Simple changes to one class often caused several test failures. Sometimes the fix was a one-liner, but often it was not. I began to find the solution when I first read Martin's paper on Mocks and Stubs. After I read that paper I began to use Mocks religiously, but that didn't really solve the problem. Using an excessive amount of mocks is just as fragile as using the actual implementations.

To me, the next distinct progression of my testing skills was following the one assertion per test guideline. Using only one assertion per test allowed me to write very focused tests that generally had less dependencies. By removing dependencies my tests became more robust.

Understanding Dependency Injection was the next major step in improving my testing skills. By using Dependency Injection I was able to further abstract my dependencies to ensure robust tests.

At this point I felt my tests had improved significantly; however, my test suite was becoming a bit of a burden to execute. The major problem was the excessive network traffic between my test suite and the database. To me, long build times are a serious problem because it represents lost developer time, several times a day. I quickly understood the distinction between Unit tests and Functional (or Integration) tests.

The next logical step was to improve test performance by not crossing boundaries while unit testing. Utilizing Dependency Injection I began to mock the data access code in my unit tests which significantly improved test processing time.

Further increasing the robustness of my tests was achieved by testing one concrete class at a time. Again, Dependency Injection helped provide the solution. By using Dependency Injection the code I wrote was significantly decoupled compared to the previous code I was writing. Injecting mocks or stubs for dependencies of my class under test did increase the robustness of the tests; however, I still had the issue of dealing with fragile mock set up code.

A common practice is to specify method names using strings when setting up expectations of a mock. This is obviously fragile when performing constant refactoring. However, using stubs always seemed like more trouble than it was worth because it required implementing an entire interface when often I only cared about a few methods. With no clear answer on the 'best' choice I used both excessively in different places to determine which approach I preferred. In the end, I found that neither should be treated as the silver bullet. In fact, using both mocks and stubs it's possible to make very clear what your test is accomplishing.

Another gray area in testing concerns what visibility to give your classes members. When I first started out testing I believed that exposing the private members for testing was acceptable. However, I quickly learned this approach was naive and a much better solution is to simply test through the public interface. Also, if a method is marked as private, but I feel it needs to be tested I likely wouldn't hesitate to simply make it public.

The reason I believed I needed to access private methods was because I was using a Code Coverage tool that told me my code was only about 80% covered. However, the answer wasn't that I needed to devise a solution to reach 100%, it was that I needed someone to tell me that 80% or higher is good enough. Should you strive for more than 80%? Sure, but don't make any compromises to get there. You will find the return on investment is generally not near as valuable.

That's the majority of my previously documented lessons learned concerning testing. I hope this entry and my previous entries can provide some guidance and help you avoid some common stumbling blocks. In part 2 of this entry I'll document some observations that I haven't previously written about.


  1. Anonymous12:23 AM

    Hi Jay. I'm just in the process of bringing myself up to speed on TDD and am planning on trying it out in my dev group in the near future.

    I've been finding your blog a great resource, thanks for all the knowledge and great links.

    I am in the process of reading the Fowler paper on Dependency Injection and am confused about his example of Constructor Injection. The code his sample uses is (I hope the comment formatting doesn't mangle this):

    private MutablePicoContainer configureContainer() {
    MutablePicoContainer pico = new DefaultPicoContainer();
    Parameter[] finderParams = {new ConstantParameter("movies1.txt")};
    pico.registerComponentImplementation(MovieFinder.class, ColonMovieFinder.class, finderParams);
    return pico;

    public void testWithPico() {
    MutablePicoContainer pico = configureContainer();
    MovieLister lister = (MovieLister) pico.getComponentInstance(MovieLister.class);
    Movie[] movies = lister.moviesDirectedBy("Sergio Leone");
    assertEquals("Once Upon a Time in the West", movies[0].getTitle());

    I'm not understanding the reason for all that extra complexity. Could you not accomplish the same thing with this code:

    public void testMoviesDirectedBy() {
    ColonMovieFinder finder = New ColonMovieFinder("movies1.txt");
    MovieList lister = new MovieLister(finder);
    Movie[] movies = lister.moviesDirectedBy("Sergio Leone");
    assertEquals("Once Upon a Time in the West", movies[0].getTitle());

    Any insight you can provide would be greatly appreciated.


  2. Anonymous10:02 AM

    In your example you are using Dependency Injection correctly. And, your example is simpler; however, Martin's example assumes that in actual production code things will be much more complicated than simply creating two classes and putting them in Pico.

    Here's the thing about PicoContainer, you really don't need it. The authors of PicoContainer will tell you the same. What they are really concerned with is that you use constructor injection for your dependencies (which you did in your example).

    However, once you use Dependency Injection throughout your codebase you will see where PicoContainer can make your life quite a bit easier. I use Containers in nearly every project I work on. The thing about Dependency Injection is that once you start using it throughout your codebase you will end up with relationships such as:
    A ctor depends on instance of B
    B ctor depends on instance of C
    C ctor depends on instance of D
    When you get to this point, it's much easier to ask PicoContainer for an instance of A than to handle construction yourself. Also, it's nice to be able to simply add a ctor argument without needing to change all the calls in the code to that ctor. If you are using Pico you'll just need to be sure that PicoContainer has reference to or can create an instance of the type you added as an argument.

    In other words, you do not need PicoContainer to do Dependency Injection; however, if you are doing Dependency Injection, PicoContainer can make your life much easier.

    Please let me know if you need more info.

  3. Anonymous1:24 PM

    Thanks for the explanation, that definately helps with my understanding. There is still something that I don't quite get about Dependency Injection. The basic idea that if class A needs to use class B in order to carry out it's work, the caller of class A should be responsible for giving class A (or telling it where to get) its instance of class B.

    My issue is that if I'm designing an SOA to provide a web service to outside consumers, I want to make my public interface as simple as possible. So if I have a method GetOvertimeHours(employeeId) it uses some other classes/services to perform its work. It has to ask the Labor service/class to give it all the labor details for that employee, then it has to ask the shift schedule service/class to give it all the scheduled shifts for that employee, then it has to do a comparison and calculate which hours are overtime.

    Well if I wanted to test the GetOvertimeHours() function in isolation, I would want to mock/stub the labor class and the shift schedule class. Dependency Injection would suggest I make the caller pass in objects to GetOvertimeHours() that represent these 2 dependencies. Something like this:
    GetOvertimeHours(employeeId, laborService, scheduleService)

    But I don't want the consumers of my service to have to worry about that, or even be aware of it. Where GetOvertimeHours() receives it's data is internal to my service, and I don't want the external callers to have to know about that, much less pass them in as parameters. The idea is to keep the external interface as simple as possible.

    How do other people deal with situations like this?

  4. Anonymous1:41 PM

    I encountered the same situation when I was working on the .norm framework. The solution is to create everything correctly using Dependency Injection first. This will allow you to unit test all the pieces of the application.

    Then, build a layer on top that pulls everything together. This layer can be tested with various functional tests.

  5. Anonymous7:40 PM

    Hey Jay. Your suggestion of doing an extra layer on top of the dependency injection layer appears like it will work out great for us.

    I have run into another problem that I'm curious how other people solve. I wish to isolate my database layer so that I can test (and implement) it separately, and have my other business logic tests isolated from it. To achieve this I created a IDatabaseLayer interface, with a GetDataReader() function that accepts a stored proc name and list of parameters - it returns a SqlDataReader.

    This allows me to pass in an instance of IDatabaseLayer to my test subject, giving me the opportunity to mock it. However, how do I easily mock a class that is supposed to return a SqlDataReader object. Trying to create a hardcoded data reader from code would not be a trivial task afaik.

  6. Anonymous9:58 PM

    It's been awhile since I've looked at .net; however, I think I remember an IDataReader class. Instead of returning the concrete SqlDataReader type, use IDataReader. Then, in your tests return a stub or mock IDataReader. Use ReSharper to create the stub (implement the interface) to avoid the majority of the work.

    Hope that helps, Jay

  7. Anonymous6:51 AM

    If you check the original literature on Mocks you will see a suggestion not to mock classes/interfaces you don't own. I would suggest you look at wrapping your SQLDataReader in something you have created that serves your needs. That will truly isolate you from the datalayer. Then you can mock this wrapper interface that you have created.


Note: Only a member of this blog may post a comment.