The High-Level Test Whisperer

Most teams have High-Level Tests in what they call Functional Tests, Integration Tests, End-to-End Tests, Smoke Tests, User Tests, or something similar. These tests are designed to exercise as much of the application as possible.

I'm a fan of high-level tests; however, back in 2009 I decided on what I considered to be a sweet-spot for high-level testing: a dozen or less. The thing about high-level tests is that they are complicated and highly fragile. It's not uncommon for an unrelated change to break an entire suite of high-level tests. Truthfully, anything related to high-level testing always comes with an implicit "here be dragons".

I've been responsible for my fair share of authoring high-level tests. Despite my best efforts, I've never found a way to write high-level tests that aren't filled with subtle and complicated tweaks. That level of complication only equals heartbreak for teammates that are less familiar with the high-level tests. The issues often arise from concurrency, stubbing external resources, configuration properties, internal state exposure and manipulation, 3rd party components, and everything else that is required to test your application under production-like circumstances.

To make things worse, these tests are your last line of defense. Most issues that these tests would catch are caught by a lower-level test that is better designed to pin-point where the issue originates. The last straw - the vast majority of the times that the tests are broken it is due to the test infrastructure (not an actual flaw in the application) and it takes a significant amount of time to figure out how to fix the infrastructure.

I've thrown away my fair share of high-level tests. Entire suites. Unmaintainable and constantly broken tests due to false negatives simply don't carry their weight. On the other hand I've found plenty of success using high-level tests I've written. For awhile I thought my success with high-level tests came from a combination of my dedication to making them as easy as possible to work with and that I never allowed more than a dozen of them.

I joined a new team back in February. My new team has a bunch of high-level tests - about 50 of them. Consider me concerned. Not long after I started adding new features did I need to dig into the high-level test infrastructure. It's very complicated. Consider me skeptical. Over the next few months I kept reevaluating whether or not I thought they were worth the amount of effort I was putting into them. I polled a few teammates to get their happiness level. After 5 months of working with them I began my attack.

Each time my functional tests broke I spent no more than 5 minutes on my own looking for an obvious issue. If I couldn't find the problem I interrupted Mike, the guy who wrote the majority of the tests and the infrastructure. More often than not Mike was able to tweak the tests quickly and we both moved on. I anticipated that Mike would be able to fix all high-level test related issues relatively quickly; however, I expected he would grow tired of the effort and we would seek a smaller and more manageable high-level test suite.

A few more weeks passed with Mike happily fielding all my high-level test issues. This result started to feel familiar, I had played the exact same role on previous projects. I realized the reason that I had been successful with high-level tests that I had written was likely solely due to the fact that I had written them. The complexity of high-level test infrastructure almost ensures that an individual will be the expert and making changes to that infrastructure are as simple as moving a few variables in their world. On my new team, Mike was that expert. I began calling Mike the High-Level Test Whisperer.

At previous points in my career I might have been revolted by the idea of having an area of the code that required an expert. However, having played that role several times I'm pretty comfortable with the associated risks. Instead of fighting the current state of affairs I decided to embrace the situation. Not only do I grab Mike when any issues arise with the high-level tests, I also ask him to write me failing tests when new ones are appropriate for features I'm working on. It takes him a few minutes to whip up a few scenarios and check them in (commented out). Then I have failing tests that I can work with while implementing a few new features. We get the benefits of high-level tests without the pain.

Obviously this set-up only works if both parties are okay with the division of responsibility. Luckily, Mike and I are both happy with our roles given the existing high-level test suite. In addition, we've added 2 more processes (a 50% increase) since I joined. For both of those processes I've created the high-level tests and the associated infrastructure - and I also deal with any associated maintenance tasks.

This is a specific case of a more general pattern I've been observing recently: if the cost of keeping 2 people educated on a piece of technology is higher than the benefit, don't do it - bus risk be damned.

Individuals Over People

I've been pondering a few different ideas lately that all center around a common theme: to be maximally effective you need to identify and allow people to focus on their strengths.

I hear you: thanks Captain Obvious.

If you are reading this it's likely that you're familiar with the phrase "Individuals and interactions over processes and tools" from the Agile Manifesto. I'm sure we agree in principle, but I'm not sure we're talking about the same thing. In fact, it's more common to hear "people over process" when discussing Agile, which I believe is more appropriate for describing the value that Agile brings.

Agile emphasizes people (as a group) over processes and tools. However, there's little room for "individuals" on the Agile teams I've been a part of. I can provide several anecdotes -

• When pair-programming with someone who prefers a Dvorak layout a compromise must be made.
• When pair-programming with someone who prefers a different IDE a compromise must be made.
• Collective code ownership implies anyone can work on anything, which often leads to inefficient story selection. (e.g. the business accidentally gives a card to someone who isn't ideally skilled for the task. Or, a developer decides to work on a card they aren't ideally skilled for despite other better suited and equally important outstanding cards)
• Collective code ownership requires a lowest common denominator technology selection. (e.g. If 3 out of 5 people know Ruby and 5 out of 5 know Clojure, selecting Ruby for any application, even when it's the appropriate choice, is likely to be met by resistance)
• Collective code ownership requires a lowest common denominator coding style selection. Let's be honest, it's easy to code in a language such as Ruby without a deep understanding of metaprogramming and evaluation. Both metaprogramming and evaluation are powerful; however, you can only take advantage of that power if you are sure everyone on the team is comfortable with both techniques.

I could ramble on a bit more, but hopefully you get my point.

I'm still a believer in Agile. It's the best way I know how to take an average performing team and put them on a path to becoming a well performing team. However, I think the Agile practices also put a ceiling on how effective a team can be. Perhaps my favorite anecdote: Ola Bini believes he is 10x faster when using Emacs as compared to IntelliJ - when writing Java! 10x is huge, so what does he use when he's pairing: IntelliJ. If there's knowledge transfer occurring then perhaps the 10x reduction in delivery speed is a good decision; however, if he's pairing with someone of a similar skill level who isn't statistically likely to maintain the code in the future it's a terrible choice. Ola is programming at 1/10 of his possible efficiency for no reason other than it's the Agile way.

Clearly, it's gray area - the knowledge transfer level will vary drastically based on who he's pairing with and what they are pairing on. That's the important point - people are more important than processes, but individuals are the most important. If you want to achieve maximum productivity you'll need to constantly reevaluate what the most effective path is based on the individuals that make up your team.

If you already agree with the ideas above then you're probably familiar with the idea that you need to learn all the rules to know when to break them. That's an old idea as well. Unfortunately, I don't see much written on this topic in the software development world. The last 3 years of my life have been lesson after lesson of how smart people can break the Agile rules and get much larger gains as a result. I've decided to tag posts that cover this subject as "Individuals over People". There are even a few historical entires available at http://blog.jayfields.com/search/label/individuals%20over%20people.

Hopefully these ideas will spark a few discussions and inspire other post-Agile developers to post their experiences as well.

Random Thoughts on Good Programmers and Good Code

I've know some really, really smart guys who write code I hate. I love using their libraries, but I don't ever want to edit their source. To me, their source is too complicated. I used to think that they were smarter than I was and I benefited by having to write clearer code, as I couldn't easily digest their complicated code.

But, I've also run into code I've written that is complicated in it's own ways. I like to find the dark corners of languages and exploit them. To me, 8 different classes being configured 10 different ways and used in 3 different situations is something I can never get comfortable with. However, metaprogramming & macros don't scare me at all, and I often (over) embrace them.

And, to me, that's the heart of the issue. Each of us have a comfort level with different styles of development. I like to write as little as possible, and embrace "magic" that allows me to write less. Other people are more comfortable with more code, as long as it follows patterns that are easy to follow through the system. We could decide which is better for "average" programmers, but I don't think there's a "better" approach for good programmers who have tried both and stick with what they prefer; they know to play to their strengths.

This also helps explain why all code we haven't written "sucks" and all code we write is "awesome"

As an example, I think test names are worthless because I don't use them. Dan North loves test names and uses them to help him understand the responsibility of code. It's as simple as a different approach for digesting code, but the result is assigning a value to an artifact. I can write all I want about how test names are worthless, but it's not going to change the fact that they are helpful to Dan. He and I could (ignorantly) argue till the end of time about the value of test names, but the actual discussion is in how we digest code.

These days, the Java, Ruby, Clojure, Patterns, Metaprogramming, Static, Dynamic, etc discussions all feel this way to me. You're either an idiot who doesn't understand what you're arguing against, or you're talented enough that you found what makes you the most productive. Either way, your opinions don't necessarily apply to anyone else.

That said, I think there is value in presenting new ideas (or restating old ideas in new ways) as it can inspire people to reevaluate their assumptions. The ideas that tend to be most helpful for me are the ones presented as experience reports and discussions on the benefits of a particular solution. The ideas that are presented as an X vs Y smackdown always tend to make me wonder if someone is still searching for a silver bullet.

Compatible Opinions on Software

I'm switching teams this month. I'll be working out of the DRW New York City office in the near future. I've been with my current team for almost 2 years, and I've learned plenty of valuable lessons. I'm not sure if the most valuable take-away centers around Compatible Opinions on Software; however, it's definitely the take-away that seems to remain in the fore-front of my memories.

The guys I work with in the DRW CT office are truly talented. Each individual on my team is someone I would consider to be a great software developer. As a result, we don't tend to have discussions on issues that where one answer is clearly right and another is clearly wrong. Instead, we often find ourselves in situations where one or more individuals have differing opinions and a "correct" answer cannot be found. Despite everyone being an excellent developer, several members of the team have incompatible opinions on software.

Here's the net result: Quality software. We aren't talking about a problem that caused projects to fail. On the contrary, our customer is very happy and the team is largely happy with the product we deliver. Unfortunately, as a teammate puts it, I seem to be immune to complacency. Even though things are good, I wont be satisfied until they are great. And, I feel that incompatible opinions on software is our largest production thief.

In general, it seems that anything can cause incompatible opinions. However, to make clear what I'm talking about, here is a short sample list of incompatible opinions I've run into:

• powerful language vs. powerful IDE
• monolithic application vs many small components
• what level of duplication is acceptable, and why
• specialization vs generalization
• team size

For the majority of my career I've believed - if I work on a team of excellent developers everything will simply fall into place. I now believe that being technically excellent is only the first requirement. I also believe that you can have two technically excellent people who have vastly different opinions on the most effective way to deliver software.

Two developers who prefer a powerful language may or may not be more productive than two developers who prefer a powerful IDE. However, both of the previous pairs will be more productive than one developer who prefers a powerful language teamed up with one developer who prefers a powerful IDE.

I'm not talking about people who love IntelliJ but are perfectly willing to learn emacs. If you're open to working in emacs then you have a compatible opinion with an emacs zealot. I'm talking about people who have tried the opposing point of view and still have a strong opinion that it's not the best way to deliver software. Putting those people on the same team only ensures a never-ending battle about why things aren't as good as they could be.

Of course, having differing views can be a good thing. Diversity ensures all kinds of benefits. I'm not suggesting that you rid yourself of all diversity. However, for the sake of productivity, I suggest that you do your best to avoid working with someone who has both an opposing view and is as inflexible as you are on the subject. The more central the subject is to the project, the more likely it is that productivity will be lost.

Let's get back to a concrete example. For me personally, I strongly prefer Clojure to Java. That's not to say that Java doesn't have it's place, it does. But, if both languages are valid choices, I'll always opt for Clojure. Given that opinion, I'm more effective if I'm working with someone else that prefers Clojure to Java. The language decision is important to me, very central to the project, and likely to be an ongoing issue.

Building off of the previous example, I don't personally care if we use IntelliJ or Emacs. As long as the IDE or editor choice is a reasonable one, I'm happy to use whatever someone else on the team prefers. I welcome diversity on compatible opinions. In fact, it's likely that diversity on compatible opinions will lead to greater productivity as better solutions are found.

So what can you do? The obvious answers: I've started making a list of opinions that I'm less flexible about. If I'm interviewing for a teammate, I'll ask questions to ensure we have compatible opinions on software. If I'm interviewing for a new job, I'll ask the interviewers about their opinions on software. To a certain extent I've always asked these questions, but I think I'll have better results if I identify which of my opinions I'm less flexible about and focus a bit more on ensuring that there aren't any conflicts.

If you already have a team in place and aren't looking to leave it's a bit harder to resolve incompatible opinions on software. Depending on the size of your team, you may be able to split the team up into smaller groups of people who have compatible opinions on software. Obviously, this will depend on having enough people to allow for a split, and having software that can be logically divided. If you can pull this off, you also get the nice benefit of seeing the results from doing things differently, without having to change your approach until it makes sense for you.

At this point I've worked on three different teams that achieved a level of productivity greater than that of any other teams I've been a part of. Of course, there are several factors that contributed to the success of all three teams. However, all three teams had compatible opinions on software, and avoided an entire class of problems that erode productivity. You'd have a hard time convincing me that those two things weren't somehow related.

Taking a Second Look at Collective Code Ownership

It's common to hear proponents of Agile discussing the benefits of collective code ownership. The benefits can be undeniable. Sharing knowledge ensures at least one other perspective and drastically reduces Bus Risk. However, sharing comes at a cost: time. The ROI of sharing with a few people can be greatly different than the ROI of sharing with 5 or more people.

I do believe in the benefits of collective code ownership. Collective code ownership is a step towards taking an underachieving team and turning them into a good team. However, I'm becoming more and more convinced that it's not the way to take good team and make them great.

(context: I believe these ideas apply to teams larger than 3. Teams of 2-3 should likely be working with everyone on everything.)

If you pair program, you will incur a context switch every time you rotate pairs. Context switches always have a non-zero cost, and the more you work on, the larger the context switch is likely to be. Teams where everyone works on everything are very likely paying for expensive context switches on a regular basis. In fact, the words Context Switch are often used specifically to point out the cost.

Working on everything also ensures a non-trivial amount of time ramping up on whatever code you are about to start working on. Sometimes you work on something you know fairly well. Other times you're working on something you've never seen before. Working on something you've never seen before creates two choices (assuming pair-programming): go along for the ride, understanding little - or - slow your pair down significantly while they explain what's going on.

Let's say you choose to slow your pair down for the full explanation: was it worth it? If you're the only other person that knows the component, it's very likely that it was worth it. What if everyone else on the team already knows that component deeply? Well, if they all die, you can maintain the app, but I don't think that's going to be the largest issue on the team.

(the same ideas apply if you don't pair-program, except you don't have the "go along for the ride" option)

I can hear some of you right now: We rotate enough that the context switch is virtually free and because we rotate so much there's little ramp up time. You might be right. Your problem domain might be so simple that jumping on and off of parts of your system is virtually free. However, if you're domain is complex in anyway, I think you're underestimating the cost of context switches and ramp up time. Also, the devil is traditionally in the details, so you're "simple domain" probably isn't as simple as you think.

Let's assume your domain is that simple: it might be cheaper to rewrite the software than take your Bus Number from 3 to 4.

Another benefit of pair programming combined with collective code ownership is bringing up everyone's skill level to that of the most skilled team member. In my opinion, that's something you need to worry about on an underachieving team, not a good team. If you're on a good team, it's likely that you can learn just as much from any member of your team; therefore, you are not losing anything by sticking to working with just a few of them in specific areas. You really only run into an education problem if you're team has more learners than mentors - and, in that case, you're not ready to worry about going from good to great.

There's also opportunity cost of not sticking to certain areas. Focusing on a problem allows you to create better solutions. Specifically, it allows you to create a vision of what needs to be done, work towards that vision and constantly revise where necessary.

Mark Twain once wrote that his letters would be shorter if he had more time. The same is often true of software. The simplest solution is not always the most obvious. If you're jumping from problem to problem, you're more likely to create an inferior solution. You'll solve problems, but you'll be creating higher maintenance costs for the project in the long term.

Instead, I often find it very helpful to ponder a problem and create a more simple and, very often, a more concise solution. In my experience, the maintenance costs are also greatly reduced by the simplified, condensed solution.

I'd like to repeat for clarity: Collective code ownership has benefits. There's no doubt that it is better to have everyone work on everything than have everyone focused on individual parts of the codebase. However, it's worth considering the cost of complete sharing if you are trying to move from good to great.

Pairing isn't the Solution

In 2007 & 2008 I wrote several blog entires on Pair Programming (tagged with pair programming). Pair programming solved a lot of problems for me: knowledge transfer, mentoring, code review, etc. It also solved another problem at the same time, even though I wasn't aware of it. Pairing helps reduce the number of cooks in the kitchen.

These days I'm working on a project with some really talented people. The pace at which we deliver features is far faster than any project I worked on before I joined DRW. However, I've seen two effects of having so many talented developers on the same team: wiki coding and spork sharing. (both metaphors coined by my tech lead, badams)

Wiki coding occurs when the churn on a class or component is so great that whoever commits last ends up deciding what it should do. Wiki coding leads to inconsistent design and lack of convention. Spork sharing occurs when a fork is designed and a spoon is also needed. Instead of creating a spoon, you want to share the handle, so you create a spork instead. Now, you have no spoon or fork, you have a spork, and sporks suck if you really want a spoon or a fork. Both problems seem to stem from differing vision for the classes or components.

It appears that the more talented programmers you put on a team, the more fractured the vision for the project becomes. Of course, complexity (as well as many other factors) can also increase the fracture.

You won't catch me advocating for 'hard-working average programmers'. What I do believe is: you should stock your team with only rockstars, between 2 and 4 of them. I've worked on teams that only had 3 people. I've worked on teams with about 16. I was a consultant, I've worked on a lot of teams - big and small. My experience was that the smaller teams were much more effective, every time.

Reflecting on the past few years, I came up with the following reasons for keeping your teams small:

• New Technology: Small teams can more easily come to a decision to try a new technology. Also, if a small team selects a new technology, the entire team is likely to learn the new technology. However, an individual on a larger team may find a way to never learn the new technology and never contribute to that area of the code. Worse, a larger team may shy away from trying new technology because they cannot come to a consensus. New technology can offer productivity boosts, and missing those boosts can hurt a teams efficiency.
• Smaller Problems: Smaller teams generally implies solving smaller problems. Most problems can be broken down into smaller, less complex problems. Once the problem is broken down, it should be easier for the small teams to craft elegant solutions that can integrate to solve the larger business need.
• Improved Maintainability: Smaller teams generate less code, which allows all team members to have depth and breadth concerning a solution. Having depth and breadth allows you to easily share vision and fix broken windows.
• Ownership: Ownership isn't bad. Single points of failure are bad, but having a small team that feels ownership over an application will generally lead to higher quality and more emotional attachment to it's success.
• Context Switching: Smaller codebases maintained by small teams solving a smaller problem will context switch less, because there's nothing else to context switch to.
• Responsibility: The Bystander Effect applies to code (e.g. production bugs) as well.
• Unified Vision: In a large team it's easy to have an opposing vision and never be proven wrong. In a small team it's likely that you will agree on a vision for the project (process, technology, etc) as an entire team.
• Adding: Adding one more person to a 2 person team will likely result in a new productive team member relatively quickly. Smaller team, smaller problem, smaller codebase = easier team to join.

Sometimes I wonder if consultancies sell large teams, and then use pairing to make the large teams more effective. It's definitely my experience that large teams pairing are much more effective than large teams that aren't pairing. However, I wonder if anyone ever stopped to ask: would a smaller team have been sufficient (or, better)?

I still believe pairing is an answer to many problems. However, the best solution to making a 8 person team more effective isn't pairing. I believe that a superior solution is finding a way to solve the problem with a smaller, more effective team (or teams).