Being a Lead Consultant

These are not my ideas, they were given to me. I wanted to store them in a safe place, which is why they can be found here. Thank you for your wisdom Scott and Sean.

• Voice of authority to the client on technical matters, meaning that I'm looking for someone who can establish their own credibility with senior client staff, including non-technical managers. This means knowing what the client values and being able to communicate the important issues, not hung up on trivial issues.
• Leading developers by example. Other developers, particularly junior developers, will pick up and mimic behaviors of their lead developer in ways that may not even be obvious to either one of them. I look for the senior devs to understand this and model the behavior that want to see from their team.
• Understanding that it's not just about coding cards. This means making sure the relationship between the development team and the BAs and testers is fruitful and productive. Obviously, the PM and IM also have a role to play here but the senior developers are even more influential with other developers.
• Herding the technical team. This means being the guy to make sure the unfun cards get done, everyone responds to broken builds, and that there is a sense of signing up to and committing to work. A lot of this can come from the leadership by example but there is also a vocal component that I'm talking about here. Not being a cheerleader but willing and able to facilitate decision making in groups and address issues one-on-one as appropriate.
• Confidence-by-proxy. That is, your teams just feels safer/better/stronger when you're there, vs when you're not.
• Escalation conduit. Your team naturally pulls you into their trouble, or invites your contributions to their challenges/achievements
• Professional reflection. Your team is eager to hear your thoughts on their individual capabilities, reach.
• An interesting indicator is often the 'are they talking about what I think when I'm not there' test. That is, do they find themselves bringing your ideas/opinions/objectives into conversations, even when you're not present. Kind of a fun 'tell' of a powerful influencer/leader.

Signs That Your Project Team Might Be Too Large

Amanda Laucher recently queried the twitterverse on the topic of team size.

In my opinion, my last project team was "too large". This blog entry will focus on the things that happened that drove me to that conclusion.

note: I was the only person on the team who believed it was too large. The rest of the team thought we 'might' be too large, and the cost of making a change was greater than the pain of remaining 'potentially' too large. Since I was the minority, it's safe to assume that my opinion is controversial at a minimum.

In many ways my last project was a very, very well funded start-up. We started small, with 3 developers; however, as the scope of our problem grew so went the size of our team. When the team had reached 7 (5 developers in one office and 2 in another), I began to advocate for a team split. By the time we reached 10 total, I was positive that a split was appropriate. As I stated, we were in start-up mode. I wanted to deliver as fast as possible and I felt that several factors were impacting our ability to produce.

Camps. Technology choices originally drove my desire to split the team. The project had speed requirements that forced the use of Java for some portions of our solution; however, other portions of the codebase could have been written using Java, Clojure, or even Ruby. I held the opinion that anything that could be written in Clojure should be written in Clojure. I felt faster writing code in Clojure (vs Java) and I wanted to deliver as fast as I possibly could. Two of my colleagues agreed with my opinion, 2 abstained, and 2 strongly disagreed with me. The two that disagreed with me thought it was easier to write everything in one language, using one toolset. I completely see their point of view, and acknowledge all of their opinions. They are not wrong, we just have incompatible opinions on software.

I need to be clear here, these were good developers. It would have been easy if they were inferior developers - we would have improved their skills or found them a new home. However, that wasn't the case. These were guys that consistently delivered as much, if not more value than I did - they just wanted to get things done is a way that clashed with my approach. The team was more or less split into 2 camps - pro Clojure and pro Java.

Countless hours were spent discussing how to deal with this difference of opinions. Retrospectives were often composed of 2 types of problems: Problems that we all agreed on and solved, and problems that we would never agree on and never solved. The 2nd types of problems appeared at every retrospective and always led to the same interaction.

• someone (usually me) bitched about the lack consistency on the team
• people from my camp agreed that it was an issue, and spoke in generic terms (not wanting to target or alienate anyone).
• people from the opposing camp made their (completely valid) points.
• the room became awkwardly silent; everyone knew we were at an impass
• someone made a joke to ease tensions
• we moved on, no resolution

This situation infuriated me, but, realistically, there was never going to be a solution that made the entire team happy. Everything that was written in Clojure annoyed the pro Java camp, and anything that wasn't performance sensitive and was written in Java annoyed the pro Clojure camp. I naively believed that the pro Java guys would "see the light", and tried everything I could think of to convince them that Clojure was the right choice - and it was for me, but it (likely) never will be for them. In the end, I decided that advocating Clojure was a waste of time and focused my efforts on getting the team split by compatible opinions.

reviewer feedback

On Jun 18, 2012, Patrick Farley wrote:

I think your points about non-compatible opinions on a team, in the saddest case, can occur on a team of two. Do you suppose two distinct camps (as opposed to a bickering mob) become more likely as team size grows?

I do believe it is more likely that camps occur as the team sizes grow. I believe disagreements are healthy; however, camps are counterproductive - due to too much time spent in arbitration. I do agree that you can have camps on a team size two, and it's possible to have no camps on a team of size 10 - which is why I don't think size X will guarantee camps, but that if you have camps then X might be too many.

If your team has camps with incompatible (yet, perfectly valid) opinions then it's a sign that it's possibly too big, or it has members that would be more effectively used if swapped with someone else in the organization that holds compatible opinions. Or, more precisely, if you have camps (where camp is defined as N or more people) with conflicting opinions, and you have a solution that can be split vertically or horizontally, then your team size may be "too big" - where too big is defined as "a split would likely increase the productivity of the members of both camps". It could be possible to replace X split-away members with Y new members where (Y > X) and as long as you do not have camps, the team would not be considered too big - which is why you cannot simply state that Z people on a team is 'too big'. note: N is defined as the number of people required to reasonably address bus risk. On a highly functional team with almost no turnover, N can equal 2. More often, N will be 3 or more.

Depth Shortage. Agile Enthusiasts loved to point out the benefits of Collective Code Ownership - and they are undeniable when contrasted with Strong Code Ownership. However, I believe that Collective Code Ownership actually hindered the delivery pace of the previously referenced development team. Collective Code Ownership combined with our team size resulted in many team members having breadth of knowledge, but lacking depth of knowledge of the system.

reviewer feedback

On Jun 18, 2012, J. B. Rainsberger wrote:

Do you believe that CCO hindered the pace of delivery, or that the code base grew to a size and level of complication that made it hard to own collectively?

Definitely the latter. Our problem had grown to a size that caused CCO to go from a positive impact to a negative one. This was not due to a flaw in CCO, this was due to applying CCO to a situation where it was no longer the most effective choice.

The problem we were solving had 3rd party, distributed system, compliance, extreme performance, and algorithmic concerns that all needed attention. On the surface it didn't seem like a tough problem to solve, and yet, not a single member of the best team I've ever worked on could explain the entire problem. In short, it wasn't just complicated, it was deceptively complicated - which made matters worse. The problem was large, and we attempted to tackle it (clearly, not intentionally) by building a team that was too large.

After a year of practicing Collective Code Ownership I found that most areas of the codebase would have a primary person who deeply understood it's purpose, and almost everyone else knew very, very little about it. Those with shallow knowledge could successfully edit a method, but a change of greater size would require pairing with the primary person. The problem was large, and we were spread too thin.

The application was already split into logical components, and my solution was to move to a Weak Code Ownership model - with the requirement that each component would have at least 3 "owners" that deeply understood each component. You would still be required to know what each component did, but you wouldn't ever be required to even look at the code for that component. I believe this solution would have solved our depth issue, but so would breaking up our large team.

If your team is so large that working in every area of the codebase results in shallow knowledge of over 40% of your codebase, that might be a sign that you could benefit from splitting the team into smaller groups with more specific responsibilities.

And, just in case you are wondering, pairing isn't the solution to having a team that is too big.

I'd like to thank the following people for reviewing and providing feedback on this entry: J. B. Rainsberger, Peter Royal, Nick Drew, Mike Rettig, John Hume, Clinton Begin, Pat Kua, Nat Pryce, Patrick Farley, & Dan North.

Life After Pair Programming

When I first joined DRW I noticed that the vast majority of developers were not pair-programming. I was surprised that a company that employed so many smart people would just simply disregard a practice that I considered to be so obviously beneficial.

note: it's so easy to judge people who don't pair-program, isn't it? You can write-off their behavior for so many reasons:

• They simply haven't done it enough to see how much value it provides.
  
• They've used the wrong setup, and turned a positive ROI into a negative ROI situation in the past.
  
• They must be anti-social or not team players.
  
• They must guard their code because they are too proud or believe it's job security.

Never once did I consider that their behavior might be correct based on their context. There was no context in which pair-programming wasn't the right choice, right?

So, I did what any responsible employee who wants to help would do: I gave a presentation. I took pictures of a traditional (read: broken) pairing station setup - the setup where one employee looked over the shoulder of the person doing all the work. Then I took a picture of a dual monitor, dual keyboard, and dual mouse setup. I made the distinction between pairing and watching someone work. I explained how ping-pong-pair-programming worked. I think I said the right things, and the presentation was well received. Management was on board and offered to setup pairing stations for any developers in the company that wanted to give it a try.

A few people embraced it, but most went back to their traditional habits. I did manage to convert the guys on my team, and we paired the vast majority of the time.

Almost 3 years later, and on a different team, I basically never pair. In fact, when my lone teammate (and boss) asks if we should pair on something my initial reaction is always 'no'. The 2 of us are the only people working on the applications we support, and when we pair I often find myself very distracted and not really participating. Sometimes we ping-pong-pair and it keeps us both involved, but I don't feel like we're going faster than if I'd been working solo. Don't get me wrong, we sit directly next to each-other, talk constantly, and even look at code together at times when we're just starting to talk about a new problem. That level of collaboration feels beneficial, but actually coding the implementation together feels wasteful.

I know how I felt 3 years ago (when I joined DRW), and I know how I feel now. The fact that I've changed my tune so drastically made me think it was worth putting a few ideas on the web. I was also motivated a bit by the recent publication of Pair Programming Benefits. The Pair Programming Benefits article actually gives me a perfect starting place, by examining the benefits of pairing and how they apply to my context.

I feel like I can safely summarize the Team and System Benefits as: transfer of knowledge, superior quality, and collaboration. The Programmer Benefits can be summarized as: expansion of knowledge, truck factor reduction, superior quality, and team support. The Management/Project Management Benefits can be summarized as: aiding addition and removal of employees, knowledge transfer, and the ability to hire any skilled generalist.

So, our full list of benefits is: expansion and transfer of knowledge, superior quality, collaboration, truck factor reduction, team support, eased hiring and firing. I would agree with these benefits of pair-programming, which explains why I was such an avid believer in pairing in the past. However, if those pros don't apply to my situation then pair-programming becomes nothing more than using twice as many man hours to get the same amount of work done.

I'll start by addressing the easiest "benefits" that are effectively irrelevant for my team: eased hiring and firing and truck factor. Like I said before, my team size is 2. Mike and I are both very happy and there's almost no chance that either of us leave anytime soon. Even if we did choose to leave, we would give enough notice that someone could easily come in and replace us - and either one of us could support the applications on our own if necessary. The only real risk is that we both are killed at the same time - at which point it wouldn't matter if we paired or not. Statistically it's much more likely that only one of us would suffer something causing us to not be able to return to work, and as Chris Zuehlke (manager at DRW) once pointed out to me: if you build small, well written processes the time to understand or rewrite any software you know nothing about is small - often smaller than the cost of keeping 2 people informed on the details of the process. If I asked the stakeholders if we should use 2x as many man hours to ensure that we're covered in case one of us is suddenly killed they'd laugh at me - and I'd deserve it.

We're also not looking to add to our team, and if we decided we did want to grow I expect we wouldn't have any trouble finding someone who was looking for a well-paying job in manhattan using very cool technology.

The previous discussion was meant to address team size, but it also touches on "transfer of knowledge". Transfer of knowledge is valuable for teaching new techniques and enabling people to do maintenance. However, if little teaching is going on (often the case as Mike and I add features to an app we know fairly well, using technologies we know fairly well) and we both tend to do the maintenance on the code we previously wrote then "transfer of knowledge" is either not occurring anyway or is unnecessary. Also, on more than one occasion I've had to take a first look at some code Mike had recently written. It's always been straightforward and taken no more than an hour to dig-in, make, and test a change. Had we paired on everything he worked on I would have unquestionably spent far more than an hour here or there. Again, the ROI on knowledge transfer doesn't look good for our situation.

The "superior quality" benefit is an interesting one. I consider myself to be a good engineer who on occasion writes something beautiful. I like to share what's beautiful with everyone. This is something that's going to happen whether or not I pair (evidenced by my blogging). I know Mike does the same. I expect we're all proud of our most elegant solutions. However, more often than not I write fairly good software that isn't excitement provoking, but solves the problem at hand. I'm not sure anyone is benefiting from working with me at those moments. And, while it's possible that we'd get a slightly better solution if Mike & I paired, spending twice as many man hours for a 10% better solution doesn't seem like a great choice in the short term. In the long term those 10%s could add up, but what if I see the superior solution on my 2nd pass through and end up there while expanding a feature anyway? If I have a massive solution that might be a big cost. However, with small, focused processes I can tend to make a massive change to a process in ~4 hours. Delete is still my favorite refactoring and I do it as often as possible on my quests for superior solutions. Therefore, I don't believe those 10%s add up, I believe the 90% superior solution is often good enough or deleted.

I believe a few additional factors allow us to write code that is "good enough" on our own.

• The application is fairly mature and many of the core design decisions have already been made.
• Mike and I have Compatible Opinions on Software.
• Mike and I are at a similar skill level.

In my current situation I find that many of the tough decisions have already been made and I can follow the existing path. And, any decisions that I do need to make I usually end up choosing what Mike would choose and solving in a way that we both think is appropriate.

While I believe "expansion of knowledge" is often undervalued and under-practiced in our profession, I don't think pair-programming is the most effective way to expand your skills. Blogs, books, and presentations allow you to work at your own pace and dig in to your desired level. My experience pairing with people using tools or languages I don't know have been painful. I did learn things, but I couldn't help but feel like there were much more effective ways for me to get up to speed.

The final 2 benefits that I need to consider are "collaboration" and "team support". As I previously stated, Mike and I collaborate quite closely. We're always both happy to stop what we're doing to provide an opinion or direction. I honestly can't imagine a more collaborative environment. Basically the same thing can be said of "team support" - we constantly watch each-other's back. We're both looking to succeed and we know we need each-other to get to that goal, so it only makes sense that we look out for each other. We don't need pairing to force us to support each-other, the desire to succeed is more than enough.

When considering my context and the "benefits" of (or, in my situation, lack of benefits of) pair-programming, I'm no longer surprised that I basically never want to pair these days. Also, looking back at the other teams I was previously judging - their context back then looks a lot like my current context. With a bit more experience under my belt I have to say I'm honestly a bit embarrassed for never even considering that they already "got it" and were already making the right choice. Oh well, live and learn.

While it's easy for me to examine my situation and make a determination I think there is a more general context where these same ideas apply. The aspects of an environment where pairing may not be beneficial seem to look something like this:

• small team (< 4 people)
• small software (components/processes can be rewritten in < 2 days)
• motivated teammates
• talented and well-seasoned teammates (similar skill level, and skilled overall)
• the design of the application is fairly mature and stable
• all team members have compatible options on software

Thanks to Michael Feathers, Alexey Verkhovsky, Pat Kua, Jeremy Lightsmith, Lance Walton, Dan Worthington-Bodart, James Richardson, and John Hume for providing feedback on the initial draft of this entry.

The High-Level Test Whisperer

Most teams have High-Level Tests in what they call Functional Tests, Integration Tests, End-to-End Tests, Smoke Tests, User Tests, or something similar. These tests are designed to exercise as much of the application as possible.

I'm a fan of high-level tests; however, back in 2009 I decided on what I considered to be a sweet-spot for high-level testing: a dozen or less. The thing about high-level tests is that they are complicated and highly fragile. It's not uncommon for an unrelated change to break an entire suite of high-level tests. Truthfully, anything related to high-level testing always comes with an implicit "here be dragons".

I've been responsible for my fair share of authoring high-level tests. Despite my best efforts, I've never found a way to write high-level tests that aren't filled with subtle and complicated tweaks. That level of complication only equals heartbreak for teammates that are less familiar with the high-level tests. The issues often arise from concurrency, stubbing external resources, configuration properties, internal state exposure and manipulation, 3rd party components, and everything else that is required to test your application under production-like circumstances.

To make things worse, these tests are your last line of defense. Most issues that these tests would catch are caught by a lower-level test that is better designed to pin-point where the issue originates. The last straw - the vast majority of the times that the tests are broken it is due to the test infrastructure (not an actual flaw in the application) and it takes a significant amount of time to figure out how to fix the infrastructure.

I've thrown away my fair share of high-level tests. Entire suites. Unmaintainable and constantly broken tests due to false negatives simply don't carry their weight. On the other hand I've found plenty of success using high-level tests I've written. For awhile I thought my success with high-level tests came from a combination of my dedication to making them as easy as possible to work with and that I never allowed more than a dozen of them.

I joined a new team back in February. My new team has a bunch of high-level tests - about 50 of them. Consider me concerned. Not long after I started adding new features did I need to dig into the high-level test infrastructure. It's very complicated. Consider me skeptical. Over the next few months I kept reevaluating whether or not I thought they were worth the amount of effort I was putting into them. I polled a few teammates to get their happiness level. After 5 months of working with them I began my attack.

Each time my functional tests broke I spent no more than 5 minutes on my own looking for an obvious issue. If I couldn't find the problem I interrupted Mike, the guy who wrote the majority of the tests and the infrastructure. More often than not Mike was able to tweak the tests quickly and we both moved on. I anticipated that Mike would be able to fix all high-level test related issues relatively quickly; however, I expected he would grow tired of the effort and we would seek a smaller and more manageable high-level test suite.

A few more weeks passed with Mike happily fielding all my high-level test issues. This result started to feel familiar, I had played the exact same role on previous projects. I realized the reason that I had been successful with high-level tests that I had written was likely solely due to the fact that I had written them. The complexity of high-level test infrastructure almost ensures that an individual will be the expert and making changes to that infrastructure are as simple as moving a few variables in their world. On my new team, Mike was that expert. I began calling Mike the High-Level Test Whisperer.

At previous points in my career I might have been revolted by the idea of having an area of the code that required an expert. However, having played that role several times I'm pretty comfortable with the associated risks. Instead of fighting the current state of affairs I decided to embrace the situation. Not only do I grab Mike when any issues arise with the high-level tests, I also ask him to write me failing tests when new ones are appropriate for features I'm working on. It takes him a few minutes to whip up a few scenarios and check them in (commented out). Then I have failing tests that I can work with while implementing a few new features. We get the benefits of high-level tests without the pain.

Obviously this set-up only works if both parties are okay with the division of responsibility. Luckily, Mike and I are both happy with our roles given the existing high-level test suite. In addition, we've added 2 more processes (a 50% increase) since I joined. For both of those processes I've created the high-level tests and the associated infrastructure - and I also deal with any associated maintenance tasks.

This is a specific case of a more general pattern I've been observing recently: if the cost of keeping 2 people educated on a piece of technology is higher than the benefit, don't do it - bus risk be damned.

Individuals Over People

I've been pondering a few different ideas lately that all center around a common theme: to be maximally effective you need to identify and allow people to focus on their strengths.

I hear you: thanks Captain Obvious.

If you are reading this it's likely that you're familiar with the phrase "Individuals and interactions over processes and tools" from the Agile Manifesto. I'm sure we agree in principle, but I'm not sure we're talking about the same thing. In fact, it's more common to hear "people over process" when discussing Agile, which I believe is more appropriate for describing the value that Agile brings.

Agile emphasizes people (as a group) over processes and tools. However, there's little room for "individuals" on the Agile teams I've been a part of. I can provide several anecdotes -

• When pair-programming with someone who prefers a Dvorak layout a compromise must be made.
• When pair-programming with someone who prefers a different IDE a compromise must be made.
• Collective code ownership implies anyone can work on anything, which often leads to inefficient story selection. (e.g. the business accidentally gives a card to someone who isn't ideally skilled for the task. Or, a developer decides to work on a card they aren't ideally skilled for despite other better suited and equally important outstanding cards)
• Collective code ownership requires a lowest common denominator technology selection. (e.g. If 3 out of 5 people know Ruby and 5 out of 5 know Clojure, selecting Ruby for any application, even when it's the appropriate choice, is likely to be met by resistance)
• Collective code ownership requires a lowest common denominator coding style selection. Let's be honest, it's easy to code in a language such as Ruby without a deep understanding of metaprogramming and evaluation. Both metaprogramming and evaluation are powerful; however, you can only take advantage of that power if you are sure everyone on the team is comfortable with both techniques.

I could ramble on a bit more, but hopefully you get my point.

I'm still a believer in Agile. It's the best way I know how to take an average performing team and put them on a path to becoming a well performing team. However, I think the Agile practices also put a ceiling on how effective a team can be. Perhaps my favorite anecdote: Ola Bini believes he is 10x faster when using Emacs as compared to IntelliJ - when writing Java! 10x is huge, so what does he use when he's pairing: IntelliJ. If there's knowledge transfer occurring then perhaps the 10x reduction in delivery speed is a good decision; however, if he's pairing with someone of a similar skill level who isn't statistically likely to maintain the code in the future it's a terrible choice. Ola is programming at 1/10 of his possible efficiency for no reason other than it's the Agile way.

Clearly, it's gray area - the knowledge transfer level will vary drastically based on who he's pairing with and what they are pairing on. That's the important point - people are more important than processes, but individuals are the most important. If you want to achieve maximum productivity you'll need to constantly reevaluate what the most effective path is based on the individuals that make up your team.

If you already agree with the ideas above then you're probably familiar with the idea that you need to learn all the rules to know when to break them. That's an old idea as well. Unfortunately, I don't see much written on this topic in the software development world. The last 3 years of my life have been lesson after lesson of how smart people can break the Agile rules and get much larger gains as a result. I've decided to tag posts that cover this subject as "Individuals over People". There are even a few historical entires available at http://blog.jayfields.com/search/label/individuals%20over%20people.

Hopefully these ideas will spark a few discussions and inspire other post-Agile developers to post their experiences as well.

Random Thoughts on Good Programmers and Good Code

I've know some really, really smart guys who write code I hate. I love using their libraries, but I don't ever want to edit their source. To me, their source is too complicated. I used to think that they were smarter than I was and I benefited by having to write clearer code, as I couldn't easily digest their complicated code.

But, I've also run into code I've written that is complicated in it's own ways. I like to find the dark corners of languages and exploit them. To me, 8 different classes being configured 10 different ways and used in 3 different situations is something I can never get comfortable with. However, metaprogramming & macros don't scare me at all, and I often (over) embrace them.

And, to me, that's the heart of the issue. Each of us have a comfort level with different styles of development. I like to write as little as possible, and embrace "magic" that allows me to write less. Other people are more comfortable with more code, as long as it follows patterns that are easy to follow through the system. We could decide which is better for "average" programmers, but I don't think there's a "better" approach for good programmers who have tried both and stick with what they prefer; they know to play to their strengths.

This also helps explain why all code we haven't written "sucks" and all code we write is "awesome"

As an example, I think test names are worthless because I don't use them. Dan North loves test names and uses them to help him understand the responsibility of code. It's as simple as a different approach for digesting code, but the result is assigning a value to an artifact. I can write all I want about how test names are worthless, but it's not going to change the fact that they are helpful to Dan. He and I could (ignorantly) argue till the end of time about the value of test names, but the actual discussion is in how we digest code.

These days, the Java, Ruby, Clojure, Patterns, Metaprogramming, Static, Dynamic, etc discussions all feel this way to me. You're either an idiot who doesn't understand what you're arguing against, or you're talented enough that you found what makes you the most productive. Either way, your opinions don't necessarily apply to anyone else.

That said, I think there is value in presenting new ideas (or restating old ideas in new ways) as it can inspire people to reevaluate their assumptions. The ideas that tend to be most helpful for me are the ones presented as experience reports and discussions on the benefits of a particular solution. The ideas that are presented as an X vs Y smackdown always tend to make me wonder if someone is still searching for a silver bullet.

The President and the CEO of an Application

The CEO is a visionary, often leaving day-to-day operations to the President -- wikipedia

I recently switched teams. As part of the transition process, I started taking on less and less of the responsibility for things that were long-term goals. I was still very interested in helping the team succeed, but not at the risk of leaving them with something that would be harder to maintain.

So, I found myself sitting with the users and looking for the lowest-risk, highest-impact features that I could work on for my final days. This seemed like the responsible thing to do for the long-term health of the project, but an interesting side-effect was the unexpected impact to the short-term success of the project.

At first I started grabbing the story-cards that fit this criteria, and I worked my way from the highest prioritized on down. As you can imagine, I moved down the list quickly - many of the cards were sufficiently complicated that working on them would have created maintenance risk. The users were pleased with the quick additions of highly prioritized features; however, as I moved down the list their happiness became very erratic. Some features were still very appreciated; while others were met largely with ambivalence. It became obvious that I needed to adapt the plan.

I stopped working in priority order. In fact, I stopped working from the story cards entirely. I sat down next to the users twice a day, for extended periods of time, listened to anything they had to say, and I watched the way the application was used. I started making tweaks to the application based on their complaints and any wasted effort that I witnessed. I looked for "every time I do A, then I do B". Sometimes the interaction was complicated and I couldn't automate it away. However, there were non-trivial amounts of tasks that I entirely eliminated.

The users were great about providing feedback. There was some feedback that I had to dismiss, because it wasn't something I could fix safely (in the short-term); however, a large amount of their complaints took 5-20 minutes to address. Those issues were nagging and annoying, but not (individually) more important than the long term goals of the project. But, as I began to quickly address the issues the application became more polished. It went from a good application, to an application that was surprisingly nice to work with. One of the most demanding users I've ever worked for sent me a few different emails letting me know how great the app was and how much he appreciated my effort. I was shocked.

Delivering all of these little, helpful features so greatly impacted the happiness and effectiveness of the users that I had to take a step back and see if this was something I could reproduce on future projects. The first thing that came to mind was the relationship between a CEO and a President. As the quote above says, The CEO is a visionary, often leaving day-to-day operations to the President. This idea seemed similar to what occurred in my final days. I was adding features that made day-to-day activities more efficient, and at the same time a pair was working on the features that solved our long term goals. The application was moving in the right direction for the long-term, while still experiencing short bursts of improvement in the short-term as well.

It wasn't unfiltered awesomeness. I delivered a few things that addressed an immediate problem, but not in a way that satisfied the users' needs. I wasn't working off story cards or deep conversations. I was often speculating on what I thought the application needed most. Sometimes I was wrong. However, each failure occurred quickly, and led to delivering something that the users did need. And, this cycle was complete within a few hours. (e.g. [in an hour] deliver a feature that isn't quite right, get immediate feedback, spend an hour fixing it, deliver what's really needed)

You could argue that I wasted that first hour, but spending 30 minutes with 2 users to get it right the first time would have cost the team more in the end. But, we're quickly approaching dangerous territory. Developers may like the idea of no requirements and delivering speculative features all-day, but I expect businesses would (rightly-so) object. I think a key variable in my scenario was the 2 hour failure-success cycle. If we tried this approach, it took 2 weeks, and it could have been avoided by a 30 minute discussion I think everyone would consider it a failure (again, rightly-so).

This led me to the observation that developers are more likely to correctly prioritize features that take less than a day to deliver and users are more likely to correctly prioritize features based on long-term goals. I've definitely seen this occur in two ways in the past.

• I often see users prioritize based solely on the value of the feature, even when an estimate is present.
• I have personally thought prioritizing some small features over a larger feature was the right choice, only to see that the impact of the large feature was significantly greater than I understood

This is also intuitive to me; developers are more likely to correctly estimate a smaller change and users are more likely to correctly estimate the impact of a larger change.

This experience has definitely left me wondering if teams would benefit from:

• developers prioritizing and keeping a separate backlog of smaller feature requests
• users prioritizing and keeping a separate backlog of larger feature requests
• ensuring there is always at least one person working on each backlog.

I was very pleased with the results of using this style, and I'm hoping to try it again in the future.

Compatible Opinions on Software

I'm switching teams this month. I'll be working out of the DRW New York City office in the near future. I've been with my current team for almost 2 years, and I've learned plenty of valuable lessons. I'm not sure if the most valuable take-away centers around Compatible Opinions on Software; however, it's definitely the take-away that seems to remain in the fore-front of my memories.

The guys I work with in the DRW CT office are truly talented. Each individual on my team is someone I would consider to be a great software developer. As a result, we don't tend to have discussions on issues that where one answer is clearly right and another is clearly wrong. Instead, we often find ourselves in situations where one or more individuals have differing opinions and a "correct" answer cannot be found. Despite everyone being an excellent developer, several members of the team have incompatible opinions on software.

Here's the net result: Quality software. We aren't talking about a problem that caused projects to fail. On the contrary, our customer is very happy and the team is largely happy with the product we deliver. Unfortunately, as a teammate puts it, I seem to be immune to complacency. Even though things are good, I wont be satisfied until they are great. And, I feel that incompatible opinions on software is our largest production thief.

In general, it seems that anything can cause incompatible opinions. However, to make clear what I'm talking about, here is a short sample list of incompatible opinions I've run into:

• powerful language vs. powerful IDE
• monolithic application vs many small components
• what level of duplication is acceptable, and why
• specialization vs generalization
• team size

For the majority of my career I've believed - if I work on a team of excellent developers everything will simply fall into place. I now believe that being technically excellent is only the first requirement. I also believe that you can have two technically excellent people who have vastly different opinions on the most effective way to deliver software.

Two developers who prefer a powerful language may or may not be more productive than two developers who prefer a powerful IDE. However, both of the previous pairs will be more productive than one developer who prefers a powerful language teamed up with one developer who prefers a powerful IDE.

I'm not talking about people who love IntelliJ but are perfectly willing to learn emacs. If you're open to working in emacs then you have a compatible opinion with an emacs zealot. I'm talking about people who have tried the opposing point of view and still have a strong opinion that it's not the best way to deliver software. Putting those people on the same team only ensures a never-ending battle about why things aren't as good as they could be.

Of course, having differing views can be a good thing. Diversity ensures all kinds of benefits. I'm not suggesting that you rid yourself of all diversity. However, for the sake of productivity, I suggest that you do your best to avoid working with someone who has both an opposing view and is as inflexible as you are on the subject. The more central the subject is to the project, the more likely it is that productivity will be lost.

Let's get back to a concrete example. For me personally, I strongly prefer Clojure to Java. That's not to say that Java doesn't have it's place, it does. But, if both languages are valid choices, I'll always opt for Clojure. Given that opinion, I'm more effective if I'm working with someone else that prefers Clojure to Java. The language decision is important to me, very central to the project, and likely to be an ongoing issue.

Building off of the previous example, I don't personally care if we use IntelliJ or Emacs. As long as the IDE or editor choice is a reasonable one, I'm happy to use whatever someone else on the team prefers. I welcome diversity on compatible opinions. In fact, it's likely that diversity on compatible opinions will lead to greater productivity as better solutions are found.

So what can you do? The obvious answers: I've started making a list of opinions that I'm less flexible about. If I'm interviewing for a teammate, I'll ask questions to ensure we have compatible opinions on software. If I'm interviewing for a new job, I'll ask the interviewers about their opinions on software. To a certain extent I've always asked these questions, but I think I'll have better results if I identify which of my opinions I'm less flexible about and focus a bit more on ensuring that there aren't any conflicts.

If you already have a team in place and aren't looking to leave it's a bit harder to resolve incompatible opinions on software. Depending on the size of your team, you may be able to split the team up into smaller groups of people who have compatible opinions on software. Obviously, this will depend on having enough people to allow for a split, and having software that can be logically divided. If you can pull this off, you also get the nice benefit of seeing the results from doing things differently, without having to change your approach until it makes sense for you.

At this point I've worked on three different teams that achieved a level of productivity greater than that of any other teams I've been a part of. Of course, there are several factors that contributed to the success of all three teams. However, all three teams had compatible opinions on software, and avoided an entire class of problems that erode productivity. You'd have a hard time convincing me that those two things weren't somehow related.

Taking a Second Look at Collective Code Ownership

It's common to hear proponents of Agile discussing the benefits of collective code ownership. The benefits can be undeniable. Sharing knowledge ensures at least one other perspective and drastically reduces Bus Risk. However, sharing comes at a cost: time. The ROI of sharing with a few people can be greatly different than the ROI of sharing with 5 or more people.

I do believe in the benefits of collective code ownership. Collective code ownership is a step towards taking an underachieving team and turning them into a good team. However, I'm becoming more and more convinced that it's not the way to take good team and make them great.

(context: I believe these ideas apply to teams larger than 3. Teams of 2-3 should likely be working with everyone on everything.)

If you pair program, you will incur a context switch every time you rotate pairs. Context switches always have a non-zero cost, and the more you work on, the larger the context switch is likely to be. Teams where everyone works on everything are very likely paying for expensive context switches on a regular basis. In fact, the words Context Switch are often used specifically to point out the cost.

Working on everything also ensures a non-trivial amount of time ramping up on whatever code you are about to start working on. Sometimes you work on something you know fairly well. Other times you're working on something you've never seen before. Working on something you've never seen before creates two choices (assuming pair-programming): go along for the ride, understanding little - or - slow your pair down significantly while they explain what's going on.

Let's say you choose to slow your pair down for the full explanation: was it worth it? If you're the only other person that knows the component, it's very likely that it was worth it. What if everyone else on the team already knows that component deeply? Well, if they all die, you can maintain the app, but I don't think that's going to be the largest issue on the team.

(the same ideas apply if you don't pair-program, except you don't have the "go along for the ride" option)

I can hear some of you right now: We rotate enough that the context switch is virtually free and because we rotate so much there's little ramp up time. You might be right. Your problem domain might be so simple that jumping on and off of parts of your system is virtually free. However, if you're domain is complex in anyway, I think you're underestimating the cost of context switches and ramp up time. Also, the devil is traditionally in the details, so you're "simple domain" probably isn't as simple as you think.

Let's assume your domain is that simple: it might be cheaper to rewrite the software than take your Bus Number from 3 to 4.

Another benefit of pair programming combined with collective code ownership is bringing up everyone's skill level to that of the most skilled team member. In my opinion, that's something you need to worry about on an underachieving team, not a good team. If you're on a good team, it's likely that you can learn just as much from any member of your team; therefore, you are not losing anything by sticking to working with just a few of them in specific areas. You really only run into an education problem if you're team has more learners than mentors - and, in that case, you're not ready to worry about going from good to great.

There's also opportunity cost of not sticking to certain areas. Focusing on a problem allows you to create better solutions. Specifically, it allows you to create a vision of what needs to be done, work towards that vision and constantly revise where necessary.

Mark Twain once wrote that his letters would be shorter if he had more time. The same is often true of software. The simplest solution is not always the most obvious. If you're jumping from problem to problem, you're more likely to create an inferior solution. You'll solve problems, but you'll be creating higher maintenance costs for the project in the long term.

Instead, I often find it very helpful to ponder a problem and create a more simple and, very often, a more concise solution. In my experience, the maintenance costs are also greatly reduced by the simplified, condensed solution.

I'd like to repeat for clarity: Collective code ownership has benefits. There's no doubt that it is better to have everyone work on everything than have everyone focused on individual parts of the codebase. However, it's worth considering the cost of complete sharing if you are trying to move from good to great.

Pairing isn't the Solution

In 2007 & 2008 I wrote several blog entires on Pair Programming (tagged with pair programming). Pair programming solved a lot of problems for me: knowledge transfer, mentoring, code review, etc. It also solved another problem at the same time, even though I wasn't aware of it. Pairing helps reduce the number of cooks in the kitchen.

These days I'm working on a project with some really talented people. The pace at which we deliver features is far faster than any project I worked on before I joined DRW. However, I've seen two effects of having so many talented developers on the same team: wiki coding and spork sharing. (both metaphors coined by my tech lead, badams)

Wiki coding occurs when the churn on a class or component is so great that whoever commits last ends up deciding what it should do. Wiki coding leads to inconsistent design and lack of convention. Spork sharing occurs when a fork is designed and a spoon is also needed. Instead of creating a spoon, you want to share the handle, so you create a spork instead. Now, you have no spoon or fork, you have a spork, and sporks suck if you really want a spoon or a fork. Both problems seem to stem from differing vision for the classes or components.

It appears that the more talented programmers you put on a team, the more fractured the vision for the project becomes. Of course, complexity (as well as many other factors) can also increase the fracture.

You won't catch me advocating for 'hard-working average programmers'. What I do believe is: you should stock your team with only rockstars, between 2 and 4 of them. I've worked on teams that only had 3 people. I've worked on teams with about 16. I was a consultant, I've worked on a lot of teams - big and small. My experience was that the smaller teams were much more effective, every time.

Reflecting on the past few years, I came up with the following reasons for keeping your teams small:

• New Technology: Small teams can more easily come to a decision to try a new technology. Also, if a small team selects a new technology, the entire team is likely to learn the new technology. However, an individual on a larger team may find a way to never learn the new technology and never contribute to that area of the code. Worse, a larger team may shy away from trying new technology because they cannot come to a consensus. New technology can offer productivity boosts, and missing those boosts can hurt a teams efficiency.
• Smaller Problems: Smaller teams generally implies solving smaller problems. Most problems can be broken down into smaller, less complex problems. Once the problem is broken down, it should be easier for the small teams to craft elegant solutions that can integrate to solve the larger business need.
• Improved Maintainability: Smaller teams generate less code, which allows all team members to have depth and breadth concerning a solution. Having depth and breadth allows you to easily share vision and fix broken windows.
• Ownership: Ownership isn't bad. Single points of failure are bad, but having a small team that feels ownership over an application will generally lead to higher quality and more emotional attachment to it's success.
• Context Switching: Smaller codebases maintained by small teams solving a smaller problem will context switch less, because there's nothing else to context switch to.
• Responsibility: The Bystander Effect applies to code (e.g. production bugs) as well.
• Unified Vision: In a large team it's easy to have an opposing vision and never be proven wrong. In a small team it's likely that you will agree on a vision for the project (process, technology, etc) as an entire team.
• Adding: Adding one more person to a 2 person team will likely result in a new productive team member relatively quickly. Smaller team, smaller problem, smaller codebase = easier team to join.

Sometimes I wonder if consultancies sell large teams, and then use pairing to make the large teams more effective. It's definitely my experience that large teams pairing are much more effective than large teams that aren't pairing. However, I wonder if anyone ever stopped to ask: would a smaller team have been sufficient (or, better)?

I still believe pairing is an answer to many problems. However, the best solution to making a 8 person team more effective isn't pairing. I believe that a superior solution is finding a way to solve the problem with a smaller, more effective team (or teams).