Monday, February 23, 2015

Experience Report: Weak Code Ownership

In 2006 Martin Fowler wrote about Code Ownership. It's a quick read, I'd recommend checking it out if you've never seen it. At the time I was working at ThoughtWorks; I remember thinking "Clearly Strong makes no sense and I have no idea what scenario would make Weak reasonable". 8 years later, I find myself advocating for Weak Code Ownership within my team.

Collective Code Ownership (CCO) served me well between 2005 and 2009. Given the make-up of the teams that I was a part of I found it to be the most effective way to deliver software. Around 2009 I joined a team that eventually grew to around 9 people, all very senior developers. The team practiced Collective Code Ownership. Everyone on the team was very talented, but that didn't translate to constant agreement. In fact, we disagreed far more often than I thought we should. That experience drove me to write about the importance of Compatible Opinions. I still believe in the importance of compatible opinions, but I now wonder if the team wouldn't have been more effective (despite incompatible opinions) if we had adopted Weak Code Ownership.

The 2009 project heavily shaped my approach to developing software. I suspect I'm not the only one who (at one time) believed: if we get a team full of massively talented people we can do anything. It turns out, it's not nearly that easy. Too many cooks in the kitchen is the obvious concern, and it does come up. However, the much larger problem is that talented people work in vastly different ways. Some meticulously refactor in small steps, others make wide reaching and large changes. Some prefer one language to rule them all, others are comfortable switching between 12-15 different languages in the same day. Monolithic vs separated codebases. Inherited vs duplicated config. It goes on and on. You try to optimize for everyone, to ensure everyone is maximally effective. Pretty quickly you run into this situation-
If you optimize everything, you will always be unhappy. --Donald Knuth
Looking back, I believe our "Collective Code Ownership" degraded to "Last in Wins". People began to cluster around shared opinions as the team grew. Inevitably the components of the project splintered and work included constant angling to ensure you only worked in areas designed to match your personal style. Perhaps that wasn't such a bad thing, but never formalizing the splinters led to constant discussion around responsibility and design. Those discussions always felt like waste to me.

I eventually left that team, and I left with the feeling that as a team we'd been ignoring Diffusion of responsibility (DOR), Bystander effect (BE), and Crowd psychology (CP). In my opinion the combination of (exclusively) talented, senior developers and collective code ownership led to suboptimal productivity. The belief that "CCO works and we're just doing it wrong" made things worse. It seemed like our solution was always: we just need to do better. You could write this team off as not as talented as I describe, but you'd be mistaken. All of the developers on the team had great success before the project, and (after that team split up) 8 out of 9 have gone on to lead very successful teams.

Several years later I found myself leading a team that was beginning to grow. We had just expanded to a 4 person team, and I could see the high level problems beginning to appear. Consistency had begun slip, and bugs had begun to appear in places where code was "correct" at the micro level, but the system didn't work as expected at the macro level. There are a few ways to manage these issues, pair programming was an obvious solution to this problem, but I believe pair programming would have introduced a different class of problems. I believe switching to pair programming (exclusively) would actually have been net negative for the team. I've written about giving up on exclusive pair programming, I'll leave it at that. The issues we were facing felt vaguely familiar, and I started to wonder if they stemmed from DOR, BE, and CP. Collective code ownership allowed developers to pop in and out of codebases, making small changes to enable features, but where was the person looking at the big picture? "It's everyone's responsibility" wasn't an answer I was comfortable with.

I found myself looking for another solution that would
  • encourage consistency
  • give equal importance to macro and micro quality
  • address the other issues caused by DOR, BE, and CP
We already had our project split in a microservices style, and I proposed to the team that we move to what I called Primary Code Ownership. Primary Code Ownership can be described as-
  • A codebase has a "primary", the person who's responsible for that process in production. You could use the word "owner", but I don't think it conveys what I'm looking for. The team still owns the code, but when problems occur the responsibility falls on the primary first.
  • The primary drives the architecture of the codebase. The primary gets final say on architectural decisions within a codebase.
  • A primary may commit to master a codebase without code review.
  • Any commit to master from a non-primary must come via a Pull Request, and only primaries can merge (or give permission for the non-primary to merge).
  • note: in emergency circumstances, anyone can commit to any codebase.
  • Primaries can change at any time. If a primary feels that another team member is better suited to become the primary they can propose a switch.
  • Rotation of primary responsibility occurs naturally as people contribute to different codebases. If you seek to become a primary of a codebase, all you need to do is focus your efforts on features that require changing said codebase. If you're doing the majority of the work in that codebase, you should become the primary in a reasonable amount of time.
The motivation for this approach is that a primary will commit keeping their vision in mind. Non-primaries can continue to commit to the same codebase, and are given the confidence that if they make a consistency or macro mistake the primary will catch it during Pull Request review.

We've been working with this approach for about a year now and I've been happy with the results. We've had 1 major bug in 12 months and our architecture has remained consistent despite losing 1 team member and gaining 2 more (team size is 5 now). It's probably worth mentioning that my team is entirely remote as well, making those results even more impressive (imo).

A nice side effect of this approach is eased on-boarding of new team-members. After an initial stretch of about a month of co-located pair programming, a new team member is free to work on whatever they want. They can work knowing that their changes will be reviewed by someone with deep understanding of the necessary changes. The rest of the team is confident that the newbie's changes shouldn't cause prod issues, and wont be merged until they're consistent with existing conventions. Another nice side effect of this process (pointed out by Dan Bodart) is that some people are more inclined to polish their code if they know it's going to be featured in a pull request.

It's not (and never is) all roses. Primaries do complain at times about the context switches required to merge pull requests. I haven't found a solution to this issue, yet. In the beginning people complained about having to create branches and create pull requests. However, as the team got comfortable with this workflow, the pain seemed to disappear. Pull requests are often fairly small; the issues do not mirror the complaints against feature branches.

While reviewing this entry Jake McCrary and others pointed out that while context switching is annoying, there are many benefits that also come from the pull request review process. I agree with this observation, and I'm always pleased when I see discussion (comments) occur on a pull request. Perhaps I'm over optimistic, but I always see the discussion as evidence that the team is learning from each other and we are further advancing on our shared goals.

12 months in, I'm happy with the results.

Unsurprisingly, there are other people using and/or experimenting with similar approaches. Paul Nasrat noted the similarities with this approach and the maintainer model of Linux; Scott Robinson pointed me at OWNERS Files; Romily Cocking noted the similarities with the underlying model used in Envy/Developer (pre Java).