Monday, February 23, 2015

Experience Report: Weak Code Ownership

In 2006 Martin Fowler wrote about Code Ownership. It's a quick read, I'd recommend checking it out if you've never seen it. At the time I was working at ThoughtWorks; I remember thinking "Clearly Strong makes no sense and I have no idea what scenario would make Weak reasonable". 8 years later, I find myself advocating for Weak Code Ownership within my team.

Collective Code Ownership (CCO) served me well between 2005 and 2009. Given the make-up of the teams that I was a part of I found it to be the most effective way to deliver software. Around 2009 I joined a team that eventually grew to around 9 people, all very senior developers. The team practiced Collective Code Ownership. Everyone on the team was very talented, but that didn't translate to constant agreement. In fact, we disagreed far more often than I thought we should. That experience drove me to write about the importance of Compatible Opinions. I still believe in the importance of compatible opinions, but I now wonder if the team wouldn't have been more effective (despite incompatible opinions) if we had adopted Weak Code Ownership.

The 2009 project heavily shaped my approach to developing software. I suspect I'm not the only one who (at one time) believed: if we get a team full of massively talented people we can do anything. It turns out, it's not nearly that easy. Too many cooks in the kitchen is the obvious concern, and it does come up. However, the much larger problem is that talented people work in vastly different ways. Some meticulously refactor in small steps, others make wide reaching and large changes. Some prefer one language to rule them all, others are comfortable switching between 12-15 different languages in the same day. Monolithic vs separated codebases. Inherited vs duplicated config. It goes on and on. You try to optimize for everyone, to ensure everyone is maximally effective. Pretty quickly you run into this situation-
If you optimize everything, you will always be unhappy. --Donald Knuth
Looking back, I believe our "Collective Code Ownership" degraded to "Last in Wins". People began to cluster around shared opinions as the team grew. Inevitably the components of the project splintered and work included constant angling to ensure you only worked in areas designed to match your personal style. Perhaps that wasn't such a bad thing, but never formalizing the splinters led to constant discussion around responsibility and design. Those discussions always felt like waste to me.

I eventually left that team, and I left with the feeling that as a team we'd been ignoring Diffusion of responsibility (DOR), Bystander effect (BE), and Crowd psychology (CP). In my opinion the combination of (exclusively) talented, senior developers and collective code ownership led to suboptimal productivity. The belief that "CCO works and we're just doing it wrong" made things worse. It seemed like our solution was always: we just need to do better. You could write this team off as not as talented as I describe, but you'd be mistaken. All of the developers on the team had great success before the project, and (after that team split up) 8 out of 9 have gone on to lead very successful teams.

Several years later I found myself leading a team that was beginning to grow. We had just expanded to a 4 person team, and I could see the high level problems beginning to appear. Consistency had begun slip, and bugs had begun to appear in places where code was "correct" at the micro level, but the system didn't work as expected at the macro level. There are a few ways to manage these issues, pair programming was an obvious solution to this problem, but I believe pair programming would have introduced a different class of problems. I believe switching to pair programming (exclusively) would actually have been net negative for the team. I've written about giving up on exclusive pair programming, I'll leave it at that. The issues we were facing felt vaguely familiar, and I started to wonder if they stemmed from DOR, BE, and CP. Collective code ownership allowed developers to pop in and out of codebases, making small changes to enable features, but where was the person looking at the big picture? "It's everyone's responsibility" wasn't an answer I was comfortable with.

I found myself looking for another solution that would
  • encourage consistency
  • give equal importance to macro and micro quality
  • address the other issues caused by DOR, BE, and CP
We already had our project split in a microservices style, and I proposed to the team that we move to what I called Primary Code Ownership. Primary Code Ownership can be described as-
  • A codebase has a "primary", the person who's responsible for that process in production. You could use the word "owner", but I don't think it conveys what I'm looking for. The team still owns the code, but when problems occur the responsibility falls on the primary first.
  • The primary drives the architecture of the codebase. The primary gets final say on architectural decisions within a codebase.
  • A primary may commit to master a codebase without code review.
  • Any commit to master from a non-primary must come via a Pull Request, and only primaries can merge (or give permission for the non-primary to merge).
  • note: in emergency circumstances, anyone can commit to any codebase.
  • Primaries can change at any time. If a primary feels that another team member is better suited to become the primary they can propose a switch.
  • Rotation of primary responsibility occurs naturally as people contribute to different codebases. If you seek to become a primary of a codebase, all you need to do is focus your efforts on features that require changing said codebase. If you're doing the majority of the work in that codebase, you should become the primary in a reasonable amount of time.
The motivation for this approach is that a primary will commit keeping their vision in mind. Non-primaries can continue to commit to the same codebase, and are given the confidence that if they make a consistency or macro mistake the primary will catch it during Pull Request review.

We've been working with this approach for about a year now and I've been happy with the results. We've had 1 major bug in 12 months and our architecture has remained consistent despite losing 1 team member and gaining 2 more (team size is 5 now). It's probably worth mentioning that my team is entirely remote as well, making those results even more impressive (imo).

A nice side effect of this approach is eased on-boarding of new team-members. After an initial stretch of about a month of co-located pair programming, a new team member is free to work on whatever they want. They can work knowing that their changes will be reviewed by someone with deep understanding of the necessary changes. The rest of the team is confident that the newbie's changes shouldn't cause prod issues, and wont be merged until they're consistent with existing conventions. Another nice side effect of this process (pointed out by Dan Bodart) is that some people are more inclined to polish their code if they know it's going to be featured in a pull request.

It's not (and never is) all roses. Primaries do complain at times about the context switches required to merge pull requests. I haven't found a solution to this issue, yet. In the beginning people complained about having to create branches and create pull requests. However, as the team got comfortable with this workflow, the pain seemed to disappear. Pull requests are often fairly small; the issues do not mirror the complaints against feature branches.

While reviewing this entry Jake McCrary and others pointed out that while context switching is annoying, there are many benefits that also come from the pull request review process. I agree with this observation, and I'm always pleased when I see discussion (comments) occur on a pull request. Perhaps I'm over optimistic, but I always see the discussion as evidence that the team is learning from each other and we are further advancing on our shared goals.

12 months in, I'm happy with the results.

Unsurprisingly, there are other people using and/or experimenting with similar approaches. Paul Nasrat noted the similarities with this approach and the maintainer model of Linux; Scott Robinson pointed me at OWNERS Files; Romily Cocking noted the similarities with the underlying model used in Envy/Developer (pre Java).

Thursday, January 08, 2015

Preview Arbitrary HTML

I'm a big fan of for sharing code. It's fantastic for quickly putting something online to talk over with someone else. I've often found myself wishing for something that allowed me to share rendered HTML in the same way.

For example, a gist of the HTML for this blog entry can be seen here: If I want someone to give me feedback on the rendered output, that gist isn't very helpful.

It turns out, it's really easy to see that HTML rendered: switch the file extension to md. Here's the same gist with a md extension: This all works due to the way Github renders markdown gists, and markdown supporting inline HTML.

I wouldn't use this trick to create any data you'd like to have online in the long term, but it's great for sharing short lived HTML.

Wednesday, January 07, 2015

LTR Org Chart

Most traditional organizations have an Org Chart that looks similar to the following:

Org Charts displayed in this way emphasize the chain of command. A quick look at a node in this Org Chart will let you know who someone reports to and who reports to them. The chart gives a clear indication of responsibility; it also gives an impression of "who's in charge" and how far away you are from the top.

Several years ago I joined ThoughtWorks (TW). TW (at least, when I was there) boasted of their Flat Org Chart and it's advantages.
flat organization (also known as horizontal organization or delayering) is an organization that has an organizational structure with few or no levels of middle management between staff and executives. wikipedia
A Flat Org Chart may look something like the following image.

I've previously written on how joining TW was the single best thing for my career. There were many things I loved about TW - the Flat Org Chart was not one of those things.

The (previously linked) wikipedia article discusses the positive aspects of Flat Organizations - there are several I would agree with. What I find generally missing from the discussion is responsibility. If I make a mistake, who, other than me, could have helped me avoid that mistake? If I need help, whose responsibility is it to ensure I get the help I need? If I'm unhappy and plan to quit, who's going to understand my value and determine what an appropriate counter offer looks like?

In a flat organization, this is me:

Reporting to, thus having access to the person who reports directly to the CEO is great. At the same time, over a hundred other people report to the same person. Would anyone honestly argue that this one individual can guide, mentor, and properly evaluate over a hundred people?

Whenever I bring up responsibility people quickly point out that: those are everyone's issues and it's everyone's responsibility. I generally point these people at Diffusion of responsibility, Bystander effect, and Crowd psychology. Personally, I'd love to see the result of, following a resignation, "everyone" determining an individual's value and producing a counter offer.

I'm thankful for the experience I gained working in a Flat Organization. I'll admit that there's something nice about being one manager away from the CEO on an Org Chart. That said, it was the access to a high level manager and the CEO that I found to be the largest benefit of working in a Flat Organization.

I eventually left TW - a move that was partially motivated by the issues I described above. When I began looking for a new job, organizational structure didn't factor into my decision making process, but I did look for something I called Awesome All the Way Up.
Awesome All the Way Up: Given an Org Chart, do you consider the CEO and every person between you and the CEO to be awesome.
I believed then, and I believe now that you'll always be best off if you put yourself in Awesome All the Way Up situations. When I left TW, I only considered 2 companies - they were the only two I could find that offered Awesome All the Way Up. Everyone will likely have their own definition of "awesome". My definition of "awesome", in this context, is: above all, someone I can trust; someone whose vision I believe in; someone I will always have access to.

One of the two previously referenced companies became my next (and current) employer: DRW Trading. DRW could have a traditional Org Chart, and I would sit towards the "bottom" of it. (note: this is an approximation, not an actual DRW Org Chart.)

Personally, I wouldn't have any complaints with this Org Chart. Every person between me and DRW himself plays an important and specific role. Since I'm in an Awesome All the Way Up situation I have access to each of those people. If I have a CSO problem, I go to the CSO; if I have a CTO problem I go to the CTO; you get the idea. I strongly prefer to know whose responsibility it is when I identify what I consider to be a problem.

You may have noticed I said DRW could have an Org Chart similar to what's above. The phrasing was intentional - if you ask our CSO Derek we have an upside down Org Chart. I'm not sure which of those articles best reflects his opinion. He once showed me a picture similar to the one below and said "part of being a leader is understanding that the shit still rolls downhill, but our hill is different".

Derek and I haven't spoken extensively on the upside down Org Chart, but I assume he's a fan of Servant Leadership. As an employee I (obviously) appreciate the upside down Org Chart and Derek's approach to leadership. As a team lead I find the upside down Org Chart to be a great reminder that I need to put the needs of my teammates first. As someone sitting above Derek on the upside down Org Chart I find the idea of my shit (and the shit of all of my "peers") rolling down to Derek to be... unpleasant.

I was once out with one of the partners from DRW and he introduced me to a friend of his (John). John asked how I knew the DRW partner, I said: he's my boss. The DRW partner interrupted and said: that's not true, we're teammates - I can't do what he does and he can't do what I do, we work together. The partner from this story isn't in my "chain of command", but I find the same team-first attitude present in each person that is.

My discomfort with my shit rolling down on anyone combined with the team-first attitude I've encountered led me to the idea of the Left-to-Right Org Chart (or LTR Org Chart). Below you'll find what would be my current LTR Org Chart.

The LTR Org Chart reminds me to be a Servant Leader while also emphasizing the teammates I can count on to help me out when I encounter issues.

It's probably not practical to generate an LTR Org Chart for every individual in your organization, but it certainly wouldn't be hard to create software to present an LTR Org Chart on a website. It's possible the subtle difference between upside down Org Charts and LTR Org Charts isn't worth the effort required to build the software. I wouldn't mind finding out.

Tuesday, January 06, 2015

Making Remote Work: Tools

I recently wrote about my experiences working on a remote team. Within that blog entry you can find a more verbose version of the following text:
Communication is what I consider to be the hardest part of remote work. I haven't found an easy, general solution. A few teammates prefer video chat, others despise it. A few teammates like the wiki as a backlog, a few haven't ever edited the wiki. Some prefer strict usage of email/chat/phone for async-unimportant/async-important/sync-urgent, others tend to use one of those 3 for all communication.
As you can tell, we have several different communication tools. When writing, I generally prefer to include concrete examples. This blog entry will list each tool referenced above. However, I cannot emphasize enough that: this list is a snapshot of what we're using, not a recommended set of tools.

app: Github
usage: We use many of the features of Github; however, the two features that help facilitate remote work are (a) pull requests with inline comments and (b) compare. A pull request with inline comments has (thus far) been the most productive way to asynchronously discuss specific pieces of code. Almost all non-trivial commits will eventually end up in a pull request that's reviewed by at least one other team member. We've found compare view to be the best solution for distilling changes for a teammate with limited context.

app: Hipchat
usage: We have 3 hipchat rooms: work, social, support. It should be pretty obvious what we use each room for. The primary driver for splitting the 3 is for keeping noise down. Most team members look at chat history for work and support, reading anything that happened between now and the last time they were logged in. Social tends to be more verbose, often off-topic, and never required reading for keeping up with what the team is up to.

app: Cisco Jabber
usage: Within the team, we primarily use Cisco Jabber for video calls; however Cisco Jabber is also a great way for people within DRW offices to reach anyone on my team without having to know their location. Cisco Jabber is significantly better than asking people to remember to call your cell, or forwarding your desk phone to your cell - it provides you 1 number that anyone can reach you at, regardless of your physical location. There's not much to say about the video capabilities, they're there, they work well. Cisco Jabber also provides desktop sharing, which we use occasionally for "remote pair-programming".

app: Confluence
usage: Our backlog resides on a Confluence wiki; it's a single page with about 150 lines. The backlog is split into 3 sections: Milestones, Now, and Soon. There are generally 3-5 Milestones, which list (as sub-bullets) their dependencies. A dependency is a reference to a line item that will live in Now or Soon. Now is the list of highest priority tasks - the things we need to get down right away. Soon is the list of things that are urgent but not important, or important but not urgent. Both Now and Soon lists contain placeholders for conversations, each placeholder is around 1-2 lines. Below you'll find a contrived, sample backlog.
  • Deploy to Billy Ray Valentine
    • (market data 1)
    • (execution 1)
  • Automated Order Matching (stakeholder: Mortimer and Randolph Duke)
    • (execution 2)
    • (reporting 1)
  • Market data
    • (1) support pork bellies
    • support orange juice
  • Execution
    • (1) support pork bellies
    • (2) internally match incoming orders from customers
  • Market data
    • support coffee
    • support wheat
  • Reporting
    • (1) Commission summary
note: some things in Now need to be done immediately, but do not support an upcoming milestone. These things are incremental changes for previously met milestones.
Obviously we use email and other tools as well, but I can't think of any remote specific usage patterns that are worth sharing.

As I previously mentioned, each member of the team uses each of these tools in their own way. None of these tools are ideal for every member of the team, and I believe a good team lead helps ensure each team member is only required to use the tools they find most helpful.

Sunday, December 28, 2014

Making Remote Work

Over 18 months ago I wrote Year Five, an experience report I never imagined I would write. I closed the blog entry by saying I look forward to writing about Year Six. A year and a half later, I'm still having a hard time deciding what (if anything) I should write. My writers block isn't the result the Remote Work experiment failing. Quite the opposite, the success of the Remote Work experiment has helped shape a team I'm very proud to be a part of, and yet I find myself unable to declare victory.

"How can you work effectively with remote teammates" has become the most common question I hear when meeting up with old colleagues for coffee. Clearly people are interested in the topic. At the same time, I prefer not to write about half-baked ideas (these days), thus my apprehension in documenting my approach.

This entry is an experience report, nothing more, nothing less. I'd be very skeptical of anyone providing recipes or best practices around remote work. Those working remotely today are breaking traditional workplace rules. Some are succeeding, most are failing, and I don't know of anyone with solid general advice. What follows are merely my observations.

My team became remote on accident. My boss and I were a 2 person co-located team in NYC, he quit, they gave me his job, and my best option to fill my previous role lived in Chicago. David Chelimsky was willing to be in NYC 2 weeks out of the month, and I was willing to be in Chicago 1 week out of the month. It was effectively co-location, and our team ran that way for a couple months.

Sometime in month three I started to feel like having David in NYC 2 weeks a month was unnecessary, and (surprisingly to me) he'd independently come to the same conclusion. From that point forward we alternated on traveling 1 week a month - thus we worked remotely for 3 weeks out of 4. We saw no drop off in our productivity or camaraderie, and traveling less definitely improved our overall happiness.  I'm not sure how it would have worked out if I'd been forced to try remote work; however, arriving there organically was surprisingly painless.

Eventually the team grew and DRW hired John Hume, who lives in Austin, Texas. I believe it's worth noting that we each lived in a different city. I've always believed that a team needs to either be completely remote or completely co-located. The team is up to 5 people at this point, and I've actively gone out of my way to ensure no two people work out of the same place on an ongoing basis.

When you join the team you'll have to start on a 6 month contract. The contract period gives both you and I the opportunity to figure out if you're a good fit. There are plenty of brilliant people who I wouldn't work with: I'm looking for people with compatible opinions on software who are able to flourish in the environment we've created.

Remote disagreements are hard to resolve. We can't get a beer and talk it out. I can't read your body language (or other subtle signs) and see something needs to change. Thus, it's not enough to be talented, you'll also need to be a philosophical match. The philosophical ideas are easy to agree on when everyone's looking to start a new endeavor, but best intentions don't always equal an ideal working environment. The 6 month contract ensures both parties know what we're getting into long before we discuss the idea of full time employment.

My team is stretched through various timezones, sometimes from London to Los Angeles. This leaves the options of work odd hours, or finding people who can take on larger tasks and don't require constant contact. My team chose to go the latter path, and we've found no notable impact on our ability to collaborate even when our hours overlap as little as 2 hours a day. This choice is another reason the 6 month contract is critical: some people want more than 2 hours of contact a day. That's neither good, nor bad, nor is it easy to predict. If someone ends up needing more contact than they find they're getting, it's better for all parties if everyone goes their separate way after 6 months.

I've found that my opinion on pair-programming and co-location is constantly evolving. In more than 2 years of working remotely, I've probably done less than 4 hours of remote pair-programming. I mention this because some people believe remote pair-programming is an essential ingredient to successful remote work. I am not one of those people.

That said, I try to see my teammates as often as is reasonable, and we do often pair-program when we are co-located. At this point, I see every team member quarterly, and the entire team spends a week together every 6 months. We've found this frequency to be a solid balance between keeping relationships strong and keeping travel to a minimum.

The above schedule works for ongoing collaboration; however, the beginning of a work relationship is almost the exact opposite. When a new contractor joins the team, they travel and co-locate as much as possible. A recent team member spent his first week in Austin, his second in Chicago, and his third in London. By the end of those 3 weeks he'd seen the codebase from 3 different perspectives and spent a week (mostly pairing) with every member of the (at that time, 4 person) team. That much travel is a high up-front cost, but it helps immensely with learning both the codebase and the team you'll be working with.

Communication is what I consider to be the hardest part of remote work. I haven't found an easy, general solution, thus I often find myself duplicating effort to ensure teammates can consume data in their preferred format. A few teammates prefer video chat each time we're on the phone, a few teammates despise video chat. A few teammates like the wiki as a backlog, a few haven't ever edited the wiki (as far as I know). Some prefer strict usage of email/chat/phone for async-unimportant/async-important/sync-urgent, others tend to use one of those 3 for all communication. There hasn't been one tool that I would recommend; instead I think it's much more valuable to note that people prefer different approaches, and it's the job of the team lead to communicate with the team members in the way that they prefer, not the other way around. The only rule I try to apply universally: I end as many conversations as I can with "is there anything I can do to make your life better". If you constantly ask that question, it should be (often painfully) obvious what you need to change to continue to improve things for the team as a whole.

The question of hardware often comes up as well, what should a company provide, what should an individual? My approach: take the cost of a 30" monitor, any laptop, a tablet, a smart phone, and anything else they'd want, then average it out over 2 years. I think you'll find the amount of money is so trivial that you'd be a fool not to buy them whatever they want. (and that you're company loses money every time you waste your time talking about such a small expenditure.)

That's more or less it. I would summarize it like so: I want to create a team that people want to be a part of for at least the next 10 years. That begins by finding people who are a great fit; not everyone will be, and we'll learn valuable lessons from those people as well. We start the relationship out right, spending many hours together getting to know each other and getting to know the ins & outs of the project. From that point on we'll see if your preferred working style (hours, communication needs, etc) fit well with the team. We'll already know if you'll be happy on the team long before either of us has to commit to any long term working relationship. From there, as long as I remember that I work for the team, not the other way around, everyone should continue to be happy and effective.