Someone recently asked me for information on starting new consulting engagements. A few years back I published Sean Doran and Scott Conley's thoughts on Being a Lead Consultant. Sean and Scott's list is great for any lead consultant, and the advice applies well for the lifetime of a project. I considered sending that list to the person looking for new engagement advice, but I'm not sure that list would be the best place to focus my attention at the beginning of a project.
The beginning of a project is a special and dangerous time. You get a mix of (at least) optimism, concern, and freedom. There are a large number of ways the project can go, and as a lead consultant you'll play a major role determining it's outcome. The specific question I recently received was: did you have a process you followed when new engagements began. The remainder of this blog post contains my (slightly edited) response.
I found being a lead at ThoughtWorks to be a nearly impossible balancing act. It's possible we didn't have a process because we weren't organized enough, but it's more likely that there's no general formula that works.
What we ran into constantly was what we called the "enablement versus delivery" issue: If you're teaching client devs (enablement), you really don't have time to meet delivery deadlines; conversely if you're delivering software you rarely have time to teach. How to balance enablement versus delivery is something that varies based on the client's skillset and the ROI of the software being developed. What makes it worse is that the client often thinks they need one thing, but if the the company didn't have issues you (likely) wouldn't be there. Sometimes the issues are with the stakeholder and they'll tell you to focus on the wrong thing, and sometimes the stakeholder knows the deal but the other people you are forced to work with are the problem.
Often it's a race to figure out what the client needs and get them to agree to fix it, before you've lost too much good will.
Another issue is, most software is worthless within a few years, and the only way to break a perpetual cycle of mediocrity is to level up the people and processes. This can lead some people to think that delivery of a specific piece of software is secondary to improving process and people. There's a major flaw to that approach though; a non-TW consultant once said that he feels guilty about his job, because when he helps people become better the most talented always end up leaving the (suboptimal environment of the) client. You can help people improve and install good processes, but when the good people leave and the remaining don't understand the foundations of the process, you end up with not enough talent and a process that's loosely followed and for none of the right reasons.
That said, you can't focus exclusively on delivery. I once led a team that beat all deadlines, created a great piece of software, and provided a great deal of content for Martin Fowler's DSL book. It was a "huge success" until the client devs took over, couldn't maintain it, and wrote their own version that had 20% of the functionality we provided. The software we wrote was classified as a "proof of concept" and thrown out.
I was somewhat oblivious to all of this in my first few years at ThoughtWorks; I would work with the talented clients while isolating the less talented. Eventually you find out that a stakeholder will usually fire you before their worst employee, regardless of how obvious it is.
If I were going into a new engagement these days, I would split my time between training and delivery initially. After figuring out which is the bigger problem, you can spend more or less time on training or delivery.
I would also keep a spreadsheet with every client employee and their talents; every one of those employees will impact your success. If you find what they're good at and get them doing that, you may have found an ally and advocate. Every employee that has no talents listed in your spreadsheet is not only slowing you down, but is also likely (consciously or unconsciously) sabotaging you in every discussion you aren't a part of. Assume you cannot get rid of them, and don't bother trying; spending political capital managing client staff is a bad investment. Keep them on your sheet and make finding their talent a top priority.
Good luck, it's not an easy gig. Then again, if things don't go well you can always move on to another client. That was what always kept me sane. I always did the best job I could, but I also knew if I failed at an impossible task it wasn't the end of the world. There's an endless stream of impossible tasks available.
Wednesday, May 17, 2017
Tuesday, June 28, 2016
Curious Customer
I currently work on a pretty small team, 4 devs (including myself). We have no one dedicated strictly to QA. A few years ago we ran into a few unexpected issues with our software. I hesitate to call them bugs, because they only appeared when you did things that made little sense. We write internal-only software, thus we expect a minimum level of competency from our users. In addition, it's tempting justify ignoring problematic nonsensical behavior in the name of not having to write and maintain additional software.
But, when I wasn't in denial, I was willing to admit that these were in fact bugs and they were costing us time.
The problems caused by these bugs were small, e.g. a burst of worthless emails, a blip in data flowing to the application. The emails could be quickly deleted, and the application was eventually consistent. Thus I pretended as though these issues were of low importance, and that the pain was low for both myself and our customers. I imagine that sounds foolish; in retrospect, it was foolish. The cost of developer context switching is often very high, higher if it's done as an interrupt. Introducing noise into your error reporting devalues your error reporting. Users can't as easily differentiate between good data, a blip of bad data due to something they did, and actual bad data, thus they begin to distrust all of the data.
The cost of these bugs created by nonsensical behavior is high, much higher than the cost of writing and maintaining the software that eliminated these bugs.
Once we eliminated these bugs, I spent notably more time happily focused on writing software. For me, delivering features is satisfying; conversely, tracking down issues stemming from nonsensical behavior always feels like a painfully inefficient task. I became very intent on avoiding that inefficiency in the future. The team brainstormed on how to address this behavior, and honestly we came up with very little. We already write unit tests, load tests, and integration tests. Between all of our tests, we catch the majority of our bugs before they hit production. However, this was a different type of bug, created by behavior a developer often wouldn't think of, thus a developer wasn't very likely to write a test that would catch this issue.
I proposed an idea I wasn't very fond of, the Curious Customer (CC): upon delivery of any feature you could ask another developer on the team to use the feature in the staging environment, acting as a user curiously toying with all aspects of the feature.
Over a year later, I'm not sure it's such a bad idea. In that year we've delivered several features, and (prior to release) I've found several bugs while playing the part of CC. I can't remember a single one of them that would have led to a notable problem in production; however all of them would have led to at least one support call, and possibly a bit less trust in our software.
My initial thought was: asking developers to context switch to QAing some software they didn't write couldn't possibly work, could it? Would they give it the necessary effort, or would they half-ass the task and get back to coding?
For fear of half-ass, thus wasted effort, I tried to define the CC's responsibilities very narrowly. CC was an option, not a requirement; if you delivered a feature you could request a CC, but you could also go to production without a CC. A CC was responsible for understanding the domain requirements, not the technical requirements. It's the developers responsibility to get the software to staging, the CC should be able to open staging and get straight to work. If the CC managed to crash or otherwise corrupt staging, it was the developers responsibility to get things back to a good state. The CC doesn't have an official form or process for providing feedback; The CC may chose email, chat, or any mechanism they prefer for providing feedback.
That's the idea, more or less. I've been surprised and pleased at the positive impact CC has had. It's not life changing, but it does reduce the number of support calls and the associated waste with tracking down largely benign bugs, at least, on our team.
You might ask how this differs from QA. At it's core, I'm not sure it does in any notable way. That said, I believe traditional QA differs in a few interesting ways. Traditional QA is often done by someone whose job is exclusively QA. With that in mind, I suppose we could follow the "devops" pattern and call this something like "devqa", but that doesn't exactly roll off the tongue. Traditional QA is also often a required task, every feature and/or build requires QA sign off. Finally, the better QA engineers I've worked with write automated tests that continually run to prevent regression; A CC may write a script or two for a single given task, but those scripts are not expected to be valuable to any other team member now or for anyone (including the author) at any point in the future.
But, when I wasn't in denial, I was willing to admit that these were in fact bugs and they were costing us time.
The problems caused by these bugs were small, e.g. a burst of worthless emails, a blip in data flowing to the application. The emails could be quickly deleted, and the application was eventually consistent. Thus I pretended as though these issues were of low importance, and that the pain was low for both myself and our customers. I imagine that sounds foolish; in retrospect, it was foolish. The cost of developer context switching is often very high, higher if it's done as an interrupt. Introducing noise into your error reporting devalues your error reporting. Users can't as easily differentiate between good data, a blip of bad data due to something they did, and actual bad data, thus they begin to distrust all of the data.
The cost of these bugs created by nonsensical behavior is high, much higher than the cost of writing and maintaining the software that eliminated these bugs.
Once we eliminated these bugs, I spent notably more time happily focused on writing software. For me, delivering features is satisfying; conversely, tracking down issues stemming from nonsensical behavior always feels like a painfully inefficient task. I became very intent on avoiding that inefficiency in the future. The team brainstormed on how to address this behavior, and honestly we came up with very little. We already write unit tests, load tests, and integration tests. Between all of our tests, we catch the majority of our bugs before they hit production. However, this was a different type of bug, created by behavior a developer often wouldn't think of, thus a developer wasn't very likely to write a test that would catch this issue.
I proposed an idea I wasn't very fond of, the Curious Customer (CC): upon delivery of any feature you could ask another developer on the team to use the feature in the staging environment, acting as a user curiously toying with all aspects of the feature.
Over a year later, I'm not sure it's such a bad idea. In that year we've delivered several features, and (prior to release) I've found several bugs while playing the part of CC. I can't remember a single one of them that would have led to a notable problem in production; however all of them would have led to at least one support call, and possibly a bit less trust in our software.
My initial thought was: asking developers to context switch to QAing some software they didn't write couldn't possibly work, could it? Would they give it the necessary effort, or would they half-ass the task and get back to coding?
For fear of half-ass, thus wasted effort, I tried to define the CC's responsibilities very narrowly. CC was an option, not a requirement; if you delivered a feature you could request a CC, but you could also go to production without a CC. A CC was responsible for understanding the domain requirements, not the technical requirements. It's the developers responsibility to get the software to staging, the CC should be able to open staging and get straight to work. If the CC managed to crash or otherwise corrupt staging, it was the developers responsibility to get things back to a good state. The CC doesn't have an official form or process for providing feedback; The CC may chose email, chat, or any mechanism they prefer for providing feedback.
That's the idea, more or less. I've been surprised and pleased at the positive impact CC has had. It's not life changing, but it does reduce the number of support calls and the associated waste with tracking down largely benign bugs, at least, on our team.
You might ask how this differs from QA. At it's core, I'm not sure it does in any notable way. That said, I believe traditional QA differs in a few interesting ways. Traditional QA is often done by someone whose job is exclusively QA. With that in mind, I suppose we could follow the "devops" pattern and call this something like "devqa", but that doesn't exactly roll off the tongue. Traditional QA is also often a required task, every feature and/or build requires QA sign off. Finally, the better QA engineers I've worked with write automated tests that continually run to prevent regression; A CC may write a script or two for a single given task, but those scripts are not expected to be valuable to any other team member now or for anyone (including the author) at any point in the future.
Thursday, June 16, 2016
Maintainability and Expect Literals
Recently, Stephen Schaub asked the following on the wewut group:
In general, given the option of having many similar strings (or any literal) vs a helper function, I would always prefer the literal. When a test is failing I only care about that single failing test. If I have to look at the helper function I no longer have the luxury of staying focused on the single test; now I need to consider what the helper function is giving me and what it's giving all other callers. Suddenly the scope of my work has shifted from one test to all of the tests coupled by this helper function. If this helper function wasn't written by me, this expansion in scope wasn't even my decision, it was forced upon me by the helper function creator. In the best case the helper function could return a single, constant string. The scope expansion becomes even worse when the helper function contains code branches.
As for alternatives, my solution would depend on the problem. If the strings were fairly consistent, I would likely simply duplicate everything knowing that any formatting changes can likely be addressed using a bulk edit via find and replace. If the strings were not consistent, I would look at breaking up the methods in a way that would allow me to verify the code branches using as little duplication as possible, e.g. if I wanted to test a string that dynamically changed based on a few variables, I would look to test those variables independently, and then only have a few tests for the formatting.
A concrete example will likely help here. Say I'm writing a trading system and I need to display messages such as
There's quite a bit of variation in those messages. You could have 1 function that creates the entire string:
However, I think you'd end up with quite a few verbose tests. Given this problem, I would look to break down those strings into smaller, more focused functions, for example:
Now that you've broken down the function, you're free to test the code paths of the more focused functions, and the test for confirmMsg becomes trivial. Something along the lines of
Several of the unit test examples in the book verify the construction of both HTML and plain text strings. Jay recommends using literal strings in the assertions. However, this strikes me as not a particularly maintainable approach. If the requirements regarding the formatting of these strings changes (a very likely scenario), every single test that verifies one of these strings using a literal must be updated. Combined with the advice that each test should check only one thing, this leads to a large number of extremely brittle tests.I responded to Stephen; however, I've seen similar questions asked a few times. Below are my extended thoughts regarding literals as expected values.
Am I missing something here? I can appreciate the reasons Jay recommends using literals in the tests. However, it seems that we pay a high maintainability price in exchange for the improved readability.
In general, given the option of having many similar strings (or any literal) vs a helper function, I would always prefer the literal. When a test is failing I only care about that single failing test. If I have to look at the helper function I no longer have the luxury of staying focused on the single test; now I need to consider what the helper function is giving me and what it's giving all other callers. Suddenly the scope of my work has shifted from one test to all of the tests coupled by this helper function. If this helper function wasn't written by me, this expansion in scope wasn't even my decision, it was forced upon me by the helper function creator. In the best case the helper function could return a single, constant string. The scope expansion becomes even worse when the helper function contains code branches.
As for alternatives, my solution would depend on the problem. If the strings were fairly consistent, I would likely simply duplicate everything knowing that any formatting changes can likely be addressed using a bulk edit via find and replace. If the strings were not consistent, I would look at breaking up the methods in a way that would allow me to verify the code branches using as little duplication as possible, e.g. if I wanted to test a string that dynamically changed based on a few variables, I would look to test those variables independently, and then only have a few tests for the formatting.
A concrete example will likely help here. Say I'm writing a trading system and I need to display messages such as
"paid 10 on 15 APPL. $7 Commission. spent: $157"
"paid 1 on 15 VTI. Commission free. spent: $15"
"sold 15 APPL at 20. $7 Commission. collected: $293"
"sold 15 VTI at 2. Commission free. collected: $30"
There's quite a bit of variation in those messages. You could have 1 function that creates the entire string:
confirmMsg(side, size, px, ticker)
However, I think you'd end up with quite a few verbose tests. Given this problem, I would look to break down those strings into smaller, more focused functions, for example:
describeOrder(side, size, px, ticker)
describeCommission(ticker)
describeTotal(side, size, px, ticker)
Now that you've broken down the function, you're free to test the code paths of the more focused functions, and the test for confirmMsg becomes trivial. Something along the lines of
assertEquals("paid 10 on 15 APPL",
describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("sell 15 APPL at 10",
describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("$7 Commission",
describeCommission({tickerName:"APPL",commission:"standard"}))
assertEquals("Commission free",
describeCommission({tickerName:"APPL",commission:"free"}))
assertEquals("spent: $157",
describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("collected: $143",
describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("spent: $150",
describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"free"}))
assertEquals("collected: $150",
describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"free"}))
assertEquals("order. commission. total",
confirmMsg("order", "commission", "total"))
I guess I could summarize it by saying, I should be able to easily find and replace my expected literals. If I cannot, then I have an opportunity to further break down a method and write more focused tests on the newly introduced, more granular tests.
Thursday, June 25, 2015
Drop Books
The vast majority of books I purchase are for my own enjoyment, but not all of them. There are a few books that I buy over and over, and drop on the desks of friends and colleagues. These books, all technical, are books that I think most programmers will benefit from reading. I call these books "Drop Books"; I drop them and never expect them to be returned.
My main motivation for dropping books is to spread what I think are great ideas. Specifically, I'm always happy to spread the ideas found in the following books:
I know a few of my friends buy Drop Books as well. Spreading solid ideas and supporting authors seems like a win/win to me; hopefully more and more people will begin to do the same.
My main motivation for dropping books is to spread what I think are great ideas. Specifically, I'm always happy to spread the ideas found in the following books:
I know a few of my friends buy Drop Books as well. Spreading solid ideas and supporting authors seems like a win/win to me; hopefully more and more people will begin to do the same.
Monday, April 06, 2015
Unit Testing Points of View, Probably
Michael Feathers, Brian Marick, and I are collaborating to create a new book: Unit Testing Points of View ... probably.
Unit testing is now mainstream, has tons of entry level books, and has a great reference book. The result is a widely accepted idea that you can quickly and easily adopt; unfortunately, I've found little in the way of documenting pattern trade-offs. I believe that combination leads to a lot of waste. To help avoid some of that waste, I'd like to see more written about why you may want to choose one Unit Testing style over another. That was the inspiration and my goal for Working Effectively with Unit Tests. Growing Object-Oriented Software is another great example of a book that documents both the hows and whys of unit testing. After that, if you're looking for tradeoff guidance... good luck.
Is this a problem? I think so. Without discussion of tradeoffs, experience reports, and concrete direction you end up with hours wasted on bad tests and proclamations of TDD's death. The report of TDD's death was an exaggeration, and the largest clue was the implication that TDD and Unit Testing were synonymous. To this day, Unit Testing is still largely misunderstood.
Today, most teams fail repeatedly with Unit Tests, leading to (at best) significant wasted time and (at worst) abandoning the idea with almost nothing gained. We aim to provide tradeoff guidance that will help teams select a Unit Testing approach that best fits their context.
As I said above: There's no reason this evolution couldn't be extended by other authors. In fact, that's our long term hope. Ideally, Brian, Michael and I write volume one of Unit Testing Points of View. Volume two could be written by Kevlin Henney, Steve Freeman, and Roy Osherove - or anyone else who has interest in the series. Of course, given the original inspiration, we're all hoping Martin Fowler clears some time in his schedule to take part in the series.
If you'd like to see this project happen, please do sign up on the interest list and encourage others to as well (here's a tweet to RT, if you'd like to help).
Origin
In 2014 Martin Fowler provided Technical Review for Working Effectively with Unit Tests. As part of his feedback he said something along the lines of: I'm glad you wrote this book, and I'll likely write a bliki entry noting what I agree with and detailing what I would do differently. I'm still looking forward to that entry, and I think the idea is worth extending beyond Martin.Unit testing is now mainstream, has tons of entry level books, and has a great reference book. The result is a widely accepted idea that you can quickly and easily adopt; unfortunately, I've found little in the way of documenting pattern trade-offs. I believe that combination leads to a lot of waste. To help avoid some of that waste, I'd like to see more written about why you may want to choose one Unit Testing style over another. That was the inspiration and my goal for Working Effectively with Unit Tests. Growing Object-Oriented Software is another great example of a book that documents both the hows and whys of unit testing. After that, if you're looking for tradeoff guidance... good luck.
Is this a problem? I think so. Without discussion of tradeoffs, experience reports, and concrete direction you end up with hours wasted on bad tests and proclamations of TDD's death. The report of TDD's death was an exaggeration, and the largest clue was the implication that TDD and Unit Testing were synonymous. To this day, Unit Testing is still largely misunderstood.
What could be
Working Effectively with Unit Tests starts with one set of opinionated Unit Tests and evolves to the types of Unit Tests that I find more productive. There's no reason this evolution couldn't be extended by other authors. This is the vision we have for Unit Testing Points of View. Michael, Brian, and I began by selecting a common domain model. The first few chapters will detail the hows and whys of how I would test the common domain model. After I've expressed what motivates my tests, Brian or Michael will evolve the tests to a style they find superior. After whoever goes second (Brian or Michael) finishes, the other will continue the evolution. The book will note where we agree, but the majority of the discussion will occur around where our styles differ and what aspect of software development we've chosen emphasize by using an alternative approach.Today, most teams fail repeatedly with Unit Tests, leading to (at best) significant wasted time and (at worst) abandoning the idea with almost nothing gained. We aim to provide tradeoff guidance that will help teams select a Unit Testing approach that best fits their context.
As I said above: There's no reason this evolution couldn't be extended by other authors. In fact, that's our long term hope. Ideally, Brian, Michael and I write volume one of Unit Testing Points of View. Volume two could be written by Kevlin Henney, Steve Freeman, and Roy Osherove - or anyone else who has interest in the series. Of course, given the original inspiration, we're all hoping Martin Fowler clears some time in his schedule to take part in the series.
Why "Probably"?
Writing a book is almost always a shot in the dark. Martin, Michael, Brian, and I all think this a book worth writing (and a series worth starting), but we have no idea if people actually desire such a book. In the past you took the leap expecting failure and praying for success. Michael, Brian, and I believe there's a better way: leanpub interest lists (here's Unit Testing Points of View's). We're looking for 15,000 people to express interest (by providing their email addresses). I'm writing the first 3 chapters, and they'll be made available when we cross the 15k mark. At that point, Michael and Brian will know that the interest is real, and their contributions need to be written. If we never cross the 15k mark then we know such a book isn't desired, and we shouldn't waste our time.If you'd like to see this project happen, please do sign up on the interest list and encourage others to as well (here's a tweet to RT, if you'd like to help).
Sunday, March 15, 2015
My Answers for Microservices Awkward Questions
Earlier this year, Ade published Awkward questions for those boarding the microservices bandwagon. I think the list is pretty solid, and (with a small push from Ade) I decided to write concise details on my experience.
I think it's reasonable to start with a little context building. When I started working on the application I'm primarily responsible for microservices were very much fringe. Fred George was already giving (great) presentations on the topic, but the idea had gained neither momentum nor hype. I never set out to write "microsevices"; I set out to write a few small projects (codebases) that collaborated to provide a (single) solid user experience.
I pushed to move the team to small codebases after becoming frustrated with a monolith I was a part of building and a monolith I ended up inheriting. I have no idea if several small codebases are the right choice for the majority, but I find they help me write more maintainable software. Practically everything is a tradeoff in software development; when people ask me what the best aspect of a small services approach is, I always respond: I find accidental coupling to be the largest productivity drain on projects with monolithic codebases. Independent codebases reduce what can be reasonably accidentally coupled.
As I said, I never set out to create microservices; I set out to reduce accidental coupling. 3 years later, it seems I have around 3 years of experience working with Microservies (according to the wikipedia definition). I have nothing to do with microservices advocacy, and you shouldn't see this entry as confirmation or condemnation of a microservices approach. The entry is an experience report, nothing more, nothing less.
On to Ade's questions (in bold).
Why isn't this a library?:
It is. Each of our services compile to a jar that can be run independently, but could just as easily be used as a library.
What heuristics do you use to decide when to build (or extract) a service versus building (or extracting) a library?:
Something that's strictly a library provides no business value on it's own.
How do you plan to deploy your microservices?:
Shell scripts copy jars around, and we have a web UI that provides easy deployment via a browser on your laptop, phone, or anywhere else.
What is your deployable unit?:
A jar.
Will you be deploying each microservice in isolation or deploying the set of microservices needed to implement some business functionality?:
In isolation. We have a strict rule that no feature should require deploying separate services at the same time. If a new feature requires deploying new versions of two services, one of them must support old and new behavior - that one can be deployed first. After it's run in production for a day the other service can be deployed. Once the second service has run in production for a day the backwards compatibility for the first service can be safely removed.
Are you capable of deploying different instances (where an instance may represent multiple processes on multiple machines) of the same microservice with different configurations?:
If I'm understanding your question correctly, absolutely. We currently run a different instance for each trading desk.
Is it acceptable for another team to take your code and spin up another instance of your microservice?:
Yes.
Can team A use team B's microservice or are they only used within rather than between teams?:
Anyone can request a supported instance or fork individual services. If a supported instance is requested the requestor has essentially become a customer, a peer of the trading desks, and will be charged the appropriate costs allocated to each customer. Forking a service is free, but comes with zero guarantees. You fork it, you own it.
Do you have consumer contacts for your microservices or is it the consumer's responsibility to keep up with the changes to your API?:
It is the consumer's responsibility.
Is each microservice a snowflake or are there common conventions?:
There are common conventions, and yet, each is also a snowflake. There are common libraries that we use, and patterns that we follow, but many services require sightly different behavior. A service owner (or primary) is free to make whatever decision best fits the problem space.
How are these conventions enforced?:
Each service has a primary, who is responsible for ensuring consistency (more here). Before you become a primary you'll have spent a significant amount of time on the team; thus you'll be equipped with the knowledge required to apply conventions appropriately.
How are these conventions documented?:
They are documented strictly by the code.
What's involved in supporting these conventions?:
When you first join the team you pair exclusively for around 4 weeks. During those 4 weeks you drive the entire time. By pairing and being the primary driver, you're forced to become comfortable working within the various codebases, and you have complete access to someone who knows the history of the overall project. Following those 4 weeks you're free to request a pair whenever you want. You're unlikely to become a service owner until you've been on the team for at least 6 months.
Are there common libraries that help with supporting these conventions?:
A few, but not many to be honest. Logging, testing, and some common functions - that's about it.
How do you plan to monitor your microservices?:
We use proprietary monitoring software managed by another team. We also have front line support that is on call and knows who to contact when they encounter issues they cannot solve on their own.
How do you plan to trace the interactions between different microservices in a production environment?:
We log as much as is reasonable. User generated interactions are infrequent enough that every single one is logged. Other interactions, such as snapshots of large pieces of data, or data that is updated 10 or more times per second, are generally not logged. We have a test environment were we can bring the system up and mirror prod, so interactions we miss in prod can (likely) be reproduced in dev if necessary. A few of our systems are also experimenting with a logging system that logs every interaction shape, and a sample per shape. For example, if the message {foo: {bar:4 baz:45 cat:["dog", "elephant"]}} is sent, the message and the shape {foo {bar:Int, baz:Int, cat:[String]}} will be logged. If that shape previous existed, neither the shape nor the message will be logged. In practice, logging what you can and having an environment to reproduce what you can't log is all you need 95% of the time.
What constitutes a production-ready microservice in your environment?:
Something that provides any value whatsoever, and has been hardened enough that it should create zero production issues under normal working circumstances.
What does the smallest possible deployable microservice look like in your environment?:
A single webpage that displays (readonly) data it's collecting from various other services.
I'm happy to share more of my experiences with a microservice architecture, if people find that helpful. If you'd like elaboration on a question above or you'd like an additional question answered, please leave a comment.
I think it's reasonable to start with a little context building. When I started working on the application I'm primarily responsible for microservices were very much fringe. Fred George was already giving (great) presentations on the topic, but the idea had gained neither momentum nor hype. I never set out to write "microsevices"; I set out to write a few small projects (codebases) that collaborated to provide a (single) solid user experience.
I pushed to move the team to small codebases after becoming frustrated with a monolith I was a part of building and a monolith I ended up inheriting. I have no idea if several small codebases are the right choice for the majority, but I find they help me write more maintainable software. Practically everything is a tradeoff in software development; when people ask me what the best aspect of a small services approach is, I always respond: I find accidental coupling to be the largest productivity drain on projects with monolithic codebases. Independent codebases reduce what can be reasonably accidentally coupled.
As I said, I never set out to create microservices; I set out to reduce accidental coupling. 3 years later, it seems I have around 3 years of experience working with Microservies (according to the wikipedia definition). I have nothing to do with microservices advocacy, and you shouldn't see this entry as confirmation or condemnation of a microservices approach. The entry is an experience report, nothing more, nothing less.
On to Ade's questions (in bold).
Why isn't this a library?:
It is. Each of our services compile to a jar that can be run independently, but could just as easily be used as a library.
What heuristics do you use to decide when to build (or extract) a service versus building (or extracting) a library?:
Something that's strictly a library provides no business value on it's own.
How do you plan to deploy your microservices?:
Shell scripts copy jars around, and we have a web UI that provides easy deployment via a browser on your laptop, phone, or anywhere else.
What is your deployable unit?:
A jar.
Will you be deploying each microservice in isolation or deploying the set of microservices needed to implement some business functionality?:
In isolation. We have a strict rule that no feature should require deploying separate services at the same time. If a new feature requires deploying new versions of two services, one of them must support old and new behavior - that one can be deployed first. After it's run in production for a day the other service can be deployed. Once the second service has run in production for a day the backwards compatibility for the first service can be safely removed.
Are you capable of deploying different instances (where an instance may represent multiple processes on multiple machines) of the same microservice with different configurations?:
If I'm understanding your question correctly, absolutely. We currently run a different instance for each trading desk.
Is it acceptable for another team to take your code and spin up another instance of your microservice?:
Yes.
Can team A use team B's microservice or are they only used within rather than between teams?:
Anyone can request a supported instance or fork individual services. If a supported instance is requested the requestor has essentially become a customer, a peer of the trading desks, and will be charged the appropriate costs allocated to each customer. Forking a service is free, but comes with zero guarantees. You fork it, you own it.
Do you have consumer contacts for your microservices or is it the consumer's responsibility to keep up with the changes to your API?:
It is the consumer's responsibility.
Is each microservice a snowflake or are there common conventions?:
There are common conventions, and yet, each is also a snowflake. There are common libraries that we use, and patterns that we follow, but many services require sightly different behavior. A service owner (or primary) is free to make whatever decision best fits the problem space.
How are these conventions enforced?:
Each service has a primary, who is responsible for ensuring consistency (more here). Before you become a primary you'll have spent a significant amount of time on the team; thus you'll be equipped with the knowledge required to apply conventions appropriately.
How are these conventions documented?:
They are documented strictly by the code.
What's involved in supporting these conventions?:
When you first join the team you pair exclusively for around 4 weeks. During those 4 weeks you drive the entire time. By pairing and being the primary driver, you're forced to become comfortable working within the various codebases, and you have complete access to someone who knows the history of the overall project. Following those 4 weeks you're free to request a pair whenever you want. You're unlikely to become a service owner until you've been on the team for at least 6 months.
Are there common libraries that help with supporting these conventions?:
A few, but not many to be honest. Logging, testing, and some common functions - that's about it.
How do you plan to monitor your microservices?:
We use proprietary monitoring software managed by another team. We also have front line support that is on call and knows who to contact when they encounter issues they cannot solve on their own.
How do you plan to trace the interactions between different microservices in a production environment?:
We log as much as is reasonable. User generated interactions are infrequent enough that every single one is logged. Other interactions, such as snapshots of large pieces of data, or data that is updated 10 or more times per second, are generally not logged. We have a test environment were we can bring the system up and mirror prod, so interactions we miss in prod can (likely) be reproduced in dev if necessary. A few of our systems are also experimenting with a logging system that logs every interaction shape, and a sample per shape. For example, if the message {foo: {bar:4 baz:45 cat:["dog", "elephant"]}} is sent, the message and the shape {foo {bar:Int, baz:Int, cat:[String]}} will be logged. If that shape previous existed, neither the shape nor the message will be logged. In practice, logging what you can and having an environment to reproduce what you can't log is all you need 95% of the time.
What constitutes a production-ready microservice in your environment?:
Something that provides any value whatsoever, and has been hardened enough that it should create zero production issues under normal working circumstances.
What does the smallest possible deployable microservice look like in your environment?:
A single webpage that displays (readonly) data it's collecting from various other services.
I'm happy to share more of my experiences with a microservice architecture, if people find that helpful. If you'd like elaboration on a question above or you'd like an additional question answered, please leave a comment.
Monday, February 23, 2015
Experience Report: Weak Code Ownership
In 2006 Martin Fowler wrote about Code Ownership. It's a quick read, I'd recommend checking it out if you've never seen it. At the time I was working at ThoughtWorks; I remember thinking "Clearly Strong makes no sense and I have no idea what scenario would make Weak reasonable". 8 years later, I find myself advocating for Weak Code Ownership within my team.
Collective Code Ownership (CCO) served me well between 2005 and 2009. Given the make-up of the teams that I was a part of I found it to be the most effective way to deliver software. Around 2009 I joined a team that eventually grew to around 9 people, all very senior developers. The team practiced Collective Code Ownership. Everyone on the team was very talented, but that didn't translate to constant agreement. In fact, we disagreed far more often than I thought we should. That experience drove me to write about the importance of Compatible Opinions. I still believe in the importance of compatible opinions, but I now wonder if the team wouldn't have been more effective (despite incompatible opinions) if we had adopted Weak Code Ownership.
The 2009 project heavily shaped my approach to developing software. I suspect I'm not the only one who (at one time) believed: if we get a team full of massively talented people we can do anything. It turns out, it's not nearly that easy. Too many cooks in the kitchen is the obvious concern, and it does come up. However, the much larger problem is that talented people work in vastly different ways. Some meticulously refactor in small steps, others make wide reaching and large changes. Some prefer one language to rule them all, others are comfortable switching between 12-15 different languages in the same day. Monolithic vs separated codebases. Inherited vs duplicated config. It goes on and on. You try to optimize for everyone, to ensure everyone is maximally effective. Pretty quickly you run into this situation-
I eventually left that team, and I left with the feeling that as a team we'd been ignoring Diffusion of responsibility (DOR), Bystander effect (BE), and Crowd psychology (CP). In my opinion the combination of (exclusively) talented, senior developers and collective code ownership led to suboptimal productivity. The belief that "CCO works and we're just doing it wrong" made things worse. It seemed like our solution was always: we just need to do better. You could write this team off as not as talented as I describe, but you'd be mistaken. All of the developers on the team had great success before the project, and (after that team split up) 8 out of 9 have gone on to lead very successful teams.
Several years later I found myself leading a team that was beginning to grow. We had just expanded to a 4 person team, and I could see the high level problems beginning to appear. Consistency had begun slip, and bugs had begun to appear in places where code was "correct" at the micro level, but the system didn't work as expected at the macro level. There are a few ways to manage these issues, pair programming was an obvious solution to this problem, but I believe pair programming would have introduced a different class of problems. I believe switching to pair programming (exclusively) would actually have been net negative for the team. I've written about giving up on exclusive pair programming, I'll leave it at that. The issues we were facing felt vaguely familiar, and I started to wonder if they stemmed from DOR, BE, and CP. Collective code ownership allowed developers to pop in and out of codebases, making small changes to enable features, but where was the person looking at the big picture? "It's everyone's responsibility" wasn't an answer I was comfortable with.
I found myself looking for another solution that would
We've been working with this approach for about a year now and I've been happy with the results. We've had 1 major bug in 12 months and our architecture has remained consistent despite losing 1 team member and gaining 2 more (team size is 5 now). It's probably worth mentioning that my team is entirely remote as well, making those results even more impressive (imo).
A nice side effect of this approach is eased on-boarding of new team-members. After an initial stretch of about a month of co-located pair programming, a new team member is free to work on whatever they want. They can work knowing that their changes will be reviewed by someone with deep understanding of the necessary changes. The rest of the team is confident that the newbie's changes shouldn't cause prod issues, and wont be merged until they're consistent with existing conventions. Another nice side effect of this process (pointed out by Dan Bodart) is that some people are more inclined to polish their code if they know it's going to be featured in a pull request.
It's not (and never is) all roses. Primaries do complain at times about the context switches required to merge pull requests. I haven't found a solution to this issue, yet. In the beginning people complained about having to create branches and create pull requests. However, as the team got comfortable with this workflow, the pain seemed to disappear. Pull requests are often fairly small; the issues do not mirror the complaints against feature branches.
While reviewing this entry Jake McCrary and others pointed out that while context switching is annoying, there are many benefits that also come from the pull request review process. I agree with this observation, and I'm always pleased when I see discussion (comments) occur on a pull request. Perhaps I'm over optimistic, but I always see the discussion as evidence that the team is learning from each other and we are further advancing on our shared goals.
12 months in, I'm happy with the results.
Unsurprisingly, there are other people using and/or experimenting with similar approaches. Paul Nasrat noted the similarities with this approach and the maintainer model of Linux; Scott Robinson pointed me at OWNERS Files; Romily Cocking noted the similarities with the underlying model used in Envy/Developer (pre Java).
Collective Code Ownership (CCO) served me well between 2005 and 2009. Given the make-up of the teams that I was a part of I found it to be the most effective way to deliver software. Around 2009 I joined a team that eventually grew to around 9 people, all very senior developers. The team practiced Collective Code Ownership. Everyone on the team was very talented, but that didn't translate to constant agreement. In fact, we disagreed far more often than I thought we should. That experience drove me to write about the importance of Compatible Opinions. I still believe in the importance of compatible opinions, but I now wonder if the team wouldn't have been more effective (despite incompatible opinions) if we had adopted Weak Code Ownership.
The 2009 project heavily shaped my approach to developing software. I suspect I'm not the only one who (at one time) believed: if we get a team full of massively talented people we can do anything. It turns out, it's not nearly that easy. Too many cooks in the kitchen is the obvious concern, and it does come up. However, the much larger problem is that talented people work in vastly different ways. Some meticulously refactor in small steps, others make wide reaching and large changes. Some prefer one language to rule them all, others are comfortable switching between 12-15 different languages in the same day. Monolithic vs separated codebases. Inherited vs duplicated config. It goes on and on. You try to optimize for everyone, to ensure everyone is maximally effective. Pretty quickly you run into this situation-
If you optimize everything, you will always be unhappy. --Donald KnuthLooking back, I believe our "Collective Code Ownership" degraded to "Last in Wins". People began to cluster around shared opinions as the team grew. Inevitably the components of the project splintered and work included constant angling to ensure you only worked in areas designed to match your personal style. Perhaps that wasn't such a bad thing, but never formalizing the splinters led to constant discussion around responsibility and design. Those discussions always felt like waste to me.
I eventually left that team, and I left with the feeling that as a team we'd been ignoring Diffusion of responsibility (DOR), Bystander effect (BE), and Crowd psychology (CP). In my opinion the combination of (exclusively) talented, senior developers and collective code ownership led to suboptimal productivity. The belief that "CCO works and we're just doing it wrong" made things worse. It seemed like our solution was always: we just need to do better. You could write this team off as not as talented as I describe, but you'd be mistaken. All of the developers on the team had great success before the project, and (after that team split up) 8 out of 9 have gone on to lead very successful teams.
Several years later I found myself leading a team that was beginning to grow. We had just expanded to a 4 person team, and I could see the high level problems beginning to appear. Consistency had begun slip, and bugs had begun to appear in places where code was "correct" at the micro level, but the system didn't work as expected at the macro level. There are a few ways to manage these issues, pair programming was an obvious solution to this problem, but I believe pair programming would have introduced a different class of problems. I believe switching to pair programming (exclusively) would actually have been net negative for the team. I've written about giving up on exclusive pair programming, I'll leave it at that. The issues we were facing felt vaguely familiar, and I started to wonder if they stemmed from DOR, BE, and CP. Collective code ownership allowed developers to pop in and out of codebases, making small changes to enable features, but where was the person looking at the big picture? "It's everyone's responsibility" wasn't an answer I was comfortable with.
I found myself looking for another solution that would
- encourage consistency
- give equal importance to macro and micro quality
- address the other issues caused by DOR, BE, and CP
- A codebase has a "primary", the person who's responsible for that process in production. You could use the word "owner", but I don't think it conveys what I'm looking for. The team still owns the code, but when problems occur the responsibility falls on the primary first.
- The primary drives the architecture of the codebase. The primary gets final say on architectural decisions within a codebase.
- A primary may commit to master a codebase without code review.
- Any commit to master from a non-primary must come via a Pull Request, and only primaries can merge (or give permission for the non-primary to merge).
- note: in emergency circumstances, anyone can commit to any codebase.
- Primaries can change at any time. If a primary feels that another team member is better suited to become the primary they can propose a switch.
- Rotation of primary responsibility occurs naturally as people contribute to different codebases. If you seek to become a primary of a codebase, all you need to do is focus your efforts on features that require changing said codebase. If you're doing the majority of the work in that codebase, you should become the primary in a reasonable amount of time.
We've been working with this approach for about a year now and I've been happy with the results. We've had 1 major bug in 12 months and our architecture has remained consistent despite losing 1 team member and gaining 2 more (team size is 5 now). It's probably worth mentioning that my team is entirely remote as well, making those results even more impressive (imo).
A nice side effect of this approach is eased on-boarding of new team-members. After an initial stretch of about a month of co-located pair programming, a new team member is free to work on whatever they want. They can work knowing that their changes will be reviewed by someone with deep understanding of the necessary changes. The rest of the team is confident that the newbie's changes shouldn't cause prod issues, and wont be merged until they're consistent with existing conventions. Another nice side effect of this process (pointed out by Dan Bodart) is that some people are more inclined to polish their code if they know it's going to be featured in a pull request.
It's not (and never is) all roses. Primaries do complain at times about the context switches required to merge pull requests. I haven't found a solution to this issue, yet. In the beginning people complained about having to create branches and create pull requests. However, as the team got comfortable with this workflow, the pain seemed to disappear. Pull requests are often fairly small; the issues do not mirror the complaints against feature branches.
While reviewing this entry Jake McCrary and others pointed out that while context switching is annoying, there are many benefits that also come from the pull request review process. I agree with this observation, and I'm always pleased when I see discussion (comments) occur on a pull request. Perhaps I'm over optimistic, but I always see the discussion as evidence that the team is learning from each other and we are further advancing on our shared goals.
12 months in, I'm happy with the results.
Unsurprisingly, there are other people using and/or experimenting with similar approaches. Paul Nasrat noted the similarities with this approach and the maintainer model of Linux; Scott Robinson pointed me at OWNERS Files; Romily Cocking noted the similarities with the underlying model used in Envy/Developer (pre Java).
Thursday, January 08, 2015
Preview Arbitrary HTML
I'm a big fan of https://gist.github.com/ for sharing code. It's fantastic for quickly putting something online to talk over with someone else. I've often found myself wishing for something that allowed me to share rendered HTML in the same way.
For example, a gist of the HTML for this blog entry can be seen here: https://gist.github.com/jaycfields/82e2cc0a588bd83a91f4. If I want someone to give me feedback on the rendered output, that gist isn't very helpful.
It turns out, it's really easy to see that HTML rendered: switch the file extension to
I wouldn't use this trick to create any data you'd like to have online in the long term, but it's great for sharing short lived HTML.
For example, a gist of the HTML for this blog entry can be seen here: https://gist.github.com/jaycfields/82e2cc0a588bd83a91f4. If I want someone to give me feedback on the rendered output, that gist isn't very helpful.
It turns out, it's really easy to see that HTML rendered: switch the file extension to
md. Here's the same gist with a md extension: https://gist.github.com/jaycfields/e383b12b6ff9bd49c40a
This all works due to the way Github renders markdown gists, and markdown supporting inline HTML.
I wouldn't use this trick to create any data you'd like to have online in the long term, but it's great for sharing short lived HTML.
Wednesday, January 07, 2015
LTR Org Chart
Most traditional organizations have an Org Chart that looks similar to the following:
Org Charts displayed in this way emphasize the chain of command. A quick look at a node in this Org Chart will let you know who someone reports to and who reports to them. The chart gives a clear indication of responsibility; it also gives an impression of "who's in charge" and how far away you are from the top.
Several years ago I joined ThoughtWorks (TW). TW (at least, when I was there) boasted of their Flat Org Chart and it's advantages.
I've previously written on how joining TW was the single best thing for my career. There were many things I loved about TW - the Flat Org Chart was not one of those things.
The (previously linked) wikipedia article discusses the positive aspects of Flat Organizations - there are several I would agree with. What I find generally missing from the discussion is responsibility. If I make a mistake, who, other than me, could have helped me avoid that mistake? If I need help, whose responsibility is it to ensure I get the help I need? If I'm unhappy and plan to quit, who's going to understand my value and determine what an appropriate counter offer looks like?
In a flat organization, this is me:
Whenever I bring up responsibility people quickly point out that: those are everyone's issues and it's everyone's responsibility. I generally point these people at Diffusion of responsibility, Bystander effect, and Crowd psychology. Personally, I'd love to see the result of, following a resignation, "everyone" determining an individual's value and producing a counter offer.
I'm thankful for the experience I gained working in a Flat Organization. I'll admit that there's something nice about being one manager away from the CEO on an Org Chart. That said, it was the access to a high level manager and the CEO that I found to be the largest benefit of working in a Flat Organization.
I eventually left TW - a move that was partially motivated by the issues I described above. When I began looking for a new job, organizational structure didn't factor into my decision making process, but I did look for something I called Awesome All the Way Up.
One of the two previously referenced companies became my next (and current) employer: DRW Trading. DRW could have a traditional Org Chart, and I would sit towards the "bottom" of it. (note: this is an approximation, not an actual DRW Org Chart.)
Derek and I haven't spoken extensively on the upside down Org Chart, but I assume he's a fan of Servant Leadership. As an employee I (obviously) appreciate the upside down Org Chart and Derek's approach to leadership. As a team lead I find the upside down Org Chart to be a great reminder that I need to put the needs of my teammates first. As someone sitting above Derek on the upside down Org Chart I find the idea of my shit (and the shit of all of my "peers") rolling down to Derek to be... unpleasant.
I was once out with one of the partners from DRW and he introduced me to a friend of his (John). John asked how I knew the DRW partner, I said: he's my boss. The DRW partner interrupted and said: that's not true, we're teammates - I can't do what he does and he can't do what I do, we work together. The partner from this story isn't in my "chain of command", but I find the same team-first attitude present in each person that is.
My discomfort with my shit rolling down on anyone combined with the team-first attitude I've encountered led me to the idea of the Left-to-Right Org Chart (or LTR Org Chart). Below you'll find what would be my current LTR Org Chart.
The LTR Org Chart reminds me to be a Servant Leader while also emphasizing the teammates I can count on to help me out when I encounter issues.
It's probably not practical to generate an LTR Org Chart for every individual in your organization, but it certainly wouldn't be hard to create software to present an LTR Org Chart on a website. It's possible the subtle difference between upside down Org Charts and LTR Org Charts isn't worth the effort required to build the software. I wouldn't mind finding out.
Org Charts displayed in this way emphasize the chain of command. A quick look at a node in this Org Chart will let you know who someone reports to and who reports to them. The chart gives a clear indication of responsibility; it also gives an impression of "who's in charge" and how far away you are from the top.
Several years ago I joined ThoughtWorks (TW). TW (at least, when I was there) boasted of their Flat Org Chart and it's advantages.
A flat organization (also known as horizontal organization or delayering) is an organization that has an organizational structure with few or no levels of middle management between staff and executives. wikipediaA Flat Org Chart may look something like the following image.
The (previously linked) wikipedia article discusses the positive aspects of Flat Organizations - there are several I would agree with. What I find generally missing from the discussion is responsibility. If I make a mistake, who, other than me, could have helped me avoid that mistake? If I need help, whose responsibility is it to ensure I get the help I need? If I'm unhappy and plan to quit, who's going to understand my value and determine what an appropriate counter offer looks like?
In a flat organization, this is me:
Reporting to, thus having access to the person who reports directly to the CEO is great. At the same time, over a hundred other people report to the same person. Would anyone honestly argue that this one individual can guide, mentor, and properly evaluate over a hundred people?
Whenever I bring up responsibility people quickly point out that: those are everyone's issues and it's everyone's responsibility. I generally point these people at Diffusion of responsibility, Bystander effect, and Crowd psychology. Personally, I'd love to see the result of, following a resignation, "everyone" determining an individual's value and producing a counter offer.
I'm thankful for the experience I gained working in a Flat Organization. I'll admit that there's something nice about being one manager away from the CEO on an Org Chart. That said, it was the access to a high level manager and the CEO that I found to be the largest benefit of working in a Flat Organization.
I eventually left TW - a move that was partially motivated by the issues I described above. When I began looking for a new job, organizational structure didn't factor into my decision making process, but I did look for something I called Awesome All the Way Up.
Awesome All the Way Up: Given an Org Chart, do you consider the CEO and every person between you and the CEO to be awesome.I believed then, and I believe now that you'll always be best off if you put yourself in Awesome All the Way Up situations. When I left TW, I only considered 2 companies - they were the only two I could find that offered Awesome All the Way Up. Everyone will likely have their own definition of "awesome". My definition of "awesome", in this context, is: above all, someone I can trust; someone whose vision I believe in; someone I will always have access to.
One of the two previously referenced companies became my next (and current) employer: DRW Trading. DRW could have a traditional Org Chart, and I would sit towards the "bottom" of it. (note: this is an approximation, not an actual DRW Org Chart.)
Personally, I wouldn't have any complaints with this Org Chart. Every person between me and DRW himself plays an important and specific role. Since I'm in an Awesome All the Way Up situation I have access to each of those people. If I have a CSO problem, I go to the CSO; if I have a CTO problem I go to the CTO; you get the idea. I strongly prefer to know whose responsibility it is when I identify what I consider to be a problem.
You may have noticed I said DRW could have an Org Chart similar to what's above. The phrasing was intentional - if you ask our CSO Derek we have an upside down Org Chart. I'm not sure which of those articles best reflects his opinion. He once showed me a picture similar to the one below and said "part of being a leader is understanding that the shit still rolls downhill, but our hill is different".
I was once out with one of the partners from DRW and he introduced me to a friend of his (John). John asked how I knew the DRW partner, I said: he's my boss. The DRW partner interrupted and said: that's not true, we're teammates - I can't do what he does and he can't do what I do, we work together. The partner from this story isn't in my "chain of command", but I find the same team-first attitude present in each person that is.
My discomfort with my shit rolling down on anyone combined with the team-first attitude I've encountered led me to the idea of the Left-to-Right Org Chart (or LTR Org Chart). Below you'll find what would be my current LTR Org Chart.
The LTR Org Chart reminds me to be a Servant Leader while also emphasizing the teammates I can count on to help me out when I encounter issues.
It's probably not practical to generate an LTR Org Chart for every individual in your organization, but it certainly wouldn't be hard to create software to present an LTR Org Chart on a website. It's possible the subtle difference between upside down Org Charts and LTR Org Charts isn't worth the effort required to build the software. I wouldn't mind finding out.
Tuesday, January 06, 2015
Making Remote Work: Tools
Communication is what I consider to be the hardest part of remote work. I haven't found an easy, general solution. A few teammates prefer video chat, others despise it. A few teammates like the wiki as a backlog, a few haven't ever edited the wiki. Some prefer strict usage of email/chat/phone for async-unimportant/async-important/sync-urgent, others tend to use one of those 3 for all communication.As you can tell, we have several different communication tools. When writing, I generally prefer to include concrete examples. This blog entry will list each tool referenced above. However, I cannot emphasize enough that: this list is a snapshot of what we're using, not a recommended set of tools.
app: Github
usage: We use many of the features of Github; however, the two features that help facilitate remote work are (a) pull requests with inline comments and (b) compare. A pull request with inline comments has (thus far) been the most productive way to asynchronously discuss specific pieces of code. Almost all non-trivial commits will eventually end up in a pull request that's reviewed by at least one other team member. We've found compare view to be the best solution for distilling changes for a teammate with limited context.
app: Hipchat
usage: We have 3 hipchat rooms: work, social, support. It should be pretty obvious what we use each room for. The primary driver for splitting the 3 is for keeping noise down. Most team members look at chat history for work and support, reading anything that happened between now and the last time they were logged in. Social tends to be more verbose, often off-topic, and never required reading for keeping up with what the team is up to.
app: Cisco Jabber
usage: Within the team, we primarily use Cisco Jabber for video calls; however Cisco Jabber is also a great way for people within DRW offices to reach anyone on my team without having to know their location. Cisco Jabber is significantly better than asking people to remember to call your cell, or forwarding your desk phone to your cell - it provides you 1 number that anyone can reach you at, regardless of your physical location. There's not much to say about the video capabilities, they're there, they work well. Cisco Jabber also provides desktop sharing, which we use occasionally for "remote pair-programming".
app: Confluence
usage: Our backlog resides on a Confluence wiki; it's a single page with about 150 lines. The backlog is split into 3 sections: Milestones, Now, and Soon. There are generally 3-5 Milestones, which list (as sub-bullets) their dependencies. A dependency is a reference to a line item that will live in Now or Soon. Now is the list of highest priority tasks - the things we need to get down right away. Soon is the list of things that are urgent but not important, or important but not urgent. Both Now and Soon lists contain placeholders for conversations, each placeholder is around 1-2 lines. Below you'll find a contrived, sample backlog.
Milestones
Now
- Deploy to Billy Ray Valentine
- (market data 1)
- (execution 1)
- Automated Order Matching (stakeholder: Mortimer and Randolph Duke)
- (execution 2)
- (reporting 1)
Soon
- Market data
- (1) support pork bellies
- support orange juice
- Execution
- (1) support pork bellies
- (2) internally match incoming orders from customers
- Market data
- support coffee
- support wheat
- Reporting
- (1) Commission summary
note: some things in Now need to be done immediately, but do not support an upcoming milestone. These things are incremental changes for previously met milestones.Obviously we use email and other tools as well, but I can't think of any remote specific usage patterns that are worth sharing.
As I previously mentioned, each member of the team uses each of these tools in their own way. None of these tools are ideal for every member of the team, and I believe a good team lead helps ensure each team member is only required to use the tools they find most helpful.
Sunday, December 28, 2014
Making Remote Work
Over 18 months ago I wrote Year Five, an experience report I never imagined I would write. I closed the blog entry by saying I look forward to writing about Year Six. A year and a half later, I'm still having a hard time deciding what (if anything) I should write. My writers block isn't the result the Remote Work experiment failing. Quite the opposite, the success of the Remote Work experiment has helped shape a team I'm very proud to be a part of, and yet I find myself unable to declare victory.
"How can you work effectively with remote teammates" has become the most common question I hear when meeting up with old colleagues for coffee. Clearly people are interested in the topic. At the same time, I prefer not to write about half-baked ideas (these days), thus my apprehension in documenting my approach.
This entry is an experience report, nothing more, nothing less. I'd be very skeptical of anyone providing recipes or best practices around remote work. Those working remotely today are breaking traditional workplace rules. Some are succeeding, most are failing, and I don't know of anyone with solid general advice. What follows are merely my observations.
My team became remote on accident. My boss and I were a 2 person co-located team in NYC, he quit, they gave me his job, and my best option to fill my previous role lived in Chicago. David Chelimsky was willing to be in NYC 2 weeks out of the month, and I was willing to be in Chicago 1 week out of the month. It was effectively co-location, and our team ran that way for a couple months.
Sometime in month three I started to feel like having David in NYC 2 weeks a month was unnecessary, and (surprisingly to me) he'd independently come to the same conclusion. From that point forward we alternated on traveling 1 week a month - thus we worked remotely for 3 weeks out of 4. We saw no drop off in our productivity or camaraderie, and traveling less definitely improved our overall happiness. I'm not sure how it would have worked out if I'd been forced to try remote work; however, arriving there organically was surprisingly painless.
Eventually the team grew and DRW hired John Hume, who lives in Austin, Texas. I believe it's worth noting that we each lived in a different city. I've always believed that a team needs to either be completely remote or completely co-located. The team is up to 5 people at this point, and I've actively gone out of my way to ensure no two people work out of the same place on an ongoing basis.
When you join the team you'll have to start on a 6 month contract. The contract period gives both you and I the opportunity to figure out if you're a good fit. There are plenty of brilliant people who I wouldn't work with: I'm looking for people with compatible opinions on software who are able to flourish in the environment we've created.
Remote disagreements are hard to resolve. We can't get a beer and talk it out. I can't read your body language (or other subtle signs) and see something needs to change. Thus, it's not enough to be talented, you'll also need to be a philosophical match. The philosophical ideas are easy to agree on when everyone's looking to start a new endeavor, but best intentions don't always equal an ideal working environment. The 6 month contract ensures both parties know what we're getting into long before we discuss the idea of full time employment.
My team is stretched through various timezones, sometimes from London to Los Angeles. This leaves the options of work odd hours, or finding people who can take on larger tasks and don't require constant contact. My team chose to go the latter path, and we've found no notable impact on our ability to collaborate even when our hours overlap as little as 2 hours a day. This choice is another reason the 6 month contract is critical: some people want more than 2 hours of contact a day. That's neither good, nor bad, nor is it easy to predict. If someone ends up needing more contact than they find they're getting, it's better for all parties if everyone goes their separate way after 6 months.
I've found that my opinion on pair-programming and co-location is constantly evolving. In more than 2 years of working remotely, I've probably done less than 4 hours of remote pair-programming. I mention this because some people believe remote pair-programming is an essential ingredient to successful remote work. I am not one of those people.
That said, I try to see my teammates as often as is reasonable, and we do often pair-program when we are co-located. At this point, I see every team member quarterly, and the entire team spends a week together every 6 months. We've found this frequency to be a solid balance between keeping relationships strong and keeping travel to a minimum.
The above schedule works for ongoing collaboration; however, the beginning of a work relationship is almost the exact opposite. When a new contractor joins the team, they travel and co-locate as much as possible. A recent team member spent his first week in Austin, his second in Chicago, and his third in London. By the end of those 3 weeks he'd seen the codebase from 3 different perspectives and spent a week (mostly pairing) with every member of the (at that time, 4 person) team. That much travel is a high up-front cost, but it helps immensely with learning both the codebase and the team you'll be working with.
Communication is what I consider to be the hardest part of remote work. I haven't found an easy, general solution, thus I often find myself duplicating effort to ensure teammates can consume data in their preferred format. A few teammates prefer video chat each time we're on the phone, a few teammates despise video chat. A few teammates like the wiki as a backlog, a few haven't ever edited the wiki (as far as I know). Some prefer strict usage of email/chat/phone for async-unimportant/async-important/sync-urgent, others tend to use one of those 3 for all communication. There hasn't been one tool that I would recommend; instead I think it's much more valuable to note that people prefer different approaches, and it's the job of the team lead to communicate with the team members in the way that they prefer, not the other way around. The only rule I try to apply universally: I end as many conversations as I can with "is there anything I can do to make your life better". If you constantly ask that question, it should be (often painfully) obvious what you need to change to continue to improve things for the team as a whole.
The question of hardware often comes up as well, what should a company provide, what should an individual? My approach: take the cost of a 30" monitor, any laptop, a tablet, a smart phone, and anything else they'd want, then average it out over 2 years. I think you'll find the amount of money is so trivial that you'd be a fool not to buy them whatever they want. (and that you're company loses money every time you waste your time talking about such a small expenditure.)
That's more or less it. I would summarize it like so: I want to create a team that people want to be a part of for at least the next 10 years. That begins by finding people who are a great fit; not everyone will be, and we'll learn valuable lessons from those people as well. We start the relationship out right, spending many hours together getting to know each other and getting to know the ins & outs of the project. From that point on we'll see if your preferred working style (hours, communication needs, etc) fit well with the team. We'll already know if you'll be happy on the team long before either of us has to commit to any long term working relationship. From there, as long as I remember that I work for the team, not the other way around, everyone should continue to be happy and effective.
"How can you work effectively with remote teammates" has become the most common question I hear when meeting up with old colleagues for coffee. Clearly people are interested in the topic. At the same time, I prefer not to write about half-baked ideas (these days), thus my apprehension in documenting my approach.
This entry is an experience report, nothing more, nothing less. I'd be very skeptical of anyone providing recipes or best practices around remote work. Those working remotely today are breaking traditional workplace rules. Some are succeeding, most are failing, and I don't know of anyone with solid general advice. What follows are merely my observations.
My team became remote on accident. My boss and I were a 2 person co-located team in NYC, he quit, they gave me his job, and my best option to fill my previous role lived in Chicago. David Chelimsky was willing to be in NYC 2 weeks out of the month, and I was willing to be in Chicago 1 week out of the month. It was effectively co-location, and our team ran that way for a couple months.
Sometime in month three I started to feel like having David in NYC 2 weeks a month was unnecessary, and (surprisingly to me) he'd independently come to the same conclusion. From that point forward we alternated on traveling 1 week a month - thus we worked remotely for 3 weeks out of 4. We saw no drop off in our productivity or camaraderie, and traveling less definitely improved our overall happiness. I'm not sure how it would have worked out if I'd been forced to try remote work; however, arriving there organically was surprisingly painless.
Eventually the team grew and DRW hired John Hume, who lives in Austin, Texas. I believe it's worth noting that we each lived in a different city. I've always believed that a team needs to either be completely remote or completely co-located. The team is up to 5 people at this point, and I've actively gone out of my way to ensure no two people work out of the same place on an ongoing basis.
When you join the team you'll have to start on a 6 month contract. The contract period gives both you and I the opportunity to figure out if you're a good fit. There are plenty of brilliant people who I wouldn't work with: I'm looking for people with compatible opinions on software who are able to flourish in the environment we've created.
Remote disagreements are hard to resolve. We can't get a beer and talk it out. I can't read your body language (or other subtle signs) and see something needs to change. Thus, it's not enough to be talented, you'll also need to be a philosophical match. The philosophical ideas are easy to agree on when everyone's looking to start a new endeavor, but best intentions don't always equal an ideal working environment. The 6 month contract ensures both parties know what we're getting into long before we discuss the idea of full time employment.
My team is stretched through various timezones, sometimes from London to Los Angeles. This leaves the options of work odd hours, or finding people who can take on larger tasks and don't require constant contact. My team chose to go the latter path, and we've found no notable impact on our ability to collaborate even when our hours overlap as little as 2 hours a day. This choice is another reason the 6 month contract is critical: some people want more than 2 hours of contact a day. That's neither good, nor bad, nor is it easy to predict. If someone ends up needing more contact than they find they're getting, it's better for all parties if everyone goes their separate way after 6 months.
I've found that my opinion on pair-programming and co-location is constantly evolving. In more than 2 years of working remotely, I've probably done less than 4 hours of remote pair-programming. I mention this because some people believe remote pair-programming is an essential ingredient to successful remote work. I am not one of those people.
That said, I try to see my teammates as often as is reasonable, and we do often pair-program when we are co-located. At this point, I see every team member quarterly, and the entire team spends a week together every 6 months. We've found this frequency to be a solid balance between keeping relationships strong and keeping travel to a minimum.
The above schedule works for ongoing collaboration; however, the beginning of a work relationship is almost the exact opposite. When a new contractor joins the team, they travel and co-locate as much as possible. A recent team member spent his first week in Austin, his second in Chicago, and his third in London. By the end of those 3 weeks he'd seen the codebase from 3 different perspectives and spent a week (mostly pairing) with every member of the (at that time, 4 person) team. That much travel is a high up-front cost, but it helps immensely with learning both the codebase and the team you'll be working with.
Communication is what I consider to be the hardest part of remote work. I haven't found an easy, general solution, thus I often find myself duplicating effort to ensure teammates can consume data in their preferred format. A few teammates prefer video chat each time we're on the phone, a few teammates despise video chat. A few teammates like the wiki as a backlog, a few haven't ever edited the wiki (as far as I know). Some prefer strict usage of email/chat/phone for async-unimportant/async-important/sync-urgent, others tend to use one of those 3 for all communication. There hasn't been one tool that I would recommend; instead I think it's much more valuable to note that people prefer different approaches, and it's the job of the team lead to communicate with the team members in the way that they prefer, not the other way around. The only rule I try to apply universally: I end as many conversations as I can with "is there anything I can do to make your life better". If you constantly ask that question, it should be (often painfully) obvious what you need to change to continue to improve things for the team as a whole.
The question of hardware often comes up as well, what should a company provide, what should an individual? My approach: take the cost of a 30" monitor, any laptop, a tablet, a smart phone, and anything else they'd want, then average it out over 2 years. I think you'll find the amount of money is so trivial that you'd be a fool not to buy them whatever they want. (and that you're company loses money every time you waste your time talking about such a small expenditure.)
That's more or less it. I would summarize it like so: I want to create a team that people want to be a part of for at least the next 10 years. That begins by finding people who are a great fit; not everyone will be, and we'll learn valuable lessons from those people as well. We start the relationship out right, spending many hours together getting to know each other and getting to know the ins & outs of the project. From that point on we'll see if your preferred working style (hours, communication needs, etc) fit well with the team. We'll already know if you'll be happy on the team long before either of us has to commit to any long term working relationship. From there, as long as I remember that I work for the team, not the other way around, everyone should continue to be happy and effective.
Wednesday, December 17, 2014
Working Effectively with Unit Tests Official Launch
Today marks the official release release of Working Effectively with Unit Tests. The book is available in various formats:
As far as the softcover edition, I had offers from a few major publishers, but in the end none of them would allow me to continue to sell on leanpub at the same time. I strongly considered caving to the demands of the major publishers, but ultimately the ability to create a high quality softcover and make it available on Amazon was too tempting to pass up.
The feedback has been almost universally positive - the reviews are quite solid on goodreads (http://review.wewut.com). I believe the book provides specific, concise direction for effective Unit Testing, and I hope it helps increase the quality of the unit tests found in the wild.
If you'd like to try before you buy, there's a sample available in pdf format or on the web.
- DRM free pdf, epub, & mobi (Kindle) at http://leanpub.com/wewut
- Softcover at http://amzn.com/1503242706
- Kindle edition at http://amzn.com/B00QS2HXUO
As far as the softcover edition, I had offers from a few major publishers, but in the end none of them would allow me to continue to sell on leanpub at the same time. I strongly considered caving to the demands of the major publishers, but ultimately the ability to create a high quality softcover and make it available on Amazon was too tempting to pass up.
The feedback has been almost universally positive - the reviews are quite solid on goodreads (http://review.wewut.com). I believe the book provides specific, concise direction for effective Unit Testing, and I hope it helps increase the quality of the unit tests found in the wild.
If you'd like to try before you buy, there's a sample available in pdf format or on the web.
Monday, August 25, 2014
The Case for Buying Technical Books
In the past few months I've seen more than a few articles encouraging programmers to write books. Each article provides at least a bit of good advice, and proceeds to conclude with the same idea:
You should write a book to build your brand.
I find this conclusion accurate and extremely disappointing. If the overwhelming reason to write a book is brand building, then the pool of potential authors is restricted to people who would benefit from brand building (and people who don't value their time).
How Did We Get Here?
The Internet, obviously. Practically everyone knows how to download any movie, song, or book at no cost. Opinions on "illegal downloading" range from opposition to pride. I'm not particularly interested in discussing those opinions; however, I believe it's worth observing the impact of the combination of ability and desire to acquire content without compensating the creator.
“Books aren't written - they're rewritten...” -- Michael Crichton
If you've never written a book, you may not be aware of colossal effort it takes to write a mediocre book. When it's all said and done, it can take well over an hour of effort per page. Great books, such as Java Concurrency in Practice, require an even greater level of attention to detail, and cost even more time to create. Brian Goetz estimates that it took them approximately 2,400 hours to create JCiP. If we also knew their royalty structure and the number of copies sold, we'd be able to calculate the hourly rate for writing a high quality book.
It turns out, one of the recent articles encouraging writing gives you royalty numbers and a hint on how many copies a quality book might sell.
Royalties for print should start at 18% of net revenues to the publisher. (Expect that figure to be around $10-20, so you're only making a few dollars on each sale.)...Selling 10 thousand copies of a print tech book these days is a solid success and should be compensated accordingly. -- Obie Fernandez
Let's assume JCiP was more than a solid success and sold 20K copies (doubling Obie's "solid success" benchmark). Assuming they negotiated royalties well, that would mean making $40,000 - thus the hourly rate for writing JCiP would be under $17 per hour.
Clearly I've made a few assumptions, but I believe all of them are based on sound logic. As long as you work 8 hours a day, 5 days a week, for 50 weeks and write a modern classic, you'll make around $34,000 per year. Anecdotal evidence among my author friends who've yet to write a modern classic is worse: the hourly rate is less than minimum wage.
The royalty structures combined with lessening sales create an environment where writing a book for (royalty) profit isn't a reasonable use of your time. As a result, the majority of today's authors are either consultants or unknown programmers. Established, non-consultant programmers gain little from the brand building aspect of writing a book, and likely make far more than $34,000 a year at their full-time jobs - why would they take on a poorly paying second job?
Around 2005 it became fairly easy to download, for free, practically any book. It might be coincidence that 10 of 13 of these Must-Read books were written prior to 2005. Despite the possibility, I don't believe it's a coincidence. Rather, I believe that at one time it paid to create a best selling technical book, and people with various backgrounds took up the challenge.
Nice Assumption Filled History Lesson, What's Your Point?
My point is fairly simple. If you're, like I am, tired of having to choose between books written decades ago and books written by those with at least a slightly ulterior motive, buy some books. Does your company have a book buying policy? If you aren't spending your entire book budget, why not? It costs you nothing to buy a book and give it to a teammate, and every royalty penny reminds an author that someone cares about all of those hours writing and rewriting.
Even if your company doesn't have a book budget, ask yourself if you'd rather your next book about Java be written by a consultant you've never heard of or Java's language architect. The average technical book costs little compared to life's other expenses, and buying a technical book is investing in your profession twice. You stand to gain knowledge both from today's book purchase and a potential future book written by the same author - a future book that may never be written given the current financial incentives.
If you're a CTO, Director or Manager, why aren't you constantly buying books for the developers you work with? They could probably use your advice on which books will best guide their careers.
Makes Sense, What Should I Buy?
There are several good books now available on leanpub - where the authors are paid significantly higher royalties. If you want to support authors you should always start there. From there I would own at least a copy of Chad's (previously referenced) Must-Read books. I'd also buy Chad's Passionate Programmer. Finally, you can't go wrong working your way through this list: Clojure Bookshelf.
Wednesday, July 16, 2014
Solitary Unit Test
Originally found in Working Effectively with Unit Tests
It’s common to unit test at the class level. The
In the same entry, Bill also defines a boundary as: ”... or even an ordinary class if that class is ‘outside’ the area your [sic] trying to work with or are responsible for”. Bill’s recommendation is a good one, but I find it too vague. Bill’s statement fails to give concrete advice on where to draw the line. My second constraint is a concrete (and admittedly restrictive) version of Bill’s recommendation. The concept of constraining a unit test such that ‘the Class Under Test should be the only concrete class found in a test’ sounds extreme, but it’s actually not that drastic if you assume a few things.
Solitary Unit Test can be defined as:
It’s common to unit test at the class level. The
Foo class will have an associated FooTests
class. Solitary Unit Tests follow two additional constraints:
- Never cross boundaries
- The Class Under Test should be the only concrete class found in a test.
In the same entry, Bill also defines a boundary as: ”... or even an ordinary class if that class is ‘outside’ the area your [sic] trying to work with or are responsible for”. Bill’s recommendation is a good one, but I find it too vague. Bill’s statement fails to give concrete advice on where to draw the line. My second constraint is a concrete (and admittedly restrictive) version of Bill’s recommendation. The concept of constraining a unit test such that ‘the Class Under Test should be the only concrete class found in a test’ sounds extreme, but it’s actually not that drastic if you assume a few things.
- You’re using a framework that allows you to easily stub most concrete classes
- This constraint does not apply to any primitive or class that has a literal (e.g. int, Integer, String, etc)
- You’re using some type of automated refactoring tool.
Solitary Unit Test can be defined as:
Solitary Unit Testing is an activity by which methods of a class or functions of a namespace are tested to determine if they are fit for use. The tests used to determine if a class or namespace is functional should isolate the class or namespace under test by stubbing all collaboration with additional classes and namespaces.
Monday, June 30, 2014
Working Effectively with Unit Tests Rough Draft Complete
I finally put the finishing touches on the rough draft of Working Effectively with Unit Tests. It's been an interesting journey thus far, and I'm hoping the attention to detail I've put into the rough draft will translate into an enjoyable read.
What I did poorly: I'd written the book's sample before I ever put it on leanpub. Before a book is published you can collect contact and price information from those who are interested. However, once you publish and begin selling, you no longer have the ability to collect the previously mentioned information. I published and began selling my book immediately - and forfeited my chance to collect that information.
What I did well: I published early and often. I can't say enough nice things about leanpub. I've gotten tons of feedback on example style, writing style, typos, and content. One reader's suggestion to switch to Kevlin Henney's Java formatting style made my book enjoyable to read on a Kindle. I had twitter followers apologizing for "being pedantic and pointing out typos", and I couldn't have been happier to get the feedback. Each typo I fix makes the book more enjoyable for everyone. If you're going to write a book, get it on leanpub asap and start interacting with your audience.
What I learned from Refactoring: Ruby Edition (RRE): RRE contains errors, far too many errors. I vowed to find a better way this time around, and I'm very happy with the results. Every example test in the book can be run, and uses classes also shown in the book. However, writing about tests is a bit tricky: sometimes "failure" is the outcome you're looking to document. Therefore, I couldn't simply write tests for everything. Instead I piped the output to files and used them as example output in the book, but also as verification that what failed once continued to fail in the future (and vice versa). WEwUT has a script that runs every test from the book and overwrites the output files. If the output files are unchanged, I know all the passing examples are still correctly passing, and all the failing examples are still correctly failing. In a way, git diff became my test suite output. I'm confident in all the code found in WEwUT, and happy to be able to say it's all "tested".
What's unclear: Using leanpub was great, but I'm not really sure how to get the word out any further at this point. I set up a goodreads.com page and many friends have been kind enough to tweet about it, but I don't really have any other ideas at this point. I've reached out to a few publishers to see about creating a paperback, and I suspect a print version will increase interest. Still, I can't help thinking there's something else I should be doing between now and paperback launch.
What's next: The rough draft is 100% complete, but I expect to continue to get feedback over the next month or so. As long as the feedback is coming in, I'll be doing updates and publishing new versions.
If you've already bought the book, thank you for the support. It takes 10 seconds to get a pdf of any book you want these days, and I can't thank you enough for monetarily supporting all the effort I've put into WEwUT. If you haven't bought the book, you're welcome to give the sample a read for free. I hope you'll find it enjoyable, and I would gladly accept any feedback you're willing to provide.
What I did poorly: I'd written the book's sample before I ever put it on leanpub. Before a book is published you can collect contact and price information from those who are interested. However, once you publish and begin selling, you no longer have the ability to collect the previously mentioned information. I published and began selling my book immediately - and forfeited my chance to collect that information.
What I did well: I published early and often. I can't say enough nice things about leanpub. I've gotten tons of feedback on example style, writing style, typos, and content. One reader's suggestion to switch to Kevlin Henney's Java formatting style made my book enjoyable to read on a Kindle. I had twitter followers apologizing for "being pedantic and pointing out typos", and I couldn't have been happier to get the feedback. Each typo I fix makes the book more enjoyable for everyone. If you're going to write a book, get it on leanpub asap and start interacting with your audience.
What I learned from Refactoring: Ruby Edition (RRE): RRE contains errors, far too many errors. I vowed to find a better way this time around, and I'm very happy with the results. Every example test in the book can be run, and uses classes also shown in the book. However, writing about tests is a bit tricky: sometimes "failure" is the outcome you're looking to document. Therefore, I couldn't simply write tests for everything. Instead I piped the output to files and used them as example output in the book, but also as verification that what failed once continued to fail in the future (and vice versa). WEwUT has a script that runs every test from the book and overwrites the output files. If the output files are unchanged, I know all the passing examples are still correctly passing, and all the failing examples are still correctly failing. In a way, git diff became my test suite output. I'm confident in all the code found in WEwUT, and happy to be able to say it's all "tested".
What's unclear: Using leanpub was great, but I'm not really sure how to get the word out any further at this point. I set up a goodreads.com page and many friends have been kind enough to tweet about it, but I don't really have any other ideas at this point. I've reached out to a few publishers to see about creating a paperback, and I suspect a print version will increase interest. Still, I can't help thinking there's something else I should be doing between now and paperback launch.
What's next: The rough draft is 100% complete, but I expect to continue to get feedback over the next month or so. As long as the feedback is coming in, I'll be doing updates and publishing new versions.
If you've already bought the book, thank you for the support. It takes 10 seconds to get a pdf of any book you want these days, and I can't thank you enough for monetarily supporting all the effort I've put into WEwUT. If you haven't bought the book, you're welcome to give the sample a read for free. I hope you'll find it enjoyable, and I would gladly accept any feedback you're willing to provide.
Wednesday, May 21, 2014
Working Effectively with Unit Tests
Unit Testing has moved from fringe to mainstream, which is a great thing. Unfortunately, as a side effect developers are creating mountains of unmaintainable tests. I've been fighting the maintenance battle pretty aggressively for years, and I've decided to write a book that captures what I believe is the most effective way to test.
I'm currently ~25% done with the book, and it's available now for $14.99. My plan is to raise the price to $19.99 when I'm 50% done, and $24.99 when I'm 75% done. Leanpub offers my book with 100% Happiness Guarantee: Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks. Therefore, if you find the above or the free sample interesting, you might want to buy it now and save a few bucks.
Buy Now here: https://leanpub.com/wewut
From the PrefaceThe book does touch on some theory and definition, but the main purpose is to show you how to take tests that are causing you pain and turn them into tests that you're happy to work with.
Over a dozen years ago I read Refactoring for the first time; it immediately became my bible. While Refactoring isn’t about testing, it explicitly states: If you want to refactor, the essential precondition is having solid tests. At that time, if Refactoring deemed it necessary, I unquestionably complied. That was the beginning of my quest to create productive unit tests.
Throughout the 12+ years that followed reading Refactoring I made many mistakes, learned countless lessons, and developed a set of guidelines that I believe make unit testing a productive use of programmer time. This book provides a single place to examine those mistakes, pass on the lessons learned, and provide direction for those that want to test in a way that I’ve found to be the most productive.
For example, the book demonstrates how to go from...As of right now, you can read the first 2 chapters for free at https://leanpub.com/wewut/read
looping test with many (built elsewhere) collaborators
.. to individual tests that expect literals, limit scope, explicitly define collaborators, and focus on readability
.. to fine-grained tests that focus on testing a single responsibility, are resistant to cascading failures, and provide no friction for those practicing ruthless Refactoring.
I'm currently ~25% done with the book, and it's available now for $14.99. My plan is to raise the price to $19.99 when I'm 50% done, and $24.99 when I'm 75% done. Leanpub offers my book with 100% Happiness Guarantee: Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks. Therefore, if you find the above or the free sample interesting, you might want to buy it now and save a few bucks.
Buy Now here: https://leanpub.com/wewut
Monday, May 19, 2014
Weighing in on Long Live Testing
DHH recently wrote a provocative piece that gave some views into how he does and doesn't test these days. While I don't think I agree with him completely, I applaud his willingness to speak out against TDD dogma. I've written publicly about not buying the pair-programming dogma, but I hadn't previously been brave enough to admit that I no longer TDD the vast majority of the time.
The truth is, I haven't been dogmatic about TDD in quite some time. Over 6 years ago I was on a ThoughtWorks project where I couldn't think of a single good reason to TDD the code I was working on. To be honest, there weren't really any reasons that motivated me to write tests at all. We were working on a fairly simple, internal application. They wanted software as fast as they could possibly get it, and didn't care if it crashed fairly often. We kept everything simple, manually tested new features through the UI, and kept our customer's very happy.
There were plenty of reasons that we could have written tests. Reasons that I expect people will want to yell at me right now. To me, that's actually the interesting, and missing part, of the latest debate on TDD. I don't see people asking: Why are we writing this test? Is TDD good or bad? That depends; TDD is just a tool, and often the individual is the determining factor when it comes to how effective a tool is. If we start asking "Why?", it's possible to see how TDD could be good for some people, and bad for DHH.
I've been quietly writing a book on Working Effectively with Unit Tests, and I'll have to admit that it was really, really hard not to jump into the conversation with some of the content I've recently written. Specifically, I think this paragraph from the Preface could go a long way to helping people understand an opposing argument.
I find myself doing both. Different development activities often require different tools; i.e. Depending on what I'm doing, different motivators apply, and what tests I write change (hopefully) appropriately.
To be honest, if you look at your tests in the context of the motivators above, that's probably all you need to help you determine whether or not your tests are making you more or less effective. However, if you want more info on what I'm describing, you can pick up the earliest version of my upcoming book. (cheaply, with a full refund guarantee)
The truth is, I haven't been dogmatic about TDD in quite some time. Over 6 years ago I was on a ThoughtWorks project where I couldn't think of a single good reason to TDD the code I was working on. To be honest, there weren't really any reasons that motivated me to write tests at all. We were working on a fairly simple, internal application. They wanted software as fast as they could possibly get it, and didn't care if it crashed fairly often. We kept everything simple, manually tested new features through the UI, and kept our customer's very happy.
There were plenty of reasons that we could have written tests. Reasons that I expect people will want to yell at me right now. To me, that's actually the interesting, and missing part, of the latest debate on TDD. I don't see people asking: Why are we writing this test? Is TDD good or bad? That depends; TDD is just a tool, and often the individual is the determining factor when it comes to how effective a tool is. If we start asking "Why?", it's possible to see how TDD could be good for some people, and bad for DHH.
I've been quietly writing a book on Working Effectively with Unit Tests, and I'll have to admit that it was really, really hard not to jump into the conversation with some of the content I've recently written. Specifically, I think this paragraph from the Preface could go a long way to helping people understand an opposing argument.
I don't actually know what motivates DHH to test, but if we assumed he cares about validating the system, preventing future regressions, and enabling refactoring (exclusively) then there truly is no reason to TDD. That doesn't mean you shouldn't; it just means, given what he values and how he works, TDD isn't valuable to him. Of course, conversely, if you value immediate feedback, problems in small pieces, and tests as clients that shape design, TDD is probably invaluable to you.Why Test?
The answer was easy for me: Refactoring told me to. Unfortunately, doing something strictly because someone or something told you to is possibly the worst approach you could take. The more time I invested in testing, the more I found myself returning to the question: Why am I writing this test?
There are many motivators for creating a test or several tests:Some of the above motivators are healthy in the right context, others are indicators of larger problems. Before writing any test, I would recommend deciding which of the above are motivating you to write a test. If you first understand why you're writing a test, you'll have a much better chance of writing a test that is maintainable and will make you more productive in the long run.
- validating the system
- immediate feedback that things work as expected
- prevent future regressions
- increase code-coverage
- enable refactoring of legacy codebase
- document the behavior of the system
- your manager told you to
- Test Driven Development
- improved design
- breaking a problem up into smaller pieces
- defining the "simplest thing that could possibly work"
- customer approval
- ping pong pair-programming
Once you start looking at tests while considering the motivator, you may find you have tests that aren't actually making you more productive. For example, you may have a test that increases code-coverage, but provides no other value. If your team requires 100% code-coverage, then the test provides value. However, if you team has abandoned the (in my opinion harmful) goal of 100% code-coverage, then you're in a position to perform my favorite refactoring: delete.
I find myself doing both. Different development activities often require different tools; i.e. Depending on what I'm doing, different motivators apply, and what tests I write change (hopefully) appropriately.
To be honest, if you look at your tests in the context of the motivators above, that's probably all you need to help you determine whether or not your tests are making you more or less effective. However, if you want more info on what I'm describing, you can pick up the earliest version of my upcoming book. (cheaply, with a full refund guarantee)
Monday, January 27, 2014
REPL Driven Development
When I describe my current workflow I use the TLA RDD, which is short for REPL Driven Development. I've been using REPL Driven Development for all of my production work for awhile now, and I find it to be the most effective workflow I've ever used. RDD differs greatly from any workflow I've used in the past, and (despite my belief that it's superior) I've often had trouble concisely describing what makes the workflow so productive. This entry is an attempt to describe what I consider RDD to be, and to demonstrate why I find it the most effective way to work.
If that isn't clear, hopefully the video below demonstrates what I'm talking about.
If you're unfamiliar with RDD, the previous video might leave you wondering: What's so impressive about RDD? To answer that question, I think it's worth making explicit what the video is: an example of a running application that needs to change, a change taking place, and verification that the application runs as desired. The video demonstrates change and verification; what makes RDD so effective to me is what's missing: (a) restarting the application, (b) running something other than the application to verify behavior, and (c) moving out of the source to execute arbitrary code. Eliminating those 3 steps allows me to focus on what's important, writing and running code that will be executed in production.
In my career I've spent significant time writing applications in C#, Ruby, & Java. While working in C# and Java, if I wanted to make and verify (in the application) any non-trivial change to an application, I would need to stop the application, rebuild/recompile, & restart the application. I found the slowness of this feedback loop to be unacceptable, and wholeheartedly embraced tools such as NUnit and JUnit.
I've never been as enamored with TDD as some of my peers; regardless, I absolutely endorsed it. The Design aspect of TDD was never that enticing to me, but tests did allow me to get feedback at a significantly superior pace. Tests also provide another benefit while working with C# & Java: They're the poorest man's REPL. Need to execute some arbitrary code? Write a test, that you know you're going to immediately delete, and execute away. Of course, tests have other pros and cons. At this moment I'm limiting my discussion around tests to the context of rapid feedback, but I'll address TDD & RDD later in this entry.
Ruby provided a more effective workflow (technically, Rails provided a more effective workflow). Rails applications I worked on were similar to my RDD experience: I was able to make changes to a running application, refresh a webpage and see the result of the new behavior. Ruby also provided a REPL, but I always ran the REPL external to my editor (I knew of no other option). This workflow was the closest, in terms of efficiency, that I've ever felt to what I have with RDD; however, there are some minor differences that do add up to an inferior experience: (a) having to switch out of a source file to execute arbitrary code is an unnecessary nuisance and (b) refreshing a webpage destroys any client side state that you've built up. I have no idea if Ruby now has editor & repl integration, if it does, then it's likely on par with the experience I have now.
When working on a feature, the short term goal is to have it working in the application as fast as possible. Arbitrary execution, live changes, and only writing what you need are 3 things that can help you complete that short term goal as fast as possible. The video above is the best example I have of how you go from a feature request to software that does what you want in the smallest amount of time. In the video, I only leave the buffer to verify that the application works as intended. If the short term goal was the only goal, RDD without writing tests would likely be the solution. However, we all know that that are many other goals in software. Good design is obviously important. If you think tests give you better design, then you should probably mix both TDD & RDD. Preventing regression is also important, and that can be accomplished by writing tests after you have a working feature that you're satisfied with. Regression tests are great for giving confidence that a feature works as intended and will continue to in the future.
REPL Driven Development doesn't need to replace your current workflow, it can also be used to extend your existing TDD workflow.
RDD Cycle
First, I'd like to address the TLA RDD. I use the term RDD because I'm relying on the REPL to drive my development. More specifically, when I'm developing, I create an s-expression that I believe will solve my problem at hand. Once I'm satisfied with my s-expression, I send that s-expression to the REPL for immediate evaluation. The result of sending an s-expression can either be a value that I manually inspect, or it can be a change to a running application. Either way, I'll look at the result, determine if the problem is solved, and repeat the process of crafting an s-expression, sending it to the REPL, and evaluating the result.If that isn't clear, hopefully the video below demonstrates what I'm talking about.
If you're unfamiliar with RDD, the previous video might leave you wondering: What's so impressive about RDD? To answer that question, I think it's worth making explicit what the video is: an example of a running application that needs to change, a change taking place, and verification that the application runs as desired. The video demonstrates change and verification; what makes RDD so effective to me is what's missing: (a) restarting the application, (b) running something other than the application to verify behavior, and (c) moving out of the source to execute arbitrary code. Eliminating those 3 steps allows me to focus on what's important, writing and running code that will be executed in production.
Feedback
I've found that, while writing software, getting feedback is the single largest time thief. Specifically, there are two types of feedback that I want to get as quickly as possible: (1) Is my application doing what I believe it is? (2) What does this arbitrary code return when executed? I believe the above video demonstrates how RDD can significantly reduce the time needed to answer both of those questions.In my career I've spent significant time writing applications in C#, Ruby, & Java. While working in C# and Java, if I wanted to make and verify (in the application) any non-trivial change to an application, I would need to stop the application, rebuild/recompile, & restart the application. I found the slowness of this feedback loop to be unacceptable, and wholeheartedly embraced tools such as NUnit and JUnit.
I've never been as enamored with TDD as some of my peers; regardless, I absolutely endorsed it. The Design aspect of TDD was never that enticing to me, but tests did allow me to get feedback at a significantly superior pace. Tests also provide another benefit while working with C# & Java: They're the poorest man's REPL. Need to execute some arbitrary code? Write a test, that you know you're going to immediately delete, and execute away. Of course, tests have other pros and cons. At this moment I'm limiting my discussion around tests to the context of rapid feedback, but I'll address TDD & RDD later in this entry.
Ruby provided a more effective workflow (technically, Rails provided a more effective workflow). Rails applications I worked on were similar to my RDD experience: I was able to make changes to a running application, refresh a webpage and see the result of the new behavior. Ruby also provided a REPL, but I always ran the REPL external to my editor (I knew of no other option). This workflow was the closest, in terms of efficiency, that I've ever felt to what I have with RDD; however, there are some minor differences that do add up to an inferior experience: (a) having to switch out of a source file to execute arbitrary code is an unnecessary nuisance and (b) refreshing a webpage destroys any client side state that you've built up. I have no idea if Ruby now has editor & repl integration, if it does, then it's likely on par with the experience I have now.
Semantics
I like Simon's description, but I don't believe that we need to break things down to two different TLAs. Quite simply, (sadly) I don't think enough people are developing in this way, and the additional specification causes a bit of confusion among people who aren't familiar with RDD. However, Simon's description is so spot on I felt the need to describe why I'm choosing to ignore his classifications.-- Simon Katz
- It's important to distinguish between two meanings of "REPL" - one is a window that you type forms into for immediate evaluation; the other is the process that sits behind it and which you can interact with from not only REPL windows but also from editor windows, debugger windows, the program's user interface, etc.
- It's important to distinguish between REPL-based development and REPL-driven development:
- REPL-based development doesn't impose an order on what you do. It can be used with TDD or without TDD. It can be used with top-down, bottom-up, outside-in and inside-out approaches, and mixtures of them.
- REPL-driven development seems to be about "noodling in the REPL window" and later moving things across to editor buffers (and so source files) as and when you are happy with things. I think it's fair to say that this is REPL-based development using a series of mini-spikes. I think people are using this with a bottom-up approach, but I suspect it can be used with other approaches too.
RDD & TDD
RDD and TDD are not in direct conflict with each other. As Simon notes above, you can do TDD backed by a REPL. Many popular testing frameworks have editor specific libraries that provide immediate feedback through REPL interaction.When working on a feature, the short term goal is to have it working in the application as fast as possible. Arbitrary execution, live changes, and only writing what you need are 3 things that can help you complete that short term goal as fast as possible. The video above is the best example I have of how you go from a feature request to software that does what you want in the smallest amount of time. In the video, I only leave the buffer to verify that the application works as intended. If the short term goal was the only goal, RDD without writing tests would likely be the solution. However, we all know that that are many other goals in software. Good design is obviously important. If you think tests give you better design, then you should probably mix both TDD & RDD. Preventing regression is also important, and that can be accomplished by writing tests after you have a working feature that you're satisfied with. Regression tests are great for giving confidence that a feature works as intended and will continue to in the future.
REPL Driven Development doesn't need to replace your current workflow, it can also be used to extend your existing TDD workflow.
Subscribe to:
Posts (Atom)