Someone recently asked me for information on starting new consulting engagements. A few years back I published Sean Doran and Scott Conley's thoughts on Being a Lead Consultant. Sean and Scott's list is great for any lead consultant, and the advice applies well for the lifetime of a project. I considered sending that list to the person looking for new engagement advice, but I'm not sure that list would be the best place to focus my attention at the beginning of a project.
The beginning of a project is a special and dangerous time. You get a mix of (at least) optimism, concern, and freedom. There are a large number of ways the project can go, and as a lead consultant you'll play a major role determining it's outcome. The specific question I recently received was: did you have a process you followed when new engagements began. The remainder of this blog post contains my (slightly edited) response.
I found being a lead at ThoughtWorks to be a nearly impossible balancing act. It's possible we didn't have a process because we weren't organized enough, but it's more likely that there's no general formula that works.
What we ran into constantly was what we called the "enablement versus delivery" issue: If you're teaching client devs (enablement), you really don't have time to meet delivery deadlines; conversely if you're delivering software you rarely have time to teach. How to balance enablement versus delivery is something that varies based on the client's skillset and the ROI of the software being developed. What makes it worse is that the client often thinks they need one thing, but if the the company didn't have issues you (likely) wouldn't be there. Sometimes the issues are with the stakeholder and they'll tell you to focus on the wrong thing, and sometimes the stakeholder knows the deal but the other people you are forced to work with are the problem.
Often it's a race to figure out what the client needs and get them to agree to fix it, before you've lost too much good will.
Another issue is, most software is worthless within a few years, and the only way to break a perpetual cycle of mediocrity is to level up the people and processes. This can lead some people to think that delivery of a specific piece of software is secondary to improving process and people. There's a major flaw to that approach though; a non-TW consultant once said that he feels guilty about his job, because when he helps people become better the most talented always end up leaving the (suboptimal environment of the) client. You can help people improve and install good processes, but when the good people leave and the remaining don't understand the foundations of the process, you end up with not enough talent and a process that's loosely followed and for none of the right reasons.
That said, you can't focus exclusively on delivery. I once led a team that beat all deadlines, created a great piece of software, and provided a great deal of content for Martin Fowler's DSL book. It was a "huge success" until the client devs took over, couldn't maintain it, and wrote their own version that had 20% of the functionality we provided. The software we wrote was classified as a "proof of concept" and thrown out.
I was somewhat oblivious to all of this in my first few years at ThoughtWorks; I would work with the talented clients while isolating the less talented. Eventually you find out that a stakeholder will usually fire you before their worst employee, regardless of how obvious it is.
If I were going into a new engagement these days, I would split my time between training and delivery initially. After figuring out which is the bigger problem, you can spend more or less time on training or delivery.
I would also keep a spreadsheet with every client employee and their talents; every one of those employees will impact your success. If you find what they're good at and get them doing that, you may have found an ally and advocate. Every employee that has no talents listed in your spreadsheet is not only slowing you down, but is also likely (consciously or unconsciously) sabotaging you in every discussion you aren't a part of. Assume you cannot get rid of them, and don't bother trying; spending political capital managing client staff is a bad investment. Keep them on your sheet and make finding their talent a top priority.
Good luck, it's not an easy gig. Then again, if things don't go well you can always move on to another client. That was what always kept me sane. I always did the best job I could, but I also knew if I failed at an impossible task it wasn't the end of the world. There's an endless stream of impossible tasks available.
Wednesday, May 17, 2017
Tuesday, June 28, 2016
Curious Customer
I currently work on a pretty small team, 4 devs (including myself). We have no one dedicated strictly to QA. A few years ago we ran into a few unexpected issues with our software. I hesitate to call them bugs, because they only appeared when you did things that made little sense. We write internal-only software, thus we expect a minimum level of competency from our users. In addition, it's tempting justify ignoring problematic nonsensical behavior in the name of not having to write and maintain additional software.
But, when I wasn't in denial, I was willing to admit that these were in fact bugs and they were costing us time.
The problems caused by these bugs were small, e.g. a burst of worthless emails, a blip in data flowing to the application. The emails could be quickly deleted, and the application was eventually consistent. Thus I pretended as though these issues were of low importance, and that the pain was low for both myself and our customers. I imagine that sounds foolish; in retrospect, it was foolish. The cost of developer context switching is often very high, higher if it's done as an interrupt. Introducing noise into your error reporting devalues your error reporting. Users can't as easily differentiate between good data, a blip of bad data due to something they did, and actual bad data, thus they begin to distrust all of the data.
The cost of these bugs created by nonsensical behavior is high, much higher than the cost of writing and maintaining the software that eliminated these bugs.
Once we eliminated these bugs, I spent notably more time happily focused on writing software. For me, delivering features is satisfying; conversely, tracking down issues stemming from nonsensical behavior always feels like a painfully inefficient task. I became very intent on avoiding that inefficiency in the future. The team brainstormed on how to address this behavior, and honestly we came up with very little. We already write unit tests, load tests, and integration tests. Between all of our tests, we catch the majority of our bugs before they hit production. However, this was a different type of bug, created by behavior a developer often wouldn't think of, thus a developer wasn't very likely to write a test that would catch this issue.
I proposed an idea I wasn't very fond of, the Curious Customer (CC): upon delivery of any feature you could ask another developer on the team to use the feature in the staging environment, acting as a user curiously toying with all aspects of the feature.
Over a year later, I'm not sure it's such a bad idea. In that year we've delivered several features, and (prior to release) I've found several bugs while playing the part of CC. I can't remember a single one of them that would have led to a notable problem in production; however all of them would have led to at least one support call, and possibly a bit less trust in our software.
My initial thought was: asking developers to context switch to QAing some software they didn't write couldn't possibly work, could it? Would they give it the necessary effort, or would they half-ass the task and get back to coding?
For fear of half-ass, thus wasted effort, I tried to define the CC's responsibilities very narrowly. CC was an option, not a requirement; if you delivered a feature you could request a CC, but you could also go to production without a CC. A CC was responsible for understanding the domain requirements, not the technical requirements. It's the developers responsibility to get the software to staging, the CC should be able to open staging and get straight to work. If the CC managed to crash or otherwise corrupt staging, it was the developers responsibility to get things back to a good state. The CC doesn't have an official form or process for providing feedback; The CC may chose email, chat, or any mechanism they prefer for providing feedback.
That's the idea, more or less. I've been surprised and pleased at the positive impact CC has had. It's not life changing, but it does reduce the number of support calls and the associated waste with tracking down largely benign bugs, at least, on our team.
You might ask how this differs from QA. At it's core, I'm not sure it does in any notable way. That said, I believe traditional QA differs in a few interesting ways. Traditional QA is often done by someone whose job is exclusively QA. With that in mind, I suppose we could follow the "devops" pattern and call this something like "devqa", but that doesn't exactly roll off the tongue. Traditional QA is also often a required task, every feature and/or build requires QA sign off. Finally, the better QA engineers I've worked with write automated tests that continually run to prevent regression; A CC may write a script or two for a single given task, but those scripts are not expected to be valuable to any other team member now or for anyone (including the author) at any point in the future.
But, when I wasn't in denial, I was willing to admit that these were in fact bugs and they were costing us time.
The problems caused by these bugs were small, e.g. a burst of worthless emails, a blip in data flowing to the application. The emails could be quickly deleted, and the application was eventually consistent. Thus I pretended as though these issues were of low importance, and that the pain was low for both myself and our customers. I imagine that sounds foolish; in retrospect, it was foolish. The cost of developer context switching is often very high, higher if it's done as an interrupt. Introducing noise into your error reporting devalues your error reporting. Users can't as easily differentiate between good data, a blip of bad data due to something they did, and actual bad data, thus they begin to distrust all of the data.
The cost of these bugs created by nonsensical behavior is high, much higher than the cost of writing and maintaining the software that eliminated these bugs.
Once we eliminated these bugs, I spent notably more time happily focused on writing software. For me, delivering features is satisfying; conversely, tracking down issues stemming from nonsensical behavior always feels like a painfully inefficient task. I became very intent on avoiding that inefficiency in the future. The team brainstormed on how to address this behavior, and honestly we came up with very little. We already write unit tests, load tests, and integration tests. Between all of our tests, we catch the majority of our bugs before they hit production. However, this was a different type of bug, created by behavior a developer often wouldn't think of, thus a developer wasn't very likely to write a test that would catch this issue.
I proposed an idea I wasn't very fond of, the Curious Customer (CC): upon delivery of any feature you could ask another developer on the team to use the feature in the staging environment, acting as a user curiously toying with all aspects of the feature.
Over a year later, I'm not sure it's such a bad idea. In that year we've delivered several features, and (prior to release) I've found several bugs while playing the part of CC. I can't remember a single one of them that would have led to a notable problem in production; however all of them would have led to at least one support call, and possibly a bit less trust in our software.
My initial thought was: asking developers to context switch to QAing some software they didn't write couldn't possibly work, could it? Would they give it the necessary effort, or would they half-ass the task and get back to coding?
For fear of half-ass, thus wasted effort, I tried to define the CC's responsibilities very narrowly. CC was an option, not a requirement; if you delivered a feature you could request a CC, but you could also go to production without a CC. A CC was responsible for understanding the domain requirements, not the technical requirements. It's the developers responsibility to get the software to staging, the CC should be able to open staging and get straight to work. If the CC managed to crash or otherwise corrupt staging, it was the developers responsibility to get things back to a good state. The CC doesn't have an official form or process for providing feedback; The CC may chose email, chat, or any mechanism they prefer for providing feedback.
That's the idea, more or less. I've been surprised and pleased at the positive impact CC has had. It's not life changing, but it does reduce the number of support calls and the associated waste with tracking down largely benign bugs, at least, on our team.
You might ask how this differs from QA. At it's core, I'm not sure it does in any notable way. That said, I believe traditional QA differs in a few interesting ways. Traditional QA is often done by someone whose job is exclusively QA. With that in mind, I suppose we could follow the "devops" pattern and call this something like "devqa", but that doesn't exactly roll off the tongue. Traditional QA is also often a required task, every feature and/or build requires QA sign off. Finally, the better QA engineers I've worked with write automated tests that continually run to prevent regression; A CC may write a script or two for a single given task, but those scripts are not expected to be valuable to any other team member now or for anyone (including the author) at any point in the future.
Thursday, June 16, 2016
Maintainability and Expect Literals
Recently, Stephen Schaub asked the following on the wewut group:
In general, given the option of having many similar strings (or any literal) vs a helper function, I would always prefer the literal. When a test is failing I only care about that single failing test. If I have to look at the helper function I no longer have the luxury of staying focused on the single test; now I need to consider what the helper function is giving me and what it's giving all other callers. Suddenly the scope of my work has shifted from one test to all of the tests coupled by this helper function. If this helper function wasn't written by me, this expansion in scope wasn't even my decision, it was forced upon me by the helper function creator. In the best case the helper function could return a single, constant string. The scope expansion becomes even worse when the helper function contains code branches.
As for alternatives, my solution would depend on the problem. If the strings were fairly consistent, I would likely simply duplicate everything knowing that any formatting changes can likely be addressed using a bulk edit via find and replace. If the strings were not consistent, I would look at breaking up the methods in a way that would allow me to verify the code branches using as little duplication as possible, e.g. if I wanted to test a string that dynamically changed based on a few variables, I would look to test those variables independently, and then only have a few tests for the formatting.
A concrete example will likely help here. Say I'm writing a trading system and I need to display messages such as
There's quite a bit of variation in those messages. You could have 1 function that creates the entire string:
However, I think you'd end up with quite a few verbose tests. Given this problem, I would look to break down those strings into smaller, more focused functions, for example:
Now that you've broken down the function, you're free to test the code paths of the more focused functions, and the test for confirmMsg becomes trivial. Something along the lines of
Several of the unit test examples in the book verify the construction of both HTML and plain text strings. Jay recommends using literal strings in the assertions. However, this strikes me as not a particularly maintainable approach. If the requirements regarding the formatting of these strings changes (a very likely scenario), every single test that verifies one of these strings using a literal must be updated. Combined with the advice that each test should check only one thing, this leads to a large number of extremely brittle tests.I responded to Stephen; however, I've seen similar questions asked a few times. Below are my extended thoughts regarding literals as expected values.
Am I missing something here? I can appreciate the reasons Jay recommends using literals in the tests. However, it seems that we pay a high maintainability price in exchange for the improved readability.
In general, given the option of having many similar strings (or any literal) vs a helper function, I would always prefer the literal. When a test is failing I only care about that single failing test. If I have to look at the helper function I no longer have the luxury of staying focused on the single test; now I need to consider what the helper function is giving me and what it's giving all other callers. Suddenly the scope of my work has shifted from one test to all of the tests coupled by this helper function. If this helper function wasn't written by me, this expansion in scope wasn't even my decision, it was forced upon me by the helper function creator. In the best case the helper function could return a single, constant string. The scope expansion becomes even worse when the helper function contains code branches.
As for alternatives, my solution would depend on the problem. If the strings were fairly consistent, I would likely simply duplicate everything knowing that any formatting changes can likely be addressed using a bulk edit via find and replace. If the strings were not consistent, I would look at breaking up the methods in a way that would allow me to verify the code branches using as little duplication as possible, e.g. if I wanted to test a string that dynamically changed based on a few variables, I would look to test those variables independently, and then only have a few tests for the formatting.
A concrete example will likely help here. Say I'm writing a trading system and I need to display messages such as
"paid 10 on 15 APPL. $7 Commission. spent: $157"
"paid 1 on 15 VTI. Commission free. spent: $15"
"sold 15 APPL at 20. $7 Commission. collected: $293"
"sold 15 VTI at 2. Commission free. collected: $30"
There's quite a bit of variation in those messages. You could have 1 function that creates the entire string:
confirmMsg(side, size, px, ticker)
However, I think you'd end up with quite a few verbose tests. Given this problem, I would look to break down those strings into smaller, more focused functions, for example:
describeOrder(side, size, px, ticker)
describeCommission(ticker)
describeTotal(side, size, px, ticker)
Now that you've broken down the function, you're free to test the code paths of the more focused functions, and the test for confirmMsg becomes trivial. Something along the lines of
assertEquals("paid 10 on 15 APPL",
describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("sell 15 APPL at 10",
describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("$7 Commission",
describeCommission({tickerName:"APPL",commission:"standard"}))
assertEquals("Commission free",
describeCommission({tickerName:"APPL",commission:"free"}))
assertEquals("spent: $157",
describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("collected: $143",
describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("spent: $150",
describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"free"}))
assertEquals("collected: $150",
describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"free"}))
assertEquals("order. commission. total",
confirmMsg("order", "commission", "total"))
I guess I could summarize it by saying, I should be able to easily find and replace my expected literals. If I cannot, then I have an opportunity to further break down a method and write more focused tests on the newly introduced, more granular tests.
Thursday, June 25, 2015
Drop Books
The vast majority of books I purchase are for my own enjoyment, but not all of them. There are a few books that I buy over and over, and drop on the desks of friends and colleagues. These books, all technical, are books that I think most programmers will benefit from reading. I call these books "Drop Books"; I drop them and never expect them to be returned.
My main motivation for dropping books is to spread what I think are great ideas. Specifically, I'm always happy to spread the ideas found in the following books:
I know a few of my friends buy Drop Books as well. Spreading solid ideas and supporting authors seems like a win/win to me; hopefully more and more people will begin to do the same.
My main motivation for dropping books is to spread what I think are great ideas. Specifically, I'm always happy to spread the ideas found in the following books:
I know a few of my friends buy Drop Books as well. Spreading solid ideas and supporting authors seems like a win/win to me; hopefully more and more people will begin to do the same.
Monday, April 06, 2015
Unit Testing Points of View, Probably
Michael Feathers, Brian Marick, and I are collaborating to create a new book: Unit Testing Points of View ... probably.
Unit testing is now mainstream, has tons of entry level books, and has a great reference book. The result is a widely accepted idea that you can quickly and easily adopt; unfortunately, I've found little in the way of documenting pattern trade-offs. I believe that combination leads to a lot of waste. To help avoid some of that waste, I'd like to see more written about why you may want to choose one Unit Testing style over another. That was the inspiration and my goal for Working Effectively with Unit Tests. Growing Object-Oriented Software is another great example of a book that documents both the hows and whys of unit testing. After that, if you're looking for tradeoff guidance... good luck.
Is this a problem? I think so. Without discussion of tradeoffs, experience reports, and concrete direction you end up with hours wasted on bad tests and proclamations of TDD's death. The report of TDD's death was an exaggeration, and the largest clue was the implication that TDD and Unit Testing were synonymous. To this day, Unit Testing is still largely misunderstood.
Today, most teams fail repeatedly with Unit Tests, leading to (at best) significant wasted time and (at worst) abandoning the idea with almost nothing gained. We aim to provide tradeoff guidance that will help teams select a Unit Testing approach that best fits their context.
As I said above: There's no reason this evolution couldn't be extended by other authors. In fact, that's our long term hope. Ideally, Brian, Michael and I write volume one of Unit Testing Points of View. Volume two could be written by Kevlin Henney, Steve Freeman, and Roy Osherove - or anyone else who has interest in the series. Of course, given the original inspiration, we're all hoping Martin Fowler clears some time in his schedule to take part in the series.
If you'd like to see this project happen, please do sign up on the interest list and encourage others to as well (here's a tweet to RT, if you'd like to help).
Origin
In 2014 Martin Fowler provided Technical Review for Working Effectively with Unit Tests. As part of his feedback he said something along the lines of: I'm glad you wrote this book, and I'll likely write a bliki entry noting what I agree with and detailing what I would do differently. I'm still looking forward to that entry, and I think the idea is worth extending beyond Martin.Unit testing is now mainstream, has tons of entry level books, and has a great reference book. The result is a widely accepted idea that you can quickly and easily adopt; unfortunately, I've found little in the way of documenting pattern trade-offs. I believe that combination leads to a lot of waste. To help avoid some of that waste, I'd like to see more written about why you may want to choose one Unit Testing style over another. That was the inspiration and my goal for Working Effectively with Unit Tests. Growing Object-Oriented Software is another great example of a book that documents both the hows and whys of unit testing. After that, if you're looking for tradeoff guidance... good luck.
Is this a problem? I think so. Without discussion of tradeoffs, experience reports, and concrete direction you end up with hours wasted on bad tests and proclamations of TDD's death. The report of TDD's death was an exaggeration, and the largest clue was the implication that TDD and Unit Testing were synonymous. To this day, Unit Testing is still largely misunderstood.
What could be
Working Effectively with Unit Tests starts with one set of opinionated Unit Tests and evolves to the types of Unit Tests that I find more productive. There's no reason this evolution couldn't be extended by other authors. This is the vision we have for Unit Testing Points of View. Michael, Brian, and I began by selecting a common domain model. The first few chapters will detail the hows and whys of how I would test the common domain model. After I've expressed what motivates my tests, Brian or Michael will evolve the tests to a style they find superior. After whoever goes second (Brian or Michael) finishes, the other will continue the evolution. The book will note where we agree, but the majority of the discussion will occur around where our styles differ and what aspect of software development we've chosen emphasize by using an alternative approach.Today, most teams fail repeatedly with Unit Tests, leading to (at best) significant wasted time and (at worst) abandoning the idea with almost nothing gained. We aim to provide tradeoff guidance that will help teams select a Unit Testing approach that best fits their context.
As I said above: There's no reason this evolution couldn't be extended by other authors. In fact, that's our long term hope. Ideally, Brian, Michael and I write volume one of Unit Testing Points of View. Volume two could be written by Kevlin Henney, Steve Freeman, and Roy Osherove - or anyone else who has interest in the series. Of course, given the original inspiration, we're all hoping Martin Fowler clears some time in his schedule to take part in the series.
Why "Probably"?
Writing a book is almost always a shot in the dark. Martin, Michael, Brian, and I all think this a book worth writing (and a series worth starting), but we have no idea if people actually desire such a book. In the past you took the leap expecting failure and praying for success. Michael, Brian, and I believe there's a better way: leanpub interest lists (here's Unit Testing Points of View's). We're looking for 15,000 people to express interest (by providing their email addresses). I'm writing the first 3 chapters, and they'll be made available when we cross the 15k mark. At that point, Michael and Brian will know that the interest is real, and their contributions need to be written. If we never cross the 15k mark then we know such a book isn't desired, and we shouldn't waste our time.If you'd like to see this project happen, please do sign up on the interest list and encourage others to as well (here's a tweet to RT, if you'd like to help).
Subscribe to:
Posts (Atom)