Maintainability and Expect Literals

Recently, Stephen Schaub asked the following on the wewut group:

Several of the unit test examples in the book verify the construction of both HTML and plain text strings. Jay recommends using literal strings in the assertions. However, this strikes me as not a particularly maintainable approach. If the requirements regarding the formatting of these strings changes (a very likely scenario), every single test that verifies one of these strings using a literal must be updated. Combined with the advice that each test should check only one thing, this leads to a large number of extremely brittle tests.

Am I missing something here? I can appreciate the reasons Jay recommends using literals in the tests. However, it seems that we pay a high maintainability price in exchange for the improved readability.

I responded to Stephen; however, I've seen similar questions asked a few times. Below are my extended thoughts regarding literals as expected values.

In general, given the option of having many similar strings (or any literal) vs a helper function, I would always prefer the literal. When a test is failing I only care about that single failing test. If I have to look at the helper function I no longer have the luxury of staying focused on the single test; now I need to consider what the helper function is giving me and what it's giving all other callers. Suddenly the scope of my work has shifted from one test to all of the tests coupled by this helper function. If this helper function wasn't written by me, this expansion in scope wasn't even my decision, it was forced upon me by the helper function creator. In the best case the helper function could return a single, constant string. The scope expansion becomes even worse when the helper function contains code branches.

As for alternatives, my solution would depend on the problem. If the strings were fairly consistent, I would likely simply duplicate everything knowing that any formatting changes can likely be addressed using a bulk edit via find and replace. If the strings were not consistent, I would look at breaking up the methods in a way that would allow me to verify the code branches using as little duplication as possible, e.g. if I wanted to test a string that dynamically changed based on a few variables, I would look to test those variables independently, and then only have a few tests for the formatting.

A concrete example will likely help here. Say I'm writing a trading system and I need to display messages such as

"paid 10 on 15 APPL. $7 Commission. spent: $157"
"paid 1 on 15 VTI. Commission free. spent: $15"
"sold 15 APPL at 20. $7 Commission. collected: $293"
"sold 15 VTI at 2. Commission free. collected: $30"

There's quite a bit of variation in those messages. You could have 1 function that creates the entire string:
confirmMsg(side, size, px, ticker)

However, I think you'd end up with quite a few verbose tests. Given this problem, I would look to break down those strings into smaller, more focused functions, for example:

describeOrder(side, size, px, ticker)
describeCommission(ticker)
describeTotal(side, size, px, ticker)

Now that you've broken down the function, you're free to test the code paths of the more focused functions, and the test for confirmMsg becomes trivial. Something along the lines of

assertEquals("paid 10 on 15 APPL",
  describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("sell 15 APPL at 10",
  describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))

assertEquals("$7 Commission", 
  describeCommission({tickerName:"APPL",commission:"standard"}))
assertEquals("Commission free", 
  describeCommission({tickerName:"APPL",commission:"free"}))

assertEquals("spent: $157", 
  describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("collected: $143", 
  describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"standard"}))
assertEquals("spent: $150", 
  describeOrder("buy", 10, 15, {tickerName:"APPL",commission:"free"}))
assertEquals("collected: $150", 
  describeOrder("sell", 10, 15, {tickerName:"APPL",commission:"free"}))

assertEquals("order. commission. total", 
  confirmMsg("order", "commission", "total"))

I guess I could summarize it by saying, I should be able to easily find and replace my expected literals. If I cannot, then I have an opportunity to further break down a method and write more focused tests on the newly introduced, more granular tests.

Working Effectively with Unit Tests Rough Draft Complete

I finally put the finishing touches on the rough draft of Working Effectively with Unit Tests. It's been an interesting journey thus far, and I'm hoping the attention to detail I've put into the rough draft will translate into an enjoyable read.

What I did poorly: I'd written the book's sample before I ever put it on leanpub. Before a book is published you can collect contact and price information from those who are interested. However, once you publish and begin selling, you no longer have the ability to collect the previously mentioned information. I published and began selling my book immediately - and forfeited my chance to collect that information.

What I did well: I published early and often. I can't say enough nice things about leanpub. I've gotten tons of feedback on example style, writing style, typos, and content. One reader's suggestion to switch to Kevlin Henney's Java formatting style made my book enjoyable to read on a Kindle. I had twitter followers apologizing for "being pedantic and pointing out typos", and I couldn't have been happier to get the feedback. Each typo I fix makes the book more enjoyable for everyone. If you're going to write a book, get it on leanpub asap and start interacting with your audience.

What I learned from Refactoring: Ruby Edition (RRE): RRE contains errors, far too many errors. I vowed to find a better way this time around, and I'm very happy with the results. Every example test in the book can be run, and uses classes also shown in the book. However, writing about tests is a bit tricky: sometimes "failure" is the outcome you're looking to document. Therefore, I couldn't simply write tests for everything. Instead I piped the output to files and used them as example output in the book, but also as verification that what failed once continued to fail in the future (and vice versa). WEwUT has a script that runs every test from the book and overwrites the output files. If the output files are unchanged, I know all the passing examples are still correctly passing, and all the failing examples are still correctly failing. In a way, git diff became my test suite output. I'm confident in all the code found in WEwUT, and happy to be able to say it's all "tested".

What's unclear: Using leanpub was great, but I'm not really sure how to get the word out any further at this point. I set up a goodreads.com page and many friends have been kind enough to tweet about it, but I don't really have any other ideas at this point. I've reached out to a few publishers to see about creating a paperback, and I suspect a print version will increase interest. Still, I can't help thinking there's something else I should be doing between now and paperback launch.

What's next: The rough draft is 100% complete, but I expect to continue to get feedback over the next month or so. As long as the feedback is coming in, I'll be doing updates and publishing new versions.

If you've already bought the book, thank you for the support. It takes 10 seconds to get a pdf of any book you want these days, and I can't thank you enough for monetarily supporting all the effort I've put into WEwUT. If you haven't bought the book, you're welcome to give the sample a read for free. I hope you'll find it enjoyable, and I would gladly accept any feedback you're willing to provide.