Showing posts with label functional testing. Show all posts
Showing posts with label functional testing. Show all posts

Wednesday, December 17, 2014

Working Effectively with Unit Tests Official Launch

Today marks the official release release of Working Effectively with Unit Tests. The book is available in various formats:
I’m very happy with the final version. Michael Feathers wrote a great foreword. I incorporated feedback from dozens of people - some that have been friends for years, and some that I’d never previously met. I can’t say enough great things about http://leanpub.comand I highly recommend it for getting an idea out there and making it easy to get fast feedback. 

As far as the softcover edition, I had offers from a few major publishers, but in the end none of them would allow me to continue to sell on leanpub at the same time. I strongly considered caving to the demands of the major publishers, but ultimately the ability to create a high quality softcover and make it available on Amazon was too tempting to pass up.

The feedback has been almost universally positive - the reviews are quite solid on goodreads (http://review.wewut.com). I believe the book provides specific, concise direction for effective Unit Testing, and I hope it helps increase the quality of the unit tests found in the wild.

If you'd like to try before you buy, there's a sample available in pdf format or on the web.


Wednesday, May 21, 2014

Working Effectively with Unit Tests

Unit Testing has moved from fringe to mainstream, which is a great thing. Unfortunately, as a side effect developers are creating mountains of unmaintainable tests. I've been fighting the maintenance battle pretty aggressively for years, and I've decided to write a book that captures what I believe is the most effective way to test.

From the Preface

Over a dozen years ago I read Refactoring for the first time; it immediately became my bible. While Refactoring isn’t about testing, it explicitly states: If you want to refactor, the essential precondition is having solid tests. At that time, if Refactoring deemed it necessary, I unquestionably complied. That was the beginning of my quest to create productive unit tests.

Throughout the 12+ years that followed reading Refactoring I made many mistakes, learned countless lessons, and developed a set of guidelines that I believe make unit testing a productive use of programmer time. This book provides a single place to examine those mistakes, pass on the lessons learned, and provide direction for those that want to test in a way that I’ve found to be the most productive.
The book does touch on some theory and definition, but the main purpose is to show you how to take tests that are causing you pain and turn them into tests that you're happy to work with.

For example, the book demonstrates how to go from...

looping test with many (built elsewhere) collaborators
.. to individual tests that expect literals, limit scope, explicitly define collaborators, and focus on readability
.. to fine-grained tests that focus on testing a single responsibility, are resistant to cascading failures, and provide no friction for those practicing ruthless Refactoring.
As of right now, you can read the first 2 chapters for free at https://leanpub.com/wewut/read

I'm currently ~25% done with the book, and it's available now for $14.99. My plan is to raise the price to $19.99 when I'm 50% done, and $24.99 when I'm 75% done. Leanpub offers my book with 100% Happiness Guarantee: Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks. Therefore, if you find the above or the free sample interesting, you might want to buy it now and save a few bucks.

Buy Now here: https://leanpub.com/wewut

Saturday, November 19, 2011

Clojure: expectations - scenarios

expectations.scenarios are now deprecated: http://blog.jayfields.com/2012/11/clojure-deprecating-expectationsscenari.html


When I set out to write expectations I wanted to create a simple unit testing framework. I'm happy with what expectations provides for unit testing; however, I also need to write the occasional test that changes values or causes a side effect. There's no way I could go back to clojure.test after enjoying better failure messages, trimmed stack traces, automatic testing running, etc. Thus, expectations.scenarios was born.

Using expectations.scenarios should be fairly natural if you already use expectations. The following example is a simple scenario (which could be a unit test, but we'll start here for simplicity).
(ns example.scenarios
  (:use expectations.scenarios))

(scenario
 (expect nil? nil))
A quick trip to the command line shows us that everything is working as expected.
Ran 1 tests containing 1 assertions in 4 msecs
0 failures, 0 errors.
As I said above, you could write this test as a unit test. However, expectations.scenarios was created for the cases in which you want to verify a value, make a change, and verify a value again. The following example shows multiple expectations verifying changing values in the same scenario.
(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 1 @a)
    (swap! a inc)
    (expect 2 @a)))
In expectations (unit tests) you can only have one expect (or given) so failures are captured, but they do not stop execution. However, due to the procedural nature of scenarios, the first failing expect stops execution.
(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 2 @a)
    (println "you'll never see this")))

failure in (scenarios.clj:4) : example.scenarios
           (expect 2 (clojure.core/deref a))
           expected: 2 
                was: 1
           on (scenarios.clj:7)
Ran 1 tests containing 1 assertions in 81 msecs
1 failures, 0 errors.
expectations.scenarios also allows you to easily verify calls to any function. I generally use interaction expects when I need to verify some type of side effect (e.g. logging or message publishing).
(scenario
 (println "1")
 (expect (interaction (println "1"))))
It's important to note the ordering of this scenario. You don't 'setup' an expectation and then call the function. Exactly the opposite is true - you call the function the same way you would in production, then you expect the interaction to have occurred. You may find this jarring if you're used to setting up your mocks ahead of time; However, I think this syntax is the least intrusive - and I think you'll prefer it in the long term.

The above example calls println directly, but your tests are much more likely to look something like this.
(defn foo [x] (println x))

(scenario
 (foo "1")
 (expect (interaction (println "1"))))
Similar to all other mocking frameworks (that I know of) the expect is using an implicit "once" argument. You can also specify :twice and :never if you find yourself needing those interaction tests.
(defn foo [x] (println x))

(scenario
 (foo "1")
 (foo "1")
 (expect (interaction (println "1")) :twice)
 (expect (interaction (identity 1)) :never))
On occasion you may find yourself interested in verifying 2 out of 3 arguments - expectations.scenarios provides the 'anything' var that can be used for arguments you don't care about.
(defn foo [x y z] (println x y z))

(scenario
 (foo "1" 2 :a)
 (expect (interaction (println "1" anything :a))))
That's about all there is to expectations.scenarios, hopefully it fills the gap for tests you want to write that simply can't be done as unit tests.

Tuesday, April 15, 2008

Story 9: Display Customer Support Number

I thought it might be beneficial to create an entry that shows my workflow when implementing a few feature.

I generally work from stories, so it seemed logical to create an example story.
As a consumer
I would like to see the customer support number
So that I can call if I have a problem
There are a few technical details that make this story a bit more complicated than expected.
  • The site is one rails application, but it needs to be branded differently for each affiliate.
  • Based on the affiliate name (which will come from the params) we will need to connect to the web service specific to that affiliate.
  • The customer support number will come from the affiliate web service.
  • The affiliate web service will require an authentication token in production, but it is optional in development mode.
Based on the story and the known technical details we come up with the following list of tasks.
  • task 1: Create a configuration file (yaml) that will be stored on the web server. This configuration file will map the affiliate name to their web service.
  • task 2: Retrieve the customer support number from the affiliate web service.
  • task 3: Add authentication since it will be required in production.
You may prefer more fine grained tasks (I usually do), but these should do for example purposes.

Task 1
The first thing I generally do is write a functional test. I want unit tests as well, and I'll probably write the unit tests and have them passing before I have the functional test passing, but starting with a functional test allows me to think at a higher level instead of getting bogged down in the details of the implementation. Therefore, the first test I write would probably look like this RSpec spec.

describe Configuration do
it "gets the support number from the client specific configuration" do
Configuration.for("localhost").support_number.should eql("212.646.9208")
end
end

[RSpec link]

Now, I know that this test isn't going to pass for awhile, but it gives me a good reminder of the direction that I want to continue to head in. At this point I would be looking to write a unit test that verifies that I can get the address from a stub configuration file. Having a unit test will allow me to get the Configuration class working without worrying about the configuration file. The unit test will also run significantly faster than the functional test, which benefits us in the long term.

Expectations do
expect "some url" do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end
end

[expectations link]

Ultimately I know that the Configuration::for method won't return the address, but my first task is to get the configuration file working, and this test is a good way to complete that task.

At this point I have failing tests, so I write the implementation of the Configuration class. Below is all of the code written at this point. (Of course, in a real application it would be broken out to appropriate files)

require 'yaml'
require 'rubygems'
require 'expectations'
require 'spec'

class Configuration < Struct.new(:support_number)
def self.for(environment)
YAML.load(File.read("config.yaml"))[environment]["configuration_url"]
end
end

Expectations do
expect "some url" do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end
end
# >> Expectations .
# >> Finished in 0.001 seconds
# >>
# >> Success: 1 fulfilled

describe Configuration do
it "gets the support number from the client specific configuration" do
Configuration.for("localhost").support_number.should eql("212.646.9208")
end
end
# >> F
# >>
# >> 1)
# >> NoMethodError in 'Configuration gets the support number from the client specific configuration'
# >> undefined method `support_number' for "127.0.0.1/config":String
# >>
# >> Finished in 0.008434 seconds
# >>
# >> 1 example, 1 failure

The unit test is passing; therefore, assuming that we've stubbed the configuration file correctly we've completed our first task.

Task 2
Our functional test is sufficient for verifying that task two is done correctly, so the next step is to write another unit test. For the purpose of the example, assume that there's a ConfigurationGateway::retrieve_from method that we can use to get the configuration from the affiliate web service. The configuration information is going to be returned by the gateway as a hash, so we'll create an appropriate stub so our unit tests don't rely on the real service. Our next unit test would look something like the following example.

Expectations do
expect "support number" do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
ConfigurationGateway.stubs(:retrieve_from).returns(:support_number => "support number")
Configuration.for("localhost").support_number
end
end

A quick change to the Configuration class and we have our new unit test passing.

class Configuration < Struct.new(:support_number)
def self.for(environment)
configuration_url = YAML.load(File.read("config.yaml"))[environment]["configuration_url"]
self.new(ConfigurationGateway.retrieve_from(configuration_url)[:support_number])
end
end

Unfortunately, this change has broken our previously created unit test.

Expectations do
expect "some url" do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end
end
# >> Expectations F
# >> Finished in 0.001 seconds
# >>
# >> Failure: 1 failed, 0 errors, 0 fulfilled
# >>
# >> --Failures--
# >> -:40:in `expect'
# >> file <->
# >> line <40>
# >> expected: <"some url"> got: <#<struct Configuration support_number="212.646.9208">>

Our first unit test no longer works because we are no longer returning the address from the Configuration::for method. In fact, we don't need to expose the address outside of the Configuration class at all, so a behavior based unit test is probably a good choice for replacing our first unit test. The example below represents the new method for verifying that the configuration file is correctly returning the address.

Expectations do
expect ConfigurationGateway.to.receive(:retrieve_from).with("some url").returns({}) do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end
end

At this point, if the ConfigurationGateway is properly working and if your configuration file is in place, your functional test should also be passing.

Here's all the code we've written so far, including a fake ConfigurationGateway to show our Configuration class is working.

require 'yaml'
require 'rubygems'
require 'expectations'
require 'spec'

class Configuration < Struct.new(:support_number)
def self.for(environment)
configuration_url = YAML.load(File.read("config.yaml"))[environment]["configuration_url"]
self.new(ConfigurationGateway.retrieve_from(configuration_url)[:support_number])
end
end

class ConfigurationGateway
def self.retrieve_from(url)
# In actual code this would go to a external service...
{:support_number => "212.646.9208"}
end
end

Expectations do
expect ConfigurationGateway.to.receive(:retrieve_from).with("some url").returns({}) do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end

expect "support number" do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
ConfigurationGateway.stubs(:retrieve_from).returns(:support_number => "support number")
Configuration.for("localhost").support_number
end
end
# >> Expectations ..
# >> Finished in 0.001 seconds
# >>
# >> Success: 2 fulfilled

describe Configuration do
it "gets the support number from the client specific configuration" do
Configuration.for("localhost").support_number.should eql("212.646.9208")
end
end
# >> .
# >>
# >> Finished in 0.008254 seconds
# >>
# >> 1 example, 0 failures

And, here's the configuration file that you'll also need to get the tests passing.

# config.yaml
localhost:
configuration_url: 127.0.0.1/config

At this point both our functional and unit tests are passing, and we are done with task two.

Task 3
For the purposes of this example, we're going to change the ConfigurationGateway to require the authentication token. If this were real code, I would expect the ConfigurationGateway to have this ability already built in, since we'd need a way to make the development server behave like production.

class ConfigurationGateway
def self.retrieve_from(url, token)
# In actual code this would go to a external service...
{:support_number => "212.646.9208"}
end
end

This change causes the functional test to break, but the unit tests do not break. That's OK. In order to get the functional tests passing we'll need to change the implementation, which will cause the unit tests to break. In the end, all of our tests will be updated to represent the correct interactions.

To make the functional tests pass we'll need to add a call to the AuthenticaionGateway to get the next authentication token. Following this change, the code below represents the new Configuration::for method.

class Configuration < Struct.new(:support_number)
def self.for(environment)
configuration_url = YAML.load(File.read("config.yaml"))[environment]["configuration_url"]
token = AuthenticationGateway.next_token
self.new(ConfigurationGateway.retrieve_from(configuration_url, token)[:support_number])
end
end

With this change in place the functional test now passes. To get it working locally, you'll need to add the fake AuthenticationGateway class that can be found below.

class AuthenticationGateway
def self.next_token
# In actual code this would go to an external service...
1979
end
end

At this point your functional test should be passing, but a unit test should be failing.

Expectations do
expect ConfigurationGateway.to.receive(:retrieve_from).with("some url").returns({}) do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end
end
# >> Expectations F.
# >> Finished in 0.00104 seconds
# >>
# >> Failure: 1 failed, 0 errors, 1 fulfilled
# >>
# >> --Failures--
# >> -:30:in `expect'
# >> file <->
# >> line <30>
# >> #<Mock:0x5168f0>.retrieve_from('some url', 1979) - expected calls: 0, actual calls: 1
# >> Similar expectations:
# >> #<Mock:0x5168f0>.retrieve_from('some url')

We need to update the ConfigurationGateway::for expectation, but we also need to remove the call to the external service from our unit tests. We do that by stubbing the AuthenticationGateway::next_token method.

Expectations do
expect ConfigurationGateway.to.receive(:retrieve_from).with("some url", "some token").returns({}) do
AuthenticationGateway.stubs(:next_token).returns("some token")
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end

expect "support number" do
AuthenticationGateway.stubs(:next_token).returns("some token")
File.stubs(:read).returns("localhost:\n configuration_url: some url")
ConfigurationGateway.stubs(:retrieve_from).returns(:support_number => "support number")
Configuration.for("localhost").support_number
end
end

At this point we are done with task three. The code below represents all the code necessary for completing all 3 of our tasks.

require 'yaml'
require 'rubygems'
require 'expectations'
require 'spec'

class Configuration < Struct.new(:support_number)
def self.for(environment)
configuration_url = YAML.load(File.read("config.yaml"))[environment]["configuration_url"]
token = AuthenticationGateway.next_token
self.new(ConfigurationGateway.retrieve_from(configuration_url, token)[:support_number])
end
end

class ConfigurationGateway
def self.retrieve_from(url, token)
# In actual code this would go to a external service...
{:support_number => "212.646.9208"}
end
end

class AuthenticationGateway
def self.next_token
# In actual code this would go to an external service...
1979
end
end

Expectations do
expect ConfigurationGateway.to.receive(:retrieve_from).with("some url", "some token").returns({}) do
AuthenticationGateway.stubs(:next_token).returns("some token")
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.for("localhost")
end

expect "support number" do
AuthenticationGateway.stubs(:next_token).returns("some token")
File.stubs(:read).returns("localhost:\n configuration_url: some url")
ConfigurationGateway.stubs(:retrieve_from).returns(:support_number => "support number")
Configuration.for("localhost").support_number
end
end

describe Configuration do
it "gets the support number from the client specific configuration" do
Configuration.for("localhost").support_number.should eql("212.646.9208")
end
end

Refactoring
We are done with our tasks, but we aren't ready to move on to the next card yet. Our functional test is great, but our unit tests specify way too much. They use stubs instead of mocks, which helps with creating more robust tests, but they suffer from High Implementation Specification.

The solution is to create smaller methods that are more test friendly.

The first refactoring is to change the configuration_url and token local variables to be method calls instead.

class Configuration < Struct.new(:support_number)
def self.for(environment)
self.new(ConfigurationGateway.retrieve_from(configuration_url(environment), authentication_token)[:support_number])
end

def self.configuration_url(environment)
YAML.load(File.read("config.yaml"))[environment]["configuration_url"]
end

def self.authentication_token
AuthenticationGateway.next_token
end
end

This change to the Configuration class makes our second unit test a bit more readable and robust.

Expectations do
expect "support number" do
Configuration.stubs(:authentication_token)
Configuration.stubs(:configuration_url)
ConfigurationGateway.stubs(:retrieve_from).returns(:support_number => "support number")
Configuration.for("localhost").support_number
end
end

At this point our second unit test looks pretty good, but the first one is still testing a bit too much. We can solve this by breaking the first test into 4 different tests. After breaking the first test up, our unit tests would look like the code below.

Expectations do
expect ConfigurationGateway.to.receive(:retrieve_from).with(nil, "token") do
Configuration.stubs(:authentication_token).returns("token")
Configuration.stubs(:configuration_url)
Configuration.configuration_results("localhost")
end

expect ConfigurationGateway.to.receive(:retrieve_from).with("configuration url", nil) do
Configuration.stubs(:authentication_token)
Configuration.stubs(:configuration_url).returns("configuration url")
Configuration.configuration_results("localhost")
end

expect AuthenticationGateway.to.receive(:next_token) do
Configuration.authentication_token
end

expect "some url" do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.configuration_url("localhost")
end

expect "support number" do
Configuration.stubs(:configuration_results).returns(:support_number => "support number")
Configuration.for("localhost").support_number
end
end

Whether or not to break the test apart would be a judgment call. The number of things I was specifying would drive me to go ahead and break it up, but if there had only been 5 values specified I might have left the test the way it was.

The Result
Below is all the code written for this entry. I hope you find it helpful.

require 'yaml'
require 'rubygems'
require 'expectations'
require 'spec'

class Configuration < Struct.new(:support_number)
def self.for(environment)
self.new(configuration_results(environment)[:support_number])
end

def self.configuration_results(environment)
ConfigurationGateway.retrieve_from(configuration_url(environment), authentication_token)
end

def self.configuration_url(environment)
YAML.load(File.read("config.yaml"))[environment]["configuration_url"]
end

def self.authentication_token
AuthenticationGateway.next_token
end
end

class ConfigurationGateway
def self.retrieve_from(url, token)
{:support_number => "212.646.9208"}
end
end

class AuthenticationGateway
def self.next_token
1979
end
end

Expectations do
expect ConfigurationGateway.to.receive(:retrieve_from).with(nil, "token") do
Configuration.stubs(:authentication_token).returns("token")
Configuration.stubs(:configuration_url)
Configuration.configuration_results("localhost")
end

expect ConfigurationGateway.to.receive(:retrieve_from).with("configuration url", nil) do
Configuration.stubs(:authentication_token)
Configuration.stubs(:configuration_url).returns("configuration url")
Configuration.configuration_results("localhost")
end

expect AuthenticationGateway.to.receive(:next_token) do
Configuration.authentication_token
end

expect "some url" do
File.stubs(:read).returns("localhost:\n configuration_url: some url")
Configuration.configuration_url("localhost")
end

expect "support number" do
Configuration.stubs(:configuration_results).returns(:support_number => "support number")
Configuration.for("localhost").support_number
end
end

describe Configuration do
it "gets the support number from the client specific configuration" do
Configuration.for("localhost").support_number.should eql("212.646.9208")
end
end