Saturday, November 19, 2011

Clojure: expectations - scenarios

clojure-deprecating-expectationsscenari.html

When I set out to write expectations I wanted to create a simple unit testing framework. I'm happy with what expectations provides for unit testing; however, I also need to write the occasional test that changes values or causes a side effect. There's no way I could go back to clojure.test after enjoying better failure messages, trimmed stack traces, automatic testing running, etc. Thus, expectations.scenarios was born.

Using expectations.scenarios should be fairly natural if you already use expectations. The following example is a simple scenario (which could be a unit test, but we'll start here for simplicity).

(ns example.scenarios
  (:use expectations.scenarios))

(scenario
 (expect nil? nil))

A quick trip to the command line shows us that everything is working as expected.

Ran 1 tests containing 1 assertions in 4 msecs
0 failures, 0 errors.

As I said above, you could write this test as a unit test. However, expectations.scenarios was created for the cases in which you want to verify a value, make a change, and verify a value again. The following example shows multiple expectations verifying changing values in the same scenario.

(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 1 @a)
    (swap! a inc)
    (expect 2 @a)))

In expectations (unit tests) you can only have one expect (or given) so failures are captured, but they do not stop execution. However, due to the procedural nature of scenarios, the first failing expect stops execution.

(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 2 @a)
    (println "you'll never see this")))

failure in (scenarios.clj:4) : example.scenarios
           (expect 2 (clojure.core/deref a))
           expected: 2 
                was: 1
           on (scenarios.clj:7)
Ran 1 tests containing 1 assertions in 81 msecs
1 failures, 0 errors.

expectations.scenarios also allows you to easily verify calls to any function. I generally use interaction expects when I need to verify some type of side effect (e.g. logging or message publishing).

(scenario
 (println "1")
 (expect (interaction (println "1"))))

It's important to note the ordering of this scenario. You don't 'setup' an expectation and then call the function. Exactly the opposite is true - you call the function the same way you would in production, then you expect the interaction to have occurred. You may find this jarring if you're used to setting up your mocks ahead of time; However, I think this syntax is the least intrusive - and I think you'll prefer it in the long term.

The above example calls println directly, but your tests are much more likely to look something like this.

(defn foo [x] (println x))

(scenario
 (foo "1")
 (expect (interaction (println "1"))))

Similar to all other mocking frameworks (that I know of) the expect is using an implicit "once" argument. You can also specify :twice and :never if you find yourself needing those interaction tests.

(defn foo [x] (println x))

(scenario
 (foo "1")
 (foo "1")
 (expect (interaction (println "1")) :twice)
 (expect (interaction (identity 1)) :never))

On occasion you may find yourself interested in verifying 2 out of 3 arguments - expectations.scenarios provides the 'anything' var that can be used for arguments you don't care about.

(defn foo [x y z] (println x y z))

(scenario
 (foo "1" 2 :a)
 (expect (interaction (println "1" anything :a))))

That's about all there is to expectations.scenarios, hopefully it fills the gap for tests you want to write that simply can't be done as unit tests.

Tuesday, November 01, 2011

Clojure: expectations unit testing wrap-up

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six (this entry)

The previous blog posts on expectations unit testing syntax cover all of the various ways that expectations can be used to write tests and what you can expect when your tests fail. However, there are a few other things worth knowing about expectations.

Stacktraces
expectations aggressively removes lines from the stacktraces. Just like many other aspects of expectations, the focus is on more signal and less noise. Any line in the stacktrace from clojure.core, clojure.lang, clojure.main, and java.lang will be removed. As a result any line appearing in your stacktrace should be relevant to your application or a third-party lib you're using. expectations also removes any duplicates that can occasionally appear when anonymous functions are part of the stacktrace. Again, it's all about improving signal by removing noise. Speaking of noise...

Test Names
You might have noticed that expectations does not require you to create a test name. This is a reflection of my personal opinion that test names are nothing more than comments and shouldn't be required. If you desire test names, feel free to drop a comment above each test. Truthfully, this is probably a better solution anyway, since you can use spaces (instead of dashes) to separate words in a comment. Comments are good when used properly, but they can become noise when they are required. The decision to simply use comments for test names is another example of improving signal by removing noise.

Running Focused Expectations
Sometimes you'll have a file full of expectations, but you only want to run a specific expectation - expectations solves this problem by giving you 'expect-focused'. If you use expect-focused only expectations that are defined using expect-focused will be run.

For example, if you have the following expectations in a file you should see the following results from 'lein expectations'.

(ns sample.test.core
  (:use [expectations]))

(expect zero? 0)
(expect zero? 1)
(expect-focused nil? nil)

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 2 msecs
IGNORED 2 EXPECTATIONS
0 failures, 0 errors.

As you can see, expectations only ran one test - the expect-focused on line 6. If the other tests had been run the test on line 5 would have created a failure. It can be easy to accidentally leave a few expect-focused calls in, so expectations prints the number of ignored expectations in capital letters as a reminder. Focused expectation running is yet another way to remove noise while working through a problem.

Tests Running
If you always use 'lein expectations' to run your tests you'll never even care; however, if you ever want to run individual test files it's important to know that your tests run by default on JVM shutdown. When I'm working with Clojure and Java I usually end up using IntelliJ, and therefore have the ability to easily run individual files. When I switched from clojure.test to expectations I wanted to make test running as simple as possible - so I removed the need to specify (run-all-tests). Of course, if you don't want expectations to run for some reason you can disable this feature by calling (expectations/disable-run-on-shutdown).

JUnit Integration
Lack of JUnit integration was a deal breaker for my team in the early days, so expectations comes with an easy way to run all tests as part of JUnit. If you want all of your tests to run in JUnit all you need to do is implement ExpectationsTestRunner.TestSource. The following example is what I use to run all the tests in expectations with JUnit.

import expectations.junit.ExpectationsTestRunner;
import org.junit.runner.RunWith;

@RunWith(expectations.junit.ExpectationsTestRunner.class)
public class SuccessTest implements ExpectationsTestRunner.TestSource{

    public String testPath() {
        return "test/clojure/success";
    }
}

As you can see from the example above, all you need to do is tell the test runner where to find your Clojure files.

That should be everything you need to know about expectations for unit testing use. If anything is unclear, please drop me a line in the comments.

Clojure: expectations - removing duplication with given

create some state
verify a bit about the state
alter the state
verify more about the state

Part of the problem is that the assertions that occurred before the altering of the state may or may not be relevant after the alteration. Additionally, if any of the assertions fail you have to stop running the entire test - thus some of your assertions will not be run (and you'll be lacking some information).

expectations takes an alternate route - embracing the idea of multiple assertions by providing a specific syntax that allows multiple verifications and the least amount of duplication.

The following example shows how you can test multiple properties of a Java object using the 'given' syntax.

(given (java.util.ArrayList.)
       (expect
         .size 0
         .isEmpty true))

jfields$ lein expectations
Ran 2 tests containing 2 assertions in 4 msecs
0 failures, 0 errors.

The syntax is simple enough: (given an-object (expect method return-value [method return-value]))
note: [method return-value] may be repeated any number of times.

This syntax allows us to expect return-values from as many methods as we care to verify, but encourages us not to change any state between our various assertions. This syntax also allows us to to run each assertion regardless of the outcome of any previous assertion.

Obviously you could call methods that change the internal state of the object and at that point you're on your own. I definitely wouldn't recommend testing that way. However, as long as you call methods that don't change any state 'given' can help you write succinct tests that verify as many aspects of an object as you need to test.

As usual, I'll show the output for tests that fail using this syntax.

(given (java.util.ArrayList.)
       (expect
         .size 1
         .isEmpty false))

jfields$ lein expectations
failure in (core.clj:4) : sample.test.core
           (expect 1 (.size (java.util.ArrayList.)))
           expected: 1 
                was: 0
failure in (core.clj:4) : sample.test.core
           (expect false (.isEmpty (java.util.ArrayList.)))
           expected: false 
                was: true

This specific syntax was created for testing Java objects, but an interesting side effect is that it actually works on any value and you can substitute method calls with any function. For example, you can test a vector or a map using the examples below as a template.

(given [1 2 3]
       (expect
         first 1
         last 3))

(given {:a 2 :b 4}
       (expect 
         :a 2
         :b 4))

jfields$ lein expectations
Ran 4 tests containing 4 assertions in 8 msecs
0 failures, 0 errors.

And, of course, the failures.

(given [1 2 3]
       (expect
         first 2
         last 1))

(given {:a 2 :b 4}
       (expect 
         :a 1
         :b 1))

jfields$ lein expectations
failure in (core.clj:4) : sample.test.core
           (expect 2 (first [1 2 3]))
           expected: 2 
                was: 1
failure in (core.clj:4) : sample.test.core
           (expect 1 (last [1 2 3]))
           expected: 1 
                was: 3
failure in (core.clj:9) : sample.test.core
           (expect 1 (:a {:a 2, :b 4}))
           expected: 1 
                was: 2
failure in (core.clj:9) : sample.test.core
           (expect 1 (:b {:a 2, :b 4}))
           expected: 1 
                was: 4
Ran 4 tests containing 4 assertions in 14 msecs
4 failures, 0 errors.

When you want to call methods on a Java object or call functions with the same instance over and over the previous given syntax is really the simplest solution. However, there are times where you want something more flexible.

expectations also has a 'given' syntax that allows you to specify a template - thus reducing code duplication. The following example shows a test that verifies + with various arguments.

(given [x y] (expect 10 (+ x y))
       4 6
       6 4
       12 -2)

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 5 msecs
0 failures, 0 errors.

The syntax for this flavor of given is: (given bindings template-form values-to-be-bound). The template form can be anything you need - just remember to put the expect in there.

Here's another example where we combine given with in to test a few different things. This example shows both the successful and failing versions.

;; successful
(given [x y] (expect x (in y))
       :a #{:a :b}
       {:a :b} {:a :b :c :d})

;; failure
(given [x y] (expect x (in y))
       :c #{:a :b}
       {:a :d} {:a :b :c :d})

lein expectations
failure in (core.clj:8) : sample.test.core
           (expect :c (in #{:a :b}))
           key :c not found in #{:a :b}
failure in (core.clj:8) : sample.test.core
           (expect {:a :d} (in {:a :b, :c :d}))
           expected: {:a :d} 
                 in: {:a :b, :c :d}
           :a expected: :d
                   was: :b
Ran 4 tests containing 4 assertions in 13 msecs
2 failures, 0 errors.

That's basically it for 'given' syntax within expectations. There are times that I use all of the various versions of given; however, there seems to be a connection with using given and interacting with Java objects. If you don't find yourself using Java objects very often then you probably wont have a strong need for given.

Clojure: expectations and Double/NaN


;; allow Double/NaN equality in a map
(expect {:a Double/NaN :b {:c Double/NaN}} {:a Double/NaN :b {:c Double/NaN}})

;; allow Double/NaN equality in a set
(expect #{1 Double/NaN} #{1 Double/NaN})

;; allow Double/NaN equality in a list
(expect [1 Double/NaN] [1 Double/NaN])

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 32 msecs
0 failures, 0 errors.

As you would expect, you can also count on Double/NaN being considered equal even if you are using the 'in' function.

;; allow Double/NaN equality when verifying values are in a map
(expect {:a Double/NaN :b {:c Double/NaN}} (in {:a Double/NaN :b {:c Double/NaN} :d "other stuff"}))

;; allow Double/NaN equality when verifying it is in a set
(expect Double/NaN (in #{1 Double/NaN}))

;; allow Double/NaN equality when verifying it's existence in a list
(expect Double/NaN (in [1 Double/NaN]))

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 32 msecs
0 failures, 0 errors.

For completeness I'll also show the examples of each of these examples failing.

;; allow Double/NaN equality in a map
(expect {:a Double/NaN :b {:c Double/NaN}} {:a nil :b {:c Double/NaN}})

;; allow Double/NaN equality with in fn and map
(expect {:a Double/NaN :b {:c nil}} (in {:a Double/NaN :b {:c Double/NaN} :d "other stuff"}))

;; allow Double/NaN equality in a set
(expect #{1 Double/NaN} #{1 nil})

;; allow Double/NaN equality with in fn and set
(expect Double/NaN (in #{1 nil}))

;; allow Double/NaN equality in a list
(expect [1 Double/NaN] [1 nil])

;; allow Double/NaN equality with in fn and list
(expect Double/NaN (in [1 nil]))


jfields$ lein expectations
failure in (core.clj:5) : sample.test.core
           (expect {:a Double/NaN, :b {:c Double/NaN}}
                   {:a nil, :b {:c Double/NaN}})
           expected: {:a NaN, :b {:c NaN}} 
                was: {:a nil, :b {:c NaN}}
           :a expected: NaN
                   was: nil
failure in (core.clj:8) : sample.test.core
           (expect {:a Double/NaN, :b {:c nil}} (in {:a Double/NaN, :b {:c Double/NaN}, :d "other stuff"}))
           expected: {:a NaN, :b {:c nil}} 
                 in: {:a NaN, :b {:c NaN}, :d "other stuff"}
           :b {:c expected: nil
                       was: NaN
failure in (core.clj:11) : sample.test.core
           (expect #{1 Double/NaN} #{nil 1})
           expected: #{NaN 1} 
                was: #{nil 1}
           nil are in actual, but not in expected
           NaN are in expected, but not in actual
failure in (core.clj:14) : sample.test.core
           (expect Double/NaN (in #{nil 1}))
           key NaN not found in #{nil 1}
failure in (core.clj:17) : sample.test.core
           (expect [1 Double/NaN] [1 nil])
           expected: [1 NaN] 
                was: [1 nil]
           nil are in actual, but not in expected
           NaN are in expected, but not in actual
failure in (core.clj:20) : sample.test.core
           (expect Double/NaN (in [1 nil]))
           value NaN not found in [1 nil]
Ran 6 tests containing 6 assertions in 66 msecs
6 failures, 0 errors.

There always seems to be downsides to using NaN, so I tend to look for the least painful path. Hopefully expectations provides the most pain-free path when your tests end up needing to include NaN.

Clojure: expectations with values in vectors, sets, and maps

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three (this entry)
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six

I've previously written about verifying equality and the various non-equality expectations that are available. This entry will focus on another type of comparison that is allowed in expectations - verifying that an 'expected' value is in an 'actual' value.

A quick recap - expectations generally look like this: (expect expected actual)

verifying an expected value is in an actual value is straightforward and hopefully not a surprising syntax: (expect expected (in actual))

If that's not clear, these examples should make the concept completely clear.

;; expect a k/v pair in a map.
(expect {:foo 1} (in {:foo 1 :cat 4}))

;; expect a key in a set
(expect :foo (in #{:foo :bar}))

;; expect a val in a list
(expect :foo (in [:foo :bar]))

As you would expect, running these expectations results in 3 passing tests.

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 8 msecs
0 failures, 0 errors.

As usual, I'll show the failures as well.

;; expect a k/v pair in a map.
(expect {:foo 2} (in {:foo 1 :cat 4}))

;; expect a key in a set
(expect :baz (in #{:foo :bar}))

;; expect a val in a list
(expect :baz (in [:foo :bar]))

jfields$ lein expectations
failure in (core.clj:18) : sample.test.core
           (expect {:foo 2} (in {:foo 1, :cat 4}))
           expected: {:foo 2} 
                 in: {:foo 1, :cat 4}
           :foo expected: 2
                     was: 1
failure in (core.clj:21) : sample.test.core
           (expect :baz (in #{:foo :bar}))
           key :baz not found in #{:foo :bar}
failure in (core.clj:24) : sample.test.core
           (expect :baz (in [:foo :bar]))
           value :baz not found in [:foo :bar]

expectations does it's best to provide you with any additional info that might be helpful. In the case of the vector and the set there's not much else that can be said; however, the map failure gives you additional information that can be used to track down the issue.

There's nothing magical going on with 'in' expectations and you could easily do the equivalent with select-keys, contains?, or some, but expectations allows you to get that behavior while keeping your tests succinct.

Jay Fields' Thoughts