Clojure: Expectations Interaction Tests For Java Objects

I recently ran into some code that forced me to integrate with a Java library. While using the library I found myself wanting to do a bit of interaction testing, which I've historically done with Mockito. As a result, I added the ability to do interaction based tests on mock Java objects, directly in expectations.

Hopefully the code is what you'd expect.


The previous example creates a mock Runnable in an expect-let, expects the run method to be run, and then calls the run method of the mock. This test is worthless in a real world context, but it's the simplest way to demonstrate the syntax for creating a mock & specifying the interaction.

The mock function defined in erajure, a minimal wrapper around mockito. All of the "times" arguments are the same as what's available for function interaction tests, examples can be found here.

Clojure: Expectations Verify Interaction Args

The expectations framework provides the ability to create interaction (or behavior) based tests. I've previously written about adding interaction based testing to expectations; however, the examples from that blog entry focused exclusively on testing interactions where each argument is matched using equality. In this entry I'll give examples of how each argument can be also be verified using a class, regex, exception, or a custom function.

When writing state based tests using expectations the type of test you're writing is inferred from the expected value. If the expected value is a regex, expectations will test the actual value to see if it matches the regex. If you passed in a class, expectations will test the actual value to see if it's an instance of that class. If you passed in an exception... you get the idea. All of what I said above, is also true for arguments of an interaction.

Let's start with a simple interaction based test:


In the example above, we're calling the spit function with exactly the arguments that we've specified in our test. This test will pass; however, we've had to specify the exact file location and the exact data. If for some reason you can't specify exactly what the argument will be, it's nice to have a way to specify as much as you possibly can.

In the example below, we're still specifying the exact data, but we're only verifying that the file is somewhere in /tmp/.


As I previously mentioned, we can also get more general and only verify the class of an argument. For example, if we knew our data was going to be a String, but we didn't want to specify exactly what that string was, the following test would do the trick.


While expectations provides you with a lot of default options, there are times when you'll want to write your own argument "matcher". As a contrived example, let's pretend that we want to test that the last argument is true or nil.


One of the best features of expectations is it's error reporting, and the same error reporting logic is applied to arguments when an interaction based test fails. Given the example above, you'll get the following error message.

failure in (success_examples.clj:204) : success.success-examples
           expected: (spit #"/tmp/" String :append true-or-nil?) 
                got: 0 times 

           -- got: (spit "/tmp/somewhere-else" "nil")
           "nil", "/tmp/somewhere-else" are in actual, but not in expected
           true_or_nil_QMARK, #"/tmp/", :append, String are in expected, but not in actual
           expected is larger than actual 

           -- got: (spit "/tmp/hello-world" "some data" :append "s")
           - arg4: not true or nil

As you can see both calls are reported, and each argument has a detailed report (if it did not match).

Finally, expectations provides and additional function that can be used to verify that certain key/value pairs are in an argument. The following example doesn't really make sense, since you'd never want to pass a map as the last argument to spit, but it's easy to follow in the context of this blog entry.


In the above example, (contains-kvs) is used to verify that the final argument to spit contains the key/value pairs :a :b :c :d.

I hope that interaction arg matching follows the principle of least surprise, since it behaves the same as expectations state based tests. I also hope that the ability to use an arbitrary function for verification will provide any necessary flexibility. If you're using expectations, give it a try and let me know.

Clojure: Expectations Interactions - Interactions Are Code, Interactions Are Data

If you read my blog you've probably heard "code is data, data is code" and at one time and you've looked up homoiconicity. You may have deeply understood the idea the first time you heard it; I definitely did not. However, a recent addition to expectations opened my eyes to how truly powerful this programming language property can be.

I'll start by admitting what I heard when I originally encountered homoiconicity. Stuart Halloway had begun promoting Clojure, and homoiconicity was one of the advantages he noted. I hit the wikipedia page, digested the words "code is data, data is code", and thought to myself: well, yeah, obviously. I'd spent plenty of time working with DSLs in Ruby, and I had plenty of experience evaluating code in various contexts. I thought something along the lines of: So you capture the code as data and evaluate it wherever it makes sense, I don't see the big deal. In short, I didn't get it.

Fast forward a few years and several hours of full time Clojure development and you'll find me adding interaction based testing to expectations. What I had in mind for testing interactions was simple, I want to write exactly the same thing for the test as what I write for the production code. Additionally, I want the format of the test to follow the same format that is used for state based testing: (expect expected actual)

Once I had a clear vision for my requirements, the format of the tests became easy to visualize. Assume I have a function that prints to standard out, and I want to test that this print occurs.

The above test looks great, but (println 5) will be evaluated, return nil, and use nil as the expected value. I needed some way for the programmer to tell the testing framework that this was an interaction test, and expectations needed to verify that the function was called with the specified parameters. After trying a few different formats, I settled on the following solution.

By wrapping the interaction I wanted to test with (interaction ...), I created an easy way to identify and capture the function and arguments that needed to be verified.

Once I'd decided on the syntax, I went about the task of adding support to expectations. If you dug into the implementation of expectations, you'd find that expect is a macro that delegates the handling of the "expected" and "actual" arguments to the doexpect macro. The first thing the doexpect macro does is check if expected is a list and (if so) if the first argument is the symbol "interaction" (source here). If the first argument is not a list that begins with 'interaction, then the data is passed to do-value-expect and expanded more or less as is. However, if the first argument is a list that begins with 'interaction, then the data is passed to do-interaction-expect, and do-interaction-expect then destructures the data, grabbing only the pieces of the list that it cares about (source here). When I wrote this code, I found it very interesting.

When I envisioned the interaction syntax, I assumed that (interaction ...) would be a call to a macro, and I would need to need to manipulate the data passed to interaction. However, once I got into the actual implementation, I found myself using the symbol "interaction", but never actually defining a macro or even a function. That's when homoiconicity really started to become clear to me. I'd written code that I was sure would need an implementation, yet it was used exclusively as data.

If you kept digging into this example you would find that anything found within (interaction ...) is never used as written, but is instead expanded in a way that allows expectations to rebind the specified function and use the expected arguments at verification time. As a result, you write the same code in the same way but within your test it's used exclusively as data and in your production code it's used exclusively as code. I'm a big fan of convention, and there's no better convention than 'use the exact same thing'.

I later added the ability to add interaction tests for calls to Java objects as well, which led to the following behavior for expectations.

• If your expected value is not an interaction, it will be expanded as is.
• If your expected value is an interaction with a Clojure function, it will be used as data exclusively and expanded to rebind the function, capture all calls to the function and verify that a call occurred with the arguments you specified.
• If your expected value is an interaction with a Java method, it wil be used as data exclusively and expanded to mockito setup and verification code.

Thus, an expected value is sometimes code, and sometimes data.

Clojure: Converting scenarios With Interleaved expect Calls To Bare expectations

Since I've deprecated scenarios, I went through all of my projects and removed any usages of expectations.scenarios. For the most part the conversion was simple; however, I did run into one instance where the scenario contained interleaved expectations.

The following code is an example of a scenario with interleaved expectations.


In the previously linked blog entry I recommend using a clojure assert to replace the interleaved expectations. That solution works, but I found an additional approach that I wanted to share.

When I encountered code similar in structure to the code above, I immediately envisioned writing 3 expectations similar to what you find below.


note: for my contrived example the first two tests could have been written without the let; however, the tests from my codebase could not - and I believe the blog entry is easier to follow if the tests are written in the way above.

While these tests verify the same expectations, the way that they are written doesn't convey to a test maintainer that they relate to each other more than they are related to the other tests within the file. While pondering this complaint, I grouped the tests in the following way more as a joke than anything else.


I would never actually use given simply to group code; however, grouping the code together did cause me to notice that there was a usage of given that would not only keep the code grouped, but it would also allow me to test what I needed with less code.

The following example is very similar in structure to the finished product within my codebase.

The above example verifies everything that the original scenario verified, does not use a scenario, and conveys to a maintainer that related logic is being tested within all three tests - in short: this felt like the right solution.

Clojure: Deprecating expectations.scenarios

I previously mentioned:

The functionality in expectations.scenarios was borne out of compromise. I found certain scenarios I wanted to test, but I wasn't sure how to easily test them using what was already available in (bare) expectations. The solution was to add expectations.scenarios, and experiment with various features that make testing as easy as possible.

Truthfully, I've never liked scenarios - I've always viewed them as a necessary evil. First of all, I hate that you can't mix them with bare expectations - this leads to having 2 files or 2 namespaces in 1 file (or you put everything in a scenario, meh). You either can't see all of your tests at the same time (2 files), or you run the risk of your tests not working correctly with other tools (expectations-mode doesn't like having both namespaces in 1 file). Secondly, I think they lead to sloppy tests.

The second complaint causes me to get on my soap-box about test writing, but never motivated me to do anything. However, as expectations-mode has become more integral to my workflow, the first issue caused me to make a change.

As of 1.4.17 you should be able to write anything that you would usually write in a scenario in a bare expect instead.

I've already published several blog entries that should help if you're interested in migrating your scenarios to bare expectations.

• Freezing Time Added To expectations
• Interaction Based Testing Added To expectations
• redef-state Added To expectations
• Using given & expect To Replace scenarios

One feature that is noticeably missing from bare expectations is the stubbing macro. I decided to leave the stubbing macro out as I believe it's just as intention revealing to use with-redefs & constantly, and I always prefer to use core functions when possible.

If you were previously using stubbing, your test can be converted in the following way.

(stubbing [a-fn true]
  (do-work))

;;; can now be written as
(with-redefs [a-fn (constantly true)]
  (do-work))

A nice side effect of removing stubbing is the reduction of indention if you are using both stubbing and with-redefs. This seems like the right trade-off for me (less indenting, relying on core functions that everyone should know); however, I'm not against adding stubbing again in the future if it becomes a painfully missing feature.

There is one type of scenario that I haven't yet addressed, interleaved expectations. I found zero of these types of scenarios in my codebases; however, I'm addressing these types of scenarios here for completeness.

(scenario
  (do-work)
  (expect a b)
  (do-more-work)
  (expect c d))

Any scenario that has interleaved expectations can be converted in the following way:

(expect c
  (do
    (do-work)
    (assert (= a b))
    (do-more-work)
    d))

expectations 1.4.17 still has support for scenarios, so you can upgrade and migrate at your own pace. I'll likely leave scenarios in until the point that I change some code that breaks them, then I'll remove them. Of course, if you prefer scenarios, you're welcome to never upgrade, or fork expectations.

If you run into issues while converting your scenarios, please open an issue on github: https://github.com/jaycfields/expectations/issues?state=open

Clojure: Using given & expect To Replace scenarios

The functionality in expectations.scenarios was borne out of compromise. I found certain scenarios I wanted to test, but I wasn't sure how to easily test them using what was already available in (bare) expectations. The solution was to add expectations.scenarios, and experiment with various features that make testing as easy as possible.

Two years later, the features that make sense have migrated back to expectations:

• Freezing Time Added To expectations
• Interaction Based Testing Added To expectations
• redef-state Added To expectations

With those features, you should be able to convert any existing scenario to a bare expectation. What isn't covered with those features is what you should do if your scenario ends with multiple expects. This blog entry demonstrates how you can use given with a bare expectation to achieve the same test coverage.

Below is an example of a scenario that ends with multiple expects.


Using given, these scenarios are actually very easy to convert. The given + bare expectation example below tests exactly the same logic.


The test coverage is the same in the second example, but it is important to note that the let will now be executed 3 times instead of 1. This isn't an issue if your tests run quickly, if they don't you may want to revisit the test to determine if it can be written in a different way.

An interesting side-effect occurred while I was converting my scenarios - I found that some of my scenarios could be broken into multiple expectations that were then easier to read and maintain.

For example, the above expectations could be written as the example below.


note: you could simplify even further and remove the given, but that's likely only due to how contrived the test is. Still, the possibility exists that some scenarios will be easily convertible to bare expectations.

Using the technique described here, I've created bare expectations for all of the scenarios in the codebase I'm currently working on - and deleted all references to expectations.scenarios.

Clojure: Use expect-let To Share A Value Between expected And actual

Most of the time you can easily divorce the values needed in an expected form and an actual form of an expectation. In those cases, nothing needs to be shared and your test can use a simple bare expect. However, there are times when you need the same value in both the expected and actual forms - and a bare expect doesn't easily provide with a way to accomplish that.

In version 1.4.16 or higher of expectations, you can now use the expect-let macro to let one or more values and reference them in both the expected and actual forms.

Below is a simple example that makes use of expect-let to compare two maps that both have a DateTime.


If possible you should prefer expect, but expect-let gives you another option for the rare cases where you absolutely need to share a value.

Clojure: Freezing Time Added To expectations

If you're using expectations and Joda Time, you now have the ability to freeze time in bare expectations (version 1.4.16 and above). The following code demonstrates how you can use the freeze-time macro to set the time, verify anything you need, and allow time to be reset for you.


Under the covers freeze-time is setting the current millis using the DateTime you specify, running your code and resetting the current millis in a finally. As a result, after your code finishes executing, even if finishing involves throwing an exception, the millis of Joda Time will be set back to working as you'd expect.

The freeze-time macro can be used in both the expected and actual forms, and can be nested if you need to set the time multiple times within a single expectation.

Clojure: Interaction Based Testing Added To expectations

The vast majority of testing I do these days is state-based; however, there are times when I need to test an interaction (e.g. writing to a file or printing to standard out). The ability to test interactions has been in expectations.scenarios for quite awhile, but there isn't any reason that you need a scenario to test an interaction - so, as of version 1.4.16, you also have the ability to test interactions with bare expectations.

The following test shows how you can specify an expected interaction. This test passes.


Writing the test should be straightforward - expect the interaction and then call the code that causes the interaction to happen.

As I was adding this behavior I enhanced the error reporting. Below you can find a failing test and the output that is produced.


As you can see, all three calls to the 'one' function are reported. If the number of args used to call 'one' are of the same size as the expected args, each arg is compared in detail; otherwise the two lists are compared in detail (but the elements are not).

As you can see in this failure the first argument, "hello", matches.

 ;          got: (one "hello" {2 3, :a 1})
 ;                   arg1: matches
 ;          expected arg2: {:a :b, :c {:ff :gg, :dd :ee}}
 ;            actual arg2: {2 3, :a 1}
 ;          2 with val 3 is in actual, but not in expected
 ;          :c {:dd with val :ee is in expected, but not in actual
 ;          :c {:ff with val :gg is in expected, but not in actual
 ;          :a expected: :b
 ;                  was: 1

Anytime an argument matches expectations will simply print "matches". You can also specify :anything as an argument, to ignore that argument and always 'match'. The following test shows an example of matching the second argument, while the first argument is no longer matching.


That's it. Hopefully these interaction tests follow the principle of least surprise, and are easy for everyone to use.

Clojure: redef-state Added To expectations

When testing functions that reference some state (atom, ref, or agent), it's nice to be able to quickly replace the value of the state in the context of the test. When your function only interacts with one piece of state, a simple call to with-redefs will do the trick. However, there are times when the function that you're calling updates many different pieces of state, and you'd like to be able to redef all of them with one call. The expectations testing framework (v 1.4.16 and above) provides you the ability to redef all atoms, refs, and agents in a namespace with one call to redef-state.

(this same feature existed in expectation.scenarios as 'localize-state')

Let's take a look at the following contrived namespace


In the above namespace we have two atoms that are both updated when you process an update. Testing that the atoms are updated is fairly simple, which the tests below demonstrate.


Unfortunately, these tests will not both pass, as they both update the same atom. We could clean up at the end of each test, but it's usually cleaner to simply redef the atoms in the context of the test. The tests below use with-redefs to ensure that the state is only manipulated in the context of the tests.


At this point the tests all pass. This solution works fine, but expectations gives you the ability to trim a bit of code and simply specify the namespace instead. The following tests specify the namespace and let expectations take care of the rest.


That's it. Now all atoms, refs, and agents that are defined in the 'blog' namespace will be redefined within the context of the (redef-state) call. It's also important to note that redef-state can take as many namespaces as you'd like to specify in the first arg vector.

Clojure: expectations & with-redefs

In general, when I'm writing tests, the pure functions end up as bare expects and the impure functions end up as scenarios. The following contrived namespace keeps a list of users and allows you to get the full name of each user.

The tests for this namespace would often look something like the following code:

It feels natural to put the with-redefs in a scenario, since scenarios also support (and often make use of) stubbing, localize-state, & freeze-time. However, there's really no reason that you need to use a scenario if you're simply looking to redef a var.

The following test provides the same level of functionality verification, without needing to use expectations.scenarios:

scenarios are great, but these days I try to keep things simple with bare expectations whenever possible.

Clojure: Freezing Time in expectations

Don't do this, this blog post is out-dated. You want this: http://blog.jayfields.com/2012/11/clojure-freezing-time-added-to.html

The current version of expectations (1.4.3) contains support for freezing time within an expectations scenario. I already put this information out in a previous blog entry, and I'm going to use the same examples here.

The following code is a test I was working on at work:

(scenario
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (first @fills)))

The test builds a PartialFill domain object, and passes it to handle-fill. The handle-fill fn converts the domain object to a map and conj's the new map onto the fills vector (which is an atom). The above test would fail due to the time not being frozen, and the traditional way to deal with this issue was to change the test to use (in ..) and ignore the :time key-value pair.

The following code would have resulted in a passing test:

(scenario
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33} (in (first @fills))))

The above code is fine, as long as you're not interested in verifying the time; however, things get a lot uglier if you do want to verify time. The following code freezes (joda) time, allowing for time verification:

(scenario
 (DateTimeUtils/setCurrentMillisFixed 100)
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 100} (first @fills))
 (DateTimeUtils/setCurrentMillisSystem))

While the above code does result in a passing test, it would cause unexpected issues if expect ever failed. In expectations a failure throws an exception (to terminate scenario execution) and the time reset would never occur. Even if that wasn't the case, the time related code takes away from the actual focus of the test.

Therefore, expectations has been modified to take a :freeze-time parameter as part of a scenario definition. The :freeze-time parameter can be a string or a boolean - when specifying a string, anything parse-able by Joda will do; if you specify a boolean time will simply be stopped at the moment of execution.

With :freeze-time available we can rewrite the above test in the following way:

(scenario
 :freeze-time "2012-4-30"
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (first @fills)))

The resulting code is easier to work with, while still allowing verification of time.

I believe the domain related code does a better job of demonstrating the point; however, the following examples are available if you'd like something simplified.

(scenario
  :freeze-time true
  (expect (DateTime.) (do (Thread/sleep 10) (DateTime.))))

(scenario
  :freeze-time "2012-4-30TZ"
  (expect (.getMillis (DateTime.)) 1335744000000))

Clojure: expectations, colorized

The current version of expectations (1.4.3) prints colorized results by default on non-windows boxes.

The following screenshot is an example of the output when no tests are failing:

And, the following screenshot is an example of the output when there are failures or errors:


Colorized output is one of those small things that is easy to de-prioritize, but once it's done you can't figure out why you didn't do it earlier. The code to colorize your results is very simple, and there's even a lib, colorize, if you prefer to simply include a dependency instead.

Of course, if you prefer to stick with non-colorized results that's possible as well - simply set the environment variable EXPECTATIONS_COLORIZE to 'false'.

Clojure: Conditionally Importing

I recently ran into a test that needed (org.joda.time.DateTime.) to always return the same time - so it could easily be asserted against. This situation is fairly common, so it makes sense to add support to expectations. However, I didn't want to force a joda-time dependency on everyone who wanted to use expectations. Luckily, Clojure gives me the ability to conditionally import dependencies.

The test looked something like the following code snippet.

(scenario
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (in (first @fills))))

note: build is a fn that creates domain objects (I do a lot of Java Interop).

The test builds a PartialFill domain object, and passes it to handle-fill. The handle-fill fn converts the domain object to a map and conj's the new map onto the fills vector (which is an atom).

The build fn creates a PartialFill who's time is (DateTime.), which means you can either not test the time or you need to freeze the time. Joda time makes it easy to freeze time by calling DateTimeUtils/setCurrentMillisFixed, so you could write the test as:

(scenario
 (DateTimeUtils/setCurrentMillisFixed 100)
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 100} (in (first @fills)))
 (DateTimeUtils/setCurrentMillisSystem))

Of course, that would cause issues if expect ever failed, as failure throws an exception and the time reset would never occur. Even if that wasn't the case, the time related code takes away from what I'm really testing - so I set out to create a declarative approach from within expectations.

The syntax for the freezing time is fairly straightforward, and the original test can be written as what follows (with expectations 1.4.3 & up)

(scenario
 :freeze-time "2012-4-30"
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (in (first @fills))))

Writing the code to freeze time with expectations is trivial enough, but it's only relevant if you actually use joda-time. Since not everyone uses joda-time, it seemed like the best option was to make expectations import joda only if you actually use :freeze-time within your code.

The code linked here shows the conditional imports: https://github.com/jaycfields/expectations/blob/f2a8687/src/clojure/expectations/scenarios.clj#L81-L86

Even if you're not familiar with macros it should be obvious that I'm importing and using the joda-time classes conditionally. At macro expansion time I'm checking for the presence of freeze-time, and if it does exist the necessary classes are imported. If freeze-time is never found, the joda-time classes will never be imported, and no joda-time dependency will need to live within your project.

I always assumed this would be possible, but I'd never actually attempted to put this type of code together. I'm definitely happy with how the code turned out - I can freeze time in expectations if I want, but I don't force anyone to use joda.

Clojure: expectations - scenarios

expectations.scenarios are now deprecated: http://blog.jayfields.com/2012/11/clojure-deprecating-expectationsscenari.html

When I set out to write expectations I wanted to create a simple unit testing framework. I'm happy with what expectations provides for unit testing; however, I also need to write the occasional test that changes values or causes a side effect. There's no way I could go back to clojure.test after enjoying better failure messages, trimmed stack traces, automatic testing running, etc. Thus, expectations.scenarios was born.

Using expectations.scenarios should be fairly natural if you already use expectations. The following example is a simple scenario (which could be a unit test, but we'll start here for simplicity).

(ns example.scenarios
  (:use expectations.scenarios))

(scenario
 (expect nil? nil))

A quick trip to the command line shows us that everything is working as expected.

Ran 1 tests containing 1 assertions in 4 msecs
0 failures, 0 errors.

As I said above, you could write this test as a unit test. However, expectations.scenarios was created for the cases in which you want to verify a value, make a change, and verify a value again. The following example shows multiple expectations verifying changing values in the same scenario.

(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 1 @a)
    (swap! a inc)
    (expect 2 @a)))

In expectations (unit tests) you can only have one expect (or given) so failures are captured, but they do not stop execution. However, due to the procedural nature of scenarios, the first failing expect stops execution.

(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 2 @a)
    (println "you'll never see this")))

failure in (scenarios.clj:4) : example.scenarios
           (expect 2 (clojure.core/deref a))
           expected: 2 
                was: 1
           on (scenarios.clj:7)
Ran 1 tests containing 1 assertions in 81 msecs
1 failures, 0 errors.

expectations.scenarios also allows you to easily verify calls to any function. I generally use interaction expects when I need to verify some type of side effect (e.g. logging or message publishing).

(scenario
 (println "1")
 (expect (interaction (println "1"))))

It's important to note the ordering of this scenario. You don't 'setup' an expectation and then call the function. Exactly the opposite is true - you call the function the same way you would in production, then you expect the interaction to have occurred. You may find this jarring if you're used to setting up your mocks ahead of time; However, I think this syntax is the least intrusive - and I think you'll prefer it in the long term.

The above example calls println directly, but your tests are much more likely to look something like this.

(defn foo [x] (println x))

(scenario
 (foo "1")
 (expect (interaction (println "1"))))

Similar to all other mocking frameworks (that I know of) the expect is using an implicit "once" argument. You can also specify :twice and :never if you find yourself needing those interaction tests.

(defn foo [x] (println x))

(scenario
 (foo "1")
 (foo "1")
 (expect (interaction (println "1")) :twice)
 (expect (interaction (identity 1)) :never))

On occasion you may find yourself interested in verifying 2 out of 3 arguments - expectations.scenarios provides the 'anything' var that can be used for arguments you don't care about.

(defn foo [x y z] (println x y z))

(scenario
 (foo "1" 2 :a)
 (expect (interaction (println "1" anything :a))))

That's about all there is to expectations.scenarios, hopefully it fills the gap for tests you want to write that simply can't be done as unit tests.

Clojure: expectations unit testing wrap-up

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six (this entry)

The previous blog posts on expectations unit testing syntax cover all of the various ways that expectations can be used to write tests and what you can expect when your tests fail. However, there are a few other things worth knowing about expectations.

Stacktraces
expectations aggressively removes lines from the stacktraces. Just like many other aspects of expectations, the focus is on more signal and less noise. Any line in the stacktrace from clojure.core, clojure.lang, clojure.main, and java.lang will be removed. As a result any line appearing in your stacktrace should be relevant to your application or a third-party lib you're using. expectations also removes any duplicates that can occasionally appear when anonymous functions are part of the stacktrace. Again, it's all about improving signal by removing noise. Speaking of noise...

Test Names
You might have noticed that expectations does not require you to create a test name. This is a reflection of my personal opinion that test names are nothing more than comments and shouldn't be required. If you desire test names, feel free to drop a comment above each test. Truthfully, this is probably a better solution anyway, since you can use spaces (instead of dashes) to separate words in a comment. Comments are good when used properly, but they can become noise when they are required. The decision to simply use comments for test names is another example of improving signal by removing noise.

Running Focused Expectations
Sometimes you'll have a file full of expectations, but you only want to run a specific expectation - expectations solves this problem by giving you 'expect-focused'. If you use expect-focused only expectations that are defined using expect-focused will be run.

For example, if you have the following expectations in a file you should see the following results from 'lein expectations'.

(ns sample.test.core
  (:use [expectations]))

(expect zero? 0)
(expect zero? 1)
(expect-focused nil? nil)

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 2 msecs
IGNORED 2 EXPECTATIONS
0 failures, 0 errors.

As you can see, expectations only ran one test - the expect-focused on line 6. If the other tests had been run the test on line 5 would have created a failure. It can be easy to accidentally leave a few expect-focused calls in, so expectations prints the number of ignored expectations in capital letters as a reminder. Focused expectation running is yet another way to remove noise while working through a problem.

Tests Running
If you always use 'lein expectations' to run your tests you'll never even care; however, if you ever want to run individual test files it's important to know that your tests run by default on JVM shutdown. When I'm working with Clojure and Java I usually end up using IntelliJ, and therefore have the ability to easily run individual files. When I switched from clojure.test to expectations I wanted to make test running as simple as possible - so I removed the need to specify (run-all-tests). Of course, if you don't want expectations to run for some reason you can disable this feature by calling (expectations/disable-run-on-shutdown).

JUnit Integration
Lack of JUnit integration was a deal breaker for my team in the early days, so expectations comes with an easy way to run all tests as part of JUnit. If you want all of your tests to run in JUnit all you need to do is implement ExpectationsTestRunner.TestSource. The following example is what I use to run all the tests in expectations with JUnit.

import expectations.junit.ExpectationsTestRunner;
import org.junit.runner.RunWith;

@RunWith(expectations.junit.ExpectationsTestRunner.class)
public class SuccessTest implements ExpectationsTestRunner.TestSource{

    public String testPath() {
        return "test/clojure/success";
    }
}

As you can see from the example above, all you need to do is tell the test runner where to find your Clojure files.

That should be everything you need to know about expectations for unit testing use. If anything is unclear, please drop me a line in the comments.

Clojure: expectations - removing duplication with given

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five (this entry)
Clojure Unit Testing with Expectations Part Six

expectations obviously has a bias towards one assertion per test; however, there are times that verifying several things at the same time does make sense. For example, if you want to verify a few different properties of the same Java object it probably makes sense to make multiple assertions on the same instance.

One of the biggest problems with multiple assertions per test is when your test follows this pattern:

1. create some state
2. verify a bit about the state
3. alter the state
4. verify more about the state

Part of the problem is that the assertions that occurred before the altering of the state may or may not be relevant after the alteration. Additionally, if any of the assertions fail you have to stop running the entire test - thus some of your assertions will not be run (and you'll be lacking some information).

expectations takes an alternate route - embracing the idea of multiple assertions by providing a specific syntax that allows multiple verifications and the least amount of duplication.

The following example shows how you can test multiple properties of a Java object using the 'given' syntax.

(given (java.util.ArrayList.)
       (expect
         .size 0
         .isEmpty true))

jfields$ lein expectations
Ran 2 tests containing 2 assertions in 4 msecs
0 failures, 0 errors.

The syntax is simple enough: (given an-object (expect method return-value [method return-value]))
note: [method return-value] may be repeated any number of times.

This syntax allows us to expect return-values from as many methods as we care to verify, but encourages us not to change any state between our various assertions. This syntax also allows us to to run each assertion regardless of the outcome of any previous assertion.

Obviously you could call methods that change the internal state of the object and at that point you're on your own. I definitely wouldn't recommend testing that way. However, as long as you call methods that don't change any state 'given' can help you write succinct tests that verify as many aspects of an object as you need to test.

As usual, I'll show the output for tests that fail using this syntax.

(given (java.util.ArrayList.)
       (expect
         .size 1
         .isEmpty false))

jfields$ lein expectations
failure in (core.clj:4) : sample.test.core
           (expect 1 (.size (java.util.ArrayList.)))
           expected: 1 
                was: 0
failure in (core.clj:4) : sample.test.core
           (expect false (.isEmpty (java.util.ArrayList.)))
           expected: false 
                was: true

This specific syntax was created for testing Java objects, but an interesting side effect is that it actually works on any value and you can substitute method calls with any function. For example, you can test a vector or a map using the examples below as a template.

(given [1 2 3]
       (expect
         first 1
         last 3))

(given {:a 2 :b 4}
       (expect 
         :a 2
         :b 4))

jfields$ lein expectations
Ran 4 tests containing 4 assertions in 8 msecs
0 failures, 0 errors.

And, of course, the failures.

(given [1 2 3]
       (expect
         first 2
         last 1))

(given {:a 2 :b 4}
       (expect 
         :a 1
         :b 1))

jfields$ lein expectations
failure in (core.clj:4) : sample.test.core
           (expect 2 (first [1 2 3]))
           expected: 2 
                was: 1
failure in (core.clj:4) : sample.test.core
           (expect 1 (last [1 2 3]))
           expected: 1 
                was: 3
failure in (core.clj:9) : sample.test.core
           (expect 1 (:a {:a 2, :b 4}))
           expected: 1 
                was: 2
failure in (core.clj:9) : sample.test.core
           (expect 1 (:b {:a 2, :b 4}))
           expected: 1 
                was: 4
Ran 4 tests containing 4 assertions in 14 msecs
4 failures, 0 errors.

When you want to call methods on a Java object or call functions with the same instance over and over the previous given syntax is really the simplest solution. However, there are times where you want something more flexible.

expectations also has a 'given' syntax that allows you to specify a template - thus reducing code duplication. The following example shows a test that verifies + with various arguments.

(given [x y] (expect 10 (+ x y))
       4 6
       6 4
       12 -2)

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 5 msecs
0 failures, 0 errors.

The syntax for this flavor of given is: (given bindings template-form values-to-be-bound). The template form can be anything you need - just remember to put the expect in there.

Here's another example where we combine given with in to test a few different things. This example shows both the successful and failing versions.

;; successful
(given [x y] (expect x (in y))
       :a #{:a :b}
       {:a :b} {:a :b :c :d})

;; failure
(given [x y] (expect x (in y))
       :c #{:a :b}
       {:a :d} {:a :b :c :d})

lein expectations
failure in (core.clj:8) : sample.test.core
           (expect :c (in #{:a :b}))
           key :c not found in #{:a :b}
failure in (core.clj:8) : sample.test.core
           (expect {:a :d} (in {:a :b, :c :d}))
           expected: {:a :d} 
                 in: {:a :b, :c :d}
           :a expected: :d
                   was: :b
Ran 4 tests containing 4 assertions in 13 msecs
2 failures, 0 errors.

That's basically it for 'given' syntax within expectations. There are times that I use all of the various versions of given; however, there seems to be a connection with using given and interacting with Java objects. If you don't find yourself using Java objects very often then you probably wont have a strong need for given.

Clojure: expectations and Double/NaN

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four (this entry)
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six

I'm not really a fan of Double/NaN in general, but sometimes it seems like the least evil choice. When I find myself in one of those cases I always hate having to write tests in a way that differs from all the other tests in the codebase. A goal I've always had with expectations is to keep the syntax consistent, and as a result I've chosen to treat Double/NaN as equal to Double/NaN when in the various Clojure data structures.

The following examples demonstrate Double/NaN being treated as equal

;; allow Double/NaN equality in a map
(expect {:a Double/NaN :b {:c Double/NaN}} {:a Double/NaN :b {:c Double/NaN}})

;; allow Double/NaN equality in a set
(expect #{1 Double/NaN} #{1 Double/NaN})

;; allow Double/NaN equality in a list
(expect [1 Double/NaN] [1 Double/NaN])

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 32 msecs
0 failures, 0 errors.

As you would expect, you can also count on Double/NaN being considered equal even if you are using the 'in' function.

;; allow Double/NaN equality when verifying values are in a map
(expect {:a Double/NaN :b {:c Double/NaN}} (in {:a Double/NaN :b {:c Double/NaN} :d "other stuff"}))

;; allow Double/NaN equality when verifying it is in a set
(expect Double/NaN (in #{1 Double/NaN}))

;; allow Double/NaN equality when verifying it's existence in a list
(expect Double/NaN (in [1 Double/NaN]))

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 32 msecs
0 failures, 0 errors.

For completeness I'll also show the examples of each of these examples failing.

;; allow Double/NaN equality in a map
(expect {:a Double/NaN :b {:c Double/NaN}} {:a nil :b {:c Double/NaN}})

;; allow Double/NaN equality with in fn and map
(expect {:a Double/NaN :b {:c nil}} (in {:a Double/NaN :b {:c Double/NaN} :d "other stuff"}))

;; allow Double/NaN equality in a set
(expect #{1 Double/NaN} #{1 nil})

;; allow Double/NaN equality with in fn and set
(expect Double/NaN (in #{1 nil}))

;; allow Double/NaN equality in a list
(expect [1 Double/NaN] [1 nil])

;; allow Double/NaN equality with in fn and list
(expect Double/NaN (in [1 nil]))


jfields$ lein expectations
failure in (core.clj:5) : sample.test.core
           (expect {:a Double/NaN, :b {:c Double/NaN}}
                   {:a nil, :b {:c Double/NaN}})
           expected: {:a NaN, :b {:c NaN}} 
                was: {:a nil, :b {:c NaN}}
           :a expected: NaN
                   was: nil
failure in (core.clj:8) : sample.test.core
           (expect {:a Double/NaN, :b {:c nil}} (in {:a Double/NaN, :b {:c Double/NaN}, :d "other stuff"}))
           expected: {:a NaN, :b {:c nil}} 
                 in: {:a NaN, :b {:c NaN}, :d "other stuff"}
           :b {:c expected: nil
                       was: NaN
failure in (core.clj:11) : sample.test.core
           (expect #{1 Double/NaN} #{nil 1})
           expected: #{NaN 1} 
                was: #{nil 1}
           nil are in actual, but not in expected
           NaN are in expected, but not in actual
failure in (core.clj:14) : sample.test.core
           (expect Double/NaN (in #{nil 1}))
           key NaN not found in #{nil 1}
failure in (core.clj:17) : sample.test.core
           (expect [1 Double/NaN] [1 nil])
           expected: [1 NaN] 
                was: [1 nil]
           nil are in actual, but not in expected
           NaN are in expected, but not in actual
failure in (core.clj:20) : sample.test.core
           (expect Double/NaN (in [1 nil]))
           value NaN not found in [1 nil]
Ran 6 tests containing 6 assertions in 66 msecs
6 failures, 0 errors.

There always seems to be downsides to using NaN, so I tend to look for the least painful path. Hopefully expectations provides the most pain-free path when your tests end up needing to include NaN.

Clojure: expectations with values in vectors, sets, and maps

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three (this entry)
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six

I've previously written about verifying equality and the various non-equality expectations that are available. This entry will focus on another type of comparison that is allowed in expectations - verifying that an 'expected' value is in an 'actual' value.

A quick recap - expectations generally look like this: (expect expected actual)

verifying an expected value is in an actual value is straightforward and hopefully not a surprising syntax: (expect expected (in actual))

If that's not clear, these examples should make the concept completely clear.

;; expect a k/v pair in a map.
(expect {:foo 1} (in {:foo 1 :cat 4}))

;; expect a key in a set
(expect :foo (in #{:foo :bar}))

;; expect a val in a list
(expect :foo (in [:foo :bar]))

As you would expect, running these expectations results in 3 passing tests.

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 8 msecs
0 failures, 0 errors.

As usual, I'll show the failures as well.

;; expect a k/v pair in a map.
(expect {:foo 2} (in {:foo 1 :cat 4}))

;; expect a key in a set
(expect :baz (in #{:foo :bar}))

;; expect a val in a list
(expect :baz (in [:foo :bar]))

jfields$ lein expectations
failure in (core.clj:18) : sample.test.core
           (expect {:foo 2} (in {:foo 1, :cat 4}))
           expected: {:foo 2} 
                 in: {:foo 1, :cat 4}
           :foo expected: 2
                     was: 1
failure in (core.clj:21) : sample.test.core
           (expect :baz (in #{:foo :bar}))
           key :baz not found in #{:foo :bar}
failure in (core.clj:24) : sample.test.core
           (expect :baz (in [:foo :bar]))
           value :baz not found in [:foo :bar]

expectations does it's best to provide you with any additional info that might be helpful. In the case of the vector and the set there's not much else that can be said; however, the map failure gives you additional information that can be used to track down the issue.

There's nothing magical going on with 'in' expectations and you could easily do the equivalent with select-keys, contains?, or some, but expectations allows you to get that behavior while keeping your tests succinct.

Clojure: Non-equality expectations

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two (this entry)
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six

In my last blog post I gave examples of how to use expectations to test for equality. This entry will focus on non-equality expectations that are also available.

Regex
expectations allows you to specify that you expect a regex, and if the string matches that regex the expectation passes. The following example shows both the successful and failing expectations that use regexes.

(expect #"in 14" "in 1400 and 92")

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 4 msecs
0 failures, 0 errors.

(expect #"in 14" "in 1300 and 92")

jfields$ lein expectations
failure in (core.clj:17) : sample.test.core
           (expect in 14 in 1300 and 92)
           regex #"in 14" not found in "in 1300 and 92"
Ran 1 tests containing 1 assertions in 5 msecs
1 failures, 0 errors.

As you can see from the previous example, writing an expectation using a regex is syntactically the same as writing an equality expectation - and this is true for all of the non-equality expectations. In expectations there is only one syntax for expect - it's always (expect expected actual).

Testing for a certain type
I basically never write tests that verify the result of a function is a certain type. However, for the once in a blue moon case where that's what I need, expectations allows me to verify that the result of a function call is a certain type simply by using that type as the expected value. The example below shows the successful and failing examples of testing that the actual is an instance of the expected type.

(expect String "in 1300 and 92")

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 6 msecs
0 failures, 0 errors.

(expect Integer "in 1300 and 92")

jfields$ lein expectations
failure in (core.clj:17) : sample.test.core
           (expect Integer in 1300 and 92)
           in 1300 and 92 is not an instance of class java.lang.Integer
Ran 1 tests containing 1 assertions in 5 msecs
1 failures, 0 errors.

Expected Exceptions
Expected exceptions are another test that I rarely write; however, when I find myself in need - expectations has me covered.

(expect ArithmeticException (/ 12 0))

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 6 msecs
0 failures, 0 errors.

(expect ClassCastException (/12 0))

jfields$ lein expectations
failure in (core.clj:19) : sample.test.core
           (expect ClassCastException (/ 12 0))
           (/ 12 0) did not throw ClassCastException
Ran 1 tests containing 1 assertions in 4 msecs
1 failures, 0 errors.

There's another non-equality expectation that I do use fairly often - an expectation where the 'expected' value is a function. The following simple examples demonstrate that if you pass a function as the first argument to expect it will be called with the 'actual' value and it will pass or fail based on what the function returns. (truthy results pass, falsey results fail).

(expect nil? nil)
(expect true? true)
(expect false? true)

jfields$ lein expectations
failure in (core.clj:19) : sample.test.core
           (expect false? true)
           true is not false?
Ran 3 tests containing 3 assertions in 4 msecs
1 failures, 0 errors.

These are the majority of the non-equality expectations; however, there is one remaining non-equality expectation - in. Using 'in' is fairly straightforward, but since it has examples for vectors, sets, and maps I felt it deserved it's own blog post - coming soon.