Ugh. Testing Javascript. Is it possible to recommend something when everything you've seen is terrible? Usually I'm cool with picking the tool that sucks the least, but when it comes to Javascript testing the only words that come to mind are: epic fail.
In the past I've failed in two different ways.
- Selenium: Too slow and to brittle for any decent size test suite. There's more on that if you wish.
- Some javascript unit testing framework. I can't even remember the name. The syntax was ugly, the tests weren't easy to write, and the runner didn't integrate with any automated tools we used -- lame.
A few days later it occurred to me what the correct answer was -- Don't use Javascript and you won't have to test it.
No, I'm not suggesting that we should all go back to mostly static websites. Static content is fine for some things, but GMail is an obvious example of a site that is better done dynamically.
However, Javascript isn't your only choice for highly dynamic websites.
These days, if I were writing a website that required any dynamic interaction I would absolutely use Flex or Silverlight. I've done Flex, and it was nice to work with, but I must admit that I'm lured to Silverlight because it's going to (or does already?) support Ruby.
I'm not sure what the Silverlight testing story is, but I found Flex (and ActionScript) to be quite testable. The single biggest win (as I've said before) is that I no longer need to do in-browser testing. Removing the browser from the equation is huge. No more IE bugs causing your tests to fail, no long start up times as the browser is run, etc.
Testing with FlexUnit (with it's drawbacks) is an order of magnitude better than any experience I've had testing Javascript.
I'm comfortable saying that I would still use Javascript for trivial features that provided so little business value that they did not warrant testing. However, any features that provide noticeable business value must be tested, and I would move to a RIA solution instead.
In my experience, the benefits of switching to a RIA solution are dramatic, one of the largest being: you no longer need to worry about testing Javascript.
If your main objection to testing JavaScript is the fact that you need to run your tests in the browser, you should give the Johnson project a look (github.com/jbarnette/johnson). You can use it to write JavaScript tests in Ruby (I've used RSpec). Granted, it doesn't take into account browser quirks, and it's not quite ready for the main stage, but it's still a fun idea.
ReplyDeleteHi Pat,
ReplyDeleteI would argue that you must test your business important Javascript features in the browser to guarantee that they work.
The same is not true of RIA since it should behave the same regardless of browser.
Cheers, Jay
Hi Jay,
ReplyDeleteI've done a lot of flex also, but... don't you miss a high level testing framework for flex? Something like Selenium, maybe (usable by non-technical users).
(I've heard of http://funfx.rubyforge.org/, but is isn't exactly what I want, and the last time I saw it was windows only).
Cheers, Bruno
Hello Bruno,
ReplyDeleteI like high level tests, but I don't like running them through slow, buggy browsers. I much prefer writing higher level tests that execute against as part of flexunit.
The only good reason for testing through a browser is because implementations change and are buggy. If there was only one browser and it behaved as expected, I'd test everything with high level tests such as Rails Integration tests.
Cheers, Jay
Went to jQuery Conf on Sunday in Boston.
ReplyDeleteFirst time I had heard about QUnit - jQuery JavaScript Library. It looks pretty good:
http://docs.jquery.com/QUnit
jQuery seems to be exploding, and its plugin architecture and growing focus on testing reminded me a lot of the Ruby and Rails communities. I wouldn't be surprised to see this library gain a lot momentum.
http://jsunittest.com is a port of the unit test lib in prototypejs without any dependency on prototypejs. Its also the basis of a future Peepcode on JavaSscript Unit Testing.
ReplyDeleteI'm glad that you're bringing up the generally-sorry state of JavaScript unit testing here, as it needs more attention. However, I don't think "use silverlight instead" is really much of a solution. It's certainly not an option in a large number of projects. Nor should it be. Eek.
ReplyDeleteI see your point, about the browser being an awful testing environment (it is!) but what's really needed, imo, is an abstraction that guarantees (or does a best effort job, anyway) that code is cross-browser compliant, thus allowing us to take the browser out of the picture. Testing is clearly much easier when we're trying to verify the behavior of some jQuery instead of raw JavaScript, for instance.
As Dan points out, jQuery has its own test runner in qUnit. If one can believe that jQuery's own qUnit tests are complete, then the browser testing problem is mitigated to some degree by using the jQuery "abstraction", within which we just test the behavior of our jQuery and not the underlying raw JavaScript that's rendered in the browser. And if you need something that isn't provided by jQuery? Write a jQuery plugin for it and test that plugin with qUnit.
It's not perfect. But it's a step in the right direction (and the same argument could be presented for any other js lib that claims to be cross-browser compliant). Integration testing, of course, is still an issue.
You may want to take a peek at limelight. It's an open source RIA where you write your apps in Ruby.
ReplyDeletehttp://limelight.8thlight.com/
It's still in development, but it's a lot more Ruby than Silverlight.
You should really try to leave the browser specific stuff out of your javascript, and leave that to the library vendors. If your library doesn't handle a particular bug, fix it and submit a patch.
ReplyDeleteFlash/Flex/Silverlight certainly have their uses, but I see a lot of places where they would be the wrong tool for the job.
Going for Flex / Silverlight, because of poor javascript test frameworks, seems like a cop-out to me. I would have expected you of all people to get the community riled up, and get some efforts underway to remedy the situation.
Oh, c'mon Jay - what about sliders with jQuery ;) You would you give up all that for flex?
ReplyDeleteHi Sarah.
ReplyDeleteFlex has sliders.
http://livedocs.adobe.com/flex/201/html/wwhelp/wwhimpl/common/html/wwhelp.htm?context=LiveDocs_Book_Parts&file=controls_059_16.html
:)
We try our best to practice design-driven development. A great user experience should be the goal of any product that reaches the hands of user or consumers. If possible, we base all of our decisions on what's best for our users.
ReplyDeleteIf that means that it makes our jobs as developers and designers a little more difficult, so be it. Unless the additional time and effort can't be justified from a cost perspective, it's worth it.
That might be a case that's defensible: the cost of Javascript development is higher than Flex/Silverlight because of the difficulty and effort to properly test. You could also argue that for a highly dynamic site the user experience with Flex/Silverlight will exceed that of an HTML/JS-based site.
But basing the choice of technology on how easy it is to test? Maybe it's the UX person in me, but that seems backwards.
So, how exactly do I test Silverlight apps if I'm on RHEL - or for that matter, OS X? While we're there, how do I run them?
ReplyDeleteSorry, but the idea that using JavaScript is inherently a Bad Thing because you don't know of any decent testing environment is just... not that insightful.
Yes, JS is fragemented in that there're a million different frameworks that try to make it manageable - and good ones come with a testing framework. These days, I'm using dojo, and that comes with DOH, which offers - guess what? a testing framework.
A few thoughts.
ReplyDeleteA testing framework provided by the framework does not ensure that everything will function properly within a browser.
If you need to deploy something that runs on RHEL, Flex will work just fine.
If you want an abstraction that guarantees duplicate behavior across browsers, Flex works well there also.
Scott, you have a good point, but I think the Flex user experience is also, more often than not, superior.
Stick with Javascript and all it's problems at your own risk. Flex is definitely better, I speak from experience.
Hey Jay,
ReplyDeleteNice post, I agree that javascript can be a pain in the ass to test. However I am quite fond of the jquery/unobtrusive javascript movement. Graceful degradation is pretty nice especially given the ever increasing mobile market. I believe that the iPhone is now the top selling handset in the USA and it unfortunately offers no support for flex.
My only other concern with the flex/silverlight approach is that of addressability. With javascript I feel that I can mix in just enough of it to give a dynamic feel while still keeping the important parts of my application addressable. How does one support addressability of a state navigated to within the flex application? Is it similarly a approach of using flex in a pragmatic fashion to retain this ability? Or does one provide users with a mechanism to get links to the current state?
Cheers,
-Nolan
Hi Nolan,
ReplyDeleteYou are absolutely correct when it comes to the iPhone not being able to run Flex. I really don't think that's a long term problem though, and the best sites, for iPhone, are rewritten without all the bells and whistles anyway. I'm not saying it's a non-issue, but it wouldn't be a barrier to entry for me.
As far as being addressable, Flex has the idea built in. It also has other things like 'view source' and the ability for it to be parsed by Google. I haven't needed any of these features, but I believe more information is available in the documentation.
Linking is an issue, but definitely a manageable one in my opinion. Take for example the http://www.omio.com site. Much of the content is static, such as a phone description page (http://www.omio.com/phones/sony-ericsson/c905>. That sort of page doesn't need Flex and will be linkable without issue. However, on the main page the code that searches for deals does not need to be linkable. It's unlikely that you'll want to send a link to search results. However, it is possible, so in that scenario you could use Flex to generate a unique url and send that to someone.
I hope that answers your question.
Cheers, Jay
I've been learning to develop a proper JavaScript stack for user interaction; here are my findings:
ReplyDelete- Always use unobtrusive JavaScript. (Which, nowadays, translates to: "use jQuery".) Do not use jRails, do not use Prototype. Selenium may have its uses, but unit-testing JavaScript is not one of them. Graceful degradation is more of a bonus than a requirement these days, but generally, writing unobtrusive JavaScript leads to simpler code which is easier to test: both on the Rails side (functional/integration tests) and on the JavaScript side.
- Current JavaScript testing frameworks are all severely lacking, but in JavaScript it is very easy to write your own. Mocking/stubbing are trivial to implement, as long as you are comfortable with JavaScript's functions-as-arguments paradigm. You don't even need setup/teardown methods: just define test methods which take a function as an argument.
- Currently, I use qUnit and open the test suite in a browser window. I am edging towards integrating a Rhino-based solution, pioneered at http://github.com/jeresig/env-js/tree/master There are no HOWTOs for this kind of thing, but savvy developers should be able to create a test-suite JS solution. (I'm working on creating an actual Rails "Unit Test" wrapper which calls Rhino in a shell.) Spidermonkey would be preferable to me than Rhino (as it loads much faster), but env-js uses several Java calls which make that trickier.
- JavaScript objects should be engineered with as much care and attention to testability as Ruby ones.
- jsl at http://www.JavaScriptLint.com can tackle a large chunk of common cross-platform complications.