I am looking for a user test automation tool. When I say ‘user’, I
mean it should mimic the user interacting with the app. That
ultimately boils down to a tool that drives a browser or is able to
make http requests and allow us to deal easily with the responses.
Our requirements are to be able to write the tests in Ruby (no need to
switch to other language for test designing) and (relative)
browser-independence (this one puts Watir out of the game, since it
seems to be IE-dependent for the time being). A nice feature (but not
a must-have) would be possibility of headless (command line)
execution.
I am aware of Selenium and WebUnit. Does anyone have any experience
with either one (or a similar tool)?
This uses the normal Ruby Test::Unit infrastructure for writing tests
and loading fixtures.
I had a few problems with the second one, so I use the
selenium_on_rails plugin and it works great.
You can install selenium itself using RubyGems but its an old version,
or download the latest version of Selenium (0.6) and extract into your
vendor directory. Follow the instructions for either of the above.
I am aware of Selenium and WebUnit. Does anyone have any experience
with either one (or a similar tool)?
Yes, I’ve used (and still using) Watir quite extensively for just the
kind of work you describe. It’s the best thing since the sliced bread,
as long as you don’t have to deal with applets, and can live with your
automated tests working on IE 5.5+ only.
I haven’t worked with Selenium, but some fellow ThoughtWorkers say it’s
a Good Thing ™, too. The tradeoff, as far as I understand, is that
with Selenium you have to have some stuff installed on the web server
(deployment is more complicated, therefore), but you can use it on any
OS, with a variety of browsers.
I’ll probably sound like a moron here, so bear with me I don’t
understand the interest in these kinds of tools.
Seems to me that decent functional tests are will make sure your actions
are doing what you want them to do.
This is not the first thread I’ve read that mention the value of
Selenium and Watir, so obviously I’m missing something. Any
enlightenment would be appreciated.
Seems to me that decent functional tests are will make sure your actions
are doing what you want them to do.
Yes, but… When you say “functional”, you obviously refer to Rails
controller tests. Well, not everybody (to put it mildly) develops on
Rails.
And even with Rails, from a developer’s point of view, controller tests
do not cover at least two big things:
Javascript.
URL parsing/generation
Also, coverage of HTML produced is in practice either non-existant or
patchy (hoe many people actually use XPath assertions against @response.body?)
Watir is great for testing end-to-end scenarios, where you click through
several screens and there are no 404s or 500s. The #1 reason for adding
Watir test battery to Instiki was that I’ve had some some fairly heinous
bugs around URL generation, escaping my unit/controller tests.
Ability to do test automation for AJAX should be useful, too.
Disclaimer: I did not personally try testing AJAX calls with Watir on
any industrial scale yet, but I spiked it once and it works.
Thats because Rails’ idea of “functional” tests aren’t really
functional tests, they are more like unit tests for your controllers.
They are great for making sure an action does what it needs to do but
its not very good for testing your app as a whole and making sure your
app is doing what it should, not just the code.
This is where acceptance testing tools like Selenium come in.
Not just that, but clients should be able to write acceptance tests,
not just programmers, as it is your clients who should define what the
acceptance tests are.
Finally, unless you’ve tested your browser-based app inside a browser,
you haven’t really done any proper acceptance testing. Selenium and
Watir let you do that, and they let you automate the process too.
I don’t