When test just can't be automated

hi there,

i do hope i’m posting at the right place.

I am supposed to test some sort of a network device let’s say. i want to
do it with ruby unit testing framework.

Basically you can send it commands using TCP/IP in a custom language
made for the device. It also sends back results.

so basically pretty much a lot of commands can be tested.

But some can not be completly automated.

for instance, this little devices beeps. There is a command to disable
sound. You can check that the command reported something like “sound
successfully disabled”

but the ruby script can’t actually “hear” that the device has stopped
emitting sound after the command has been issued.

So i was thinking of having the test script do something like

init device
send a command that should make the device beep
–> at this point have the script stop and ask tester for confirmation
that a sound has been heard (ex: by hitting SPACE the test confirms that
she heard the sound)
send a command to disable sound
send a command that should normally have the device beeping
–> again ask the test to confirm no sound was heard

what’s the best way to implement this ? do you have a suggestion ? (the
testers using the scripts aren’t techies… :())

thanks
fourchette

Best way:
Automate the tests that are automatable, have them generate a report
that
the tester reviews/signs off on.

Then have the remaining tests performed manually, or semi-assisted with
scripts (in the manner that
you describe below). Its rarely useful to have tests that can be fully
automated, interrupted by manual
prompts (it will prevent other tests from running).

regards,
Richard.

On Thu, Aug 19, 2010 at 1:46 PM, Fourchette Fourchette <

what’s the best way to implement this ? do you have a suggestion ? (the
testers using the scripts aren’t techies… :())

If they can’t cope with a text script with Y/N prompts, maybe you could
pop up a dialog box (e.g. with Tk). Google “ruby tk examples” or
similar.

Maybe you’re worried about CLI interaction because you’re using a test
framework like Test::Unit. Tk should work fine there too. Otherwise you
can write a standalone test program which doesn’t use the test runner,
so you’re in control of the console at all times. (You could use the
framework’s assertion methods still)

Thank you for the feedback. the trouble is i think most tests are 10%
manual + 90% automatic. Basically it’s a complicated setup but
quick&easy tests (“setup” as in “setup VS tearDown”). I think setup can
be nearly totally automated, most of the tests too, but the expected
behavior (assertions) can only be observed by a human person.

i’m not very familiar with tk and i’m affraid it’s going to be hard to
have it adopted internally (heavy history of Microsft Visual Basic
GUIs).

Can the GUI be easily web-based? or such an approach would just be
overkill ? (compared to a tk UI i mean)

thanks

alex