So I’ve been hunting for a way to stress test a web application and
haven’t been terribly satisfied so far.
I’d really love something where I could define a script that
determined the way an agent would behave. Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.
My brain is slowly churning around the possibilities of putting
ruby’s benchmark.rb, mechanize, maybe drb to manage multiple test
hosts… Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?
I’m looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don’t see any
facility to do computations on the returned content from a previous
request.
too ahead of myself: Does anyone here know of a tool does this?
I’m looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don’t see any
facility to do computations on the returned content from a previous
request.
rfuzz is nice but isn’t really what he’s asking about.
What he is asking about is something that I have wanted for a long time.
I want to be able to quickly and easily setup a test script for a site.
Go to a URL. Follow a link on the page returned. Maybe pick randomly
from a few choices of links to follow. Repeat a few times to simulate
people clicking around a site. Go to a page with a form, and fill in
form
fields, etc…
Basically, a DSL to write scripts for driving interactions to a site or
application that would somewhat simulate what real traffic is expected
to
look like.
It seems like it would be fairly easy to set up something with
WWW::Mechanize, although there might be performance problems if you
wanted to use it for stress testing.
Yeah, possibly. Mechanize or hpricot could be part of a solution, but
the
real challenge is in devising a DSL that makes it easy for me to, for
example:
Then, if I can run that on two or three machines at the same time, and
can
collate their logs and run an analysis on the collated logs, I’d be a
happy fellow.
Another thing that I’d love to be able to do is to run something like
the
above set of actions, but do it more slowly, with pauses between each
request, so that the site may only get, on average, one request a second
but I can let it run for days.
I have encountered failures on occasion that didn’t seem to happen as a
result of short term heavy loading, but more as a result of steady hits
over a longer period of time.
I’d like to be able to easily simulate that usage pattern in a
development
environment.
hosts… Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?
I’m looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don’t see any
facility to do computations on the returned content from a previous
request.
Thoughts? Suggestions?
I’m not really maintaining it at this point, but RWB might help. it’s
available as a gem. I’ve depracated it in favor of RFuzz, but RFuzz
still hasn’t caught all the features.
It seems like it would be fairly easy to set up something with
WWW::Mechanize, although there might be performance problems if you
wanted to use it for stress testing.
I’m not really maintaining it at this point, but RWB might help. it’s
available as a gem. I’ve depracated it in favor of RFuzz, but RFuzz
still hasn’t caught all the features.
Both RFuzz and RWB look like really good starts toward what I’m after.
Tough to say if I’ll decide to really code something at this point.
But I’ll certainly start browsing the source of both projects to get
some ideas.
I’d love to have something that made it easy to take a script (like in
rfuzz or rwb), then just drop it straight into a harness that was
responsible for multi-threading, distribution across slave nodes,
aggregating data and generating reports.
But given the state of things, it’d be a big problem to tackle.
Possibly too big for me at this point.
Thanks for the feedback. This list never disappoints!
-Mat
Go to a URL. Follow a link on the page returned. Maybe pick
randomly from a few choices of links to follow. Repeat a few times
to simulate people clicking around a site. Go to a page with a
form, and fill in form fields, etc…
Basically, a DSL to write scripts for driving interactions to a
site or application that would somewhat simulate what real traffic
is expected to look like.
I can’t answer Kirk’s question, but personally I’m looking for full
stack stress testing. Mostly so I can give my superiors some evidence
rather than blinding estimating how many concurrent users we can
handle. Not exactly mission-critical, but warm fuzzies are always
good.
-Mat
I’m not really maintaining it at this point, but RWB might help. it’s
available as a gem. I’ve depracated it in favor of RFuzz, but RFuzz
still hasn’t caught all the features.
Both RFuzz and RWB look like really good starts toward what I’m after.
Tough to say if I’ll decide to really code something at this point.
But I’ll certainly start browsing the source of both projects to get
some ideas.
If you’re planning on writing something, either coordinate with Zed
to work on RFuzz, or take over RWB and go for it.
I’d love to have something that made it easy to take a script (like in
rfuzz or rwb), then just drop it straight into a harness that was
responsible for multi-threading, distribution across slave nodes,
aggregating data and generating reports.
That’s sort of where I was headed with RWB. It’s already multithreaded,
but it really should use dRb to coordinate across multiple nodes. I
don’t think there’s too much code needed to do that.
But given the state of things, it’d be a big problem to tackle.
Possibly too big for me at this point.
If you decide to go with RWB, I’d be happy to answer any questions.
It might be a plateful, but you can take small bites and it won’t be
that hard. (Same is true of RFuzz, except that I can’t answer the
questions.)
View fund information for some random funds.
I don’t need to drive an actual IE or Firefox session, and for this
sort of thing that’s impractical.
What you are describing is a load testing tool. I work with one such
tool, Segue (now Borland) SilkPerformer. Another well-known one is
Mercury LoadRunner. As far as I know, there are no open source tools
that have the flexibility in script capture these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.
There are also tools for load test monitoring, server monitoring and
performance/bottleneck analysis. These things are expensive, though.
The closest thing I’ve found in the open source world is jMeter. But as
far as I know, you have to code your own scripts, your own benchmarks
and your own performance analysis tools with jMeter. It’s a tradeoff of
money versus effort. Quite frankly, I’d recommend spending the money.
Email me off-line if you want the details.
Now if someone wants to write a full-featured web application load
testing tool – and I mean full-featured – in Ruby and open source
it, allow me to encourage them. I sure as heck don’t have the time to do
it.
… As far as I know, there are no open source tools
that have the flexibility in script capture these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.
Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? Yes, it’s Java, but if you’re considering
JMeter…
Caveat: I haven’t used either of the commercial tools you mentioned,
so grain of salt, YMMV, etc.
Those actually drive a real browser session, though. That’s not really
what I need.
Um, then what DO you need that Rails unit and integration testing can’t
handle? Any other type of testing requires a browser instance.
See below. Rails unit testing is irrelevant. I don’t use Rails. It’s
inadequate for my needs.
I can’t answer Kirk’s question, but personally I’m looking for full
stack stress testing. Mostly so I can give my superiors some evidence
rather than blinding estimating how many concurrent users we can
handle. Not exactly mission-critical, but warm fuzzies are always
good.
What I would like, and what Mat wants, I think, have a great deal of
overlap.
To repeat some of what I said in an earlier post, I’d like to.
Hit the homepage of a site.
Go to a random URL on the site.
Repeat 2-5 times.
Go to the fund information page.
4 Click through to the distributions display.
Login (an action that requires sending a form response, and receiving
a
cookie in response that will be sent on further requests).
View fund information for some random funds.
Go into advisor tools and feed some numbers into a calculator.
Goto the logout link.
If I can create scripts that define actions like those outlined above,
which would can simulate expected real world usage over a period of
days,
or can simulate hordes of users in a more realistic manner than the
ab/httperf mechanism of hammering a single URL, it would be very
valuable.
I don’t need to drive an actual IE or Firefox session, and for this sort
of thing that’s impractical.
WebTestRecorder? Yes, it’s Java, but if you’re considering
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P) http://borasky-research.blogspot.com/
If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.
Could you explain why this is a “deal killer”? I would hope that
per-browser
functionality testing is a seperate suite from load testing, unless
you’re
sending different websites depending on the browser type.
Caveat: I haven’t used either of the commercial tools you mentioned,
so grain of salt, YMMV, etc.
Yeah … I missed it when I first posted but I have just looked at it.
Unfortunately, the recorder apparently only works with Firefox, which is
a deal-killer. However much you may hate IE and Microsoft …
