Forum: Ruby Ruby replacement for ab?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Fc784eadb3b54531fdc3d2053db6f83f?d=identicon&s=25 Mat Schaffer (Guest)
on 2006-12-21 23:09
(Received via mailing list)
So I've been hunting for a way to stress test a web application and
haven't been terribly satisfied so far.

I'd really love something where I could define a script that
determined the way an agent would behave.  Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting
ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
hosts....   Some sort of ruby DSL would be sweet.  But before I get
too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now.  It at least allows for assertions
on HTTP requests as part of an agent script, but I don't see any
facility to do computations on the returned content from a previous
request.

Thoughts?  Suggestions?

Thanks in advance,
Mat
87e41d0d468ad56a3b07d9a6482fd6d5?d=identicon&s=25 Hemant Kumar (gnufied)
on 2006-12-21 23:21
(Received via mailing list)
On 12/22/06, Mat Schaffer <schapht@gmail.com> wrote:
> hosts....   Some sort of ruby DSL would be sweet.  But before I get
> Mat
>
>

Did you see, : http://rfuzz.rubyforge.org/
1b5341b64f7ce0244366eae17f06c801?d=identicon&s=25 unknown (Guest)
on 2006-12-21 23:52
(Received via mailing list)
On Fri, 22 Dec 2006, hemant wrote:

>> too ahead of myself: Does anyone here know of a tool does this?
>>
>> I'm looking at JMeter right now.  It at least allows for assertions
>> on HTTP requests as part of an agent script, but I don't see any
>> facility to do computations on the returned content from a previous
>> request.
>>
>
> Did you see, : http://rfuzz.rubyforge.org/

rfuzz is nice but isn't really what he's asking about.

What he is asking about is something that I have wanted for a long time.
I want to be able to quickly and easily setup a test script for a site.

Go to a URL.  Follow a link on the page returned.  Maybe pick randomly
from a few choices of links to follow.  Repeat a few times to simulate
people clicking around a site.  Go to a page with a form, and fill in
form
fields, etc...

Basically, a DSL to write scripts for driving interactions to a site or
application that would somewhat simulate what real traffic is expected
to
look like.

That would be dandy.


Kirk Haines
4f75675671b8fb1d81666c753e391cc4?d=identicon&s=25 Fred Wulff (Guest)
on 2006-12-21 23:56
(Received via mailing list)
It seems like it would be fairly easy to set up something with
WWW::Mechanize, although there might be performance problems if you
wanted to use it for stress testing.

-Fred
1b5341b64f7ce0244366eae17f06c801?d=identicon&s=25 unknown (Guest)
on 2006-12-22 00:18
(Received via mailing list)
On Fri, 22 Dec 2006, Fred Wulff wrote:

> It seems like it would be fairly easy to set up something with
> WWW::Mechanize, although there might be performance problems if you
> wanted to use it for stress testing.

Yeah, possibly.  Mechanize or hpricot could be part of a solution, but
the
real challenge is in devising a DSL that makes it _easy_ for me to, for
example:

1) Hit the homepage.

2) Go to a random URL on the site.
    Repeat 2-5 times.

3) Go to the fund information page.

4  Click through to the distributions display.

5)  Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information,
etc....



Then, if I can run that on two or three machines at the same time, and
can
collate their logs and run an analysis on the collated logs, I'd be a
happy fellow.

Another thing that I'd love to be able to do is to run something like
the
above set of actions, but do it more slowly, with pauses between each
request, so that the site may only get, on average, one request a second
but I can let it run for days.

I have encountered failures on occasion that didn't seem to happen as a
result of short term heavy loading, but more as a result of steady hits
over a longer period of time.

I'd like to be able to easily simulate that usage pattern in a
development
environment.


Kirk Haines
Bb6ecee0238ef2461bef3416722b35c5?d=identicon&s=25 pat eyler (Guest)
on 2006-12-22 00:24
(Received via mailing list)
On 12/21/06, Mat Schaffer <schapht@gmail.com> wrote:
> hosts....   Some sort of ruby DSL would be sweet.  But before I get
> too ahead of myself: Does anyone here know of a tool does this?
>
> I'm looking at JMeter right now.  It at least allows for assertions
> on HTTP requests as part of an agent script, but I don't see any
> facility to do computations on the returned content from a previous
> request.
>
> Thoughts?  Suggestions?

I'm not really maintaining it at this point, but RWB might help.  it's
available as a gem.  I've depracated it in favor of RFuzz, but RFuzz
still hasn't caught all the features.
Fc784eadb3b54531fdc3d2053db6f83f?d=identicon&s=25 Mat Schaffer (Guest)
on 2006-12-22 01:02
(Received via mailing list)
On 12/21/06, pat eyler <pat.eyler@gmail.com> wrote:
> I'm not really maintaining it at this point, but RWB might help.  it's
> available as a gem.  I've depracated it in favor of RFuzz, but RFuzz
> still hasn't caught all the features.

Both RFuzz and RWB look like really good starts toward what I'm after.
 Tough to say if I'll decide to really code something at this point.
But I'll certainly start browsing the source of both projects to get
some ideas.

I'd love to have something that made it easy to take a script (like in
rfuzz or rwb), then just drop it straight into a harness that was
responsible for multi-threading, distribution across slave nodes,
aggregating data and generating reports.

But given the state of things, it'd be a big problem to tackle.
Possibly too big for me at this point.

Thanks for the feedback.  This list never disappoints!
-Mat
Bb6ecee0238ef2461bef3416722b35c5?d=identicon&s=25 pat eyler (Guest)
on 2006-12-22 01:48
(Received via mailing list)
On 12/21/06, Mat Schaffer <schapht@gmail.com> wrote:
> On 12/21/06, pat eyler <pat.eyler@gmail.com> wrote:
> > I'm not really maintaining it at this point, but RWB might help.  it's
> > available as a gem.  I've depracated it in favor of RFuzz, but RFuzz
> > still hasn't caught all the features.
>
> Both RFuzz and RWB look like really good starts toward what I'm after.
>  Tough to say if I'll decide to really code something at this point.
> But I'll certainly start browsing the source of both projects to get
> some ideas.

If you're planning on writing something, either coordinate with Zed
to work on RFuzz, or take over RWB and go for it.


>
> I'd love to have something that made it easy to take a script (like in
> rfuzz or rwb), then just drop it straight into a harness that was
> responsible for multi-threading, distribution across slave nodes,
> aggregating data and generating reports.
>

That's sort of where I was headed with RWB.  It's already multithreaded,
but it really should use dRb to coordinate across multiple nodes.  I
don't think there's too much code needed to do that.

> But given the state of things, it'd be a big problem to tackle.
> Possibly too big for me at this point.
>

If you decide to go with RWB, I'd be happy to answer any questions.
It might be a plateful, but you can take small bites and it won't be
that hard.  (Same is true of RFuzz, except that I can't answer the
questions.)
9d1db4326350cea545c52f11a2bfc3b2?d=identicon&s=25 Nathan Smith (Guest)
on 2006-12-22 04:58
(Received via mailing list)
On 12/21/06, khaines@enigo.com <khaines@enigo.com> wrote:

> 3) Go to the fund information page.
> Then, if I can run that on two or three machines at the same time, and can
> over a longer period of time.
>
> I'd like to be able to easily simulate that usage pattern in a development
> environment.
>
>
> Kirk Haines
>
>
>
Watir (http://wtr.rubyforge.org/) and FireWatir (which uses jSSh) look
promising. Those along with the tools mentioned earlier might be a
start.

Nate
7223c62b7310e164eb79c740188abbda?d=identicon&s=25 Xavier Noria (Guest)
on 2006-12-22 08:08
(Received via mailing list)
On Dec 21, 2006, at 11:51 PM, khaines@enigo.com wrote:

> Go to a URL.  Follow a link on the page returned.  Maybe pick
> randomly from a few choices of links to follow.  Repeat a few times
> to simulate people clicking around a site.  Go to a page with a
> form, and fill in form fields, etc...
>
> Basically, a DSL to write scripts for driving interactions to a
> site or application that would somewhat simulate what real traffic
> is expected to look like.

Do you know Watir?

-- fxn
1b5341b64f7ce0244366eae17f06c801?d=identicon&s=25 unknown (Guest)
on 2006-12-22 17:03
(Received via mailing list)
On Fri, 22 Dec 2006, Nathan Smith wrote:

> Watir (http://wtr.rubyforge.org/) and FireWatir (which uses jSSh) look
> promising. Those along with the tools mentioned earlier might be a start.

Those actually drive a real browser session, though.  That's not really
what I need.


Kirk Haines
83ca41657a99b65d99889abe712ba5e2?d=identicon&s=25 Jason Roelofs (Guest)
on 2006-12-22 17:19
(Received via mailing list)
On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:
>
> Kirk Haines
>
>

Um, then what DO you need that Rails unit and integration testing can't
handle? Any other type of testing *requires* a browser instance.

Jason
Bee69cfed999cd13e3bff73d472a39ee?d=identicon&s=25 Hassan Schroeder (Guest)
on 2006-12-22 17:59
(Received via mailing list)
On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:

> Those actually drive a real browser session, though.  That's not really
> what I need.

For a non-Ruby example of a scriptable test system: WebTest:
<http://webtest.canoo.com/>

There's also a Firefox plugin to capture actions to use as the basis
for a test script: <http://webtestrecorder.canoo.com/>

Once your script is done, you run it from the command line, or from
your build environment.

FWIW!
Fc784eadb3b54531fdc3d2053db6f83f?d=identicon&s=25 Mat Schaffer (Guest)
on 2006-12-22 18:16
(Received via mailing list)
On 12/22/06, Jason Roelofs <jameskilton@gmail.com> wrote:
> >
>
I can't answer Kirk's question, but personally I'm looking for full
stack stress testing.  Mostly so I can give my superiors some evidence
rather than blinding estimating how many concurrent users we can
handle.  Not exactly mission-critical, but warm fuzzies are always
good.
-Mat
1b5341b64f7ce0244366eae17f06c801?d=identicon&s=25 unknown (Guest)
on 2006-12-22 19:44
(Received via mailing list)
On Sat, 23 Dec 2006, Mat Schaffer wrote:

> On 12/22/06, Jason Roelofs <jameskilton@gmail.com> wrote:
>> On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:
>> >
>> > Those actually drive a real browser session, though.  That's not really
>> > what I need.
>>
>> Um, then what DO you need that Rails unit and integration testing can't
>> handle? Any other type of testing *requires* a browser instance.

See below.  Rails unit testing is irrelevant.  I don't use Rails.  It's
inadequate for my needs.

> I can't answer Kirk's question, but personally I'm looking for full
> stack stress testing.  Mostly so I can give my superiors some evidence
> rather than blinding estimating how many concurrent users we can
> handle.  Not exactly mission-critical, but warm fuzzies are always
> good.

What I would like, and what Mat wants, I think, have a great deal of
overlap.

To repeat some of what I said in an earlier post, I'd like to.

1) Hit the homepage of a site.
2) Go to a random URL on the site.
    Repeat 2-5 times.
3) Go to the fund information page.
4  Click through to the distributions display.
5)  Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information,
etc....

Or, even more ambitiously:

1) Hit the homepage of a site.
2) Follow the link to the advisor login page.
3) Login (an action that requires sending a form response, and receiving
a
cookie in response that will be sent on further requests).
4) View fund information for some random funds.
5) Go into advisor tools and feed some numbers into a calculator.
6) Goto the logout link.


If I can create scripts that define actions like those outlined above,
which would can simulate expected real world usage over a period of
days,
or can simulate hordes of users in a more realistic manner than the
ab/httperf mechanism of hammering a single URL, it would be very
valuable.

I don't need to drive an actual IE or Firefox session, and for this sort
of thing that's impractical.


Thanks,

Kirk Haines
3bb23e7770680ea44a2d79e6d10daaed?d=identicon&s=25 M. Edward (Ed) Borasky (Guest)
on 2006-12-22 20:04
(Received via mailing list)
khaines@enigo.com wrote:
> 5)  Randomly select 1-4 fund information links.
> 4) View fund information for some random funds.
> I don't need to drive an actual IE or Firefox session, and for this
> sort of thing that's impractical.
What you are describing is a load testing tool. I work with one such
tool, Segue (now Borland) SilkPerformer. Another well-known one is
Mercury LoadRunner. As far as I know, there are no open source tools
that have the flexibility in script *capture* these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.
There are also tools for load test monitoring, server monitoring and
performance/bottleneck analysis. These things are expensive, though.

The closest thing I've found in the open source world is jMeter. But as
far as I know, you have to code your own scripts, your own benchmarks
and your own performance analysis tools with jMeter. It's a tradeoff of
money versus effort. Quite frankly, I'd recommend spending the money.
Email me off-line if you want the details.

Now if someone wants to *write* a full-featured web application load
testing tool -- and I mean *full-featured* -- in Ruby and open source
it, allow me to encourage them. I sure as heck don't have the time to do
it. :)

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.
Bee69cfed999cd13e3bff73d472a39ee?d=identicon&s=25 Hassan Schroeder (Guest)
on 2006-12-22 20:14
(Received via mailing list)
On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:

> ....                As far as I know, there are no open source tools
> that have the flexibility in script *capture* these tools have.
> Essentially, you fire up a script recorder, it fires up a browser, you
> exercise your app, you close the recorder and it generates a script.

Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? :-)  Yes, it's Java, but if you're considering
JMeter...

Caveat: I haven't used either of the commercial tools you mentioned,
so grain of salt, YMMV, etc.
3bb23e7770680ea44a2d79e6d10daaed?d=identicon&s=25 M. Edward (Ed) Borasky (Guest)
on 2006-12-22 20:38
(Received via mailing list)
Hassan Schroeder wrote:
> Caveat: I haven't used either of the commercial tools you mentioned,
> so grain of salt, YMMV, etc.
>
Yeah ... I missed it when I first posted but I have just looked at it.
Unfortunately, the recorder apparently only works with Firefox, which is
a deal-killer. However much you may hate IE and Microsoft ...

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.
83ca41657a99b65d99889abe712ba5e2?d=identicon&s=25 Jason Roelofs (Guest)
on 2006-12-22 20:50
(Received via mailing list)
On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:
> > WebTestRecorder? :-)  Yes, it's Java, but if you're considering
> M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
> http://borasky-research.blogspot.com/
>
> If God had meant for carrots to be eaten cooked, He would have given
> rabbits fire.
>
>
>
Could you explain why this is a "deal killer"? I would hope that
per-browser
functionality testing is a seperate suite from load testing, unless
you're
sending different websites depending on the browser type.

Jason
3bb23e7770680ea44a2d79e6d10daaed?d=identicon&s=25 M. Edward (Ed) Borasky (Guest)
on 2006-12-22 21:24
(Received via mailing list)
Jason Roelofs wrote:
>> >
>>
> per-browser
> functionality testing is a seperate suite from load testing, unless
> you're
> sending different websites depending on the browser type.
>
> Jason
>
If the application only works with Internet Explorer, I can't record it
with Firefox.

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.
1b5341b64f7ce0244366eae17f06c801?d=identicon&s=25 unknown (Guest)
on 2006-12-22 21:26
(Received via mailing list)
On Sat, 23 Dec 2006, Hassan Schroeder wrote:

> On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:
>
>> ....                As far as I know, there are no open source tools
>> that have the flexibility in script *capture* these tools have.
>> Essentially, you fire up a script recorder, it fires up a browser, you
>> exercise your app, you close the recorder and it generates a script.
>
> Perhaps you missed my prior post about WebTest and associated
> WebTestRecorder? :-)  Yes, it's Java, but if you're considering JMeter...

WebTest, from my cursory look, seems to be more oriented at validating
the
correctness of a web site/application than stress testing, though.  Did
I
miss something?


Kirk Haines
Bee69cfed999cd13e3bff73d472a39ee?d=identicon&s=25 Hassan Schroeder (Guest)
on 2006-12-22 21:34
(Received via mailing list)
On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:

> WebTest, from my cursory look, seems to be more oriented at validating the
> correctness of a web site/application than stress testing, though.  Did I
> miss something?

If you want more complex interactions than JMeter provides, there's
no reason you can't create WebTest scripts and set off multiple copies
running simultaneously. But no, it's not intrinsically designed for
stress
testing.
3bb23e7770680ea44a2d79e6d10daaed?d=identicon&s=25 M. Edward (Ed) Borasky (Guest)
on 2006-12-22 22:28
(Received via mailing list)
Hassan Schroeder wrote:
> running simultaneously. But no, it's not intrinsically designed for
> stress
> testing.
>
OK ... let's get our definitions straight.

1. QA/Correctness testing: if I enter request A, does the web app return
the correct response? If I buy a gizmo, does the database show one less
gizmo available to the next buyer? Can Joe Visitor log in and read
Francine User's credit card numbers?

2. Stress test: how gracelessly does the web app behave under an
attempted DoS attack?

3. Load and scalability testing: Assuming it returns the correct
responses, what kind of realistic user load can the web app and its
servers sustain? Where are the bottlenecks? Are they software or
hardware? If software, how can I fix them? How can I inform my designers
how to do things better the next time? If it's hardware, what kind of
hardware? Network? Disk? Server?

Number 3 presupposes you've done Number 1! In fact, the load test
scripts must, to be valid, check the data coming back from the
application to make sure the response is correct. In any event,
SilkPerformer and LoadRunner are specifically designed and evolved to do
3. I didn't dig into Web Test to see if it could be made to do 3 or how
much effort would be involved, since it doesn't record IE. Considering
how long SilkPerformer and LoadRunner have been around, I rather doubt
that Web Test has the kind of "industrial strength" load and scalability
testing capabilities.

I realize I sound like a salesman for a couple of rather expensive tools
-- if that offends the denizens of this open source list, well, I'm
sorry. As I said before, if somebody wants to go out and build an open
source tool in Ruby that does industrial strength load and scalability
testing, I'll be very encouraging, at least with "moral support". But
it's a big job -- it's not as simple as it looks. It's certainly more
than I'd be willing to take on as a one-man open source hacker.


--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.
1b5341b64f7ce0244366eae17f06c801?d=identicon&s=25 unknown (Guest)
on 2006-12-22 23:57
(Received via mailing list)
On Sat, 23 Dec 2006, M. Edward (Ed) Borasky wrote:

> I realize I sound like a salesman for a couple of rather expensive tools --
> if that offends the denizens of this open source list, well, I'm sorry. As I
> said before, if somebody wants to go out and build an open source tool in
> Ruby that does industrial strength load and scalability testing, I'll be very
> encouraging, at least with "moral support". But it's a big job -- it's not as
> simple as it looks. It's certainly more than I'd be willing to take on as a
> one-man open source hacker.

I think there is a LARGE amount of room in between ab and an open
source,
feature complete competitor for SilkPerformer for something that fills a
niche and is useful, yet manageable as an open source project.


Kirk Haines
3bb23e7770680ea44a2d79e6d10daaed?d=identicon&s=25 M. Edward (Ed) Borasky (Guest)
on 2006-12-23 00:11
(Received via mailing list)
khaines@enigo.com wrote:
>
> I think there is a LARGE amount of room in between ab and an open
> source, feature complete competitor for SilkPerformer for something
> that fills a niche and is useful, yet manageable as an open source
> project.
Yes ... there is indeed. And Ruby seems to me to be the ideal language
to create such tools with. There actually wouldn't be a need at all for
load test tools if the application designers were enabled and encouraged
to build load testing into the application when it was being designed,
rather than as an afterthought a few months before deployment, or even
worse, after deployment.

So ... we have test-driven development and we have behavior-driven
development and we have "software performance engineering". Why not
"benchmark-driven development" too? It should be easy, for example, to
write something in RSpec like "this web page should display the shopping
cart withing five seconds".



--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.
This topic is locked and can not be replied to.