Hoping for guidance here... We are running a Sinatra app under the the Glassfish gem. I have remote processes that send it requests every 10 seconds - so in a given minute I see 6 requests per remote process. It takes < .5 second to handle these requests. As the remote processes go up, the app starts crawling. I find that with merely 5 remote processes connecting, the log shows them being processed every 23 seconds, not 10! If I up the jruby-runtime-pools to a min/max of 5 or so, everything works fine again with the requests being processed at their incoming rate of 10 seconds. I don't understand why these very quick requests are processed so slowly. I would hope we could handle hundreds of such devices (without resorting to giant jruby-runtime-pool values). How might I debug this? Jconsole shows nothing close to max capacity (or rising toward it). It appears that everything from rack down is snappy, but the requests themselves appear 'queued' in some way. Maybe I should try the glassfish server instead of the gem?
Ken at Inbox256 (Guest)
on 2010-03-09 04:12
Jay McGaffigan (Guest)
on 2010-03-09 15:20
If it looks like the ruby stuff is responding quickly... you might try playing with YourKit (a profiler) to see if you can discern anything? I've used YourKit very successfully to uncover memory leaks. But it also provides profiling capabilities. If by upping runtimes you are getting better results... it could be things are not being processed in a multithreaded manner... but I would think if it takes .5 seconds to process a request you'd be getting better responses with 1 runtime. Sorry if this doesn't help much... but without seeing any code examples it's the best I Can come up with. On Mon, Mar 8, 2010 at 10:12 PM, Ken at Inbox256 <inbox256@gmail.com> wrote: > > the requests themselves appear 'queued' in some way. Maybe I should try the > glassfish server instead of the gem? > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
Brandon Hauff (Guest)
on 2010-03-09 15:28
Can you load your app w/ mongrel instead of glassfish to see if that makes a difference. We could nail down if it is glassfish or not. > also provides profiling capabilities. > On Mon, Mar 8, 2010 at 10:12 PM, Ken at Inbox256 <inbox256@gmail.com> > > > > seconds. > snappy, but > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
On Tue, Mar 9, 2010 at 8:11 AM, Jay McGaffigan <hooligan495@gmail.com> wrote: > If by upping runtimes you are getting better results... it could be > things are not being processed in a multithreaded manner... but I > would think if it takes .5 seconds to process a request you'd be > getting better responses with 1 runtime. Yeah, this was my thought too. It sounds like there's some contention in the GF gem or in the app that's preventing it from handling requests more quickly. A Sinatra app should probably run fine "threadsafe" with a single runtime handling all requests, so it's peculiar that it improves with more runtimes. And we do know of other users running GF gem + Sinatra for *really* high load applications, in the neighborhood of thousands of requests per second. So something's bogging this particular case down. Ken: So the problems seem to arise when you have many concurrent clients requesting things, yes? There is another possibility...it could be spending all its time in GC. Perhaps you could run JRuby + GF gem like this and see if the GC times get really bad: jruby -J-verbose:gc -S glassfish ... If the times get really high (especially for the "Full" collections) it's possible your application is generating too much garbage per request. Large amounts of garbage can be handled pretty efficiently by one thread, but when you have many generating that much garbage, the system can start spending all its time GCing. Try both of these separately.. * Try to get GF running "threadsafe" for your Sinatra app (perhaps min/max runtimes = 1?) and see how it looks. * Get some GC results and examine your app usage to see if it's creating too much garbage (like loading a giant object graph and throwing it away every request) - Charlie --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
Vivek Pandey (Guest)
on 2010-03-11 17:49
Grizzly starts with max 5 threads in the thread pool serving requests in the queue. See glassfish.yml (gfrake config) and you can tune the grizzly threadpool as well as request-timeout settings. to fix such long-held http requests, glassfish gem should provide continuation (comet/async support) so that the http worked thread is released immediately. -vivek.
Ken at Inbox256 (Guest)
on 2010-03-11 18:35
Guys, thanks for the updates. I believe the problem is related to 64 bit windows. I cannot recreate on 32 bit systems. So it is either win2008 or 64 bit, and I'm guessing the latter. I used the default jruby install. Perhaps I need to manually grab a 64 bit jre?
So it runs fast on 32-bit? That's rather peculiar...the only things that would really be different under 64-bit would be the native libraries. Perhaps there's some odd bottleneck in one of them? Have you been able to narrow it down to a simpler case or to a specific API or call in your application? Perhaps you can get request-processing and most of Rails out of the way? On Thu, Mar 11, 2010 at 11:35 AM, Ken at Inbox256 <inbox256@gmail.com> wrote: >> Grizzly starts with max 5 threads in the thread pool serving requests in >>> Hoping for guidance here... >>> If I up the jruby-runtime-pools to a min/max of 5 or so, everything works >>> glassfish server instead of the gem? >>> >> > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
Sorry, you're not running Rails...I meant get all request-processing and <web framework> (Sinatra) out of the way so we can start to narrow down what's actually slow. On Mon, Mar 15, 2010 at 10:55 AM, Charles Oliver Nutter <headius@headius.com> wrote: >> windows. I cannot recreate on 32 bit systems. So it is either win2008 or >>> threadpool as well as request-timeout settings. >>>> We are running a Sinatra app under the the Glassfish gem. I have remote >>>> seconds. >>> >> >> > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
Enable email notification
Enable multi-page view