Forum: JRuby JRuby mongrel process hanging

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
87d8380f6f6dab48174fbe667a70d75c?d=identicon&s=25 Chris Walters (Guest)
on 2009-03-17 19:59
(Received via mailing list)
I'm experiencing some weird behavior with my JRuby on Rails app. Every
now
and then one of my mongrel processes will peg the CPU, and the request
never
finishes or writes log information. I have only been able to repro it
once
by slamming the box with ab, but that only worked once for the many
times
I've tried it. It seemed to start happening after I introduced a small
bit
of threading code into my app (an alternative possibility is that
whatever
the issue is existed before the threading code but our users' behavior
changed once we launched the new code). My threading code is pretty
simple,
but as a precaution, I've migrated it all to use java.util.concurrency
primitives. jconsole is not able to detect any deadlocks in the hung
process. I thought it might have something to do with the
memcache-client
concurrency issues that some forums have discussed, so I've moved my
client
to use jruby-memcache-client. I've also upgraded my app to use JRuby
1.2.0,
but it's still happening. Does anyone have any tool suggestions for how
I
can diagnose this issue?

Thanks,
Chris
08264fc914cfb45adfea6c2f36f70b21?d=identicon&s=25 Jacob Kessler (Guest)
on 2009-03-17 20:06
(Received via mailing list)
The obvious thing to do would be to grab a thread dump (probably using
kill -QUIT on the hung mongrel) and see where the mongrel process is
when you kill it. That will probably be a good starting point. My
personal guess is a race condition somewhere that allows the mongrel
into an invalid state, from which it never recovers.

Chris Walters wrote:
> primitives. jconsole is not able to detect any deadlocks in the hung
> process. I thought it might have something to do with the
> memcache-client concurrency issues that some forums have discussed, so
> I've moved my client to use jruby-memcache-client. I've also upgraded
> my app to use JRuby 1.2.0, but it's still happening. Does anyone have
> any tool suggestions for how I can diagnose this issue?
>
> Thanks,
> Chris


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
87d8380f6f6dab48174fbe667a70d75c?d=identicon&s=25 Chris Walters (Guest)
on 2009-03-17 20:58
(Received via mailing list)
It looks like most of the threads are stuck on :

   java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.get(HashMap.java:303)
    at
org.codehaus.xfire.aegis.AegisBindingProvider.getParameterType(AegisBindingProvider.java:270)
    at
org.codehaus.xfire.aegis.AegisBindingProvider.initializeMessage(AegisBindingProvider.java:145)
    at
org.codehaus.xfire.service.binding.AbstractBindingProvider.initialize(AbstractBindingProvider.java:54)
    at
org.codehaus.xfire.aegis.AegisBindingProvider.initialize(AegisBindingProvider.java:133)
    at
org.codehaus.xfire.service.binding.ObjectServiceFactory.create(ObjectServiceFactory.java:469)
    at
org.codehaus.xfire.service.binding.ObjectServiceFactory.create(ObjectServiceFactory.java:374)
    at
org.codehaus.xfire.service.binding.ObjectServiceFactory.create(ObjectServiceFactory.java:355)
    ...

which suggests to me that the multiple XFire 1.2.6 clients I'm using are
not
thread-safe?

Has anyone run into this issue before?

The other thing that's weird, is that sometimes it pegs the CPU at 100%
and
sometimes at 200% (we're running Ubuntu 8.04 on an EC2 c1.medium
instance).
08264fc914cfb45adfea6c2f36f70b21?d=identicon&s=25 Jacob Kessler (Guest)
on 2009-03-17 21:04
(Received via mailing list)
Yep, that looks like it. If you have multiple threads modifying a
hashmap, they can cause a state like that. I suspect that the 100%/200%
is either one or two threads hanging in there. However, synchronizing
around the Xfire client should make thing work (though that might also
defeat the point of the threading).

Chris Walters wrote:
>     at
> are not thread-safe?
>     The obvious thing to do would be to grab a thread dump (probably
>         the CPU, and the request never finishes or writes log
>         have something to do with the memcache-client concurrency
>     ---------------------------------------------------------------------
>     To unsubscribe from this list, please visit:
>
>       http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2009-03-17 22:43
(Received via mailing list)
If there's a top-level object you can create per thread, you could put
them in a pool and pull one out for each request you're handling
concurrently. You'd have to maintain multiple xfire clients in-memory,
but that would likely keep them from stepping on each other and avoid
synchronizing all consumers against one instance.

Jacob Kessler wrote:
>>     at java.util.HashMap.get(HashMap.java:303)
>> org.codehaus.xfire.aegis.AegisBindingProvider.initialize(AegisBindingProvider.java:133)
>>     ...
>> On Tue, Mar 17, 2009 at 12:30 PM, Jacob Kessler <Jacob.Kessler@sun.com
>>
>>         but as a precaution, I've migrated it all to use
>>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
87d8380f6f6dab48174fbe667a70d75c?d=identicon&s=25 Chris Walters (Guest)
on 2009-03-17 22:50
(Received via mailing list)
Yeah, I'm implementing PoolableObjectFactory and synchronizing the
makeObject method. This serializes the clients' creations but not their
use,
so the bottleneck is minimal.

Thanks for your help, all.

On Tue, Mar 17, 2009 at 2:42 PM, Charles Oliver Nutter <
This topic is locked and can not be replied to.