Forum: JRuby JRuby hanging just prior to exit on successful operations

Hey guys,

I'm using JRuby in local development mode (because I intend to deploy 
with it - yay threads!) and having one very odd issue that I wanted to 
see if anyone else has run into before.

Basically I'm working on a Rails 3.2.11 app with the latest JRuby out 
(1.7.2 as of now). If I issue something like "bundle exec rake spec" or 
"rake db:migrate", sometimes the operation finishes (usually, in fact), 
and sometimes it doesn't, but in about 90% of cases it gets nearly done, 
and then right before actually exiting it just plain hangs there doing 
nothing.

For example, I can throw down "rake db:migrate", see the migration 
happen, see the statements in development.log, access my PostgreSQL 
console, \d <tablename> and see that it certainly did work, but the 
actual process is just hung indefinitely. I've been CTRL+C'ing out, but 
this is definitely a bit odd.

I do also have drip installed to try to speed up load times, and I have 
a suspicion that's involved somehow, but before removing it (since it 
really does help), I wanted to see if anyone else has run into this 
issue or has any ideas on what I can do to keep drip and just get rid of 
this odd behavior.

Thanks for any help you can offer!

Hi,

When you get the hang I would use jstack to dump threads for the java 
process. This should give you an idea of what's going on when it's hung. 
I've seen issues with non-thread safe code.


http://docs.oracle.com/javase/1.5.0/docs/tooldocs/...

Sent from my iPhone

What Java are you running? "jruby -v" will show you both the JRuby
version and underlying Java information.

Not too long ago there was a crash-at-exit issue running JRuby on IBM's
Java. A recent JRuby change (1.7.1 I think) avoids that specific
problem. There might be other quirks on unusual Java implementations.

Getting a thread dump is a good idea. I find "kill -quit <pid>" easier
to use than jstack. The kill will give you a thread dump immediately,
and let the process continue to run. It is great for hung or high CPU
servers. You can get several thread dumps in sequence and see what
changes between them. ("kill" is only available on Unix-like systems,
including OSX and Linux; not available on Windows.)

I don't expect a "rake db:migrate" to do any problematic threading.
Still, a thread dump will show where it's stuck.

- Bruce

Thanks for the help here guys. I've tried both the kill -quit <pid> 
option and jstack, and both are very odd.

I first tried the kill -quit <pid> to get that stack trace, yet the 
process still runs and I'm not seeing any stack trace on stdout (is 
there a log of this anywhere you guys can think of? OS X with jruby 
installed via rvm).

So next I tried jstack <pid> which couldn't attach, so I tried jstack -F 
<pid> which failed out:
└─[$] jstack -F 72656
Attaching to process ID 72656, please wait...
attach: task_for_pid(72656) failed (5)
Error attaching to process: Error attaching to process, or no such 
process


No such process? I ran ps and sure enough it's there, it's running, and 
it isn't zombied. In fact it's listed as sleeping (S+). Since I'm on OS 
X and don't have strace, I used dtruss to attach to the pid and see what 
it's doing. After letting it run for around ~5-10 minutes, this is all 
dtruss is showing me:

└─[$] sudo dtruss -p 72656
Password:
SYSCALL(args) = return


Seriously, one line of output.

I think my next phase is going to be ripping out drip and seeing if that 
makes things a bit smoother. I'll let ya'll know if that helps.

Thanks again for the suggestions!

Oh and to answer an earlier question:
└─[$] jruby -v
jruby 1.7.2 (1.9.3p327) 2013-01-04 302c706 on Java HotSpot(TM) 64-Bit 
Server VM 1.6.0_37-b06-434-11M3909 [darwin-x86_64]


--
J. Austin Hughey
jaustinhughey@me.com

Hey guys,

Thanks a lot for all your help again. I initially went with the 
sledgehammer approach earlier today (been working with this prettymuch 
all day) and nuked my entire RVM install, drip, everything, then brought 
it all back from scratch. Once that was done, I had a working "normal", 
non-drip jruby going. I replied to a tweet from @headius this morning 
(thanks man) saying as much. He stated that if it was drip, he really 
wanted to know, so I put drip back in place following directions here:

https://github.com/flatland/drip/wiki/JRuby

Under the "Using Drip with an RVM installed JRuby" section. I just 
created a new file, copy/pasta'd that code in, chmod +x'd it, and went 
off about my business. This is after 'brew install drip' and reloading 
my shell, by the way.

Then I got the same behavior as before. Some operations would hang just 
prior to their normal point of exit. For example, I couldn't even get 
ruby -v (which RVM aliases to jruby) to exit. Same behavior with jstack 
and dtruss.

So I removed drip again (deleted that rvm hook, kill -9'd all drip/java 
processes, brew uninstall drip, reloaded terminal) and now things are 
working normally.

I fully admit that this could be an Error 18 (error is 18 inches from 
the screen) but so far the behavior here is quite erratic and I do 
confess to being rather confused.

At the moment I'm just going to run dripless (is that even a word?). If 
anyone has any thoughts, I'd love to hear it, but so far this *looks* to 
be something funky with drip.

Thanks again all.

--
J. Austin Hughey
jaustinhughey@me.com