Using Thread.new and performing ActiveRecord operations causes occasional "Connection.close" excepti

Ruby JRuby
1

Has anyone seen this? Here¹s an example of what I¹m talking about:

def some_method
Thread.new do
1000.times { Model.create :field => value }
end
end

The offending code has been rolled back, but while it was running it was
causing exceptions to be thrown intermittently stating that
³Connection.close() has already been called². Also ­ I¹ve seen other
exceptions stating that a database connection could not be acquired.
Increasing pool size gave us some breathing room there but doesn¹t seem
like
a fix.

Does anyone have an explanation of how thread pooling actually works in
Rails 2.2? I feel like I¹m cargo culting here and it¹s resulting in
problems
in production.

Ikai

2

Attached is an example stack trace from our test environment where we
have
reproduced the error.

3

I am not able to reproduce the error in my app that embeds Rails 2.2
with
JRuby 1.2.

However, looking at your stack trace, you seems to be spawning thread
when
handling a http request. Are you sure that that is what you intended?
From
what I understand, it is not advisable to spawn threads from a request
handler, because the connection and all the states are managed by the
container, so when your request handler returns, the container may
either
reuse the connection, or close it.

To see if this is the problem, wait on the new thread that you created
before
returning from your some_method. If the problem goes away, then the
issue is
most likely because you are using a connection that the container
thought you
have relinquished.

Peter

From: Ikai L. [mailto:[email protected]]
Sent: Monday, March 16, 2009 1:09 PM
To: [email protected]
Subject: Re: [jruby-user] Using Thread.new and performing ActiveRecord
operations causes occasional “Connection.close” exceptions

Attached is an example stack trace from our test environment where we
have
reproduced the error.

On 3/16/09 11:02 AM, “LinkedIn Corporation” [email protected] wrote:
Has anyone seen this? Here’s an example of what I’m talking about:

def some_method
Thread.new do
1000.times { Model.create :field => value }
end
end

The offending code has been rolled back, but while it was running it was
causing exceptions to be thrown intermittently stating that
“Connection.close() has already been called”. Also – I’ve seen other
exceptions stating that a database connection could not be acquired.
Increasing pool size gave us some breathing room there but doesn’t seem
like a
fix.

Does anyone have an explanation of how thread pooling actually works in
Rails
2.2? I feel like I’m cargo culting here and it’s resulting in problems
in
production.

Ikai

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
4

Yes. I don’t know much about it, but what you are probably looking for
is some
kind of background processing framework, which are available for both
java or
jruby. One example would be http://backgroundrb.rubyforge.org/, although
there
are many others.

Spinning off threads at startup may work, assuming that you have a way
for
those threads to get a connection without the container.

Peter

5

Peter, that makes perfect sense. That was exactly what I was doing.
Basically I was looking for the cheapest way to do long operations
asynchronously.

Looks like the best way is to have some kind of queue. Based on your
explanation below, it sounds like if I spin off worker threads on
application load and work off a queue, this issue will disappear. Is
this
correct?

Ikai

On 3/16/09 11:29 AM, “Peter K Chan” [email protected] wrote:

To see if this is the problem, wait on the new thread that you created before
operations causes occasional “Connection.close” exceptions
1000.times { Model.create :field => value }
Does anyone have an explanation of how thread pooling actually works in Rails

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
6

Ikai L. wrote:

is going on.
I’ve been seeing a couple JMS-based backgrounding options discussed
lately, which might be useful for you. I know backgrounding some
operation is a very fuzzy area for JRuby at the moment, and when “just
spin up a thread” doesn’t work, you have to turn to some other kind of
queueing system.

Anyone else tried out the various backgrounding options for JRuby?

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
7

Hey Peter,

The problem here is that most background processing packages are going
to
require spinning up another process which is downright wasteful since
you’ll
basically instantiate Java again. I’ve been looking at various solutions
like embedding starling server and spinning off threads as clients, but
again, this depends on the thread not causing the connection.close()
issue
again.

Maybe it’s time to dig into the Rails code again and figure out what
exactly
is going on.

Ikai

On 3/16/09 11:41 AM, “Peter K Chan” [email protected] wrote:

-----Original Message-----
Looks like the best way is to have some kind of queue. Based on your

JRuby 1.2.
returning from your some_method. If the problem goes away, then the issue is
operations causes occasional “Connection.close” exceptions
1000.times { Model.create :field => value }

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
8

That’s funny, I’m already using the cron job mechanism for something
else. I
was looking into quartz or rufus scheduler, but it just sounds like
since
those plugins will have the connection.close() issue again.

Ikai

On 3/16/09 12:29 PM, “Peter K Chan” [email protected] wrote:

a container-managed connection.
Hey Peter,

there

To: [email protected]
correct?

handling a http request. Are you sure that that is what you intended? From
you
Attached is an example stack trace from our test environment where we have
end
Does anyone have an explanation of how thread pooling actually works in
http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
9

I am not sure if digging into Rails code helps. This is a generic
limitation
of using a connection outside of a request handler; nothing specific to
rails.

Keep in mind that just because your app server is taking a 4 GB heap to
serve
thousands of requests per second doesn’t mean that your background
processing
framework will need to do the same. Be sure to profile before dismissing
that
option.

Failing that, you can always just use a cron job to periodically invoke
a
request handler in your web app, and have background processing handled
using
a container-managed connection.

Peter

10

Seeing this MySQLNonTransientConnectionException error a lot as well…

Our solution for backgrounding work in our single-threaded rails 2.2.2
app
has been the JMS support in JRuby Rack, which lets you queue work into
JMS
listeners that will evaluate messages sent to them in the context of
another
member of the runtime pool – allowing you to amortize the cost of
long-running work over your total runtimes.

It’s a beautiful solution but every now and then our app will start
spitting
these exceptions out (only when serving HTTP requests, notably). After
cycling our Glassfish instances the errors go away.

If we weren’t using Cache Money I would have switched back to AR 2.1 by
now
as we only started seeing this behavior in 2.2. : /

I wish there was a way to disable connection pooling all together in AR
2.2
and just let the app server deal with managing db connections; It is a
real
thorn for JRuby users, especially those still running single-threaded.

Let us know if you find a good work-around solution for this… i’m
afraid a
separate process might be the only way to circumvent these connection
close
errors.

Mike

On Mon, Mar 16, 2009 at 2:05 PM, Ikai L. [email protected] wrote:

I am not sure if digging into Rails code helps. This is a generic
option.
Sent: Monday, March 16, 2009 2:02 PM
like embedding starling server and spinning off threads as clients, but

there

Sent: Monday, March 16, 2009 1:33 PM
application load and work off a queue, this issue will disappear. Is

JRuby 1.2.

Peter

The offending code has been rolled back, but while it was running it
Rails
http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

–
Best,

Mike W.
http://metasaur.us

11

The latter i.e. using dynamic runtime pooling inside Glassfish with
several
single-threaded rails environments.

On Tue, Mar 17, 2009 at 5:05 PM, Ikai L. [email protected] wrote:

Seeing this MySQLNonTransientConnectionException error a lot as well…

thousands of requests per second doesn’t mean that your background
Peter
The problem here is that most background processing packages are going to
is going on.
java

Peter
asynchronously.
On 3/16/09 11:29 AM, “Peter K Chan” [email protected] wrote:

handler, because the connection and all the states are managed by the
thought

  1000.times { Model.create :field => value }

a

http://xircles.codehaus.org/manage_email

–
Best,

Mike W.
http://metasaur.us

12

I’m not sure how different it would be with jruby, but creating a thread
pool in environment.rb is something I’ve used before.

Chris

13

Mike, when you say single-threaded, do you mean deploying a single
instance
of Rails and turning config.threadsafe! on, or do you mean deploying
several
instances of Rails?

Ikai

On 3/17/09 2:53 PM, “Mike W.” [email protected] wrote:

our Glassfish instances the errors go away.
errors.

processing

require spinning up another process which is downright wasteful since
Ikai

On 3/16/09 11:41 AM, “Peter K Chan” [email protected] wrote:

Yes. I don’t know much about it, but what you are probably looking for
is
those threads to get a connection without the container.
Peter, that makes perfect sense. That was exactly what I was doing.

From
is

Subject: Re: [jruby-user] Using Thread.new and performing ActiveRecord
def some_method
  Thread.new do
   1000.times { Model.create :field => value }
 end
end

The offending code has been rolled back, but while it was running it
was
causing exceptions to be thrown intermittently stating that
³Connection.close() has already been called². Also ­ I¹ve seen other
exceptions stating that a database connection could not be acquired.
Increasing pool size gave us some breathing room there but doesn¹t
seem
like
a
fix.

Does anyone have an explanation of how thread pooling actually works
in
Rails
2.2? I feel like I¹m cargo culting here and it¹s resulting in problems
in