Forum: JRuby Using Thread.new and performing ActiveRecord operations causes occasional "Connection.close" excepti

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Ikai L. (Guest)
on 2009-03-16 20:03
(Received via mailing list)
Has anyone seen this? Here¹s an example of what I¹m talking about:

def some_method
   Thread.new do
      1000.times { Model.create :field => value }
  end
end

The offending code has been rolled back, but while it was running it was
causing exceptions to be thrown intermittently stating that
³Connection.close() has already been called². Also ­ I¹ve seen other
exceptions stating that a database connection could not be acquired.
Increasing pool size gave us some breathing room there but doesn¹t seem
like
a fix.

Does anyone have an explanation of how thread pooling actually works in
Rails 2.2? I feel like I¹m cargo culting here and it¹s resulting in
problems
in production.

Ikai
Ikai L. (Guest)
on 2009-03-16 20:10
(Received via mailing list)
Attachment: connection_close_stack_trace.txt (0 Bytes)
Attached is an example stack trace from our test environment where we
have
reproduced the error.
Peter K Chan (Guest)
on 2009-03-16 20:29
(Received via mailing list)
I am not able to reproduce the error in my app that embeds Rails 2.2
with
JRuby 1.2.

However, looking at your stack trace, you seems to be spawning thread
when
handling a http request. Are you sure that that is what you intended?
From
what I understand, it is not advisable to spawn threads from a request
handler, because the connection and all the states are managed by the
container, so when your request handler returns, the container may
either
reuse the connection, or close it.

To see if this is the problem, wait on the new thread that you created
before
returning from your some_method. If the problem goes away, then the
issue is
most likely because you are using a connection that the container
thought you
have relinquished.

Peter

From: Ikai L. [mailto:removed_email_address@domain.invalid]
Sent: Monday, March 16, 2009 1:09 PM
To: removed_email_address@domain.invalid
Subject: Re: [jruby-user] Using Thread.new and performing ActiveRecord
operations causes occasional "Connection.close" exceptions

Attached is an example stack trace from our test environment where we
have
reproduced the error.


On 3/16/09 11:02 AM, "LinkedIn Corporation" 
<removed_email_address@domain.invalid> wrote:
Has anyone seen this? Here’s an example of what I’m talking about:

def some_method
   Thread.new do
      1000.times { Model.create :field => value }
  end
end

The offending code has been rolled back, but while it was running it was
causing exceptions to be thrown intermittently stating that
“Connection.close() has already been called”. Also – I’ve seen other
exceptions stating that a database connection could not be acquired.
Increasing pool size gave us some breathing room there but doesn’t seem
like a
fix.

Does anyone have an explanation of how thread pooling actually works in
Rails
2.2? I feel like I’m cargo culting here and it’s resulting in problems
in
production.

Ikai


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ikai L. (Guest)
on 2009-03-16 20:35
(Received via mailing list)
Peter, that makes perfect sense. That was exactly what I was doing.
Basically I was looking for the cheapest way to do long operations
asynchronously.

Looks like the best way is to have some kind of queue. Based on your
explanation below, it sounds like if I spin off worker threads on
application load and work off a queue, this issue will disappear. Is
this
correct?

Ikai


On 3/16/09 11:29 AM, "Peter K Chan" <removed_email_address@domain.invalid> 
wrote:

> To see if this is the problem, wait on the new thread that you created before
> operations causes occasional "Connection.close" exceptions
>       1000.times { Model.create :field => value }
> Does anyone have an explanation of how thread pooling actually works in Rails
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Peter K Chan (Guest)
on 2009-03-16 20:42
(Received via mailing list)
Yes. I don't know much about it, but what you are probably looking for
is some
kind of background processing framework, which are available for both
java or
jruby. One example would be http://backgroundrb.rubyforge.org/, although
there
are many others.

Spinning off threads at startup may work, assuming that you have a way
for
those threads to get a connection without the container.

Peter
Ikai L. (Guest)
on 2009-03-16 21:02
(Received via mailing list)
Hey Peter,

The problem here is that most background processing packages are going
to
require spinning up another process which is downright wasteful since
you'll
basically instantiate Java again. I've been looking at various solutions
like embedding starling server and spinning off threads as clients, but
again, this depends on the thread not causing the connection.close()
issue
again.

Maybe it's time to dig into the Rails code again and figure out what
exactly
is going on.

Ikai


On 3/16/09 11:41 AM, "Peter K Chan" <removed_email_address@domain.invalid> 
wrote:

> -----Original Message-----
> Looks like the best way is to have some kind of queue. Based on your
>> JRuby 1.2.
>> returning from your some_method. If the problem goes away, then the issue is
>> operations causes occasional "Connection.close" exceptions
>>       1000.times { Model.create :field => value }
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2009-03-16 21:21
(Received via mailing list)
Ikai L. wrote:
> is going on.
I've been seeing a couple JMS-based backgrounding options discussed
lately, which might be useful for you. I know backgrounding some
operation is a very fuzzy area for JRuby at the moment, and when "just
spin up a thread" doesn't work, you have to turn to some other kind of
queueing system.

Anyone else tried out the various backgrounding options for JRuby?

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Peter K Chan (Guest)
on 2009-03-16 21:29
(Received via mailing list)
I am not sure if digging into Rails code helps. This is a generic
limitation
of using a connection outside of a request handler; nothing specific to
rails.

Keep in mind that just because your app server is taking a 4 GB heap to
serve
thousands of requests per second doesn't mean that your background
processing
framework will need to do the same. Be sure to profile before dismissing
that
option.

Failing that, you can always just use a cron job to periodically invoke
a
request handler in your web app, and have background processing handled
using
a container-managed connection.

Peter
Ikai L. (Guest)
on 2009-03-16 23:06
(Received via mailing list)
That's funny, I'm already using the cron job mechanism for something
else. I
was looking into quartz or rufus scheduler, but it just sounds like
since
those plugins will have the connection.close() issue again.

Ikai


On 3/16/09 12:29 PM, "Peter K Chan" <removed_email_address@domain.invalid> 
wrote:

> a container-managed connection.
> Hey Peter,
>
> there
>> To: removed_email_address@domain.invalid
>> correct?
>>> handling a http request. Are you sure that that is what you intended? From
>> you
>>> Attached is an example stack trace from our test environment where we have
>>> end
>>> Does anyone have an explanation of how thread pooling actually works in
>>>     http://xircles.codehaus.org/manage_email
>>
> To unsubscribe from this list, please visit:
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Mwadhera M. (Guest)
on 2009-03-18 01:43
(Received via mailing list)
Seeing this MySQLNonTransientConnectionException error a lot as well...

Our solution for backgrounding work in our single-threaded rails 2.2.2
app
has been the JMS support in JRuby Rack, which lets you queue work into
JMS
listeners that will evaluate messages sent to them in the context of
another
member of the runtime pool -- allowing you to amortize the cost of
long-running work over your total runtimes.

It's a beautiful solution but every now and then our app will start
spitting
these exceptions out (only when serving HTTP requests, notably). After
cycling our Glassfish instances the errors go away.

If we weren't using Cache Money I would have switched back to AR 2.1 by
now
as we only started seeing this behavior in 2.2. : /

I wish there was a way to disable connection pooling all together in AR
2.2
and just let the app server deal with managing db connections; It is a
real
thorn for JRuby users, especially those still running single-threaded.

Let us know if you find a good work-around solution for this... i'm
afraid a
separate process might be the only way to circumvent these connection
close
errors.

Mike

On Mon, Mar 16, 2009 at 2:05 PM, Ikai L. <removed_email_address@domain.invalid> 
wrote:

> > I am not sure if digging into Rails code helps. This is a generic
> > option.
> > Sent: Monday, March 16, 2009 2:02 PM
> > like embedding starling server and spinning off threads as clients, but
> >
> > there
> >> Sent: Monday, March 16, 2009 1:33 PM
> >> application load and work off a queue, this issue will disappear. Is
> >>> JRuby 1.2.
> >>>
> >>> Peter
> >>>
> >>> The offending code has been rolled back, but while it was running it
> >> Rails
> >>>     http://xircles.codehaus.org/manage_email
> >>
> > To unsubscribe from this list, please visit:
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>


--
Best,

Mike W.
http://metasaur.us
Ikai L. (Guest)
on 2009-03-18 02:06
(Received via mailing list)
Mike, when you say single-threaded, do you mean deploying a single
instance
of Rails and turning config.threadsafe! on, or do you mean deploying
several
instances of Rails?

Ikai


On 3/17/09 2:53 PM, "Mike W." <removed_email_address@domain.invalid> wrote:

> our Glassfish instances the errors go away.
> errors.
>>
>>> processing
>>> >
>>> > require spinning up another process which is downright wasteful since
>>> > Ikai
>>> >
>>> >
>>> > On 3/16/09 11:41 AM, "Peter K Chan" <removed_email_address@domain.invalid> wrote:
>>> >
>>>> >> Yes. I don't know much about it, but what you are probably looking for
is
>>>> >> those threads to get a connection without the container.
>>>> >> Peter, that makes perfect sense. That was exactly what I was doing.
>>>> >>
>>>>> From
>>> > is
>>>>> >>> Subject: Re: [jruby-user] Using Thread.new and performing ActiveRecord
>>>>> >>> def some_method
>>>>> >>>    Thread.new do
>>>>> >>>       1000.times { Model.create :field => value }
>>>>> >>>   end
>>>>> >>> end
>>>>> >>>
>>>>> >>> The offending code has been rolled back, but while it was running it
was
>>>>> >>> causing exceptions to be thrown intermittently stating that
>>>>> >>> ³Connection.close() has already been called². Also ­ I¹ve seen other
>>>>> >>> exceptions stating that a database connection could not be acquired.
>>>>> >>> Increasing pool size gave us some breathing room there but doesn¹t
>>>>> seem
>>> > like
>>>> >> a
>>>>> >>> fix.
>>>>> >>>
>>>>> >>> Does anyone have an explanation of how thread pooling actually works
in
>>>> >> Rails
>>>>> >>> 2.2? I feel like I¹m cargo culting here and it¹s resulting in problems
in
Mwadhera M. (Guest)
on 2009-03-18 03:44
(Received via mailing list)
The latter i.e. using dynamic runtime pooling inside Glassfish with
several
single-threaded rails environments.

On Tue, Mar 17, 2009 at 5:05 PM, Ikai L. <removed_email_address@domain.invalid> 
wrote:

> Seeing this MySQLNonTransientConnectionException error a lot as well...
>
>
>
> > thousands of requests per second doesn't mean that your background
> > Peter
> > The problem here is that most background processing packages are going to
> > is going on.
> java
> >> Peter
> >> asynchronously.
> >> On 3/16/09 11:29 AM, "Peter K Chan" <removed_email_address@domain.invalid> wrote:
> >>> handler, because the connection and all the states are managed by the
> thought
> >>>
> >>>       1000.times { Model.create :field => value }
> >> a
> >>>
> >>
> >>
> > ---------------------------------------------------------------------
>     http://xircles.codehaus.org/manage_email
>
>
>
>
>


--
Best,

Mike W.
http://metasaur.us
snacktime (Guest)
on 2009-03-18 22:34
(Received via mailing list)
I'm not sure how different it would be with jruby, but creating a thread
pool in environment.rb is something I've used before.

Chris
This topic is locked and can not be replied to.