Rails 2.2.2 upgrade (threadsafe and connection pooling)


#1

Hi all. I’ve been upgrading one of my apps to Rails 2.2.2 and started
playing with the threadsafety option. I had to modify a fair bit of how
certain ruby files load since Rails’ threadsafety option means that you
have
to load your non-standard (i.e., presenters and other ruby
classes/modules
that aren’t in known paths) explicitly. Anyway, I’ve gotten most of it
working, but have started noticing that memcache is an issue when it
comes
to threadsafety. For some reason, I’m getting IO errors with memcache.
I’m
not sure if it has anything to do with my use of memcache-client or
cache_fu
yet. Have any of you hit upon this issue?

BTW, if I turn caching off, I’m getting around 40reqs/sec (ab -n 1000 -c
100) on a single instance on a somewhat sql heavy page. Not bad. Can’t
wait to see the results after I get caching worked out.

Rich


#2

On Wed, Dec 3, 2008 at 5:43 PM, Rich M. removed_email_address@domain.invalid
wrote:

Hi all. I’ve been upgrading one of my apps to Rails 2.2.2 and started
playing with the threadsafety option. I had to modify a fair bit of how
certain ruby files load since Rails’ threadsafety option means that you have
to load your non-standard (i.e., presenters and other ruby classes/modules
that aren’t in known paths) explicitly. Anyway, I’ve gotten most of it
working, but have started noticing that memcache is an issue when it comes
to threadsafety. For some reason, I’m getting IO errors with memcache. I’m
not sure if it has anything to do with my use of memcache-client or cache_fu
yet. Have any of you hit upon this issue?

No, but it smacks of multiple threads using the same socket. Since
Rails.cache is a single cache wrapper, I’m guessing (without looking)
that the memcache implementation creates and reuses a single socket?
If so, sounds like something we should fix and send a patch back to
rails core.

/Nick


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#3

Is this specific to Jruby or Rails in general? With the widespread
usage of
memcache i would have to imagine this has come up before?
That being said I am a pretty avid user of memcache about to migrate so
hope
this is not an issue…Does this only happen under certain scenarios or
is
Rails.cache just deemed “unsafe” right now ?

Adam


#4

AD wrote:

Is this specific to Jruby or Rails in general? With the widespread usage
of memcache i would have to imagine this has come up before?

If I had to guess I’d say it’s a Rails thing. But that’s a guess. Can
you see if you’re able to reproduce the problem using MRI 1.9 (which
uses native threads)? I suppose even MRI 1.8 would be a valid test. If
you can repro it with one of those, it’s not JRuby. What would be
really cool is if you could produce some simple test case that others
can use to reproduce the problem.

On Wed, Dec 3, 2008 at 11:17 PM, Charles Oliver N.
removed_email_address@domain.invalid wrote:

Never before could requests actually be running concurrently, and I imagine
merb hasn’t had enough use yet for people to see concurrency issues with
memcache?

memcache has been around for a good long while, though, and is
commonly used outside of the Rails/Merb worlds, including in Java
webapps which have been multi-threaded for a long time. I don’t see
this being a memcache problem, but more likely something in the
Rails->memcache interface. Does Merb use that, or does it have its
own?


Bill K.

http://bkocik.net


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#5

AD wrote:

Is this specific to Jruby or Rails in general? With the widespread
usage of memcache i would have to imagine this has come up before?

Never before could requests actually be running concurrently, and I
imagine merb hasn’t had enough use yet for people to see concurrency
issues with memcache?

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#6

I think it has something to do with the memcache-client. That’s the one
common thing that shows up in all of my stack traces. I’ll post my
stack
trace soon… working on another project right now. After I changed my
cache_store to memory_store, everything worked fine. Anyway, I think
Nick’s
hunch is correct, but I need to verify.

Rich


#7

Bill K. wrote:

Rails->memcache interface. Does Merb use that, or does it have its
own?

I concur.

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#8

We have caching turned on and are handling about 240 requests per second
on
our app.

However we are getting a fair bit of 500 errors in our logs that might
be
related to caching. I am trying to figure that out now.

Jay

From: Rich M. [mailto:removed_email_address@domain.invalid]
Sent: Wednesday, December 03, 2008 6:44 PM
To: removed_email_address@domain.invalid
Subject: [jruby-user] Rails 2.2.2 upgrade (threadsafe and connection
pooling)

Hi all. I’ve been upgrading one of my apps to Rails 2.2.2 and started
playing with the threadsafety option. I had to modify a fair bit of how
certain ruby files load since Rails’ threadsafety option means that you
have
to load your non-standard (i.e., presenters and other ruby
classes/modules
that aren’t in known paths) explicitly. Anyway, I’ve gotten most of it
working, but have started noticing that memcache is an issue when it
comes
to threadsafety. For some reason, I’m getting IO errors with memcache.
I’m
not sure if it has anything to do with my use of memcache-client or
cache_fu
yet. Have any of you hit upon this issue?

BTW, if I turn caching off, I’m getting around 40reqs/sec (ab -n 1000 -c
100) on a single instance on a somewhat sql heavy page. Not bad. Can’t
wait to see the results after I get caching worked out.

Rich

No virus found in this incoming message.
Checked by AVG - http://www.avg.com
Version: 8.0.176 / Virus Database: 270.9.0/1775 - Release Date:
12/3/2008
9:34 AM


#9

I was just picking around inside the Ruby MemCache client and found a
bunch
of methods prefixed with “threadsafe”. Basically, all these so is wrap
the
calls with a Mutex. The default setting for MemCache is NOT to be
multithreaded. In my testing, when I turn this on, I don’t see these
problems with MemCache anymore.

To enable this, make sure the :multithread => true option is set when
initializing a new instance of the MemCache client. For those of us
using
cache_fu, this can be done by adding multithread: true to the YAML file.

Ikai


#10

Turns out that’s not the only place that matters. If you are using the
new
Rails cache:

Rails.cache.read/write

Then this uses the cache store you define in the configs since this uses
the
version of the memcache-client that comes in ActiveSupport 2.2.2, not
the
memcache-client gem.

To force thread safety, you would include this in production.rb (or
whatever
file you need it in):

config.cache_store = :mem_cache_store, MEMCACHED_CONFIG[‘servers’], {
:multithread => true }

That being said, it turns out that we’re still having problems with
MemCache
and JRuby in production blocking threads, even with 1.1.6 - if I
remember
correctly Charlie removed a synchronize block from the implementation of
RubyIO as the fix. Well - all threadsafe MemCache does is wrap the calls
with a Mutex, which probably reintroduces some kind of synchronize. The
relevant part of the thread dump is attached.

Do you guys think it makes sense to wrap the Whalin MemCache library? If
so
I’ll start a Github project sometime this week for it.

Ikai


#11

Ikai L. wrote:

file you need it in):

Do you guys think it makes sense to wrap the Whalin MemCache library? If so
I’ll start a Github project sometime this week for it.

Hmm, might not be a bad idea; I’d wager it handles concurrency better
than the Ruby version.

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#12

That’s a good find! Thank you! Maybe this should go on the wiki
somewhere?

Ikai L. wrote:

Jay
fair bit of how certain ruby files load since Rails' threadsafety
bad.  Can't wait to see the results after I get caching worked out.

Rich
No virus found in this incoming message.
Checked by AVG - http://www.avg.com
Version: 8.0.176 / Virus Database: 270.9.0/1775 - Release Date:
12/3/2008 9:34 AM

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#13

I’m attempting to wrap the Whalin MemCache client with JRuby but I’m
running
into some problems. Here’s my code so far:

require ‘java’

require File.dirname(FILE) + ‘/java_memcached-release_2.0.1.jar’

class JMemCache
def initialize(*args)

include_class "com.danga.MemCache.MemCachedClient"
@client = MemCachedClient.new

end
end

I get:
cannot load Java class com.danga.MemCache.MemCachedClient

I’ve tried variants of this as documented here:
http://wiki.jruby.org/wiki/Calling_Java_from_JRuby

Defining a module and using include package:

module Whalin
include_package ‘com.danga.MemCache’
end

Isn’t working, and unfortunately, because of the way CamelCase works,
when I
use:

Java::ComDangaMemCache::MemCacheClient, this turns it into:

com.danga.mem.cache.MemCacheClient.

Interestingly enough when I fire up jruby -S irb, include_class and
instantiating MemCacheClient works fine.

Any ideas?

Ikai

On 12/5/08 1:59 PM, “Charles Oliver N.” removed_email_address@domain.invalid
wrote:

initializing a new instance of the MemCache client. For those of us
However we are getting a fair bit of 500 errors in our logs that


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#14

Learning experience, after going into “trial and error” mode, this
worked:

require File.dirname(FILE) + ‘/java_memcached-release_2.0.1.jar’

class JMemCache
def initialize(*args)
@client = Java::MemCachedClient.new

end
end

On 12/18/08 2:24 PM, “Ikai L.” removed_email_address@domain.invalid wrote:

include_class "com.danga.MemCache.MemCachedClient"

On 12/5/08 1:59 PM, “Charles Oliver N.” removed_email_address@domain.invalid wrote:

To enable this, make sure the :multithread => true option is set when

connection pooling)
memcache-client or cache_fu yet.  Have any of you hit upon this issue?

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email