Jruby-memcache-client project started on GitHub

Ruby JRuby
1

If you’ve been following the thread about MemCache issues in production,
you
know that this has been bothering me.

Well, hopefully, no more! I’ve started a GitHub project to track my work
here:

Obviously, do not use this build. I’ll let everyone know when the basic
functionality is hammered down and usable.

Ikai

2

It’s at a state where it is functional. I’m still testing it in my
application to see if I can find problems. I’m going to push my
application
to staging and bang on it.

Basically there’s one thing might might be problematic: Marshal.dump and
Marshal.load act weird. Some objects serialize using Marshal with a \222
character, which, after being stored on MemCache, on a subsequent get
comes
back as a weird ASCII character, and Marshal.load blows up. For this
reason
I’m using YAML serialization enough though YAML serialization is an
order of
magnitude slower. YAML deserialization is about 4 times slower.

3

What’s wrong with the standard memcache-client? We use it in production
(since last year) with JRuby/Rails 2.2.2. If it has to do with the
thread
safety issues, we got around it by setting the threadsafety flag to true
in
memcache-client.

Rich

4

Ah… I see. We haven’t seen the issue yet since it’s an internal app
that
hasn’t seen the volume of traffic that yours has. Good to know though.

Rich

5

The threadsafety option only wraps the calls in a Mutex. There’s no
connection pooling, and all the connections use the same socket. In a
high
traffic application, the Ruby MemCache client ends up in a state where
it
blocks a connector thread but doesn’t timeout, and the JRuby Timeout
implementation should not be relied on to recover from IO blocking.
Enabling
Multithread support exacerbates this problem because all connectors
threads
become blocked waiting for the Mutex to be released. We sent the thread
dumps to Sun, and Charlie and team put together a fix in 1.1.6. It seems
to
have helped a lot but we are still seeing issues.

The dump is below. Our application servers completely freeze up and we
end
up needing to restart them. This happens to us about once every two days
per
server. The Ruby MemCache client implementation works pretty well in an
MRI
environment because you can use Timeout as a fallback, and because it
was
never designed for multithreading.

Ikai

Thread “httpSSLWorkerThread-8080-2” thread-id 86
thread-stateBLOCKEDWaiting
on lock: org.jruby.util.io.ChannelStream@1a230ce
Owned by: httpSSLWorkerThread-8080-0 Id: 42 at:
org.jruby.util.io.ChannelStream.feof(ChannelStream.java:486)
at: org.jruby.util.io.OpenFile.checkWritable(OpenFile.java:128)
at: org.jruby.RubyIO.write(RubyIO.java:1004)
at:
org.jruby.RubyIO$i_method_1_0$RUBYINVOKER$write.call(org/jruby/RubyIO$i_meth
od_1_0$RUBYINVOKER$write.gen)
at:
org.jruby.runtime.CallSite$InlineCachingCallSite.call(CallSite.java:391)
at:
ruby.jit.ruby.opt.local.glassfish.domains.domain1.applications.j2ee_minus_mo
dules.polls.WEB_minus_INF.gems.gems.activesupport_minus_2_dot_2_dot_2.lib.ac
tive_support.vendor.memcache_minus_client_minus_1_dot_5_dot_1.memcache.cache
_get20982185_4606416.file(memcache.rb:585)
at:
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:286
)
at:
org.jruby.runtime.CallSite$InlineCachingCallSite.call(CallSite.java:442)
at:
ruby.jit.ruby.opt.local.glassfish.domains.domain1.applications.j2ee_minus_mo
dules.polls.WEB_minus_INF.gems.gems.activesupport_minus_2_dot_2_dot_2.lib.ac
tive_support.vendor.memcache_minus_client_minus_1_dot_5_dot_1.memcache.get73
49425_4606416.rescue_1$RUBY$rescue(memcache.rb:236)

6

What volume is considered “high volume” where this might occur ?

7

has anyone looked at ehcache for Jruby ?

On Tue, Dec 23, 2008 at 3:26 AM, Albert R.
[email protected] wrote:

The threadsafety option only wraps the calls in a Mutex. There’s no
end up needing to restart them. This happens to us about once every two days
org.jruby.util.io.ChannelStream.feof(ChannelStream.java:486)
at:
memcache-client.
Marshal.load act weird. Some objects serialize using Marshal with a \222
you know that this has been bothering me.

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
8

There’s GitHub - dylanz/ehcache: ehcache which I believe
Dylan got resonably complete.

/Nick

On Wed, Feb 11, 2009 at 10:02 PM, AD [email protected] wrote:

Albert

Rich

dumps to Sun, and Charlie and team put together a fix in 1.1.6. It seems to

     at:

application to see if I can find problems. I’m going to push my application

Obviously, do not use this build. I’ll let everyone know when the basic

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
9

The symptoms in this thread sound a lot like what we have seen with
Rails 2.2.2 on JRuby in Glassfish. However, we are running with
multiple instances each running single-threaded, and thus we thought
this would not be an issue. Can someone confirm if the issues reported
above refer only to running with Rails multi-threading enabled, or if
they occur in single-threaded, multi-instance glassfish environments
too?

Also, has anyone been using Ikai’s Java-based client in production? Any
issues? Anyone use it with cache-money?

10

Thank you for this, we are using memcache-client too, and have been
experiencing issues when a memcache-node dies, and we traced back the
issue to Timeout. So even in a low-volume app this might be a problem
when you want to service one of your nodes. We’ll be looking forward
to using your memcache-client, Ikai :D. Thanks again!

Albert

On Tue, Dec 23, 2008 at 4:18 AM, AD [email protected] wrote:

On Mon, Dec 22, 2008 at 5:53 PM, Ikai L. [email protected] wrote:

thread-stateBLOCKEDWaiting on lock: org.jruby.util.io.ChannelStream@1a230ce
at:
(since last year) with JRuby/Rails 2.2.2. If it has to do with the thread

Ikai

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email