Trouble inserting large objects into memcached

Rails
1

All,

Platform:
Mandriva Linux release 2006.0 (Official) for i586
Kernel 2.6.12-27mdksmp on a 4-processor i686 / \l

Background:
I’ve got an install of memcached that I’m attempting to use to store
data across requests. My application allows users to upload
spreadsheets of any size (up to 65K rows given Excel limitations), and
then I parse these spreadsheets and store their data in the database.
I’m attempting to use memcached to hold an intermediate object that I
use to display the data from the spreadsheet. I can successfully store
an object built from spreadsheet data of 1000 rows.

Problem:
I cannot successfully store an object built from 10000 rows or greater
in memcached. I have definitely narrowed it down to the cache put
action in the memcache client code. When I print out the size of the
marshaled object (from Marshal.dump) that is being put into the
memcache, I get the following:

1000 rows: 33935 (successfully stored)
10000 rows: 1559254 (not stored)
20000 rows: 3098224 (not stored)

Question:
Granted, these are big numbers, but I would expect that memcache could
take objects of this size. I’m not sure here if it’s memcache that is
choking or the Ruby Marshal library. Does anyone have any insight into
this? Just as a fun aside, everything works fine on my Windows box :slight_smile:
(using the pre-compiled memcached code).

I was hoping to avoid some crappy temporary table solution, but it looks
like I will be forced into that.

Thanks,
Wes

2

It turns out that the maximum allowed size of an object in memcache is 1
MB. I am looking into either how to increase that maximum or break up
my object into pieces so that I can effectively store it.

What’s a little annoying is that the memcache library suppresses the
message that comes back from the socket, which is a pretty clear
message:

SERVER_ERROR object too large for cache

Thanks,
Wes

3

Some more info.:

I’ve verified that I can Marshal.dump and Marshal.load my giant objects
without incident.

I am led to assume that the failure occurs on the sock.write command
(see below snippet from the “put” method of the memcache.rb file in the
memcache-client gem). I think that sock.write can’t handle a string of
that size.

Can anyone tell me if I’m on the right track? I’m going to attempt to
see if I can’t do the write in reasonably sized blocks instead of doing
it all at once.

Thanks,
Wes

===================

BEGIN code snippet

sock = server.socket
raise MemCacheError, “No connection to server” if sock.nil?

marshaled_value = Marshal.dump value
command = “set #{cache_key} 0 #{expiry}
#{marshaled_value.size}\r\n#{marshaled_value}\r\n”

begin
sock.write command
sock.gets
rescue SystemCallError, IOError => err
server.close
raise MemCacheError, err.message
end

END code snippet

4

Wes G. wrote in post #216702:

SOLUTION: I modified memcache-client to break up objects larger than
1048510 bytes (just under 1MB, you can’t successfully store more than
this in the memcache) into pieces internally so that they can be stored.
When a get is done, the pieces are reconstituted before the
Marshal.load. The delete also handles cleanup.

Also, in my OP, I said that everything worked fine on Windows, but that
wasn’t true.

Wes

Sorry for bringing out this post from dead. However, i thought it would
be helpful to others hence this.

The maximum size of data which memcache can load depends on a macro
POWER_BLOCK (in slabs.c). If we modify this macro to higher values, say
2 MB of more, memcache would be able to handle data larger than 1 MB but
less than equals 2MB.

A better approach would be to make it a parameter to memcache hence
eliminating the need to compile every time you need to increase memory.

5

Ha ha - now it is an option! But only for the last two years so I don’t
feel stupid for doing what I did.

From the memcached man page on OS X:

" -I
Override the default size of each slab page. Default is
1mb. Default is 1m, minimum is 1k, max is 128m. Adjusting this value
changes the item size limit. Beware that this also increases the number
of slabs (use -v to view), and the overal memory usage of
memcached."

Wes

6

SOLUTION: I modified memcache-client to break up objects larger than
1048510 bytes (just under 1MB, you can’t successfully store more than
this in the memcache) into pieces internally so that they can be stored.
When a get is done, the pieces are reconstituted before the
Marshal.load. The delete also handles cleanup.

Also, in my OP, I said that everything worked fine on Windows, but that
wasn’t true.

Wes