Localmemcache-0.3.0: A persistent key-value database based on mmap()'ed shared memory


#1

Hi, Localmemcache-0.3.0 is ready! Persistence is officially supported
now so it’s now a real key-value database like GDBM and the like.

Localmemcache is a library for C and ruby that aims to provide an
interface similar to memcached but for accessing local data instead
of remote data. It’s based on mmap()'ed shared memory for maximum
speed. Since version 0.3.0 it supports persistence, also making it a
fastalternative to GDBM and Berkeley DB.

FEATURES

  • blazingly fast: On my machine it’s about 10% slower to store strings
    in
    LocalMemCache than storing them in a Ruby Hash of strings
  • persistent
  • parallel writes are supported by default
  • uses transactions internally to avoid data corruption
  • lightweight: the core library is just about 1400 lines of C code

EXAMPLE

require ‘localmemcache’

1. the memcached way

$lm = LocalMemCache.new :namespace => :viewcounters

2. the GDBM way

$lm = LocalMemCache.new :filename => “./viewcounters.lmc”
$lm[:foo] = 1
$lm[:foo]
$lm.delete(:foo)

INSTALL

gem install localmemcache

(In case rubyforge has not yet updated the mirrors, fetch the 0.3.0 gem
from here: http://github.com/sck/localmemcache/downloads and then do

gem install localmemcache-0.3.0.gem )

CONTACT

Please contact me with bugs, suggestions and patches at: schween + snafu

de

LINKS

Localmemcache: http://localmemcache.rubyforge.org/
Rubyforge project: http://localmemcache.rubyforge.org/

Source code is hosted on github: http://github.com/sck/localmemcache/

Best,

Sven C. Koehler

3100e63c98860d2393bb79e6493a415b localmemcache-0.3.0.gem
c817f4eadc86aae7435bb91b52039197 localmemcache-0.3.0.tar.gz


#2

Localmemcache is a library for C and ruby that aims to provide an
interface similar to memcached but for accessing local data instead
of remote data. It’s based on mmap()'ed shared memory for maximum
speed. Since version 0.3.0 it supports persistence, also making it a
fastalternative to GDBM and Berkeley DB.

Wow that’s an interesting idea. I assume it’s a ruby wrapper for a C
library that does the storage?
-=r


#3

On Sat, Apr 18, 2009 at 12:07:23AM +0900, Roger P. wrote:

Localmemcache is a library for C and ruby that aims to provide an
interface similar to memcached but for accessing local data instead
of remote data. It’s based on mmap()'ed shared memory for maximum
speed. Since version 0.3.0 it supports persistence, also making it a
fastalternative to GDBM and Berkeley DB.

Wow that’s an interesting idea.

Thank you!

I assume it’s a ruby wrapper for a C
library that does the storage?

Yeah, that’s the case. It also has a C API. My original motivation for
writing localmemcache was that I did not want to serve requests for
images in Ruby just to update a viewcounter in the database. I wanted
to
do this right in the webserver. So with localmemcache I just can do it
very easily in a webserver module in C, but I still can access the
counters when I do more complex operations in Ruby for other requests.

Best,

Sven


#4

From README

"
SUPPORTED SYSTEMS

  • a >=64bit Unix
    "

Does it mean that it can’t run on 32bit?


Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog


#5

On Fri, Apr 17, 2009 at 2:52 AM, Sven C. Koehler removed_email_address@domain.invalid
wrote:

Hi, Localmemcache-0.3.0 is ready! Persistence is officially supported
now so it’s now a real key-value database like GDBM and the like.

It means that all data written to localmemcache store are later
available and never disappear (like in memcached)? I greped for
‘persistence’ and didn’t find too much. Also one line from C source is
strange for me:
./src/lmc_shm.c: mc->use_persistence = 0;

I looked at the source and don’t see where it can assigned with 1
(true) value etc.


Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog


#6

On Sat, Apr 18, 2009 at 12:49:59PM +0200, Rados?aw Bu?at wrote:

From README

"
SUPPORTED SYSTEMS

  • a >=64bit Unix
    "

Does it mean that it can’t run on 32bit?

I think it can run with 32bit, but this means your maximum memory pool
size will be about 4G, depending on how much memory you already use for
other stuff…
64Bit is more than enough for setting the memory pool size to the
maximum
file size of your file system… So yeah, you might be able to use
LocalMemCache, but at one point you just could run out of virtual
address
space in 32bit, and then other software which doesn’t fully rely on
mmap() would probably a better solution for you.

-S.


#7

On Sat, Apr 18, 2009 at 01:11:57PM +0200, Rados?aw Bu?at wrote:

I looked at the source and don’t see where it can assigned with 1
(true) value etc.

It’s persistent by default, this done by mmap(). Yes, it means your
data
doesn’t go away. That’s also the reason why LocalMemCache doesn’t need
a
server process like memcached.

[lmc_shm.c: lmc_shm_create]
| lmc_file_path_for_namespace((char *)&fn, mc->namespace);
| if (!lmc_handle_error((mc->fd = open(fn, O_RDWR, (mode_t)0777)) == -1,
| “open”, “ShmError”, e)) goto open_failed;
| if (!lmc_handle_error(lseek(mc->fd, mc->size - 1, SEEK_SET) == -1,
| “lseek”, “ShmError”, e)) goto failed;
| if (!lmc_handle_error(write(mc->fd, “”, 1) != 1, “write”,
| “ShmError”, e)) goto failed;
| mc->base = mmap(0, mc->size, PROT_READ | PROT_WRITE, MAP_SHARED, mc->fd,
| (off_t)0);

-S.


#8

On Sun, Apr 19, 2009 at 12:41:13AM +0900, Roger P. wrote:

It’s persistent by default, this done by mmap(). Yes, it means your
data
doesn’t go away. That’s also the reason why LocalMemCache doesn’t need

As long as you don’t reboot you’re persistent, I suppose :slight_smile:

Nope, the file will be still there if you do a normal reboot and the
buffer cache will be purged before that happens. :wink:

The only thing that could happen is that your computer crashes and the
buffer cache is not purged. If that happens you might lose some data,
but the file should not be corrupted (If we suppose that the buffer
cache
is purged in an LRU fashion, which should be the default on Unix iirc).
But these problems you will also have with Tokyo Tyrant and GDBM, so
there’s no difference.

-S.


#9

It’s persistent by default, this done by mmap(). Yes, it means your
data
doesn’t go away. That’s also the reason why LocalMemCache doesn’t need

As long as you don’t reboot you’re persistent, I suppose :slight_smile:
-=r