Forum: Ruby Localmemcache-0.3.0: A persistent key-value database based on mmap()'ed shared memory

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Sven C. Koehler (Guest)
on 2009-04-17 04:53
(Received via mailing list)
Hi, Localmemcache-0.3.0 is ready!  Persistence is officially supported
now so it's now a real key-value database like GDBM and the like.

Localmemcache is a library for C and ruby that aims to provide an
interface similar to memcached but for accessing local data instead
of remote data.  It's based on mmap()'ed shared memory for maximum
speed. Since version 0.3.0 it supports persistence, also making it a
fastalternative to GDBM and Berkeley DB.

FEATURES
========

- blazingly fast: On my machine it's about 10% slower to store strings
in
  LocalMemCache than storing them in a Ruby Hash of strings
- persistent
- parallel writes are supported by default
- uses transactions internally to avoid data corruption
- lightweight: the core library is just about 1400 lines of C code

* http://localmemcache.rubyforge.org/

EXAMPLE
=======

require 'localmemcache'
# 1. the memcached way
# $lm = LocalMemCache.new :namespace => :viewcounters
# 2. the GDBM way
$lm = LocalMemCache.new :filename => "./viewcounters.lmc"
$lm[:foo] = 1
$lm[:foo]
$lm.delete(:foo)

INSTALL
=======

# gem install localmemcache

(In case rubyforge has not yet updated the mirrors, fetch the 0.3.0 gem
from here: http://github.com/sck/localmemcache/downloads and then do
# gem install localmemcache-0.3.0.gem )

CONTACT
=======

Please contact me with bugs, suggestions and patches at: schween + snafu
# de

LINKS
=====

Localmemcache: http://localmemcache.rubyforge.org/
Rubyforge project: http://localmemcache.rubyforge.org/

Source code is hosted on github: http://github.com/sck/localmemcache/

Best,

Sven C. Koehler

3100e63c98860d2393bb79e6493a415b  localmemcache-0.3.0.gem
c817f4eadc86aae7435bb91b52039197  localmemcache-0.3.0.tar.gz
Roger P. (Guest)
on 2009-04-17 19:07
> Localmemcache is a library for C and ruby that aims to provide an
> interface similar to memcached but for accessing local data instead
> of remote data.  It's based on mmap()'ed shared memory for maximum
> speed. Since version 0.3.0 it supports persistence, also making it a
> fastalternative to GDBM and Berkeley DB.

Wow that's an interesting idea.  I assume it's a ruby wrapper for a C
library that does the storage?
-=r
Sven C. Koehler (Guest)
on 2009-04-17 21:33
(Received via mailing list)
On Sat, Apr 18, 2009 at 12:07:23AM +0900, Roger P. wrote:
> > Localmemcache is a library for C and ruby that aims to provide an
> > interface similar to memcached but for accessing local data instead
> > of remote data.  It's based on mmap()'ed shared memory for maximum
> > speed. Since version 0.3.0 it supports persistence, also making it a
> > fastalternative to GDBM and Berkeley DB.
>
> Wow that's an interesting idea.

Thank you!

> I assume it's a ruby wrapper for a C
> library that does the storage?

Yeah, that's the case.  It also has a C API.  My original motivation for
writing localmemcache was that I did not want to serve requests for
images in Ruby just to update a viewcounter in the database.  I wanted
to
do this right in the webserver.  So with localmemcache I just can do it
very easily in a webserver module in C, but I still can access the
counters when I do more complex operations in Ruby for other requests.

Best,

Sven
Radosław B. (Guest)
on 2009-04-18 14:50
(Received via mailing list)
From README

"
SUPPORTED SYSTEMS
=================

- a >=64bit Unix
"

Does it mean that it can't run on 32bit?

--
Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog
Radosław B. (Guest)
on 2009-04-18 15:13
(Received via mailing list)
On Fri, Apr 17, 2009 at 2:52 AM, Sven C. Koehler 
<removed_email_address@domain.invalid>
wrote:
> Hi, Localmemcache-0.3.0 is ready!  Persistence is officially supported
> now so it's now a real key-value database like GDBM and the like.

It means that all data written to localmemcache store are later
available and never disappear (like in memcached)? I greped for
'persistence' and didn't find too much. Also one line from C source is
strange for me:
./src/lmc_shm.c:  mc->use_persistence = 0;

I looked at the source and don't see where it can assigned with 1
(true) value etc.

--
Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog
Sven C. Koehler (Guest)
on 2009-04-18 16:04
(Received via mailing list)
On Sat, Apr 18, 2009 at 12:49:59PM +0200, Rados?aw Bu?at wrote:
> >From README
>
> "
> SUPPORTED SYSTEMS
> =================
>
> - a >=64bit Unix
> "
>
> Does it mean that it can't run on 32bit?

I think it can run with 32bit, but this means your maximum memory pool
size will be about 4G, depending on how much memory you already use for
other stuff...
64Bit is more than enough for setting the memory pool size to the
maximum
file size of your file system...  So yeah, you might be able to use
LocalMemCache, but at one point you just could run out of virtual
address
space in 32bit, and then other software which doesn't fully rely on
mmap() would probably a better solution for you.

-S.
Sven C. Koehler (Guest)
on 2009-04-18 16:07
(Received via mailing list)
On Sat, Apr 18, 2009 at 01:11:57PM +0200, Rados?aw Bu?at wrote:
> I looked at the source and don't see where it can assigned with 1
> (true) value etc.

It's persistent by default, this done by mmap().  Yes, it means your
data
doesn't go away.  That's also the reason why LocalMemCache doesn't need
a
server process like memcached.

[lmc_shm.c: lmc_shm_create]
|  lmc_file_path_for_namespace((char *)&fn, mc->namespace);
|  if (!lmc_handle_error((mc->fd = open(fn, O_RDWR, (mode_t)0777)) == -1,
|      "open", "ShmError", e)) goto open_failed;
|  if (!lmc_handle_error(lseek(mc->fd, mc->size - 1, SEEK_SET) == -1,
|      "lseek", "ShmError", e)) goto failed;
|  if (!lmc_handle_error(write(mc->fd, "", 1) != 1, "write",
|      "ShmError", e)) goto failed;
|  mc->base = mmap(0, mc->size, PROT_READ | PROT_WRITE, MAP_SHARED, mc->fd,
|      (off_t)0);


-S.
Roger P. (Guest)
on 2009-04-18 19:41
> It's persistent by default, this done by mmap().  Yes, it means your
> data
> doesn't go away.  That's also the reason why LocalMemCache doesn't need

As long as you don't reboot you're persistent, I suppose :)
-=r
Sven C. Koehler (Guest)
on 2009-04-18 20:07
(Received via mailing list)
On Sun, Apr 19, 2009 at 12:41:13AM +0900, Roger P. wrote:
>
> > It's persistent by default, this done by mmap().  Yes, it means your
> > data
> > doesn't go away.  That's also the reason why LocalMemCache doesn't need
>
> As long as you don't reboot you're persistent, I suppose :)

Nope, the file will be still there if you do a normal reboot and the
buffer cache will be purged before that happens. ;-)

The only thing that could happen is that your computer crashes and the
buffer cache is not purged.  If that happens you might lose some data,
but the file should not be corrupted (If we suppose that the buffer
cache
is purged in an LRU fashion, which should be the default on Unix iirc).
But these problems you will also have with Tokyo Tyrant and GDBM, so
there's no difference.

-S.
This topic is locked and can not be replied to.