FFI Memory Leak

I’ve found a memory leak in some FFI code. I’ll give example code
below, but I can explain the issue easily enough first.

When calling a function provided by Tokyo Cabinet, it returns a value in
the form of a pointer and a length to read at that address. The
documentation for the function (tchdbget() at
http://1978th.net/tokyocabinet/spex-en.html#tchdbapi, if you are
interested) warns that the returned region was allocated with malloc()
and needs to be free()d. I can’t figure out how to free the value, so
there is a memory leak. What I would like to know is how to free the
value at the end of the pointer.

Here’s some code showing what I just explained.

#!/usr/bin/env ruby -wKU

require “rubygems”
require “ffi”

map the C interface

module Lib
extend FFI::Library
ffi_lib(
Array(
ENV.fetch(
“TOKYO_CABINET_LIB”,
Dir["/{opt,usr}/{,local/}lib{,64}/libtokyocabinet.{dylib,so
}"]
)
)
)

attach_function :tchdbnew, [ ],
:pointer
attach_function :tchdbopen, [:pointer, :string, :int],
:bool
attach_function :tchdbput, [:pointer, :pointer, :int, :pointer,
:int],
:bool
attach_function :tchdbget, [:pointer, :pointer, :int, :pointer],
:pointer
attach_function :tchdbclose, [:pointer],
:bool
end

translate the interface to Ruby

class TokyoCabinet
def self.open(*args)
db = new(*args)
yield db
ensure
db.close if db
end

def initialize(path)
@db = Lib.tchdbnew
Lib.tchdbopen(@db, path, (1 << 1) | (1 << 2)) # write create mode
end

def []=(key, value)
k, v = key.to_s, value.to_s
Lib.tchdbput(@db, k, k.size, v, v.size)
end

def
k = key.to_s
size = FFI::MemoryPointer.new(:int)
value = Lib.tchdbget(@db, k, k.size, size)
value.address.zero? ? nil : value.get_bytes(0, size.get_int(0))
ensure
size.free if size
# FIXME: How do I free value here?
end

def close
Lib.tchdbclose(@db)
end
end

show the problem

def show_memory
3.times { GC.start } # try to clean up
mem = ps -o rss -p #{Process.pid}[/\d+/]
puts “Current memory: #{mem}”
end

TokyoCabinet.open(“leak.tch”) do |db|
db[:some_key] = “X” * 1024
10.times do
5000.times do
db[:some_key] # reading causes the memory leak
end
show_memory
end
end

Sample Run:

Current memory: 30324

Current memory: 37828

Current memory: 45364

Current memory: 52896

Current memory: 60428

Current memory: 67960

Current memory: 75488

Current memory: 83020

Current memory: 90552

Current memory: 98080

END

Thanks in advance for any help provided.

James Edward G. II

On Thu, Dec 24, 2009 at 12:41 PM, James Edward G. II
[email protected] wrote:

I’ve found a memory leak in some FFI code. I’ll give example code below, but I can explain the issue easily enough first.

When calling a function provided by Tokyo Cabinet, it returns a value in the form of a pointer and a length to read at that address. The documentation for the function (tchdbget() at http://1978th.net/tokyocabinet/spex-en.html#tchdbapi, if you are interested) warns that the returned region was allocated with malloc() and needs to be free()d. I can’t figure out how to free the value, so there is a memory leak. What I would like to know is how to free the value at the end of the pointer.

Hello,

maybe

jeg.rb · GitHub

or

jeg.rb · GitHub

Seems to work.

Merry Christmas,

On Dec 23, 2009, at 11:26 PM, John M. wrote:

jeg.rb · GitHub

or

jeg.rb · GitHub

Seems to work.

Yeah, those do seem to work. Awesome. Thanks for the help!

James Edward G. II

On Fri, Dec 25, 2009 at 2:00 AM, James Edward G. II
[email protected] wrote:

Yeah, those do seem to work. Awesome. Thanks for the help!

Great news, I will then release a fresh rufus-tokyo with the fix (I’ll
use free() directly, as in Tokyo Cabinet’s C examples).

Merry Xmas,

On Dec 24, 2009, at 5:47 PM, John M. wrote:

On Fri, Dec 25, 2009 at 2:00 AM, James Edward G. II
[email protected] wrote:

Yeah, those do seem to work. Awesome. Thanks for the help!

Great news, I will then release a fresh rufus-tokyo with the fix (I’ll
use free() directly, as in Tokyo Cabinet’s C examples).

Yeah, there doesn’t seem to be any advantage to using tcfree() that I
can glean from the documentation.

Merry Xmas,

Same to you.

James Edward G. II