Forum: Ruby FFI 0.1.1 (Foreign Function Interface) for Ruby 1.8.6/7 and 1.9

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Charles Oliver N. (Guest)
on 2008-11-01 01:18
(Received via mailing list)
The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7
and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic
libraries, binding functions within them, and calling those functions
from Ruby code. Here's a quick sample of binding and calling the getpid
C library function:


require 'ffi'

module GetPid
   extend FFI::Library

   attach_function :getpid, [], :uint
end

puts GetPid.getpid


Here's another, calling qsort and passing a Ruby block as a C callback:


require 'ffi'

module LibC
   extend FFI::Library
   callback :qsort_cmp, [ :pointer, :pointer ], :int
   attach_function :qsort, [ :pointer, :int, :int, :qsort_cmp ], :int
end

p = MemoryPointer.new(:int, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
   i1 = p1.get_int32(0)
   i2 = p2.get_int32(0)
   i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"


I posted a blog entry with a longer description of the library,
additional examples, and links to some other documentation and posts.
Docs are a little slim at this point, so feel free to experiment and
update the JRuby wiki page:

http://wiki.jruby.org/wiki/Calling_C_from_JRuby

I'm sure docs from here will filter back into the library and out into
the general cosmos.

Finally, there's no need to write a C extension to call C libraries, and
the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby 1.1.4+, and
Rubinius (though Rubinius has no callback support yet).

Don't be an extension stooge! Use FFI!

- Charlie
Thomas H. (Guest)
on 2008-11-01 03:43
(Received via mailing list)
* Charles Oliver N. (removed_email_address@domain.invalid) wrote:

> The JRuby team is proud to announce the release of FFI for Ruby
> 1.8.6/7 and 1.9!

> Finally, there's no need to write a C extension to call C libraries,
> and the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby
> 1.1.4+, and Rubinius (though Rubinius has no callback support yet).

Awesome.  I've used DL to link up some custom libs to a Ruby service,
will give FFI a go and see how it compares :)

Are things like structs likely to be supported in future, ala Python
ctypes?
Charles Oliver N. (Guest)
on 2008-11-01 07:42
(Received via mailing list)
Thomas H. wrote:
> will give FFI a go and see how it compares :)
>
> Are things like structs likely to be supported in future, ala Python
> ctypes?

Actually structs are already supported! See the blog post, and I believe
there's some examples shipped with the gem. There needs to be more docs,
certainly, and hopefully they'll get some TLC soon.

Also, I forgot to call out Evan P. for coming up with the API and
initial design, and he or someone else on Rubinus wrote up the
templating/header-file-processing stuff as well. And of course a huge
thanks to Wayne M. for implementing FFI not just once (for JRuby)
but twice (for C Ruby). His work will mean a huge leap forward in
cross-impl portability.

- Charlie
Radosław B. (Guest)
on 2008-11-01 10:41
(Received via mailing list)
On Sat, Nov 1, 2008 at 6:40 AM, Charles Oliver N.
<removed_email_address@domain.invalid> wrote:
> Actually structs are already supported! See the blog post, and I believe
> there's some examples shipped with the gem.

Your blog is preaty known but for clarity:
http://blog.headius.com/2008/10/ffi-for-ruby-now-a...
:-)

--
Rados³aw Bu³at

http://radarek.jogger.pl - mój blog
Charles Oliver N. (Guest)
on 2008-11-01 12:19
(Received via mailing list)
Radosław Bułat wrote:
> On Sat, Nov 1, 2008 at 6:40 AM, Charles Oliver N.
> <removed_email_address@domain.invalid> wrote:
>> Actually structs are already supported! See the blog post, and I believe
>> there's some examples shipped with the gem.
>
> Your blog is preaty known but for clarity:
> http://blog.headius.com/2008/10/ffi-for-ruby-now-a...
> :-)

Oh right, a link would have been useful. Thank you :)

- Charlie
Charles Oliver N. (Guest)
on 2008-11-01 12:21
(Received via mailing list)
Charles Oliver N. wrote:
> I posted a blog entry with a longer description of the library,
> additional examples, and links to some other documentation and posts.

And then I completely forgot to include the blog post URL...

http://blog.headius.com/2008/10/ffi-for-ruby-now-a...

- Charlie
Luc H. (Guest)
on 2008-11-01 12:27
(Received via mailing list)
On 1 nov. 08, at 11:19, Charles Oliver N. wrote:

> And then I completely forgot to include the blog post URL...

Looks cool, great work.

Two questions:

- Are variadic functions supported?
- Do you have any idea or measurements of the overhead of calling
through FFI as opposed to using a compiled extension?
Charles Oliver N. (Guest)
on 2008-11-01 13:18
(Received via mailing list)
Luc H. wrote:
> through FFI as opposed to using a compiled extension?
Wayne answers the latter, sorta, on his followup blog post:

http://blog.headius.com/2008/10/ffi-for-ruby-now-a...

He doesn't have specific numbers for performance at the moment, but the
short story is that FFI introduces a bit of overhead; ultimately I
believe that the overhead gets lost in the flow of a Ruby application,
especially when you're tossing units of work across like SQL queries or
arrays. Wayne probably can fill in more details on what the actual
overhead is like.

And I'd also expect that any small amount of overhead is vastly
outweighed by the ability to write an FFI-based library once and use it
across implementations.

- Charlie
Luc H. (Guest)
on 2008-11-01 23:26
(Received via mailing list)
On 1 nov. 08, at 12:16, Charles Oliver N. wrote:

> He doesn't have specific numbers for performance at the moment, but
> the short story is that FFI introduces a bit of overhead; ultimately
> I believe that the overhead gets lost in the flow of a Ruby
> application, especially when you're tossing units of work across
> like SQL queries or arrays.

# --- [begin unscientific test] --------

require 'rubygems'
require 'benchmark'
require 'zlib'
require 'ffi'
require 'dl/import'

module Zlib_ffi
     extend FFI::Library
     attach_function :zlib_version, :zlibVersion, [], :string
end

module Zlib_dl
     extend DL::Importable
     dlload "libz.dylib"
     extern "const char* zlibVersion()"
end

puts Zlib.zlib_version
puts Zlib_ffi.zlib_version
puts Zlib_dl.zlibVersion

Benchmark.bm(3) do |bm|
     bm.report("ext") { 500_000.times { Zlib.zlib_version } }
     bm.report("ffi") { 500_000.times { Zlib_ffi.zlib_version } }
     bm.report("dl")  { 500_000.times { Zlib_dl.zlibVersion } }
end

# --- [end unscientific test] --------

This gives the following results:

1.2.3
1.2.3
1.2.3
          user     system      total        real
ext  1.050000   0.320000   1.370000 (  1.373800)
ffi  2.160000   0.660000   2.820000 (  2.818966)
dl   3.500000   1.060000   4.560000 (  4.552789)

All this using MacPorts MRI 1.8.7-p72 under OS X 10.5.5. The observed
overhead is slightly over 2x for ffi, probably not a big deal unless
ffi calls are used in tight loops I guess.


PS: haven't seen any trace of variadic function support in the code.
Radosław B. (Guest)
on 2008-11-02 01:22
(Received via mailing list)
Where can I file an issue? I have trouble to build in with ruby1.9 on
64bit.

--
Rados³aw Bu³at

http://radarek.jogger.pl - mój blog
Radosław B. (Guest)
on 2008-11-02 01:25
(Received via mailing list)
> This gives the following results:
>
> 1.2.3
> 1.2.3
> 1.2.3
>         user     system      total        real
> ext  1.050000   0.320000   1.370000 (  1.373800)
> ffi  2.160000   0.660000   2.820000 (  2.818966)
> dl   3.500000   1.060000   4.560000 (  4.552789)
>

Ubuntu 8.10 64bit

$ ruby --version && ruby ffi_bench.rb
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
1.2.3.3
1.2.3.3
1.2.3.3
         user     system      total        real
ext  0.320000   0.070000   0.390000 (  0.396774)
ffi  0.770000   0.120000   0.890000 (  0.895093)
dl   2.090000   0.270000   2.360000 (  2.365029)




--
Rados³aw Bu³at

http://radarek.jogger.pl - mój blog
Charles Oliver N. (Guest)
on 2008-11-02 03:20
(Received via mailing list)
Luc H. wrote:
>          user     system      total        real
> ext  1.050000   0.320000   1.370000 (  1.373800)
> ffi  2.160000   0.660000   2.820000 (  2.818966)
> dl   3.500000   1.060000   4.560000 (  4.552789)
>
> All this using MacPorts MRI 1.8.7-p72 under OS X 10.5.5. The observed
> overhead is slightly over 2x for ffi, probably not a big deal unless ffi
> calls are used in tight loops I guess.

Seems like that pretty well seals the deal for ffi over dl, at the very
least. I'm also glad to see FFI wasn't even that bad, especially
considering it hasn't received any optimization.

- Charlie
Charles Oliver N. (Guest)
on 2008-11-02 03:22
(Received via mailing list)
Radosław Bułat wrote:
>
> dl   2.090000   0.270000   2.360000 (  2.365029)
Seems like about the same ratio...probably can be improved too!

- Charlie
Charles Oliver N. (Guest)
on 2008-11-02 03:22
(Received via mailing list)
Radosław Bułat wrote:
> Where can I file an issue? I have trouble to build in with ruby1.9 on 64bit.

Hop on the ruby-ffi project mailing lists, and I guess you can file a
bug in the tracker there too:

http://kenai.com/projects/ruby-ffi

- Charlie
Sylvain J. (Guest)
on 2008-11-03 11:28
(Received via mailing list)
On Sat, Nov 01, 2008 at 08:16:39AM +0900, Charles Oliver N. wrote:
> The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7
> and 1.9!
>
> FFI (gem install ffi) is a library for programmatically loading dynamic
> libraries, binding functions within them, and calling those functions
> from Ruby code. Here's a quick sample of binding and calling the getpid
> C library function:

Interesting. On what kind of architectures is the binding part working ?
I'm
using dyncall to do the actual interfacing work
(http://www.dyncall.org/) in a
DL-replacement library, but my problem is that dyncall does not like
Linux-PPC.
What are you using on your side ?

Sylvain
Charles Oliver N. (Guest)
on 2008-11-03 12:03
(Received via mailing list)
Sylvain J. wrote:
> using dyncall to do the actual interfacing work (http://www.dyncall.org/) in a
> DL-replacement library, but my problem is that dyncall does not like Linux-PPC.
> What are you using on your side ?

Ruby FFI uses libffi, as does JNA which ships with JRuby. I'm not
certain about libffi specifically. but JNA claims to support OSX (ppc,
x86, x86_64), linux (x86, amd64), FreeBSD/OpenBSD (x86, amd64), Solaris
(x86, amd64, sparc, sparcv9) and Windows (x86, amd64).

I'm not sure if linux-ppc support is not provided because it's not
supported or because nobody has a linux-ppc machine to build on. The
latter has been the case for several entries on the list; I myself was
the build monkey for Solaris/AMD64 and Linux/AMD64 for a short time,
before which there was no shipped support.

There's certainly one way to find out...gem install ffi. Report back
here or on ruby-ffi mailing lists what you learn :)

http://kenai.com/projects/ruby-ffi

- Charlie
unknown (Guest)
on 2008-11-03 13:40
(Received via mailing list)
On Mon, 3 Nov 2008 05:01:11 -0500, Charles Oliver N. wrote:

>I'm not sure if linux-ppc support is not provided because it's not
>supported or because nobody has a linux-ppc machine to build on. The
>latter has been the case for several entries on the list; I myself was
>the build monkey for Solaris/AMD64 and Linux/AMD64 for a short time,
>before which there was no shipped support.
>
>There's certainly one way to find out...gem install ffi. Report back
>here or on ruby-ffi mailing lists what you learn :)

Ubuntu 8.10 provides libffi on ppc. The gem builds, but a simple ruby
program hangs (rather uses 100% CPU).

I can investigate further (e.g building my own libffi), but that may
not happen immediately.

-jh
Sylvain J. (Guest)
on 2008-11-03 20:59
(Received via mailing list)
>> Interesting. On what kind of architectures is the binding part working ? I'm
>> using dyncall to do the actual interfacing work (http://www.dyncall.org/) in a
>> DL-replacement library, but my problem is that dyncall does not like Linux-PPC.
>> What are you using on your side ?
>
> Ruby FFI uses libffi, as does JNA which ships with JRuby. I'm not
> certain about libffi specifically. but JNA claims to support OSX (ppc,
> x86, x86_64), linux (x86, amd64), FreeBSD/OpenBSD (x86, amd64), Solaris
> (x86, amd64, sparc, sparcv9) and Windows (x86, amd64).
I did not know about libffi ... I'll have to look if I should not
replace
dyncall by libffi then. Thanks for the info.

> I'm not sure if linux-ppc support is not provided because it's not
> supported or because nobody has a linux-ppc machine to build on.
I guess it is a bit of both. Given that libffi is supported on
Linux/PPC64, I
guess having it on Linux/PPC should not be that much of a problem -- but
still,
I think the calling convention can be slightly different between the two
architectures.

Sylvain
Charles Oliver N. (Guest)
on 2008-11-03 21:10
(Received via mailing list)
We (JRuby and Rubinius contributors) designed in the ability to
specify calling convention for a given library bound through FFI.
Largely this was needed to support Win32's stdcall, but if Linux/PPC
has a different convention and libffi supports it, then FFi will too
(though we may have to add it to the list of accepted conventions).

I'll check on the syntax we're supporting and get back to ya.

- Charlie (mobile)

On Nov 3, 2008, at 12:56, Sylvain J.
<removed_email_address@domain.invalid
Charles Oliver N. (Guest)
on 2008-11-03 21:45
(Received via mailing list)
removed_email_address@domain.invalid wrote:
>
> Ubuntu 8.10 provides libffi on ppc. The gem builds, but a simple ruby
> program hangs (rather uses 100% CPU).
>
> I can investigate further (e.g building my own libffi), but that may
> not happen immediately.

Hey, please do. Sounds like it *should* work, at least. I sure don't
have a PPC box to test on and I don't think anyone else involved in Ruby
FFI does either.

- Charlie
Charles Oliver N. (Guest)
on 2008-11-03 21:49
(Received via mailing list)
Charles Oliver N. wrote:
> We (JRuby and Rubinius contributors) designed in the ability to specify
> calling convention for a given library bound through FFI. Largely this
> was needed to support Win32's stdcall, but if Linux/PPC has a different
> convention and libffi supports it, then FFi will too (though we may have
> to add it to the list of accepted conventions).
>
> I'll check on the syntax we're supporting and get back to ya.

In the body of the module you're binding:

ffi_convention :stdcall # :default is for normal C convention

We could add whatever convention Linux/PPC needs.

- Charlie
Charles Oliver N. (Guest)
on 2008-11-03 21:56
(Received via mailing list)
Charles Oliver N. wrote:
>
> ffi_convention :stdcall # :default is for normal C convention
>
> We could add whatever convention Linux/PPC needs.

Or, duh, have FFI use the appropriate default convention on the target
platform. Either way, it should be simple to deal with.

- Charlie
Sylvain J. (Guest)
on 2008-11-03 22:26
(Received via mailing list)
On Tue, Nov 04, 2008 at 04:54:35AM +0900, Charles Oliver N. wrote:
>> In the body of the module you're binding:
>>
>> ffi_convention :stdcall # :default is for normal C convention
>>
>> We could add whatever convention Linux/PPC needs.
>
> Or, duh, have FFI use the appropriate default convention on the target
> platform. Either way, it should be simple to deal with.

Thanks Charlie, but I have already my own library which does more than
Ruby-FFI
(typelib.sf.net and typelib.sf.net/html/ruby) so I'll probably be using
libffi
directly.

Sylvain
Charles Oliver N. (Guest)
on 2008-11-03 23:47
(Received via mailing list)
Sylvain J. wrote:
>>> In the body of the module you're binding:
>>>
>>> ffi_convention :stdcall # :default is for normal C convention
>>>
>>> We could add whatever convention Linux/PPC needs.
>> Or, duh, have FFI use the appropriate default convention on the target
>> platform. Either way, it should be simple to deal with.
>
> Thanks Charlie, but I have already my own library which does more than Ruby-FFI
> (typelib.sf.net and typelib.sf.net/html/ruby) so I'll probably be using libffi
> directly.

Is your library compatible with JRuby? I think that's a key point for
Ruby FFI.

- Charlie
Sylvain J. (Guest)
on 2008-11-04 10:32
(Received via mailing list)
On Tue, Nov 04, 2008 at 06:45:27AM +0900, Charles Oliver N. wrote:
>>>>>
>> (typelib.sf.net and typelib.sf.net/html/ruby) so I'll probably be using libffi
>> directly.
>
> Is your library compatible with JRuby? I think that's a key point for
> Ruby FFI.
It is not -- it is mainly C++ -- but given that most of the
functionality is
written in C++ *and* I use that C++ interface in other C++ programs,
moving to
Ruby-FFI is not practical for me. So ...

Sylvain
Ken B. (Guest)
on 2008-11-04 16:42
(Received via mailing list)
On Fri, 31 Oct 2008 18:16:39 -0500, Charles Oliver N. wrote:

>
>
> p.put_array_of_int32(0, [ 2, 1 ])
> additional examples, and links to some other documentation and posts.
> Rubinius (though Rubinius has no callback support yet).
>
> Don't be an extension stooge! Use FFI!
>
> - Charlie

Based on all of the examples I've seen, there's something that's still
not clear to me. How does ruby-ffi know which library to load to find
the
specified function?

--Ken
Sean O. (Guest)
on 2008-11-04 16:48
(Received via mailing list)
On Tue, Nov 4, 2008 at 2:38 PM, Ken B. <removed_email_address@domain.invalid> 
wrote:
>
> Based on all of the examples I've seen, there's something that's still
> not clear to me. How does ruby-ffi know which library to load to find the
> specified function?
>
> --Ken
>
Use ffi_lib in the extended module, e.g.

require 'rubygems'
require 'ffi'

module NCursesFFI
 extend FFI::Library
 ffi_lib 'ncurses'
 attach_function 'clear', [], :int
 attach_function 'endwin', [], :int
 attach_function 'getch', [], :int
 attach_function 'initscr', [], :int
 attach_function 'printw', [ :string ], :int
 attach_function 'refresh', [], :int
end

NCursesFFI.initscr
NCursesFFI.printw("Hello again")
NCursesFFI.refresh
NCursesFFI.getch
NCursesFFI.endwin

Regards,
Sean
R. K. (Guest)
on 2008-11-04 17:11
Sean O'halpin wrote:
> On Tue, Nov 4, 2008 at 2:38 PM, Ken B. <removed_email_address@domain.invalid> wrote:

>
> module NCursesFFI
>  extend FFI::Library
>  ffi_lib 'ncurses'
>  attach_function 'clear', [], :int
>  attach_function 'endwin', [], :int
>  attach_function 'getch', [], :int
>  attach_function 'initscr', [], :int
>  attach_function 'printw', [ :string ], :int
>  attach_function 'refresh', [], :int
> end
>

Sean,
Going by the above, and the samples of structs on the blog, for a medium
sized library such as ncurses, there can be some effort in getting this
module ready. Will there be some kind of repository of such modules, so
we are not duplicating each others effort ?

I'd like to also add that the above sample is working on osx
10.5.5/darwin/ppc after a small modification in platform.rb above line
28.

    ARCH = case CPU.downcase
    when /i?86|x86|i86pc|powerpc/
      "i386"
    when /amd64|x86_64/
      "x86_64"

I have added "powerpc" to the first "when". btw, i had a clean gem
install.
Thanks.
Sean O. (Guest)
on 2008-11-04 17:53
(Received via mailing list)
On Tue, Nov 4, 2008 at 3:10 PM, Nit K. <removed_email_address@domain.invalid> 
wrote:
>>  attach_function 'initscr', [], :int
>>  attach_function 'printw', [ :string ], :int
>>  attach_function 'refresh', [], :int
>> end
>>
>
> Sean,
> Going by the above, and the samples of structs on the blog, for a medium
> sized library such as ncurses, there can be some effort in getting this
> module ready. Will there be some kind of repository of such modules, so
> we are not duplicating each others effort ?

It's a fairly tedious (and largely thankless) task to convert a lib
like ncurses. However, I'm in the mood right now so expect a github
repo in the near future :) Your point about avoiding duplication of
effort is a good one. The main problem in tackling something like this
is fatigue setting in when you've got enough for your immediate
requirement and so ending up with a half-baked conversion (I speak
from experience). I believe another pitfall is trying to rubyify the
interface. I think it's better to provide a thin bridge and then
abstract on top of that. Then you get the benefit of being able to use
existing experience and documentation (with the usual translations)
and others can take or leave your abstraction as they see fit. (I do
think Ruby needs a portable console library but I'd rather build that
on top of ruby versions of ncurses and Win32 console than try to make
ncurses work on Windows. I'm hoping FFI will make that possible.)

I'm wondering if it would be useful to have these FFI-enabled libs
under an ffi/ namespace, e.g. you'd require 'ffi/ncurses' and possibly
include FFI::NCurses. We could put shared structs, constants, etc.
under ffi/include. Any thoughts anyone?

>
> I have added "powerpc" to the first "when". btw, i had a clean gem
> install.

Excellent news.

Regards,
Sean
Luis L. (Guest)
on 2008-11-04 19:00
(Received via mailing list)
On Nov 4, 12:51 pm, Sean O'Halpin <removed_email_address@domain.invalid> wrote:
> >>  attach_function 'initscr', [], :int
> It's a fairly tedious (and largely thankless) task to convert a lib
> think Ruby needs a portable console library but I'd rather build that
> on top of ruby versions of ncurses and Win32 console than try to make
> ncurses work on Windows. I'm hoping FFI will make that possible.)
>

Yeah, first layer of implementation and then abstraction (Rubify) is
the way to go :-)

> I'm wondering if it would be useful to have these FFI-enabled libs
> under an ffi/ namespace, e.g. you'd require 'ffi/ncurses' and possibly
> include FFI::NCurses. We could put shared structs, constants, etc.
> under ffi/include. Any thoughts anyone?
>

Something like the win32utils team does:

gem: win32-service
gem win32-eventlog

require 'win32/service'
require 'win32/eventlog'

All these live inside Win32 namespace

I think these could be named ffi-ncourses but the name implies "FFI
implementation of ncurses" where I will love to have "cross platform
curses" (note the lack of 'n').

These should depend on ruby-ffi gem and we are set to go :-)

My to cents
Dominic S. (Guest)
on 2008-11-04 19:06
(Received via mailing list)
>
> On windows, I can't get it to work for jruby or ruby


C:\jruby-1.1.5>jruby -v
jruby 1.1.5 (ruby 1.8.6 patchlevel 114) (2008-11-03 rev 7996) [x86-java]

C:\jruby-1.1.5>cd samples\ffi

C:\jruby-1.1.5\samples\ffi>dir
 Volume in drive C has no label.
 Volume Serial Number is F4EA-F50A

 Directory of C:\jruby-1.1.5\samples\ffi

11/03/2008  05:04 PM    <DIR>          .
11/03/2008  05:04 PM    <DIR>          ..
11/03/2008  04:42 PM               301 ffi.rb
11/03/2008  04:42 PM               367 gettimeofday.rb
11/03/2008  04:42 PM             2,340 pty.rb
11/03/2008  04:42 PM               476 qsort.rb
11/03/2008  04:42 PM             2,078 win32api.rb
               5 File(s)          5,562 bytes
               2 Dir(s)  18,840,158,208 bytes free

C:\jruby-1.1.5\samples\ffi>jruby win32api.rb
null:1:in `const_missing': uninitialized constant POSIX::FFI (NameError)
        from ffi.rb:4
        from ffi.rb:1:in `require'
        from win32api.rb:1

C:\jruby-1.1.5\samples\ffi>

For MRI, it tries to compile

C:\jruby-1.1.5\samples\ffi>gem install ffi
Building native extensions.  This could take a while...
ERROR:  Error installing ffi:
        ERROR: Failed to build gem native extension.

c:/ruby/bin/ruby.exe extconf.rb install ffi
creating Makefile

nmake
'nmake' is not recognized as an internal or external command,
operable program or batch file.


Gem files will remain installed in
c:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.1
for inspection.
Results logged to
c:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.1/ext/gem_make.out

C:\jruby-1.1.5\samples\ffi>
R. K. (Guest)
on 2008-11-04 19:10
Sean O'halpin wrote:
> On Tue, Nov 4, 2008 at 3:10 PM, Nit K. <removed_email_address@domain.invalid> wrote:
>>>  attach_function 'initscr', [], :int
>>>  attach_function 'printw', [ :string ], :int
>>>  attach_function 'refresh', [], :int
>>> end
>>>
>>
>> Sean,
>> Going by the above, and the samples of structs on the blog, for a medium
>> sized library such as ncurses, there can be some effort in getting this
>> module ready. Will there be some kind of repository of such modules, so
>> we are not duplicating each others effort ?
>
> It's a fairly tedious (and largely thankless) task to convert a lib
> like ncurses. However, I'm in the mood right now so expect a github
> repo in the near future :) Your point about avoiding duplication of
> effort is a good one. The main problem in tackling something like this
> is fatigue setting in when you've got enough for your immediate
> requirement and so ending up with a half-baked conversion (I speak
> from experience). I believe another pitfall is trying to rubyify the
> interface.

I suppose some of these conversions/wrappers would be "approved" and
sort of become a standard, so we don't have a plethora of them. Then
they could be placed in one repo (after some sort of review). Just a
thought.

Would hate to have find 5 different ncurses wrappers and have to study
each to figure out which to use.
Charles Oliver N. (Guest)
on 2008-11-04 21:04
(Received via mailing list)
Nit K. wrote:
>>  attach_function 'printw', [ :string ], :int
>>  attach_function 'refresh', [], :int
>> end
>>
>
> Sean,
> Going by the above, and the samples of structs on the blog, for a medium
> sized library such as ncurses, there can be some effort in getting this
> module ready. Will there be some kind of repository of such modules, so
> we are not duplicating each others effort ?

I'd recommend folks start releasing FFI-based "thin wrappers" for each
library they come across, and then depend on that gem. Don't go over the
top...just get the basic bindings in there, structs, tests to make sure
it's all working, and release a gem. Then build nice APIs on top of
that, separately.

One of my biggest gripes about C extensions in Ruby is that everyone
feels the need to release their own wrapper PLUS Ruby-like API in a
single gem, when just releasing the wrapper would allow a lot more
experimentation. So you end up with multiple people writing their own
extensions with their own APIs and little sharing.

If we just get a whole ton of, simple, clean wrappers out there, the
APIs will surely follow.

- Charlie
Charles Oliver N. (Guest)
on 2008-11-04 21:10
(Received via mailing list)
Dominic S. wrote:
> C:\jruby-1.1.5\samples\ffi>jruby win32api.rb
> null:1:in `const_missing': uninitialized constant POSIX::FFI (NameError)
>         from ffi.rb:4
>         from ffi.rb:1:in `require'
>         from win32api.rb:1

That sample could be behind the times...play with it a bit and see what
you can see. None of the core JRuby devs use Windows on a regular
basis...I think this was just a quick experiment created early in FFI
dev cycle.

FWIW, I believe someone (Dan B., probably) is working on a full FFI
win32api impl. He could use some help.

> nmake
> 'nmake' is not recognized as an internal or external command,
> operable program or batch file.

We need to get a win32 binary FFI gem released. I think one or two
people are working on that. Jump on Ruby FFI mailing lists if you are
interested in helping.

- Charlie
Charles Oliver N. (Guest)
on 2008-11-04 21:21
(Received via mailing list)
Charles Oliver N. wrote:
> I posted a blog entry with a longer description of the library,
> additional examples, and links to some other documentation and posts.
> Docs are a little slim at this point, so feel free to experiment and
> update the JRuby wiki page:
>
> http://wiki.jruby.org/wiki/Calling_C_from_JRuby
>
> I'm sure docs from here will filter back into the library and out into
> the general cosmos.

FYI, I've moved the content of the page above to the Ruby FFI wiki here:

http://kenai.com/projects/ruby-ffi/pages/Home

Please make updates and additions to that copy. I'll modify the JRuby
version to link to Ruby FFI's wiki.

- Charlie
Ken B. (Guest)
on 2008-11-05 04:40
(Received via mailing list)
On Tue, 04 Nov 2008 14:02:50 -0500, Charles Oliver N. wrote:

>>>  attach_function 'initscr', [], :int
>
> extensions with their own APIs and little sharing.
>
> If we just get a whole ton of, simple, clean wrappers out there, the
> APIs will surely follow.
>
> - Charlie

The good news is that unlike C libraries, if someone bundles their own
wrapper with their own API, you can still get at the wrapper, by finding
the module that imports FFI and just using that.
Sean O. (Guest)
on 2008-11-05 06:15
(Received via mailing list)
On Tue, Nov 4, 2008 at 7:02 PM, Charles Oliver N.
<removed_email_address@domain.invalid> wrote:
> APIs and little sharing.
>
> If we just get a whole ton of, simple, clean wrappers out there, the APIs
> will surely follow.
>
> - Charlie
>

Maybe I wasn't clear - what you're proposing is exactly what I
intended to convey. Thin wrapper in a single gem. Any abstractions
should be separate gems.

Regards,
Sean
Charles Oliver N. (Guest)
on 2008-11-05 06:37
(Received via mailing list)
Sean O'Halpin wrote:
> Maybe I wasn't clear - what you're proposing is exactly what I
> intended to convey. Thin wrapper in a single gem. Any abstractions
> should be separate gems.

Perfect then :)

Tom Enebo actually suggested just naming them the library name, as in
libncurses, libtommath, whatever.

- Charlie
Sean O. (Guest)
on 2008-11-05 18:57
(Received via mailing list)
On Wed, Nov 5, 2008 at 4:34 AM, Charles Oliver N.
<removed_email_address@domain.invalid> wrote:
>
> Tom Enebo actually suggested just naming them the library name, as in
> libncurses, libtommath, whatever.

That's a little *nix-centric, no? If we had everything under ffi/
you'd be able to find everything in one place (e.g. easier to find
examples to copy from, easier to set up conventions for where to put
shared structs, constants, etc.). As for OS-specific libs, there could
be ffi/win32/ ffi/unix, where appropriate.

It would be useful to have conventions on how to implement thin layer
ffis (e.g. how to handle return values in *ptrs, how to implement
functons defined as macros, namespacing constants, etc.). This would
help POLS. Also, guidelines on how to portably implement layers for
32-bit/64-bit and endian-ness would be handy (I know I'd like some
tips).

While there is no general solution to converting header files,
projects such as SWIG have shown that you can get a lot done
automatically. It would be worthwhile to work on tools to help with
this - even an 80% solution removes a lot of the donkey work (as I'm
finding with my ncurses port).

My general opinion is that the more boring, mechanical, choice-free
and thoughtless we can make this work, the easier it will be to do and
so the more likely to get done.

Your cross-implementation FFI initiative is very welcome - it's a
liberating move. Now we can get on with the mindless (but worthy)
drudgery :)

Regards,
Sean
R. K. (Guest)
on 2008-11-05 19:13
Sean O'halpin wrote:
> On Wed, Nov 5, 2008 at 4:34 AM, Charles Oliver N.
> <removed_email_address@domain.invalid> wrote:
>>
> automatically. It would be worthwhile to work on tools to help with
> this - even an 80% solution removes a lot of the donkey work (as I'm
> finding with my ncurses port).
>
>
> Regards,
> Sean

Eagerly awaiting your ncurses port...

Is it possible for you to document the procedure, how you go about doing
it, so one can follow that for another port, rather than be totally
lost.
Sean O. (Guest)
on 2008-11-05 20:30
(Received via mailing list)
On Wed, Nov 5, 2008 at 5:11 PM, Nit K. <removed_email_address@domain.invalid> 
wrote:
>
> Eagerly awaiting your ncurses port...

I'm aiming to get a first cut out this week (just the very basics) to
get things moving, but it will take a while to get it all done,
especially the fiddly bits. I'm going to concentrate on the core
ncurses functionality. I'm not interested in panels or forms (better
done in Ruby IMHO).

> Is it possible for you to document the procedure, how you go about doing
> it, so one can follow that for another port, rather than be totally
> lost.

Sure, as long as you don't mind long-winded stream of consciousness :)

Regards,
Sean
Charles Oliver N. (Guest)
on 2008-11-06 02:55
(Received via mailing list)
Sean O'Halpin wrote:
>> it, so one can follow that for another port, rather than be totally
>> lost.
>
> Sure, as long as you don't mind long-winded stream of consciousness :)

Oh and please, please guys, update anything that's not well-documented
here:

http://kenai.com/projects/ruby-ffi/pages/Home

It's wide open...let me know if you can't edit.

- Charlie
R. K. (Guest)
on 2008-11-06 05:47
Sean O'halpin wrote:
>>
>
> I'm aiming to get a first cut out this week (just the very basics) to
> get things moving, but it will take a while to get it all done,
> especially the fiddly bits. I'm going to concentrate on the core
> ncurses functionality. I'm not interested in panels or forms (better
> done in Ruby IMHO).

I am using panels and forms - *heavily*. I understand your last
statement to mean implementing p & f's in ruby itself, avoiding what
ncurses provides. Has anyone actually done that -- i would like to see
it and see how its better and if it can be reused.
Dominic S. (Guest)
on 2008-11-06 19:33
(Received via mailing list)
>  This blog posting has the most information I have seen so far on FFI and
ruby


http://pluskid.lifegoo.com/?p=370
Daniel DeLorme (Guest)
on 2008-11-11 02:05
(Received via mailing list)
Nit K. wrote:
>     ARCH = case CPU.downcase
>     when /i?86|x86|i86pc|powerpc/
>       "i386"
>     when /amd64|x86_64/
>       "x86_64"

hmm... that code is broken. "x86_64" will match the first regex and
result in "i386"
Clifford H. (Guest)
on 2008-11-11 06:50
(Received via mailing list)
Charles Oliver N. wrote:
> Actually structs are already supported!

Sorry if you see this twice, posting difficulties.

I'm having a bash at wrapping freetds. This library
has a Context struct which contains three callback
function pointers. POLS says these should look like
this:

 callback :handle_message, [ :pointer, :pointer, :pointer ], :in   #
TDSCONTEXT, TSSSOCKET, TDSMESSAGE
 callback :handle_int, [ :pointer ], :in     # void*

 class Context < FFI:Struct
   layout \
     :locale, :pointer, 0,                   # TDSLOCALE
     :parent, :pointer, 4,                   # void *
     :msg_handler, :handle_message, 8,       # callback(TDSCONTEXT,
TDSSOCKET, TDSMESSAGE)
     :err_handler, :handle_message, 12,      # callback(TDSCONTEXT,
TDSSOCKET, TDSMESSAGE)
     :int_handler, :handle_int, 16           # callback(void*)
 end

Am I on the right track, or is this not possible yet?

Clifford H..
Charles Oliver N. (Guest)
on 2008-11-11 06:55
(Received via mailing list)
Clifford H. wrote:
> callback :handle_message, [ :pointer, :pointer, :pointer ], :in   #
> TDSSOCKET, TDSMESSAGE)
>     :int_handler, :handle_int, 16           # callback(void*)
> end
> Am I on the right track, or is this not possible yet?

There's no support at the moment for callbacks-in-structs, but it's just
a bug report (and ideally a patch) away :)

http://kenai.com/projects/ruby-ffi

- Charlie
Clifford H. (Guest)
on 2008-11-11 07:30
(Received via mailing list)
Charles Oliver N. wrote:
> There's no support at the moment for callbacks-in-structs, but it's just
> a bug report (and ideally a patch) away :)
> http://kenai.com/projects/ruby-ffi

Thanks Charlie. I registered at Kenai.com and can log in there
though only by emaill address, username login doesn't work), but
the Bugzilla instalce doesn't recognise the login details at all.
I'll try again later, in case there's a delayed processing step.
I can't see any separate registration for Bugzilla...

Clifford H..
Charles Oliver N. (Guest)
on 2008-11-11 07:58
(Received via mailing list)
Clifford H. wrote:
> Charles Oliver N. wrote:
>> There's no support at the moment for callbacks-in-structs, but it's
>> just a bug report (and ideally a patch) away :)
>> http://kenai.com/projects/ruby-ffi
>
> Thanks Charlie. I registered at Kenai.com and can log in there
> though only by emaill address, username login doesn't work), but
> the Bugzilla instalce doesn't recognise the login details at all.
> I'll try again later, in case there's a delayed processing step.
> I can't see any separate registration for Bugzilla...

I'll make sure the Kenai guys know about the problem. Thanks for giving
it a shot. If nothing else, email the dev mailing list.

- Charlie
Luc H. (Guest)
on 2008-11-11 10:58
(Received via mailing list)
On 11 nov. 08, at 06:27, Clifford H. wrote:

> Thanks Charlie. I registered at Kenai.com and can log in there
> though only by emaill address, username login doesn't work)

Speaking of that Kenai thing, am I the only one who think it's a
*major* pain in the derrière to use ?
Clifford H. (Guest)
on 2008-11-11 12:15
(Received via mailing list)
Charles Oliver N. wrote:
> Clifford H. wrote:
>> callback :handle_message, [ :poi...
>> class Context < FFI:Struct
>>   layout \
>>     :msg_handler, :handle_message, 8,
> There's no support at the moment for callbacks-in-structs

While I'm on a roll, my example above creates a new type
(handle_message),
and the ability to define other kinds of types similarly would be
excellent.
For example, FreeTDS also has a string struct that contains a char* and
a
length, and it uses that type liberally. A typedef would rock, and the
above
example would just create a function-pointer typedef.

In a previous life, we implemented a generic structure pack/unpack
subsystem.
It had a table for every type containing the alignment and size of that
type.
Some architectures need more than one table depending on compile
options, but
we supported 30 different architectures using tables whose values were
filled
in by the C compiler. The rule of C don't allow re-ordering structure
elements,
or at least, the require that two structures that start with the same
elements
use the same alignment for those elements.

You can discover the alignment of a type by declaring a struct z { char
x; TYPE y; }
and asking for offsetof(struct z, y); Mere byte-counting and padding
suffices
to calculate alignment for the members of any structure. Nested
structures are
padded to the largest of either the maximum alignment of the members, or
some
size specific to the architecture. This can also be populated into the
alignment
tables by the C compiler.

If FFI could include such a facility for Struct definitions, that would
be magic,
as the current system is almost impossible to use for cross-platform
support.
Unless you use the awkward templating system, of course... That requires
that
you have the development versions of the target libraries when you want
to gem
install a wrapper.

Just some random thoughts, anyhow. I might have time to contribute code
to this
effort in December.

Clifford H..
brett (Guest)
on 2008-11-11 18:36
(Received via mailing list)
I get the following when I try to compile on windows - any
suggestions?  Not sure what srcdir should be or where I should set it.

========================================================================

C:\ffi>gem install ffi-0.1.0.gem
Building native extensions.  This could take a while...
ERROR:  Error installing ffi-0.1.0.gem:
        ERROR: Failed to build gem native extension.

C:/ruby/bin/ruby.exe extconf.rb install ffi-0.1.0.gem
creating Makefile

nmake

Microsoft (R) Program Maintenance Utility Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

makefile(182) : fatal error U1052: file '$(srcdir)/ffi.mk' not found
Stop.


Gem files will remain installed in C:/ruby/lib/ruby/gems/1.8/gems/
ffi-0.1.0 for inspection.
Results logged to C:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.0/ext/
gem_make.out
Charles Oliver N. (Guest)
on 2008-11-11 21:01
(Received via mailing list)
Clifford H. wrote:
> Just some random thoughts, anyhow. I might have time to contribute code
> to this
> effort in December.

Sounds like you've dealt with a lot of this stuff before. Can you
re-post this to the main FFI mailing list?

- Charlie
Charles Oliver N. (Guest)
on 2008-11-11 21:41
(Received via mailing list)
brett wrote:
> I get the following when I try to compile on windows - any
> suggestions?  Not sure what srcdir should be or where I should set it.

There needs to be a pre-built gem for win32, but we haven't published
one yet. I think someone's working on getting it wired up.

- Charlie
Luis L. (Guest)
on 2008-11-13 15:05
(Received via mailing list)
On Nov 11, 5:53 am, Luc H. <removed_email_address@domain.invalid> wrote:
> On 11 nov. 08, at 06:27, Clifford H. wrote:
>
> > Thanks Charlie. I registered at Kenai.com and can log in there
> > though only by emaill address, username login doesn't work)
>
> Speaking of that Kenai thing, am I the only one who think it's a  
> *major* pain in the derrière to use ?
>

Nop, count me on that too.

Tried to use bugzilla to provide some patches, and is a no go, only
fall back to mailing list seems the alternative.
Paul B. (Guest)
on 2008-11-13 19:32
(Received via mailing list)
Very cool!

I don't know how I missed this before, somehow slipped past my mental
filter (and now everything you were talking about at RubyConf starts to
make a lot more sense... :).

One question: a complaint I've had with DL is that it requires that I
specify the dynamic library that contains symbols I'm interested in.  Is
this necessary with FFI?  Can FFI be used to call functions in the Ruby
standard library when Ruby is not build with --enable-shared?

If so, I'd be interested in using this for Ludicrous to avoid having to
hard-code all the function pointers to the Ruby API functions I want to
call.  That would mean, though, I would need FFI to be able to give me a
real function pointer (so I can pass the pointer to libjit.. I can
imagine this being useful for other cases as well).

Paul
Sean O. (Guest)
on 2008-11-18 16:56
(Received via mailing list)
On Wed, Nov 5, 2008 at 5:11 PM, Nit K. <removed_email_address@domain.invalid> 
wrote:
>
> Eagerly awaiting your ncurses port...
>
> Is it possible for you to document the procedure, how you go about doing
> it, so one can follow that for another port, rather than be totally
> lost.

Hi,

I've just uploaded the ~very raw~ initial port of the core ncurses lib
to ruby-ffi. You can find it at:

git://github.com/seanohalpin/ffi-ncurses.git

This works with ffi-0.0.2 (which I got from the kenai project) - 0.0.1
left out typedefs for short ints.

To use:

require 'rubygems'
require 'ffi'
require 'ffi-ncurses

notes:
- example.rb shows basic usage
- to access stdscr, use the return value from NCurses.initscr
- I have not implemented any variadic functions (not supported by libffi
AFAIK)
- there's also a wrapper for libdl called dlfcn.rb
- there's an experimental script (ex01-ffi-dl-stdscr.rb) which uses
dlfcn.rb to access the exported variables from the ncurses shared lib
(stdscr, curscr, etc.). This also shows how to access values returned
through pointers (see NCurses.wattr_get).

There are some rambling notes on implementation in
notes-on-ffi-curses.org. I'll tidy these up when I get some spare
time. Basically, I wrote a script
(generate-ffi-ncurses-function-signatures.rb) that mechanically
recovered the function signatures from the ncurses.h on my machine and
added some constants. I wasted a lot of time figuring out how to
access exported variables.

I've only just done a final test on OS X Tiger - haven't checked Linux
yet (should work but...)

I have not set up a project for this yet so please send any bug
reports, questions, etc. directly to me for the moment.

There are no tests - I wonder if anyone has any ideas how best to
write specs/tests for checking this kind of output?

Regards,
Sean
Radosław B. (Guest)
on 2008-11-19 00:50
(Received via mailing list)
I tried to run all examples but no one worked. It prints tons of:
/usr/lib/ruby/gems/1.8/gems/ffi-0.1.1/lib/ffi/ffi.rb:232: warning:
instance variable @ffi_convention not initialized

and last line:
example-printw-variadic.rb:7: undefined method `printw' for
NCurses:Module (NoMethodError)

--
Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog
Sean O. (Guest)
on 2008-11-19 01:56
(Received via mailing list)
2008/11/18 Rados³aw Bu³at <removed_email_address@domain.invalid>:
>
> Rados³aw Bu³at
> http://radarek.jogger.pl - mój blog
>
Hi Rados³aw,

my wrapper requires ffi-0.2.0 which you'll need to install from
source. You can get the source from the kenai project site at
http://kenai.com/projects/ruby-ffi (though you'll need to install
mercurial to get it).

The example ex01-ffi-dl-stdscr.rb does not currently work under Linux
- I'm looking into that. The only platform I know it works on is Mac
OS X Tiger. The other examples work on both OS X and Linux (with
ffi-0.2.0).

Regards,
Sean
Sean O. (Guest)
on 2008-11-19 02:02
(Received via mailing list)
On Tue, Nov 18, 2008 at 11:56 PM, Sean O'Halpin 
<removed_email_address@domain.invalid>
wrote:
> my wrapper requires ffi-0.2.0
I notice I got the version number wrong in my earlier post (I said
0.0.2) - apologies for any confusion caused.

Regards,
Sean
Charles Oliver N. (Guest)
on 2008-11-19 06:58
(Received via mailing list)
Sean O'Halpin wrote:
> I've just uploaded the ~very raw~ initial port of the core ncurses lib
> to ruby-ffi. You can find it at:
>
> git://github.com/seanohalpin/ffi-ncurses.git
>
> This works with ffi-0.0.2 (which I got from the kenai project) - 0.0.1
> left out typedefs for short ints.

Very cool :) The early examples worked fine in JRuby but this one seems
to blow up in MemoryPointer. I'll make sure wmeissner knows about it.

I know he's taken a step back from adding new features in 0.2.0 to make
sure both JRuby and ruby-ffi are compatible again. Hopefully 0.2.0 will
be out soon along with JRuby 1.1.6.

- Charlie
Sean O. (Guest)
on 2008-12-03 04:36
(Received via mailing list)
2008/11/18 Rados³aw Bu³at <removed_email_address@domain.invalid>:
> I tried to run all examples but no one worked.
Hi Rados³aw,

If you're still interested in ffi-ncurses, please install the new
version of ffi (0.2.0), update your copy of ffi-ncurses and try the
examples again.

I've tested on Mac OSX Tiger (terminal, iTerm, xterm & urxvt) and
Ubuntu 8.04 (gnome-terminal, xterm) with ruby 1.8.6. With the usual
provisos about different terminal types supporting different feature
sets, all the examples work as they should.

Regards,
Sean
This topic is locked and can not be replied to.