Forum: IronRuby IronRuby FFI

I know that we've discussed this in the past, but I'm interested in 
doing it
for two reasons:
1. We use mono with a bridge to ObjectiveC and Cocoa, and we'd like to
investigate libffi via mono as a potential replacement for our current
bridge.
2. I'm interested just for the sake of learning more about FFI.

Mono appears to have had a libffi implementation that was later removed, 
so
I think I have a place to start. However, I'm not sure that's the right
starting point. Does anyone have a suggestion for how to get started? 
I've
been taking a look at libffi and DllImport, but I'm not sure if those 
are
the right directions, something else, or what.

Thanks,

Ryan Riley

Email: ryan.riley@panesofglass.org
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/

Ryan,

I'm right there with you, only I looked at JFFI for inspiration (didn't 
know
mono had anything - could you share more about that?). In fact, In my
infinite laziness, I posted a job for a couple hundred bucks on
Rent-A-Coder, hoping someone could essentially port JFFI to C#, so I 
could
focus on writing the actually IronRuby library... but nothing came of 
that.

I'm tempted to suck it up and start coding this myself. Would you be
interested in working together?  I figured I'd take the approach of
essentially writing "NFFI", and then write an IronRuby lib around that.

-Charles

Couldn't this be implemented via P/Invoke? Would P/Invoke work without 
the
attributes required in C#?

--
Will Green
http://hotgazpacho.org/


On Fri, Aug 20, 2010 at 4:49 PM, Charles Strahan <

That's something I'm curious about too. I don't know whether or not 
P/Invoke
alone can match the functionality of libffi. I'd have to look through 
some
of libffi and see if there's anything can't be accomplished in C#.

If everything can be accomplished in C#, I'd go that route. We could 
emit
types at runtime that have the necessary DLLImport attributes and such
(something I've already done for little projects of mine).

-Charles

I would also recommend to look at Python's c-types and especially 
IronPython's implementation:

http://github.com/ironruby/ironruby/tree/master/La...

Many of the features might be similar to Ruby's FFI and it would be good 
to share the implementation with IronPython when possible.

Tomas

From: ironruby-core-bounces@rubyforge.org 
[mailto:ironruby-core-bounces@rubyforge.org] On Behalf Of Charles 
Strahan
Sent: Friday, August 20, 2010 2:35 PM
To: ironruby-core@rubyforge.org
Subject: Re: [Ironruby-core] IronRuby FFI

That's something I'm curious about too. I don't know whether or not 
P/Invoke alone can match the functionality of libffi. I'd have to look 
through some of libffi and see if there's anything can't be accomplished 
in C#.

If everything can be accomplished in C#, I'd go that route. We could 
emit types at runtime that have the necessary DLLImport attributes and 
such (something I've already done for little projects of mine).

-Charles
On Fri, Aug 20, 2010 at 4:14 PM, Will Green 
<will@hotgazpacho.org<mailto:will@hotgazpacho.org>> wrote:
Couldn't this be implemented via P/Invoke? Would P/Invoke work without 
the attributes required in C#?

--
Will Green
http://hotgazpacho.org/

On Fri, Aug 20, 2010 at 4:49 PM, Charles Strahan 
<charles.c.strahan@gmail.com<mailto:charles.c.strahan@gmail.com>> wrote:
Ryan,

I'm right there with you, only I looked at JFFI for inspiration (didn't 
know mono had anything - could you share more about that?). In fact, In 
my infinite laziness, I posted a job for a couple hundred bucks on 
Rent-A-Coder, hoping someone could essentially port JFFI to C#, so I 
could focus on writing the actually IronRuby library... but nothing came 
of that.

I'm tempted to suck it up and start coding this myself. Would you be 
interested in working together?  I figured I'd take the approach of 
essentially writing "NFFI", and then write an IronRuby lib around that.

-Charles

On Fri, Aug 20, 2010 at 2:33 PM, Ryan Riley 
<ryan.riley@panesofglass.org<mailto:ryan.riley@panesofglass.org>> wrote:
I know that we've discussed this in the past, but I'm interested in 
doing it for two reasons:
1. We use mono with a bridge to ObjectiveC and Cocoa, and we'd like to 
investigate libffi via mono as a potential replacement for our current 
bridge.
2. I'm interested just for the sake of learning more about FFI.

Mono appears to have had a libffi implementation that was later removed, 
so I think I have a place to start. However, I'm not sure that's the 
right starting point. Does anyone have a suggestion for how to get 
started? I've been taking a look at libffi and DllImport, but I'm not 
sure if those are the right directions, something else, or what.

Thanks,

Ryan Riley

Email: ryan.riley@panesofglass.org<mailto:ryan.riley@panesofglass.org>
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/
_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org<mailto:Ironruby-core@rubyforge.org>
http://rubyforge.org/mailman/listinfo/ironruby-core


_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org<mailto:Ironruby-core@rubyforge.org>
http://rubyforge.org/mailman/listinfo/ironruby-core

I'll have to take a look at that. If it's cross platform, it could be a
great resource.

-Charles

On Fri, Aug 20, 2010 at 4:57 PM, Tomas Matousek <

Charles, I'm happy to work with you to get this done. I'm getting close 
to finishing some projects and will have more time to work on it in a 
few weeks. I will send the info I got from the mono-devel list. 
Where/how do you want to start?

Ryan

Sent from my iPhone

From the mono-devel list:

On Thu, 2010-08-19 at 10:35 -0700, Ryan Riley wrote:
> Has anyone created or investigated Mono support for libffi?

Once upon a time, Mono used libffi directly.  It was removed in r724
(git SHA1 ID d0cd6059c1b2edad12eb67cb8e64b3cd187be1b1) on 2001-09-05
(and earlier).  Unfortunately, the commit message is useless, but iirc
the reason for removing it was because it was significantly slower than
what Mono could do itself.

> I'd like to contribute this to help support Ruby-FFI for IronRuby.

I imagine IronRuby doesn't require it's own FFI, it would just use the
usual .NET FFI of DllImport, no?

As for supporting Ruby-FFI, perhaps you can extend the existing Mono FFI
support to support Ruby?  I don't know what would be involved...

- Jon


Sent from my iPhone

Ryan,

Sorry for the long delay - I meant to give it some thought before I got 
back
to you... and know it's been quite some time.

I think it would be a good idea to replicate JFFI, using P/Invoke 
directly,
if possible (as opposed to P/Invoking libffi 
<http://www.cygwin.org/libffi/>).
That would give us a good separation of concerns and a reusable library, 
and
possibly an easy way to port any Java/JRuby code that uses JFFI to 
C#/.NET
too.

I'm about to set up an NFFI repo at http://github.com/cstrahan/nffi. - I
suppose you could fork it and send me pull requests (unless you have a
better workflow in mind - I'm definitely not a git guru). I've been 
learning
C/C++ the last couple months, so I should be able to write simple DLL to 
run
our tests against. I think I'll take a TDD approach to driving out the 
C#
lib. Once we have NFFI working, it should be relatively straightforward 
to
expose that to the IronRuby runtime. I'll try to get something pushed 
out to
my repo by the end of tomorrow - I'll keep you in the loop.

That's what I have in mind, but I'm open to suggestions.

-Charles

Sounds good to me!

Sent from my iPhone

I'm also taking a look at IronPython's CTypes implementation, under 
Tomas'
advice. I've noticed that their
MemoryHolder<http://github.com/ironruby/ironruby/blob/master/La...
derives from
CriticalFinalizerObject<http://msdn.microsoft.com/en-us/library/system.run...,
which led me to the discovery of Constrained Execution
Regions<http://msdn.microsoft.com/en-us/library/ms228973.aspx>
.

I sent an inquiry to the IronPython mailing
list<http://lists.ironpython.com/pipermail/users-ironpy...
the use CFO, and about CER in general, as I haven't had any
exposure to either, and the MSDN docs are a little daunting. If anyone 
here
would like to give an explanation of either one, that would be awesome.

Any experience with either of those, Ryan?

-Charles

I've not heard of any of those. I started looking at ctypes but never 
got far.

Sent from my iPhone

I've decided to not be lazy and do a little spelunking into CER's - it's
rather interesting stuff. I found a pretty good article here:
http://msdn.microsoft.com/en-us/magazine/cc163716.aspx

In laymen's terms, it looks like CER's provide reliability where
asynchronous exceptions may be thrown: OutOfMemoryException,
StackOverflowException, and ThreadAbortException. In the case of
MemoryHolder, this is important because such exceptions could preempt 
the
storage of IntPtrs corresponding to allocated memory and/or the 
deallocation
of memory within finalizers - both resulting in memory leaks. As I 
imagined,
this will be something we'll want to incorporate in our FFI impl.

<http://msdn.microsoft.com/en-us/magazine/cc163716....

Here's Dino Viehland's response regarding the CERs:

~~~~~~~~

Objects which inherit from CriticalFinalizerObject basically have their
finalize method turned into a CER.  The finalizer method is also JITed
before any instances are created so the finalizer is guaranteed to be
runnable.



In generally CERs are all about ensuring that the VM won’t cause any
unexpected failure points in your code.  These can be introduced because 
the
VM does something lazily (including JITing methods) and doing the work 
might
fail due to insufficient resources.  Or it can also be due to thread 
abort.
Because these objects are responsible for freeing up resources we don’t 
want
any unexpected failures to be injected otherwise the resources would 
leak.



So for example MemoryHolder also has a CER in the constructor – this 
ensures
that we don’t take a ThreadAbort between the CallocCall, storing the 
value
in _data, or assigning to _ownsData.  This will all complete or not 
complete
so that our state is consistent when the finalizer is run.  It also 
makes
sure that any work the CLR needs to perform to call 
NativeFunctions.Calloc
is all performed before we enter the CER so that we don’t get an out of
memory exception while calling or returning from it.



For most environments it’s not super important that this is gotten right 
–
but if you run in a process which needs long uptime, is resource
constrained, and/or uses thread abort a lot (SQL server being an example 
of
all 3) it’s important that this is correct.  I happened to work on this
feature when I was on the CLR team so it came rather naturally to me to 
get
it right J
~~~~~~~~~

-Charles


On Mon, Oct 4, 2010 at 12:11 AM, Charles Strahan <

@Charles

Have you made any progress here? If so, what's your repo? I'm looking 
into
this again this morning.

Cheers,

Ryan Riley

Email: ryan.riley@panesofglass.org
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/


On Sun, Oct 3, 2010 at 11:47 PM, Charles Strahan <

Charles,



Have you made any progress with FFI? I have finally finished with all of 
my
craziness. Of course, the holidays are approaching, but I would love to
investigate this further in the new year.



~ Ryan



From: ironruby-core-bounces@rubyforge.org
[mailto:ironruby-core-bounces@rubyforge.org] On Behalf Of Charles 
Strahan
Sent: Sunday, October 03, 2010 11:47 PM
To: ironruby-core@rubyforge.org
Subject: Re: [Ironruby-core] IronRuby FFI



Here's Dino Viehland's response regarding the CERs:



~~~~~~~~

Objects which inherit from CriticalFinalizerObject basically have their
finalize method turned into a CER.  The finalizer method is also JITed
before any instances are created so the finalizer is guaranteed to be
runnable.



In generally CERs are all about ensuring that the VM won't cause any
unexpected failure points in your code.  These can be introduced because 
the
VM does something lazily (including JITing methods) and doing the work 
might
fail due to insufficient resources.  Or it can also be due to thread 
abort.
Because these objects are responsible for freeing up resources we don't 
want
any unexpected failures to be injected otherwise the resources would 
leak.



So for example MemoryHolder also has a CER in the constructor - this 
ensures
that we don't take a ThreadAbort between the CallocCall, storing the 
value
in _data, or assigning to _ownsData.  This will all complete or not 
complete
so that our state is consistent when the finalizer is run.  It also 
makes
sure that any work the CLR needs to perform to call 
NativeFunctions.Calloc
is all performed before we enter the CER so that we don't get an out of
memory exception while calling or returning from it.



For most environments it's not super important that this is gotten right 
-
but if you run in a process which needs long uptime, is resource
constrained, and/or uses thread abort a lot (SQL server being an example 
of
all 3) it's important that this is correct.  I happened to work on this
feature when I was on the CLR team so it came rather naturally to me to 
get
it right J

~~~~~~~~~



-Charles





On Mon, Oct 4, 2010 at 12:11 AM, Charles Strahan
<charles.c.strahan@gmail.com> wrote:

I've decided to not be lazy and do a little spelunking into CER's - it's
rather interesting stuff. I found a pretty good article here:

http://msdn.microsoft.com/en-us/magazine/cc163716.aspx



In laymen's terms, it looks like CER's provide reliability where
asynchronous exceptions may be thrown: OutOfMemoryException,
StackOverflowException, and ThreadAbortException. In the case of
MemoryHolder, this is important because such exceptions could preempt 
the
storage of IntPtrs corresponding to allocated memory and/or the 
deallocation
of memory within finalizers - both resulting in memory leaks. As I 
imagined,
this will be something we'll want to incorporate in our FFI impl.



-Charles



On Sun, Oct 3, 2010 at 11:51 PM, Ryan Riley 
<ryan.riley@panesofglass.org>
wrote:

I've not heard of any of those. I started looking at ctypes but never 
got
far.

Sent from my iPhone


On Oct 3, 2010, at 9:27 PM, Charles Strahan 
<charles.c.strahan@gmail.com>
wrote:

I'm also taking a look at IronPython's CTypes implementation, under 
Tomas'
advice. I've noticed that their MemoryHolder
<http://github.com/ironruby/ironruby/blob/master/La...
thon.Modules/_ctypes/MemoryHolder.cs>  class derives from
CriticalFinalizerObject
<http://msdn.microsoft.com/en-us/library/system.run...
.criticalfinalizerobject.aspx> , which led me to the discovery of
Constrained Execution Regions
<http://msdn.microsoft.com/en-us/library/ms228973.aspx> .



I sent an inquiry to the IronPython mailing list
<http://lists.ironpython.com/pipermail/users-ironpy...
757.html>  regarding the use CFO, and about CER in general, as I haven't 
had
any exposure to either, and the MSDN docs are a little daunting. If 
anyone
here would like to give an explanation of either one, that would be 
awesome.



Any experience with either of those, Ryan?



-Charles



On Sun, Oct 3, 2010 at 10:39 PM, Ryan Riley 
<ryan.riley@panesofglass.org>
wrote:

Sounds good to me!

Sent from my iPhone


On Oct 3, 2010, at 8:04 PM, Charles Strahan 
<charles.c.strahan@gmail.com>
wrote:

Ryan,



Sorry for the long delay - I meant to give it some thought before I got 
back
to you... and know it's been quite some time.



I think it would be a good idea to replicate JFFI, using P/Invoke 
directly,
if possible (as opposed to P/Invoking libffi 
<http://www.cygwin.org/libffi/>
). That would give us a good separation of concerns and a reusable 
library,
and possibly an easy way to port any Java/JRuby code that uses JFFI to
C#/.NET too.



I'm about to set up an NFFI repo at http://github.com/cstrahan/nffi. - I
suppose you could fork it and send me pull requests (unless you have a
better workflow in mind - I'm definitely not a git guru). I've been 
learning
C/C++ the last couple months, so I should be able to write simple DLL to 
run
our tests against. I think I'll take a TDD approach to driving out the 
C#
lib. Once we have NFFI working, it should be relatively straightforward 
to
expose that to the IronRuby runtime. I'll try to get something pushed 
out to
my repo by the end of tomorrow - I'll keep you in the loop.



That's what I have in mind, but I'm open to suggestions.



-Charles





On Sat, Aug 21, 2010 at 9:23 PM, Ryan Riley 
<ryan.riley@panesofglass.org>
wrote:

Charles, I'm happy to work with you to get this done. I'm getting close 
to
finishing some projects and will have more time to work on it in a few
weeks. I will send the info I got from the mono-devel list. Where/how do 
you
want to start?



Ryan

Sent from my iPhone


On Aug 20, 2010, at 1:49 PM, Charles Strahan 
<charles.c.strahan@gmail.com>
wrote:

Ryan,

I'm right there with you, only I looked at JFFI for inspiration (didn't 
know
mono had anything - could you share more about that?). In fact, In my
infinite laziness, I posted a job for a couple hundred bucks on
Rent-A-Coder, hoping someone could essentially port JFFI to C#, so I 
could
focus on writing the actually IronRuby library... but nothing came of 
that.

I'm tempted to suck it up and start coding this myself. Would you be
interested in working together?  I figured I'd take the approach of
essentially writing "NFFI", and then write an IronRuby lib around that.

-Charles



On Fri, Aug 20, 2010 at 2:33 PM, Ryan Riley 
<ryan.riley@panesofglass.org>
wrote:

I know that we've discussed this in the past, but I'm interested in 
doing it
for two reasons:

1. We use mono with a bridge to ObjectiveC and Cocoa, and we'd like to
investigate libffi via mono as a potential replacement for our current
bridge.

2. I'm interested just for the sake of learning more about FFI.



Mono appears to have had a libffi implementation that was later removed, 
so
I think I have a place to start. However, I'm not sure that's the right
starting point. Does anyone have a suggestion for how to get started? 
I've
been taking a look at libffi and DllImport, but I'm not sure if those 
are
the right directions, something else, or what.



Thanks,


Ryan Riley

Email: ryan.riley@panesofglass.org
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/

_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core



_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core


_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core



_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core


_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core



_______________________________________________
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core

Ryan,

Have you made any progress with FFI? I have finally finished with all of 
my
> craziness. Of course, the holidays are approaching, but I would love to
> investigate this further in the new year.


I haven't written any code yet (I've been rather swamped lately). I took 
a
look at the API (http://rdoc.info/github/ffi/ffi) that we'd have to 
provide
for a full implementation, and it looks rather daunting. I'm still 
reading
through JFFI, JRuby and RubyFFI trying to wrap my head around all of it.

I'm not sure how I would get started right now. However, I'm going to 
spend
some time the next couple days trying to figure out how I might attack 
this.
I'll be sure to keep this thread up-to-date with my thoughts/feelings on
this.

Feel free to take the lead if you have a clearer vision of how this 
needs to
get done; otherwise, I'll update you on Tuesday or Wednesday.


Cheers,
-Charles