Dynamic code generation and linking


#1

I’m very interested in being able to dynamically generate, compile,
and link in c code (and potentially other languages, including
assembly - but c is my primary interest), from within Ruby. When I
say “link in”, I mean have the compiled code immediately become
available for use by the Ruby script that asked for the compilation.

So, I’m imagining something like…

Creating a parameterless c function.

compiler.compile( ‘printf( “Rock on, dudes!” );’ ).call

A c function with an integer argument.

proc = compiler.compile( ‘x’ => Int, ‘printf( “%d * %d = %d”, x, x,
x*x);’ )
(1…10).each {|x| proc.call(x)}

A module that encapsulated this (particularly as part of the Ruby
standard distribution) would allow a “pure ruby” module to have
native speed [1].

It would also be awesome for dynamic code generation for building an
algorithm optimised for a very specific situation. Coming from an
image processing background, I find the idea quite exciting :slight_smile: As an
example, I could be using artificial evolution to grow functions that
evaluate to a pretty picture. To evaluate these functions at maximum
speed, I could compile them down to c, and execute them.

Another example would be creating a parser from a BNF like notation.
It should be possible (making use of this module) to specify a
grammar with a Rubyish domain specific language, and then compile
that to native code and immediately make use of it from within the
same Ruby program.

A further possibility is that it could pull all that pesky linking
with external libraries (well I don’t get on with it anyway, but it’s
not really my bag) in to the domain of Ruby, and make it a fun thing
to explore from within IRB, for example. An offshoot from that would
be an explorative SWIG like system: a simple c parser that you can
point at a library’s header file, allowing you to start hacking about
with the c library as if you were back programming BASIC on your BBC
[2] :slight_smile:

Has anyone come across work in this direction, or is anyone working
on it themselves? Could anyone comment on difficulties involved, or
serious practical problems? I think the major issues are likely to be:

Interfacing with the native linker.
Passing stuff in and out of the c easily.
Giving the c compiler header files (or bundles, or whatever it needs

on OS X).

It’s likely I’m completely missing many problems though - I’m coming
to this having never written a c extension for Ruby!

My dream would be to encapsulate all of the platform specific nitty
gritty in one place, and so make it reasonably simple for anyone with
some c and Ruby knowledge to just dive in and start writing super
fast code, or linking with a library they want to play with.

Sorry for a long post. I hope it will give you ideas, or stimulate
interest.

Thank you for your time,
Benjohn B.

[1] Although one would want a mechanism to be able to cache compiled
code between runs eventually, as the compile time could become annoying.

[2] I think Ruby is such an enjoyable experience for me because it
recreates the fun, and frankly joy, I used to have with BASIC, when I
first learnt to program, but it allows me to concisely specify my
ideas in ways that I’ve come to need after 20 years of programming.
What I want to do here is get close to having BBC BASIC’s build in
assembly :slight_smile: Total expressiveness, and raw touching the metal speed
and control.


#2

Benjohn B. wrote:

Another example would be creating a parser from a BNF like notation.
It should be possible (making use of this module) to specify a
grammar with a Rubyish domain specific language, and then compile
that to native code and immediately make use of it from within the
same Ruby program.

I had exactly this idea recently! I was particularly interested in
lexing, because that’s the bottleneck in some parsers I’ve written
recently. And conveniently, re2c (google it) is an existing re->c
compiler that generates optimized C from a regular expression
description. Sounds perfect for the task.

A further possibility is that it could pull all that pesky linking
with external libraries (well I don’t get on with it anyway, but it’s
not really my bag) in to the domain of Ruby, and make it a fun thing
to explore from within IRB, for example. An offshoot from that would
be an explorative SWIG like system: a simple c parser that you can
point at a library’s header file, allowing you to start hacking about
with the c library as if you were back programming BASIC on your BBC
[2] :slight_smile:

Wow, I had this idea too – except I had the idea of getting function
signatures from a .so’s debugging information, so you wouldn’t need the
C header file.

I’m really interested in any work you do in this area. I’d like to be
working on this, but I am heavily invested in other problems at the
moment.

Regards,
Josh


#3

On Thu, 23 Feb 2006, Benjohn B. wrote:

could be using artificial evolution to grow functions that evaluate to a
my bag) in to the domain of Ruby, and make it a fun thing to explore from
Passing stuff in and out of the c easily.

the fun, and frankly joy, I used to have with BASIC, when I first learnt to
program, but it allows me to concisely specify my ideas in ways that I’ve
come to need after 20 years of programming. What I want to do here is get
close to having BBC BASIC’s build in assembly :slight_smile: Total expressiveness, and
raw touching the metal speed and control.

you want to check out rubyinline and ruby/dl. the later is an interface
to
the dynamic linker which allows you load *.so files and call routines in
them
directly. the former spews c code, compiles it, and loads it from pure
ruby.
fyi - ‘require’ is an interface to the dynamic linker, so long as
the dll
being loaded is of a certain type.

cheers.

-a


#4

On Feb 22, 2006, at 4:51 PM, Benjohn B. wrote:

an algorithm optimised for a very specific situation. Coming from
within the same Ruby program.
Has anyone come across work in this direction, or is anyone working

assembly :slight_smile: Total expressiveness, and raw touching the metal speed
and control.

Sounds like you’re looking for RubyInline:
class MyTest

def factorial(n)
f = 1
n.downto(2) { |x| f *= x }
f
end

inline do |builder|
builder.c "
long factorial_c(int max) {
int i=max, result=1;
while (i >= 2) { result *= i–; }
return result;
}"
end
end

http://www.zenspider.com/ZSS/Products/RubyInline/


#5

Benjohn B. wrote:

Another example would be creating a parser from a BNF like notation.
It should be possible (making use of this module) to specify a
grammar with a Rubyish domain specific language, and then compile
that to native code and immediately make use of it from within the
same Ruby program

I worked on a compiler backend, which actually does this. It takes a
symbolic expression represented by a nested array and compiles it to
assembly code, which is processed by gas. This can be required
immediately as ruby extension. The compiler is stack based as this
simplifies the algorithm a lot. At the time local variables,
parameters, literal floats and many other things are missing, but it
works.

be an explorative SWIG like system: a simple c parser that you can
point at a library’s header file, allowing you to start hacking about
with the c library as if you were back programming BASIC on your BBC
[2] :slight_smile:

Implementing a c parser is cumbersome.

Wow, I had this idea too – except I had the idea of getting function
signatures from a .so’s debugging information, so you wouldn’t need the
C header file.

This approach seems much easier. Maybe something similar to Ruby/DL.

I’m really interested in any work you do in this area. I’d like to be
working on this, but I am heavily invested in other problems at the moment.

Regards,
Josh

Cheers,
Matthias


#6

If you’re interested in the CLR / .NET, my RubyCLR bridge uses Ruby to
generate CIL (the CLR’s intermediate language) instructions on the fly
to
create shims that glue Ruby to the CLR. It also lets me do cool stuff
like
refactor my CIL, generate macros etc.

-John
http://www.iunknown.com