# Dynamic code generation and linking

**URL:** <https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060>\
**Category:** Ruby\
**Created:** [February 22, 2006, 10:54pm UTC](https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060 "2006-02-22T22:54:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benjohn\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e5b9ba/32.png) [@Benjohn\_B](https://www.ruby-forum.com/u/Benjohn_B)\
**Post date:** [February 22, 2006, 10:54pm UTC](https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060/1 "2006-02-22T22:54:14Z")

</div>

I’m very interested in being able to dynamically generate, compile,  
and link in c code (and potentially other languages, including  
assembly - but c is my primary interest), from within Ruby. When I  
say “link in”, I mean have the compiled code immediately become  
available for use by the Ruby script that asked for the compilation.

So, I’m imagining something like…

# Creating a parameterless c function.

compiler.compile( ‘printf( “Rock on, dudes!” );’ ).call

# A c function with an integer argument.

proc = compiler.compile( ‘x’ =\> Int, ‘printf( “%d \* %d = %d”, x, x,  
x\*x);’ )  
(1…10).each {|x| proc.call(x)}

A module that encapsulated this (particularly as part of the Ruby  
standard distribution) would allow a “pure ruby” module to have  
native speed [1].

It would also be awesome for dynamic code generation for building an  
algorithm optimised for a very specific situation. Coming from an  
image processing background, I find the idea quite exciting 🙂 As an  
example, I could be using artificial evolution to grow functions that  
evaluate to a pretty picture. To evaluate these functions at maximum  
speed, I could compile them down to c, and execute them.

Another example would be creating a parser from a BNF like notation.  
It should be possible (making use of this module) to specify a  
grammar with a Rubyish domain specific language, and then compile  
that to native code and immediately make use of it from within the  
same Ruby program.

A further possibility is that it could pull all that pesky linking  
with external libraries (well I don’t get on with it anyway, but it’s  
not really my bag) in to the domain of Ruby, and make it a fun thing  
to explore from within IRB, for example. An offshoot from that would  
be an explorative SWIG like system: a simple c parser that you can  
point at a library’s header file, allowing you to start hacking about  
with the c library as if you were back programming BASIC on your BBC  
[2] 🙂

Has anyone come across work in this direction, or is anyone working  
on it themselves? Could anyone comment on difficulties involved, or  
serious practical problems? I think the major issues are likely to be:

```
Interfacing with the native linker.
Passing stuff in and out of the c easily.
Giving the c compiler header files (or bundles, or whatever it needs

```

on OS X).

It’s likely I’m completely missing many problems though - I’m coming  
to this having never written a c extension for Ruby!

My dream would be to encapsulate all of the platform specific nitty  
gritty in one place, and so make it reasonably simple for anyone with  
some c and Ruby knowledge to just dive in and start writing super  
fast code, or linking with a library they want to play with.

Sorry for a long post. I hope it will give you ideas, or stimulate  
interest.

Thank you for your time,  
Benjohn B.

[1] Although one would want a mechanism to be able to cache compiled  
code between runs eventually, as the compile time could become annoying.

[2] I think Ruby is such an enjoyable experience for me because it  
recreates the fun, and frankly joy, I used to have with BASIC, when I  
first learnt to program, but it allows me to concisely specify my  
ideas in ways that I’ve come to need after 20 years of programming.  
What I want to do here is get close to having BBC BASIC’s build in  
assembly 🙂 Total expressiveness, and raw touching the metal speed  
and control.

---

<div class="post-metadata">

**Author:** ![Benjohn\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e5b9ba/32.png) [@Benjohn\_B](https://www.ruby-forum.com/u/Benjohn_B)\
**Post date:** [February 22, 2006, 11:00pm UTC](https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060/2 "2006-02-22T23:00:19Z")

</div>

Benjohn B. wrote:

> Another example would be creating a parser from a BNF like notation.  
> It should be possible (making use of this module) to specify a  
> grammar with a Rubyish domain specific language, and then compile  
> that to native code and immediately make use of it from within the  
> same Ruby program.

I had exactly this idea recently! I was particularly interested in  
lexing, because that’s the bottleneck in some parsers I’ve written  
recently. And conveniently, re2c (google it) is an existing re-\>c  
compiler that generates optimized C from a regular expression  
description. Sounds perfect for the task.

> A further possibility is that it could pull all that pesky linking  
> with external libraries (well I don’t get on with it anyway, but it’s  
> not really my bag) in to the domain of Ruby, and make it a fun thing  
> to explore from within IRB, for example. An offshoot from that would  
> be an explorative SWIG like system: a simple c parser that you can  
> point at a library’s header file, allowing you to start hacking about  
> with the c library as if you were back programming BASIC on your BBC  
> [2] 🙂

Wow, I had this idea too – except I had the idea of getting function  
signatures from a .so’s debugging information, so you wouldn’t need the  
C header file.

I’m really interested in any work you do in this area. I’d like to be  
working on this, but I am heavily invested in other problems at the  
moment.

Regards,  
Josh

---

<div class="post-metadata">

**Author:** ![Benjohn\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e5b9ba/32.png) [@Benjohn\_B](https://www.ruby-forum.com/u/Benjohn_B)\
**Post date:** [February 22, 2006, 11:53pm UTC](https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060/3 "2006-02-22T23:53:56Z")

</div>

On Thu, 23 Feb 2006, Benjohn B. wrote:

> could be using artificial evolution to grow functions that evaluate to a  
> my bag) in to the domain of Ruby, and make it a fun thing to explore from  
> Passing stuff in and out of the c easily.
> 
> the fun, and frankly joy, I used to have with BASIC, when I first learnt to  
> program, but it allows me to concisely specify my ideas in ways that I’ve  
> come to need after 20 years of programming. What I want to do here is get  
> close to having BBC BASIC’s build in assembly 🙂 Total expressiveness, and  
> raw touching the metal speed and control.

you want to check out rubyinline and ruby/dl. the later is an interface  
to  
the dynamic linker which allows you load \*.so files and call routines in  
them  
directly. the former spews c code, compiles it, and loads it from pure  
ruby.  
fyi - ‘require’ **is** an interface to the dynamic linker, so long as  
the dll  
being loaded is of a certain type.

cheers.

-a

---

<div class="post-metadata">

**Author:** ![Benjohn\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e5b9ba/32.png) [@Benjohn\_B](https://www.ruby-forum.com/u/Benjohn_B)\
**Post date:** [February 22, 2006, 11:35pm UTC](https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060/4 "2006-02-22T23:35:41Z")

</div>

On Feb 22, 2006, at 4:51 PM, Benjohn B. wrote:

> an algorithm optimised for a very specific situation. Coming from  
> within the same Ruby program.  
> Has anyone come across work in this direction, or is anyone working
> 
> assembly 🙂 Total expressiveness, and raw touching the metal speed  
> and control.

Sounds like you’re looking for RubyInline:  
class MyTest

def factorial(n)  
f = 1  
n.downto(2) { |x| f \*= x }  
f  
end

inline do |builder|  
builder.c "  
long factorial\_c(int max) {  
int i=max, result=1;  
while (i \>= 2) { result \*= i–; }  
return result;  
}"  
end  
end

> **[rubyinline | software projects | by ryan davis](https://www.zenspider.com/projects/rubyinline.html)**
>
> Inlined foreign code for ultra-fast and flexible ruby.

---

<div class="post-metadata">

**Author:** ![Benjohn\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e5b9ba/32.png) [@Benjohn\_B](https://www.ruby-forum.com/u/Benjohn_B)\
**Post date:** [February 23, 2006, 5:08pm UTC](https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060/5 "2006-02-23T17:08:59Z")

</div>

> Benjohn B. wrote:
> 
> > Another example would be creating a parser from a BNF like notation.  
> > It should be possible (making use of this module) to specify a  
> > grammar with a Rubyish domain specific language, and then compile  
> > that to native code and immediately make use of it from within the  
> > same Ruby program

I worked on a compiler backend, which actually does this. It takes a  
symbolic expression represented by a nested array and compiles it to  
assembly code, which is processed by gas. This can be required  
immediately as ruby extension. The compiler is stack based as this  
simplifies the algorithm a lot. At the time local variables,  
parameters, literal floats and many other things are missing, but it  
works.  
…

> > be an explorative SWIG like system: a simple c parser that you can  
> > point at a library’s header file, allowing you to start hacking about  
> > with the c library as if you were back programming BASIC on your BBC  
> > [2] 🙂

Implementing a c parser is cumbersome.

> Wow, I had this idea too – except I had the idea of getting function  
> signatures from a .so’s debugging information, so you wouldn’t need the  
> C header file.

This approach seems much easier. Maybe something similar to Ruby/DL.

> I’m really interested in any work you do in this area. I’d like to be  
> working on this, but I am heavily invested in other problems at the moment.
> 
> Regards,  
> Josh

Cheers,  
Matthias

---

<div class="post-metadata">

**Author:** ![Benjohn\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e5b9ba/32.png) [@Benjohn\_B](https://www.ruby-forum.com/u/Benjohn_B)\
**Post date:** [February 23, 2006, 3:44am UTC](https://www.ruby-forum.com/t/dynamic-code-generation-and-linking/72060/6 "2006-02-23T03:44:40Z")

</div>

If you’re interested in the CLR / .NET, my RubyCLR bridge uses Ruby to  
generate CIL (the CLR’s intermediate language) instructions on the fly  
to  
create shims that glue Ruby to the CLR. It also lets me do cool stuff  
like  
refactor my CIL, generate macros etc.

-John

> **[John Lam's Blog](https://www.iunknown.com/)**
>
> Since 1997