RubyMacros 0.1.0 Released

pandemic · October 24, 2008, 10:23pm

RubyMacros is a lisp-like macro pre-processor for Ruby. More than just a
purely textual substitution scheme, RubyMacros can manipulate and morph
Ruby parse trees (in the form of RedParse Nodes) at parse time in just
about
any way you see fit.

Macros are programmed in ruby itself. And since parse trees are
represented
in RedParse format, they’re easier to use (programatically) and more
object-
oriented than other available ruby parsetree formats. (RedParse Node
format
is actually designed to be straightforward to use and to represent the
structure of ruby source code very closely.)

== Benefits:

Powerful and easy metaprogramming
Create better DSLs
Manipulate syntax trees to suit yourself
Access local variables and other caller context unavailable to
methods
Macros as inline methods: should be slightly faster than equivalent
methods

== Drawbacks:
Although in theory already as powerful as lisp macros, the current
implementation has a number of problems which added together make it
merely
a proof of concept or toy at this point:

pre-processing is very, very slow (because of RedParse)
macro calls must be inside some sort of method;
straight out macro calls at the top level won’t work
macros can’t have blocks or receivers
some ruby syntax is unsupported in files using macros
files using macros must be loaded via Macro.require;
Kernel#require will not recognize macros
RedParse Node tree format will be changing slightly
macros cannot be scoped
no variable (or other) hygiene

== Requirements:
RubyMacros requires RedParse.

== Install:
gem install rubymacros

== Examples:
macro simple(a,b)
:(^a+^b)
end
def simple_user
p simple(1,2) #prints 3
end

#loop as a macro, should be a bit faster than the #loop method
macro loop(body)
:(while true
^body
end
)
end

#for more examples, see the examples/ directory

== New Syntax:
I have invented 3 new syntactical constructions in order to allow
reasonably
easy to use macros. Macros themselves look just like methods except that
‘macro’ instead of ‘def’ is used to start the macro definition off. A
form
literal is an expression surrounded by ‘:(’ and ‘)’. The form escape
operator
is ‘^’. ‘^’ is a unary operator of fairly high precedence.

== Forms and Form Escapes:
Forms are an essential adjunct to macros. Forms represent quoted source
code, which has been parsed but not evaled yet. When a form literal is
executed, it returns a RedParse::Node representing the parse tree for
the
enclosed source code. Within a form literal, a ^, used as a unary
operator,
will escape the expression it controls, so that instead of being part of
the
form’s data, it is executed at the same time as the form literal, and
the
result of an escaped expression (which should be a Node) is interpolated
into the form at that point. The whole effect is much like that of
string
interpolations (#{}) inside string literals.

== How Macros Work
Typically, macros return a single form literal, which contains form
escape
expressions within it which make use of the macro’s parameters. However,
macro bodies may contain anything at all; more complicated macros will
likely not contain any forms. (Likewise, form literals may be used
outside
macros, but the utility of doing so may be minimal.)

At parse time (well, really at method definition time, but in effect
it’s
much the same thing) method bodies are scanned for callsites which have
the
names of known macros. When such a call is found, it is expanded as
follows.
The parsetrees for the arguments to the callsite are passed as arguments
to
the macro. The macro is expected to return a parsetree, which replaces
the
macro callsite in the parsetree which contained it.

== License:
LGPL

pandemic · October 25, 2008, 10:50am

2008/10/24 Caleb C. [email protected]:

RubyMacros is a lisp-like macro pre-processor for Ruby.

Sounds interesting. When will you release some code to look at?

Regards,
Pit

pandemic · October 25, 2008, 11:45am

When will you release some code to look at?

I was close to asking the very same question. But then I saw “gem
install”, so it’s available as gem.

Anyway, how does this approach compare to:

http://blog.drewolson.org/2008/06/ruby-and-macros-experiment.html
http://weblog.raganwald.com/2008/06/macros-hygiene-and-call-by-name-in-ruby.html

May I humbly ask, how do I run the examples? I tried calling running
the example/*_wrap.rb scripts with ruby18 and ruby19 but only got
Syntax errors:

ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-cygwin]
ruby 1.9.0 (2008-08-26 revision 18849) [i386-cygwin]

pandemic · October 25, 2008, 7:44pm

On 10/25/08, toomln [email protected] wrote:

When will you release some code to look at?

I was close to asking the very same question. But then I saw “gem
install”, so it’s available as gem.

Sorry, I didn’t include any urls since I didn’t have the rubyforge
project yet. Maybe I should have posted one anyway… but now the
project has been approved, and the code is up there, so try these:

http://rubymacros.rubyforge.org/README.txt
http://rubyforge.org/projects/rubymacros

Anyway, how does this approach compare to:

http://blog.drewolson.org/2008/06/ruby-and-macros-experiment.html

I have to confess, I haven’t looked into this project very much. This
one is using strings to represent sexps… you can do a lot with that,
but eventually you’ll run into a glass ceiling of things you want to
find in those strings that just can’t be parsed readily with regexps.
(My ‘with’ macro, (in the examples/ dir) is an example of the kind of
thing that would be impossible.) The macros themselves require a
distressingly large number of calls to eval.

Macros, Hygiene, and Call By Name in Ruby

This is the best attempt I’ve seen to make something macro-like using
ParseTree’s syntax tree format. I’m amazed at Reg’s ability to
persuade ruby to do feats of gymnastics that I wouldn’t have thought
possible… but the result here is still slightly to very clunky,
depending on which abstraction you’re using.

Here’s the shorter of his two of his attempts to write an andand macro
that operates like the && operator:

called_by_name(:our_and) { |x,y|
    if temp = x
        y
    else
        temp
    end
}

Not so bad in itself, but all called_by_name’s must be wrapped in a
‘with’, which must get tiresome.

The longer one is too horrible to contemplate. The equivalent in
RubyMacros is something like this:

macro andand(a,b)
if temp = ^a
^b
else
temp
end
)
#or maybe just ^a && ^b )
end

If you want to write a sexp processor, rewrite can help you do that,
and it should be possible to do things even macros can’t that way. But
it’s very hard to use.

This called_by_name looks considerably easier to use, but I think Reg
says somewhere that it’s only useful for a subset of what macros do…

May I humbly ask, how do I run the examples? I tried calling running
the example/*_wrap.rb scripts with ruby18 and ruby19 but only got
Syntax errors:

Aaaaaag! Don’t tell me I published broken code… I did! Ok, I just
fixed it, but gems hasn’t found the new version yet, hopefully that
won’t take long, but til then, here’s a couple of direct links to the
fixed version:
http://rubyforge.org/frs/download.php/45707/rubymacros-0.1.1.tgz
http://rubyforge.org/frs/download.php/45706/rubymacros-0.1.1.gem

(This is all because of a misguided attempt to keep rdoc from
crashing… I give up on rdoc for the time being.)

You were correct to try invoking the *_wrap scripts. “ruby -rubygems
example/simple_wrap.rb” should do the trick.

pandemic · October 26, 2008, 2:07am

Caleb C. wrote:

Here’s the shorter of his two of his attempts to write an andand macro
that operates like the && operator:
called_by_name(:our_and) { |x,y|
    if temp = x
        y
    else
        temp
    end
}
Not so bad in itself, but all called_by_name’s must be wrapped in a
‘with’, which must get tiresome.

The use of “with” is a deliberate design choice. Rather than making
macros global and “automagical,” you state what you are using and where
you are using it. This is somewhat akin to writing ‘require
such-and-such’ in each ruby source file.

Of course, some people like magic, and if you look at Rails, the
initializers and environment.rb file allow you to sprinkle magic
throughout your project implicitly. My feeling when I designed rewrite
was that that if I started with explicit “with,” it would easy to build
implicit into a project or framework later.

The longer one is too horrible to contemplate.

Speak for yourself

The equivalent in RubyMacros is something like this:

macro andand(a,b)
if temp = ^a
^b
else
temp
end
)
#or maybe just ^a && ^b )
end

^a && ^b ) is a little too metacircular for my taste, but I put it
to you that rewrite allows you to define your own syntactic replacement
using && if you want to. Now to get more specific. called_by_name is
actually not a way of doing macros, it’s a way of writing functions with
call by name semantics.

Rewrite actually provides a facility for code rewriting, which is one
level above simple unhygienic macros. A traditional unhygienic macro
is a way of saying “when you see something that looks like a method
call, replace it with the following code, performing substitutions here
and here and here.” Rewrite supports this as well as a number of other
arbitrary rewriting rules.

For example, you can say "when you see foo.select { …blah… }.map {
…blah-blah }, replace it with a single call to .each that performs the
selction and mapping with out iterating over the collection twice.

Now, called_by_name is actually a macro written using rewrite. So it’s a
meta-macro. I would say that gievn your example, the macro is better
because it does not “compile” into a function call, whereas anything
built with called_by_name will be rewritten as a function call. If what
you want is the fastest, tightest code, use a ruby macro or use rewrite
to directly rewrite the function call as an if statement.

If you want to compare rewrite and ruby macros more directly, there’s a
little thing I wrote called Unhygienic. It does code rewriting “by
example.” Now, I use the term “andand” to refer to
http://andand.rubyforge.org/, so here is how to write part of that gem
using Rewriting by example:

    Unhygienic.from(:receiver, :message, [:parameters]) {
      receiver.andand.message(parameters)
    }.to {
      lambda { |andand_temp|
        andand_temp.message(parameters) if andand_temp
      }.call(receiver)
    }

By the way, I use lambdas a lot in my rewrites to try to alleviate the
pain of Ruby’s scoping rules. If you like to live a little more
dangerously (and the example above does), this rule can be made shorter:

    Unhygienic.from(:receiver, :message, [:parameters]) {
      receiver.andand.message(parameters)
    }.to {
      temp.message(parameters) if (temp = receiver)
    }

Both examples are longer than the macro or called_by_name definitions.
The “from” says "here is a snippet of code where receiver, message, and
parameters are placehilders for an expression, and expression, and a
list of expressions. The “to” says “when you find that, replace it with
this, plugging in the placeholders.”

The idea here is that you can use any arbitrary ruby expression, not
just something that looks like a function call. In this case, you are
making something that looks like a method call expand into something
else entirely.

My motivation with Rewrite was very specific: I was trying to show that
we have alternatives to wide scale opening of core classes to implement
DSLs and syntactic abstractions. This goal necessarily encompassed
providing an alternative to existing idioms like #andand or
Symbol#to_proc or #try. The goal of Ruby macros seems to be a little
different, and thus the two gems work in different ways.

Overall, I wish Ruby macros every success and hope that people get
excited about syntactic abstractions.

pandemic · November 2, 2008, 11:42pm

On 10/26/08, ara.t.howard [email protected] wrote:

equivalent methods

only manipulating the syntax tree seems like a real advantage from
here, generating a dsl is already as nearly painless as it could be in
ruby, and methods do in fact have access to local variables and other
caller context

Sorry for the late reply… I thought this thread had died.

You seem to have left out inline methods, for whatever they’re worth.

Yes, DSLs in ruby are currently very easy to create and use; I
anticipate that macros will make DSLs slightly harder to write, but
easier to use. Ruby DSLs tend to be very natural looking to users, but
there are often small compromises to usability that won’t make much
sense to domain users, such as the need to begin some words with a
colon and the need to use ‘do’ at certain places in the language for
no apparent reason. If you use macros to define your DSL, it should be
possible to rid oneself of those features. The situation is already
pretty good, but macros will make it slightly better. I say “will”,
because, to be honest, at the moment most of the sugary convenience
features needed for nicer DSLs are not present.

As an example of a DSL (perhaps the wrong word in this case…) that
could be written in macros, there is iterate for common lisp:

Iterate is a looping mini-language with special syntax for many common
looping tasks. It looks kind of like what list comprehensions do for
you in python, but more powerful. Now I imagine that something like
this could be written entirely with methods… but it would be too
slow. (I admit, tho, that I don’t understand iterate – or lisp in
general – very well. If someone out there wants to correct my
misapprehensions, please feel free.)

cfp:~ > cat a.rb
def context &block
eval ‘a += 40’, block
block.call
end

This is a slick way of getting to your caller’s lvars. I would have
passed in a Binding myself, but this way is probably a little cleaner.

But, the caller must pass a block (or binding) in order to make this
work. Sometimes, that’s not a problem. Sometimes it is. For instance
(I’ve run into this) if you want to create an api that works exactly
like Regexp#match, you’ll find that it can’t be done. #match sets its
caller’s $~ (a local variable); methods can’t do that. In the past,
I’ve passed in an optional Binding to handle this case, but
practically speaking, it was a little too clumsy. Using a block
instead is a better idea, but you’re still changing the interface used
by your custom #match. The whole point is to re-use your user’s
existing knowledge about #match… if he has to remember, “oh yeah,
and if you use $~ or other matching variables, you have to pass an
extra block to #match”, then that’s not an effective re-use of
existing knowledge; it might as well be a new interface.

can you show us something that cannot be done using ruby currently,
which macros make possible?

A recently requested new feature on ruby-core was DIR, which acts
like FILE, but returns the directory containing the current source
file. As a macro, that is:

macro DIR
File.dirname FILE )
end

Now maybe (now that I’ve seen your block-as-binding trick) you can
actually write this as a method, something like,

def DIR(&ctx)
File.dirname(eval(“FILE”,ctx))
end

I have no ruby ATM, and can’t check if that works or not, sorry. But
if it does, it will have to be called like DIR{}, instead of
DIR. I’d find that a little jarring.

Another recent request was a ‘with’ keyword, which operates like
instance_eval, but only changes the default receiver for code in the
block passed in, and not self as seen by instance variables. I have an
implementation of this as well (in the example directory of
RubyMacros), but for various reasons I’m unsatisfied with it right
now, so I’d rather not post it.

I’m not claiming that either of these macros is actually a good idea;
I’m just trying to illustrate the possible.

It’s likely that quite a few of the features for ruby that get
requested could actually be implemented by macros. It’s probably
appropriate that most of these requests are rejected; we don’t really
need a lot of global changes to the language. However, if users can
write their own macros to scratch some of these itches, that’s a
better solution. They get the feature they want in just the program
that needs it, and the rest of us get a stable, predictable language
without a lot of weird new features in all other ruby programs.

pandemic · November 3, 2008, 3:40am

On Nov 2, 5:40 pm, “Caleb C.” [email protected] wrote:

It’s likely that quite a few of the features for ruby that get
requested could actually be implemented by macros. It’s probably
appropriate that most of these requests are rejected; we don’t really
need a lot of global changes to the language. However, if users can
write their own macros to scratch some of these itches, that’s a
better solution. They get the feature they want in just the program
that needs it, and the rest of us get a stable, predictable language
without a lot of weird new features in all other ruby programs.

Or we could just write our programs in “Macro-Ruby” and to put a death
nail in so called predictable language.

        _._                           _._
       ||||                           ||||
       ||||_           ___           _||||
       |  ||        .-'___`-.        ||  |
       \   /      .' .'_ _'. '.      \   /
       /~~|       | (| b d |) |       |~~\
      /'  |       |  |  '  |  |       |  `\

, /.-: ,| | -' | |, :-.__\ , |'-------( \-''""/.| /\___/\ |.\""''-/ )------'| | \_.-'\ / '-._____.-' \ /'-._/ | |.---------\ /'._| _ .---. === |_.'\ /--------.| ' \ / | |\_\ _ \=v=/ _ | | \ / ' . | | __\ ~~~ (_) | | .’
'"'|'–..^..–‘|'"'
\ /
,..---'"'---..,' :--..___..--: TO DSL... \ / |. .’| AND BEYOND!
| :: |
| | | |
| | | |
|.-.| |.-.|
|-'| |-'|
| | | |
/ | |
|| |_|
‘:—:-’-:—:’
/ | |
jgs /.—.| |.—.
`.; :.’

T.

pandemic · October 26, 2008, 10:25pm

On Oct 24, 2008, at 2:23 PM, Caleb C. wrote:

== Benefits:

Powerful and easy metaprogramming

Create better DSLs

Manipulate syntax trees to suit yourself

Access local variables and other caller context unavailable to
methods

Macros as inline methods: should be slightly faster than
equivalent methods

only manipulating the syntax tree seems like a real advantage from
here, generating a dsl is already as nearly painless as it could be in
ruby, and methods do in fact have access to local variables and other
caller context

cfp:~ > cat a.rb
def context &block
eval ‘a += 40’, block
block.call
end

a = 2
context{ p a }
p a

cfp:~ > ruby a.rb
42
42

can you show us something that cannot be done using ruby currently,
which macros make possible?

cheers.

a @ http://codeforpeople.com/

pandemic · November 3, 2008, 10:24am

2008/11/3 ara.t.howard [email protected]:

hrrrrm. could we possibly use it to skin the

self.ivar = value

problem?

Of course:

irb(main):001:0> require “nolocal-spike”
=> true

irb(main):002:0> class C
irb(main):003:1> attr_accessor :a, :b, :c
irb(main):004:1> def initialize &blk
irb(main):005:2> instance_eval( &nolocal( &blk ) )
irb(main):006:2> end
irb(main):007:1> end
=> nil

irb(main):008:0> C.new do
irb(main):009:1* a = b = 5
irb(main):010:1> c = a + 2
irb(main):011:1> end
=> #<C:0x2fce8e0 @c=7, @b=5, @a=5>

Two years ago I was experimenting with modifying the AST using plain
Ruby. (On Windows, I couldn’t use parsetree for example back then.)
The code above is one of the results. The same should be easy with the
actual libraries.

Regards,
Pit

pandemic · November 3, 2008, 8:05am

On Nov 2, 2008, at 3:40 PM, Caleb C. wrote:

Sorry for the late reply… I thought this thread had died.

You seem to have left out inline methods, for whatever they’re worth.

yeah - if they are really a lot faster it’s worth considering but, for
now, i’ll assume they’re not…

pretty good, but macros will make it slightly better. I say “will”,
because, to be honest, at the moment most of the sugary convenience
features needed for nicer DSLs are not present.

that’s an interesting point. not sure about it though - if people are
writing dsls that are not ruby support debugging becomes quite
difficult. still, i get it.

But, the caller must pass a block (or binding) in order to make this
extra block to #match", then that’s not an effective re-use of
existing knowledge; it might as well be a new interface.

i wouldn’t strictly agree with your analysis, but i do agree that it’s
very hard to do so. the key is having a context or marker which
allows the method to be safely used on all objects, this is what tagz
does for html/xml generation to avoid this - basically methods called
on any ‘self’ have easy access to the caller. for instance

class Object
def LikeRegexp
LikeRegexpObject.new(self)
end
ed

LikeRegexp.match …

is one workaround. i do clearly see the value of being inside an
object though. however, is the implication that all macros are global?

def DIR
filename = caller[0][/^(.*):/, 1]
File.expand_path(File.dirname(filename))
end

stolen wholesale from Ramaze (lot’s of good stuff in there )

Another recent request was a ‘with’ keyword, which operates like
instance_eval, but only changes the default receiver for code in the
block passed in, and not self as seen by instance variables. I have an
implementation of this as well (in the example directory of
RubyMacros), but for various reasons I’m unsatisfied with it right
now, so I’d rather not post it.

def with &block
scope = Scope.new

instance_variables.each do |ivar|
scope.instance_variable_set ivar, instance_variable_get(ivar)
end

scope.instance_eval &block
end

hacky? yes. but it works well enough for ActionView…

without a lot of weird new features in all other ruby programs.

well now that’s something everyone can agree on! seriously, the
project looks super interesting - just trying to think of a real use
case.

hrrrrm. could we possibly use it to skin the

self.ivar = value

problem?

and, to repeat from above, are the global? i’m hoping macros can be
scoped to an object like instance methods…

cheers.

a @ http://codeforpeople.com/

pandemic · November 3, 2008, 2:24pm

Caleb C. wrote:

A recently requested new feature on ruby-core was DIR, which acts
like FILE, but returns the directory containing the current source
file. As a macro, that is:

macro DIR
File.dirname FILE )
end

Now maybe (now that I’ve seen your block-as-binding trick) you can
actually write this as a method, something like,

def DIR(&ctx)
File.dirname(eval(“FILE”,ctx))
end

I have no ruby ATM, and can’t check if that works or not, sorry.

It seems to, as long you provide a dummy block: DIR{}.

I think there’s a simpler solution though:

def DIR
# filename:nnn[:in `method’]
File.dirname(caller.first.split(’:’).first)
end

Actually, it’s a pain that ‘caller’ doesn’t provide a proper composite
object with filename, line number, method name. Windows people may have
C:… at the front of their filename. A regexp split should fix that.

pandemic · November 3, 2008, 11:30am

just trying to think of a real use case.

Macros help lisp to adapt to about every new idea/trend without making
changes to the language as such.

One simple use case that comes to mind is conditional compilation, eg
inserting some code only if a certain flag is set.

i’m hoping macros can be scoped to an object like instance methods…

I personally think it would be a good idea to somehow restrict macro
expansion to specified classes/namespaces. Otherwise, just of what
would happen if you define a macro foo and use some library that
defines a method of the same name.

pandemic · November 3, 2008, 2:37pm

On Nov 3, 8:23 am, Brian C. [email protected] wrote:

Actually, it’s a pain that ‘caller’ doesn’t provide a proper composite
object with filename, line number, method name. Windows people may have
C:… at the front of their filename. A regexp split should fix that.

+1

And that has been requested since as long as I can remember.

T.

pandemic · November 3, 2008, 8:44pm

On Nov 3, 2008, at 02:28 , Tom L. wrote:

One simple use case that comes to mind is conditional compilation, eg
inserting some code only if a certain flag is set.

You mean like:

if $DEBUG then
def x; debug_version; end
else
def x; end
end

?

pandemic · November 3, 2008, 3:02pm

On 11/3/08, ara.t.howard [email protected] wrote:

ed

LikeRegexp.match …

Ok, maybe I’m just being really dense here, but how does that help
with setting the caller’s $~? LikeRegexpObject has a reference to its
caller’s self (clever trick for that, yet again), which surely is
useful for mucking with the caller’s instance variables… but
locals?

Binding.of_caller solves this, of course.

is one workaround. i do clearly see the value of being inside an
object though. however, is the implication that all macros are global?

At the moment, yes. I want to have macros be scoped (statically) to
the class or module they are defined in. (And then maybe a facility to
import them to another class…) But for now, all macros are
effectively globals. The lack of scoping is one of the reasons I
consider the current implementation a toy.

     def __DIR__
       filename = caller[0][/^(.*):/, 1]
       File.expand_path(File.dirname(filename))
     end

Whoa. Good one. I guess you don’t need a macro for that…

scope = Scope.new

instance_variables.each do |ivar|
scope.instance_variable_set ivar, instance_variable_get(ivar)
end

scope.instance_eval &block
end

hacky? yes. but it works well enough for ActionView…

This looks like its the implements the complement of my with; instance
variables of the current self are available in the block, but methods
on the current self aren’t. So now you’ll no doubt come up with a
clever implementation of the other semantics, knocking down another of
my use cases… Maybe I’ll even have a crack at it myself:

def with(oldobj,&block) #UNTESTED
obj=oldobj.class.allocate

… #some magic to forward obj’s method calls to oldobj

obj.instance_eval &block
end

I still like the macro better, tho.

well now that’s something everyone can agree on! seriously, the
project looks super interesting - just trying to think of a real use
case.

iterate, from lisp.

hrrrrm. could we possibly use it to skin the

self.ivar = value

problem?

As Pit C. pointed out, once you have parse trees, it’s fairly
easy to rewrite them in whatever way you want. This feature should
definitely be possible as a macro. The trick is to get it to control
the syntax tree at a high enough level…