Best gem to parse Ruby with?

I’ve been considering rewriting my require_all gem:

http://github.com/tarcieri/require_all/

…to parse the Ruby source code, extracting a list of constants a
particular file defines and a list of constants that are expected to be
defined before a file is loaded.

The gem can iterate through all files requested to be loaded, generate
these
lists of constants per file, and from there resolve the proper order
that
the files should be loaded in, provided all constants are resolvable.

This would solve a number of problems with require_all’s approach.

What’s the best gem to do this with? ruby_parser?

On 7/21/09, Tony A. [email protected] wrote:

I’ve been considering rewriting my require_all gem:

http://github.com/tarcieri/require_all/

…to parse the Ruby source code, extracting a list of constants a
particular file defines and a list of constants that are expected to be
defined before a file is loaded.
[snip]
What’s the best gem to do this with? ruby_parser?

No doubt its author will say it is, but I’d like to recommend my own
library, RedParse. If you can stand its painful slowness, it outputs
the nicest parsetrees.

see: GitHub - coatl/redparse: RedParse is a ruby parser written in pure ruby.
or simply: gem install redparse

You should also consider ParseTree, which is likely to be faster than
either of the others.

On Tue, Jul 21, 2009 at 9:28 PM, Caleb C. [email protected]
wrote:

No doubt its author will say it is, but I’d like to recommend my own
library, RedParse. If you can stand its painful slowness, it outputs
the nicest parsetrees.

I’m definitely looking for a pure Ruby solution, and in that regard
RedParse
looks nice.

You should also consider ParseTree, which is likely to be faster than
either of the others.

ParseTree is great but it seems to only work with 1.8.x MRI which is
unacceptable to me. I guess I should’ve stated that outright. I’m fine
with an FFI solution but not something which needs a non-FFI native
extension.

On Jul 22, 1:18 am, Tony A. [email protected] wrote:

You should also consider ParseTree, which is likely to be faster than
either of the others.

ParseTree is great but it seems to only work with 1.8.x MRI which is
unacceptable to me. I guess I should’ve stated that outright. I’m fine
with an FFI solution but not something which needs a non-FFI native
extension.

Pure ruby to parse Ruby doesn’t fit you?

see ruby_parser:

http://rubyforge.org/projects/parsetree

Non-FFI extension is impossible. At the end you’re going to need link
somehow to Ruby C level, and FFI is actually a C extension.

On Wed, Jul 22, 2009 at 7:00 AM, Caleb C. [email protected]
wrote:

I forgot to mention RubyNode before. That fills out the list of ruby
parsing libraries I know of.

So you want something that:
works in 1.9

1.9 support would be nice, but really I’m looking for something that
works
on JRuby. If it doesn’t work on 1.9 I’m not too worried for the time
being.

written in pure ruby
doesn’t use extensions (except maybe thru FFI)

Taken together, I think these requirements eliminate all the
reasonable candidates.

Why do you care whether it uses FFI or the traditional extension api?
For that matter, why do you care what language it’s written in?

I don’t care what language it’s written in, however I don’t want to
depend
on the old extension API because I’d like for my gem to work on JRuby.
I
plan on using this gem on JRuby in the very near future and in that
regard
very much want to support it.

On 7/22/09, Tony A. [email protected] wrote:

1.9 support would be nice, but really I’m looking for something that works
on JRuby. If it doesn’t work on 1.9 I’m not too worried for the time being.

Ah, ok. I don’t know if RedParse works on JRuby or not; last I knew,
there were a number of strange things it (and its dependencies) were
doing that exposed bugs on JRuby. Those may well have been fixed by
now, tho.

In the JRuby world, there is JParseTree, the JRuby version of
ParseTree. Probably, you could write your code to use ParseTree on MRI
and JParseTree on JRuby. You should try it first, to see if it works
for you.

On Jul 22, 2009, at 11:35 , Caleb C. wrote:

In the JRuby world, there is JParseTree, the JRuby version of
ParseTree. Probably, you could write your code to use ParseTree on MRI
and JParseTree on JRuby. You should try it first, to see if it works
for you.

Nope. Last I looked JParseTree wasn’t compatible with the Unified Ruby
sexp format.

On 7/21/09, Tony A. [email protected] wrote:

ParseTree is great but it seems to only work with 1.8.x MRI which is
unacceptable to me. I guess I should’ve stated that outright. I’m fine
with an FFI solution but not something which needs a non-FFI native
extension.

RedParse doesn’t work in 1.9 yet either, tho.

I forgot to mention RubyNode before. That fills out the list of ruby
parsing libraries I know of.

So you want something that:
works in 1.9
written in pure ruby
doesn’t use extensions (except maybe thru FFI)

Taken together, I think these requirements eliminate all the
reasonable candidates.

Why do you care whether it uses FFI or the traditional extension api?
For that matter, why do you care what language it’s written in?

On Jul 22, 2009, at 06:00, Caleb C. wrote:

parsing libraries I know of.

So you want something that:
works in 1.9
written in pure ruby
doesn’t use extensions (except maybe thru FFI)

Taken together, I think these requirements eliminate all the
reasonable candidates.

Um, ruby_parser works great on 1.9, is written in pure ruby and can
not use extensions. (It sits atop racc which is part of the standard
library and has a non-extension version. I imagine that racc would
ship with JRuby.)

On 7/22/09, Eric H. [email protected] wrote:

Um, ruby_parser works great on 1.9, is written in pure ruby and can
not use extensions. (It sits atop racc which is part of the standard
library and has a non-extension version. I imagine that racc would
ship with JRuby.)

I did not realize that. I thought the extension was required.

On 7/22/09, Ryan D. [email protected] wrote:

On Jul 22, 2009, at 11:35 , Caleb C. wrote:

In the JRuby world, there is JParseTree, the JRuby version of
ParseTree. Probably, you could write your code to use ParseTree on MRI
and JParseTree on JRuby. You should try it first, to see if it works
for you.

Nope. Last I looked JParseTree wasn’t compatible with the Unified Ruby
sexp format.

For what Tony is doing, I doubt he cares about the difference between
unified and raw trees.

On Jul 22, 2009, at 17:55 , Caleb C. wrote:

Ruby
sexp format.

For what Tony is doing, I doubt he cares about the difference between
unified and raw trees.

Well the raw trees are incompatible… so unless they unify, there is
no way he can write code for ParseTree and then use it for jruby.

On Wed, Jul 22, 2009 at 6:55 PM, Caleb C. [email protected]
wrote:

For what Tony is doing, I doubt he cares about the difference between
unified and raw trees.

Interoperability is always nice

On 7/23/09, Ryan D. [email protected] wrote:

Well the raw trees are incompatible… so unless they unify, there is
no way he can write code for ParseTree and then use it for jruby.

Tony is primarily interested in constants. It looks to me like
constants have the same representation in both variants. Correct me if
I’m wrong.