Ripper on array access operation

Ruby
1

Ripper.sexp(‘a[1]’)

yeilds the following output

[:program, [[:aref, [:var_ref, [:@ident, “a”, [1, 0]]],
[:args_add_block, [[:@int, “1”, [1, 2]]], false]]]]

i am trying to figure out where the information that the methods :[] was
called is present.

But when i do

Ripper.sexp(‘a.’)

[:program, [[:method_add_arg, [:call, [:var_ref, [:@ident, “a”, [1,
0]]], :".", [:@op, “[]”, [1, 2]]], [:arg_paren, [:args_add_block,
[[:@int, “0”, [1, 5]]], false]]]]]

I get the information about which method was called.

2

Quoting “Surya G.” [email protected]:

called is present.
I get the information about which method was called.

–
Posted via http://www.ruby-forum.com/.

I am pretty sure it’s the :aref part. Ripper is making [] a special
case.

-Justin

3

unknown wrote in post #1019486:

Quoting “Surya G.” [email protected]:

called is present.
I get the information about which method was called.

–
Posted via http://www.ruby-forum.com/.

I am pretty sure it’s the :aref part. Ripper is making [] a special
case.

-Justin

Thats what it looks like but why not treat aref as any-other method
call.

4

Quoting “Surya G.” [email protected]:

I don’t know. Ripper’s output looks like a mess to me when compared to
RubyParser:

s(:call, s(:call, nil, :a, s(:arglist)), :[], s(:arglist, s(:lit, 1)))

-Justin

5

On Aug 31, 2011, at 7:00 PM, [email protected] wrote:

Thats what it looks like but why not treat aref as any-other method
call.

–
Posted via http://www.ruby-forum.com/.

Aref is parsed differently from other method calls. Ripper delivers a
parse tree, and by providing aref as a different AST node, a program
consuming the parse tree can distinguish between a.[](1, 2) and a[1,2].
There are plenty of reasons (such as pretty-printing) that these two
should not be conflated.

I don’t know. Ripper’s output looks like a mess to me when compared to
RubyParser:

s(:call, s(:call, nil, :a, s(:arglist)), :[], s(:arglist, s(:lit, 1)))

-Justin

That’s because RubyParser doesn’t provide an abstract syntax tree, it
provides a tree interpretation of Ruby programs. While Ripper comes
close to a concrete syntax tree at times, at least it sticks to syntax.
t discuss my choice of Ripper over RubyParser in my undergraduate thesis
[0]; while it is perhaps suitable for an interpreter, its output
discards so much of the syntax. It even inserts nodes which have nothing
to do with syntax and everything to do with semantics. In this example,
if I want my static analyzer to find examples of a., I can’t using
RubyParser (at least with the default settings).

It adds :scope nodes which don’t have anything to do with the syntax, it
evaluates constant literals, from integers to regular expressions, which
is not the job of a general-purpose AST. It even inserts a local
assignment node when it parses “rescue Foo => x” (x = $!).

RubyParser is a good library, and its value as a pure-ruby parser cannot
be understated. It’s not suitable for many uses because of the shortcuts
it takes, the over-interpretation it does (this aref example is a clear
one), and the overall focus on semantics and not on just parsing.

[0] PDF:
http://www.cs.dartmouth.edu/cms_file/SYS_techReport/532/TR2011-686.pdf

6

On 08/31/2011 08:41 PM, Michael E. wrote:

-Justin

-Justin

That’s because RubyParser doesn’t provide an abstract syntax tree, it provides a
tree interpretation of Ruby programs. While Ripper comes close to a concrete
syntax tree at times, at least it sticks to syntax. t discuss my choice of Ripper
over RubyParser in my undergraduate thesis [0]; while it is perhaps suitable for
an interpreter, its output discards so much of the syntax. It even inserts nodes
which have nothing to do with syntax and everything to do with semantics. In this
example, if I want my static analyzer to find examples of a., I can’t using
RubyParser (at least with the default settings).

It adds :scope nodes which don’t have anything to do with the syntax, it
evaluates constant literals, from integers to regular expressions, which is not
the job of a general-purpose AST. It even inserts a local assignment node when it
parses “rescue Foo => x” (x = $!).

RubyParser is a good library, and its value as a pure-ruby parser cannot be
understated. It’s not suitable for many uses because of the shortcuts it takes,
the over-interpretation it does (this aref example is a clear one), and the
overall focus on semantics and not on just parsing.

[0] PDF: http://www.cs.dartmouth.edu/cms_file/SYS_techReport/532/TR2011-686.pdf

I suppose it depends on what you want out of it. For what I use it for,
I don’t care about the exact syntax used when they mean the same thing,
and the fewer node types I need to check for, the better. And I
appreciate that RubyParser takes care of much of the semantics for me.
In your case, you may have needed different information.

So, different strokes for different folks.

-Justin

7

On Sep 1, 2011, at 9:37 PM, Justin C. wrote:

For what I use it for, I don’t care about the exact syntax used when they mean
the same thing, and the fewer node types I need to check for, the better.

I suspect this is actually only true at the margins, which is why parse
trees are valuable and why I argue the corners RubyParser cuts are
arbitrary and silly.

Would you prefer that RubyParser turn “@foo = bar” into a call to
“instance_variable_set”?

Or “class A < B; end” into a series of conditions, constant lookups,
assignments, “Class.new(B)”, and so on?

Should “class << foo; end” turn into “foo.singleton_class.class_eval
do… end”?

Michael E.
[email protected]
http://carboni.ca/

8

On Fri, Sep 2, 2011 at 12:54 AM, Justin C.
[email protected]wrote:

and silly.
Or “class A< B; end” into a series of conditions, constant lookups,
suggesting.
Will they always mean exactly the same thing? There are very few cases I
=> [:program, [[:unary, :!, [:var_ref, [:@ident, “a”, [1, 1]]]]]]

They aren’t quite the same, they bind differently:

!false && false # => false
not false && false # => true

Probably you can come up with some instances where RubyParser changes the
semantics of the original code it is representing, which would be a flaw in
RubyParser. It may or may not matter for my use case, though.

I think this is the ultimate point here. You have different needs, and
different libs to address those different needs.

9

On 09/01/2011 06:57 PM, Michael E. wrote:

On Sep 1, 2011, at 9:37 PM, Justin C. wrote:

For what I use it for, I don’t care about the exact syntax used when they mean
the same thing, and the fewer node types I need to check for, the better.
I suspect this is actually only true at the margins, which is why parse trees
are valuable and why I argue the corners RubyParser cuts are arbitrary and silly.

Would you prefer that RubyParser turn “@foo = bar” into a call to
“instance_variable_set”?

Kind of… :slight_smile: But “instance_variable_set” might have a different meaning
than “@foo = bar”. Whereas, as far as I know, “a[]” and “a.” will
always be the same.

Or “class A< B; end” into a series of conditions, constant lookups,
assignments, “Class.new(B)”, and so on?

I think this might be going in the other direction (but you may also be
correct). In “a[]” vs. “a.”, RubyParser makes it easier because I
only have to look for a call to “[]”. How the call is represented in the
actual code is not important (to me!). If it were always valid to assume
that the above-mentioned conditions, lookups, etc., would mean the same
as “class A < B; end” then I (in my case!) would not mind if I just got
back something representing “class A < B; end”. In other words, the
reverse of what you are suggesting.

Should “class<< foo; end” turn into “foo.singleton_class.class_eval do… end”?

Michael E.
[email protected]
http://carboni.ca/

Will they always mean exactly the same thing? There are very few cases I
can think of where two bits of Ruby code will always be the same
thing, because you can override nearly anything.

Here’s another case:

ruby-1.9.2-p290 :003 > RubyParser.new.parse “!a”
=> s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :004 > RubyParser.new.parse “not a”
=> s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :005 > Ripper.sexp “!a”
=> [:program, [[:unary, :!, [:var_ref, [:@ident, “a”, [1, 1]]]]]]
ruby-1.9.2-p290 :006 > Ripper.sexp “not a”
=> [:program, [[:unary, :not, [:var_ref, [:@ident, “a”, [1, 4]]]]]]

Again, if I want to check for a negative of “a”, then with Ripper I must
look for (at least) two alternatives, but with RubyParser there is just
one. Semantically there is no difference. (There is a difference in the
interpretation because RubyParser doesn’t know about Ruby 1.9, but that
is a different difference. What I mean is there is no difference between
Ripper’s “!” and “not”, because they will do the same thing.)

Probably you can come up with some instances where RubyParser changes
the semantics of the original code it is representing, which would be a
flaw in RubyParser. It may or may not matter for my use case, though.

Whew. I’m only responding because you asked such direct questions :slight_smile: As
I’ve said, RubyParser works better for me, while Ripper is a bit more
difficult to manage. Clearly you needed more information about the
program, so Ripper made sense to use. As you have pointed out before,
Ripper has its own issues. There are use cases for either one depending
on what your needs are. If Ripper were available for 1.8, I might have
used it instead.

-Justin

10

Quoting Josh C. [email protected]:

than “@foo = bar”. Whereas, as far as I know, “a[]” and “a.” will always
code is not important (to me!). If it were always valid to assume that the

[email protected]
=> s(:not, s(:call, nil, :a, s(:arglist)))
interpretation because RubyParser doesn’t know about Ruby 1.9, but that is a
different difference. What I mean is there is no difference between Ripper’s
“!” and “not”, because they will do the same thing.)

They aren’t quite the same, they bind differently:

!false && false # => false
not false && false # => true

Yes, but precedence is handled by the parser, so that difference
doesn’t matter once you have the s-expressions:

ruby-1.9.2-p290 :005 > RubyParser.new.parse “!false && false”
=> s(:and, s(:not, s(:false)), s(:false))
ruby-1.9.2-p290 :006 > RubyParser.new.parse “not false && false”
=> s(:not, s(:and, s(:false), s(:false)))
ruby-1.9.2-p290 :007 > Ripper.sexp “!false && false”
=> [:program, [[:binary, [:unary, :!, [:var_ref, [:@kw, “false”, [1,
1]]]], :“&&”, [:var_ref, [:@kw, “false”, [1, 10]]]]]]
ruby-1.9.2-p290 :008 > Ripper.sexp “not false && false”
=> [:program, [[:unary, :not, [:binary, [:var_ref, [:@kw, “false”,
[1, 4]]], :“&&”, [:var_ref, [:@kw, “false”, [1, 13]]]]]]]

Probably you can come up with some instances where RubyParser changes the
semantics of the original code it is representing, which would be a flaw in
RubyParser. It may or may not matter for my use case, though.

I think this is the ultimate point here. You have different needs, and
different libs to address those different needs.

Yes.

-Justin