Forum: Ruby-core [Ruby 1.9-Bug#3765][Open] Ripper::Lexer missed out tokens after '=>' operator

Bug #3765: Ripper::Lexer missed out tokens after '=>' operator
http://redmine.ruby-lang.org/issues/show/3765

Author: TzeYang Ng
Status: Open, Priority: Normal
ruby -v: ruby 1.9.1p376 (2009-12-07 revision 26041) [x86_64-linux]

Given the following ripper use case:

pp Ripper.lex(':x => 1')

I'm getting:
>> [[[1, 0], :on_symbeg, ":"],
>>  [[1, 1], :on_ident, "x"],
>>  [[1, 2], :on_sp, " "],
>>  [[1, 3], :on_op, "=>"]]

I'm thinking the output should be:
>> [[[1, 0], :on_symbeg, ":"],
>>  [[1, 1], :on_ident, "x"],
>>  [[1, 2], :on_sp, " "],
>>  [[1, 3], :on_op, "=>"],
>>  [[1, 4], :on_sp, " "],
>>  [[1, 5], :on_int, "1"]]

Somehow the trailing ' 1' is missed out. Seems like a bug ?

Issue #3765 has been updated by Eito Katagiri.


I think that it is because the provided statement was illegal and Ripper 
stopped parsing it.

  % ruby -ve ":x => 1"
  ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin9.8.0]
  -e:1: syntax error, unexpected tASSOC, expecting $end
  :x => 1
       ^

----------------------------------------
http://redmine.ruby-lang.org/issues/show/3765

Issue #3765 has been updated by Yusuke Endoh.

Status changed from Open to Rejected

Eito is right.

Ruby lexer has very complex states.
Such an incomplete code fragment may not parse.
You should write complete code:

  $ ruby -rripper -e 'p *Ripper.lex("foo :x => 1")'
  [[1, 0], :on_ident, "foo"]
  [[1, 3], :on_sp, " "]
  [[1, 4], :on_symbeg, ":"]
  [[1, 5], :on_ident, "x"]
  [[1, 6], :on_sp, " "]
  [[1, 7], :on_op, "=>"]
  [[1, 9], :on_sp, " "]
  [[1, 10], :on_int, "1"]

--
Yusuke Endoh <mame@tsg.ne.jp>
----------------------------------------
http://redmine.ruby-lang.org/issues/show/3765