rubylexer version 0.7.4 has been released!
RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is
meant
as a lexer for Ruby that’s complete and correct; all legal Ruby
code should be lexed correctly by RubyLexer as well. Just enough parsing
capability is included to give RubyLexer enough context to tokenize
correctly
in all cases. (This turned out to be more parsing than I had thought or
wanted to take on at first.) RubyLexer handles the hard things like
complicated strings, the ambiguous nature of some punctuation characters
and
keywords in ruby, and distinguishing methods and local variables.
Changes:
0.7.4/5-20-2009
-
2 Major Enhancements:
- preliminary support for ruby 1.9
- utf8 inputs should now work… more or less
-
5 Bugfixes:
- better detection of illegal escapes and interpolations in strings
- indicate error on unterminated here body
- fixed pattern of keywords that can’t start a param list (ignores ?,!
now) - in is_var_name?, check for global/instance vars first
- comma and star in a true lhs should be correctly marked as such, now
-
2 Minor Enhancements:
- added tag field to Token; I hope many flags can be coalesced into
tag. - note line that all strings (and here docs) start and end on
- added tag field to Token; I hope many flags can be coalesced into