# Rubylexer 0.7.0 Released

**URL:** <https://www.ruby-forum.com/t/rubylexer-0-7-0-released/129826>\
**Category:** Ruby\
**Created:** [February 21, 2008, 10:11pm UTC](https://www.ruby-forum.com/t/rubylexer-0-7-0-released/129826 "2008-02-21T22:11:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pandemic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/8491ac/32.png) [@pandemic](https://www.ruby-forum.com/u/pandemic)\
**Post date:** [February 21, 2008, 10:11pm UTC](https://www.ruby-forum.com/t/rubylexer-0-7-0-released/129826/1 "2008-02-21T22:11:21Z")

</div>

rubylexer version 0.7.0 has been released!

[http://rubyforge.org/projects/rubylexer](http://rubyforge.org/projects/rubylexer)

RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is  
meant  
as a lexer for Ruby that’s complete and correct; all legal Ruby  
code should be lexed correctly by RubyLexer as well. Just enough parsing  
capability is included to give RubyLexer enough context to tokenize  
correctly  
in all cases. (This turned out to be more parsing than I had thought or  
wanted to take on at first.) RubyLexer handles the hard things like  
complicated strings, the ambiguous nature of some punctuation characters  
and

keywords in ruby, and distinguishing methods and local variables.

This version (long overdue) fixes most known minor problems with the  
previous release. The output is now more suitable for use by a parser.  
The next major goal is the Big String Reform which will remove the  
tokens-inside-a-token hack needed to support string inclusions  
currently.

Changes:

### 0.7.0/2-15-2008

- implicit tokens are now emitted at the right times (need more test  
code)
- local variables are now temporarily hidden by class, module, and def
- line numbers should always be correct now (=begin…=end caused this)  
(??)
- fixed meth/var-name confusion in idents after ‘def’ but before params
- FileAndLineToken now emitted on all new lines (0.8)
- offset of **END** now works(??)
- put files into lib/\* eof in unusual cases is better handled
- improvements in parsing unary \* and & (??)
- input can now come from a string
- local vars (defs and uses) are recognized in string inclusions
- explicitly decimal numbers, eg: 0d123456789
- eof in unusual cases is better handled
- **END** is not treated as a keyword
- ‘|’ as goalpost is now better handled
- a number of things have been renamed internally
- no more implicit parens for setter method calls
- ‘{’ after return, break, and next is now the start of a hash.
- ambiguous cases of ‘::’,‘(’,‘:’,and ‘?’ are now better handled.
- more start/end hint tokens (for ‘when’, ‘rescue’, and ‘for’)
- bugfixes in rhs hint tokens
- parsing of def headers for singleton methods is improved
- rescue as operator is now handled
- block param list lookahead is simplified
- unary ops (including \* and &) can be easily distinguished in output
- here document bodies better handled, esp after escaped newline
- symbols like %s{symbol} now actually work
- implicit parens around =\> better handled…
- different cases of ‘{’ can now be easily distinguished
- ImplicitParamList Start and End are now Keyword, not Ignore tokens.

To get rubylexer, type:  
gem install rubylexer  
Or download:  
[http://rubyforge.org/frs/download.php/32769/rubylexer-0.7.0.tgz](http://rubyforge.org/frs/download.php/32769/rubylexer-0.7.0.tgz)

project homepage:  
[http://rubyforge.org/projects/rubylexer](http://rubyforge.org/projects/rubylexer)