# Rubylexer 0.7.1 Released

**URL:** <https://www.ruby-forum.com/t/rubylexer-0-7-1-released/147465>\
**Category:** Ruby\
**Created:** [September 2, 2008, 5:52am UTC](https://www.ruby-forum.com/t/rubylexer-0-7-1-released/147465 "2008-09-02T05:52:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![pandemic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/8491ac/32.png) [@pandemic](https://www.ruby-forum.com/u/pandemic)\
**Post date:** [September 2, 2008, 5:52am UTC](https://www.ruby-forum.com/t/rubylexer-0-7-1-released/147465/1 "2008-09-02T05:52:43Z")

</div>

rubylexer version 0.7.1 has been released!

- [http://rubyforge.org/projects/rubylexer/](http://rubyforge.org/projects/rubylexer/)
- [http://rubylexer.rubyforge.org/](http://rubylexer.rubyforge.org/)

RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is  
meant  
as a lexer for Ruby that’s complete and correct; all legal Ruby  
code should be lexed correctly by RubyLexer as well. Just enough parsing  
capability is included to give RubyLexer enough context to tokenize  
correctly  
in all cases. (This turned out to be more parsing than I had thought or  
wanted to take on at first.) RubyLexer handles the hard things like  
complicated strings, the ambiguous nature of some punctuation characters  
and  
keywords in ruby, and distinguishing methods and local variables.

Users of rubygems can install with this command:  
gem update rubylexer

Or download it here:

- [http://rubyforge.org/frs/download.php/42434/rubylexer-0.7.1.gem](http://rubyforge.org/frs/download.php/42434/rubylexer-0.7.1.gem)
- [http://rubyforge.org/frs/download.php/42433/rubylexer-0.7.1.tgz](http://rubyforge.org/frs/download.php/42433/rubylexer-0.7.1.tgz)

Changes:

### 0.7.1/10-29-2008

- 6 Major Enhancements:

- 5 Minor Enhancements

- 26 Bugfixes

- 18 Internal Changes (not user visible)

- 15 Enhancements and bug fixes to tests:

- [http://rubyforge.org/projects/rubylexer/](http://rubyforge.org/projects/rubylexer/)

- [http://rubylexer.rubyforge.org/](http://rubylexer.rubyforge.org/)

---

<div class="post-metadata">

**Author:** ![pandemic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/8491ac/32.png) [@pandemic](https://www.ruby-forum.com/u/pandemic)\
**Post date:** [September 3, 2008, 7:29am UTC](https://www.ruby-forum.com/t/rubylexer-0-7-1-released/147465/2 "2008-09-03T07:29:48Z")

</div>

> RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is  
> meant  
> as a lexer for Ruby that’s complete and correct; all legal Ruby

What’s its relationship to ruby\_parser and racc?  
Thanks!  
-=R

---

<div class="post-metadata">

**Author:** ![pandemic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/8491ac/32.png) [@pandemic](https://www.ruby-forum.com/u/pandemic)\
**Post date:** [September 5, 2008, 8:16pm UTC](https://www.ruby-forum.com/t/rubylexer-0-7-1-released/147465/3 "2008-09-05T20:16:20Z")

</div>

On 9/2/08, Roger P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is  
> > meant  
> > as a lexer for Ruby that’s complete and correct; all legal Ruby
> 
> What’s its relationship to ruby\_parser and racc?  
> Thanks!

Sorry for the late reply.

Racc is a ruby-based compiler compiler. As far as I know, it’s a  
variant of yacc; but using ruby in actions and generated code instead  
of c. Ruby\_parser is a parser for ruby written in racc/ruby.  
Ruby\_parser is a port of the yacc/c code used to parse ruby in MRI. It  
has it’s own lexer, a port of the MRI c lexer. There is no dependency  
between them. The two libraries are trying to accomplish similar  
goals, but RubyParser doesn’t take the process quite as far.

---

<div class="post-metadata">

**Author:** ![pandemic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/8491ac/32.png) [@pandemic](https://www.ruby-forum.com/u/pandemic)\
**Post date:** [October 8, 2008, 7:20am UTC](https://www.ruby-forum.com/t/rubylexer-0-7-1-released/147465/4 "2008-10-08T07:20:16Z")

</div>

Caleb C. wrote:

> rubylexer version 0.7.1 has been released!  
> RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is  
> meant  
> as a lexer for Ruby that’s complete and correct; all legal Ruby

Looks like it works with 1.8 [though not 1.9]

Here’s a working example [the docs’ is a little off]

lexer=RubyLexer.new(nil, ‘1+1′); until  
RubyLexer::EoiToken===(token=lexer.get1token) do; print token, ‘ ‘; end

Thanks!  
-=R