Forum: Ruby lightweigth lexer?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Ff884553e8061a725a64458afd710be3?d=identicon&s=25 Meador Inge (Guest)
on 2006-05-13 02:37
(Received via mailing list)
Is there lightweight lexer for Ruby? I have seen a few lexer
generators out there, but I want something simple that can be used
without generating the lexer first. I really just need the basic
functionaliy of matching a pattern and associating an action with that
pattern. For example, I was thinking something like:
    class MyLexer < Lexer
       rule /a/ do
          # action for pattern a
       end
       rule /b/ do
          # action for pattern b
       end
       rule /c/ do
          # action for pattern c
       end
    end

    l = MyLexer.new(input_stream)
    l.each_token do |token|
       # do something with token
    end

I have something similar to the above that I have created. However, I
wanted to check and see what is out there first before I invested too
much time in it. If no one knows of anything else and\or is interested
in what I have described let me know and I will clean it up and
publish it.

--Meador
E34b5cae57e0dd170114dba444e37852?d=identicon&s=25 Logan Capaldo (Guest)
on 2006-05-13 03:04
(Received via mailing list)
On May 12, 2006, at 8:35 PM, Meador Inge wrote:

>          # action for pattern b
>
> I have something similar to the above that I have created. However, I
> wanted to check and see what is out there first before I invested too
> much time in it. If no one knows of anything else and\or is interested
> in what I have described let me know and I will clean it up and
> publish it.
>
> --Meador
>

StringScanner is close if you combine it with ifs.
481b8eedcc884289756246e12d1869c1?d=identicon&s=25 Francis Cianfrocca (Guest)
on 2006-05-13 04:47
(Received via mailing list)
If you look closely at what traditional tools like lex and flex do, you
could almost take the input to lex and execute it directly in Ruby! It's
not
much more than a case statement from regular expressions. I'm not sure
what
you mean by "lightweight" but if you mean "fast": a lot of people who
wrote
language compilers back in the day (I'm one of them) would often write
scanners in hand-tuned C rather than using lex.
Ff884553e8061a725a64458afd710be3?d=identicon&s=25 Meador Inge (Guest)
on 2006-05-13 06:03
(Received via mailing list)
> If you look closely at what traditional tools like lex and flex do, you
> could almost take the input to lex and execute it directly in Ruby!
They do look very similar.

> It's not much more than a case statement from regular expressions.
It is a bit more that a case statement on regular expressions:). You
have to
do a bit of work to manage the input buffer and ensure that the longest
match is made (e.g. greedy) for each token. Also, if you want some of
the
other traditional features of lex and flex you have to add features such
as
start states, putting characters back into the input stream, reject
matches,
etc...

> I'm not sure what you mean by "lightweight"
I more or less meant simple and easy to use (maybe I should have just
said
so:)) without the full range of features of a typical lex\flex knockoff.
481b8eedcc884289756246e12d1869c1?d=identicon&s=25 Francis Cianfrocca (Guest)
on 2006-05-13 06:15
(Received via mailing list)
Well, yes, that's what I meant by "not much more." In my experience
(writing
scanners for programming languages) the start states and greedy matching
were the most important features beyond the basic NFA. Unshifting input
seems like more of a job for the parser than the scanner, but different
strokes, I guess. Are you going to give this a try?
Ff884553e8061a725a64458afd710be3?d=identicon&s=25 Meador Inge (Guest)
on 2006-05-13 06:24
(Received via mailing list)
> Are you going to give this a try?
I think I just may. I will be sure to post to ruby-talk if I come up
with
anything.
481b8eedcc884289756246e12d1869c1?d=identicon&s=25 Francis Cianfrocca (Guest)
on 2006-05-13 06:39
(Received via mailing list)
Might be fun to integrate it with racc.
0b561a629b87f0bbf71b45ee5a48febb?d=identicon&s=25 Dave Burt (Guest)
on 2006-05-13 10:20
(Received via mailing list)
Meador Inge wrote:
> Is there lightweight lexer for Ruby?

There's a library called LittleLexer.

http://littlelexer.rubyforge.org/

Cheers,
Dave
61d446522cd03aa337029d5fdb4ad1d1?d=identicon&s=25 Scott Barron (Guest)
on 2006-05-13 15:44
(Received via mailing list)
On 5/13/06, Meador Inge <meadori@gmail.com> wrote:
>
> > Are you going to give this a try?
> I think I just may. I will be sure to post to ruby-talk if I come up with
> anything.


There is rex, http://raa.ruby-lang.org/project/rex/ which takes a very
lex-like input and produces a ruby lexer that can be used with racc.

-Scott
F4ae9e7822cb761275aff25a0341f443?d=identicon&s=25 Phil Jackson (Guest)
on 2006-05-14 14:18
(Received via mailing list)
On Sat, May 13, 2006 at 05:19:45 +0900, Dave Burt wrote:
> Meador Inge wrote:
> > Is there lightweight lexer for Ruby?
>
> There's a library called LittleLexer.
>
> http://littlelexer.rubyforge.org/

The fact he uses the word "cute" on his site puts me off!

Cheers,
Phil
0b561a629b87f0bbf71b45ee5a48febb?d=identicon&s=25 Dave Burt (Guest)
on 2006-05-15 14:08
(Received via mailing list)
Phil Jackson wrote:
> On Sat, May 13, 2006 at 05:19:45 +0900, Dave Burt wrote:
>> Meador Inge wrote:
>>> Is there lightweight lexer for Ruby?
>> There's a library called LittleLexer.
>>
>> http://littlelexer.rubyforge.org/
>
> The fact he uses the word "cute" on his site puts me off!

But he's right!

Cheers,
Dave
This topic is locked and can not be replied to.