# "yield" and "old-way iteration"

**URL:** <https://www.ruby-forum.com/t/yield-and-old-way-iteration/77586>\
**Category:** Ruby\
**Created:** [November 28, 2006, 11:20pm UTC](https://www.ruby-forum.com/t/yield-and-old-way-iteration/77586 "2006-11-28T23:20:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bjarke\_Walling](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e19b73/32.png) [@Bjarke\_Walling](https://www.ruby-forum.com/u/Bjarke_Walling)\
**Post date:** [November 28, 2006, 11:20pm UTC](https://www.ruby-forum.com/t/yield-and-old-way-iteration/77586/1 "2006-11-28T23:20:59Z")

</div>

Hi,

I am new to Ruby, but I find the language easy to learn and use.  
However I have some code that I think could be written smarter.

I am writing a lexer and parser for a small language I have created.  
The first part splitting some input into tokens was easy to write using  
“yield” (20 lines or so). I was actually a little overwhelmed how easy  
it was. The next part is to examine these tokens and parse them into  
language structures. I want to create a class with “current” and “next”  
methods to get the current token and fetch the next (advance the  
pointer). I have solved it by using the first lexer yielding tokens and  
collecting them in an array. Afterwards I can fetch tokens from the  
array. But could it be done in a smarter way?

My code is like this:

class FirstParse  
def initialize  
…  
end  
def each  
… yield tokens …  
end  
end

class SecondParse  
def initialize  
@tokens = []  
@index = 0  
first\_parse = FirstParse.new  
for token in first\_parse  
@tokens.add token  
end  
end  
def current  
@tokens[@index]  
end  
def next  
@index++  
self.current  
end  
def read\_structure1  
… read structures …  
end  
def read\_structure2  
… read structures …  
end  
end

Am I being to “Java’ish” or what do you think. It is not a big problem  
since the code works, but do I really need to load the tokens into an  
array first?

- Bjarke Walling

---

<div class="post-metadata">

**Author:** ![Bjarke\_Walling](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e19b73/32.png) [@Bjarke\_Walling](https://www.ruby-forum.com/u/Bjarke_Walling)\
**Post date:** [November 29, 2006, 1:41am UTC](https://www.ruby-forum.com/t/yield-and-old-way-iteration/77586/2 "2006-11-29T01:41:21Z")

</div>

HI –

On Wed, 29 Nov 2006, Bjarke Walling wrote:

> methods to get the current token and fetch the next (advance the  
> def each  
> @tokens.add token  
> end  
> end  
> def current  
> @tokens[@index]  
> end  
> def next  
> @index++

That won’t parse 🙂 There’s no ++ operator in Ruby; you’ll want to  
do:  
@index += 1

> Am I being to “Java’ish” or what do you think. It is not a big problem  
> since the code works, but do I really need to load the tokens into an  
> array first?

I would say that if you’re going to load the tokens into an array,  
don’t do it by yielding; do it by returning an array. (All the code  
that follows is very sketchy and just intended to illustrate the broad  
picture.)

class FirstParse  
attr\_reader :tokens  
def initialize  
# put all tokens in @tokens array  
end  
end

class SecondParse  
def initialize  
@tokens = FirstParse.new.tokens  
end  
end

If you want to, you can just yield from the first parse to the second  
parse.

class FirstParse  
def initialize  
while (# get token from stream)  
yield token  
end  
end  
end

class SecondParse  
def initialize  
FirstParse.new.each do |token|  
# do something with token here – don’t save it  
end  
end  
end

That way you don’t have to maintain your own pointer (though you could  
add it back in if needed it for some other reason).

David

---

<div class="post-metadata">

**Author:** ![Bjarke\_Walling](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e19b73/32.png) [@Bjarke\_Walling](https://www.ruby-forum.com/u/Bjarke_Walling)\
**Post date:** [November 29, 2006, 1:55am UTC](https://www.ruby-forum.com/t/yield-and-old-way-iteration/77586/3 "2006-11-29T01:55:24Z")

</div>

Bjarke Walling wrote:

> I am writing a lexer and parser for a small language I have created.  
> The first part splitting some input into tokens was easy to write using  
> “yield” (20 lines or so). I was actually a little overwhelmed how easy  
> it was. The next part is to examine these tokens and parse them into  
> language structures. I want to create a class with “current” and “next”  
> methods to get the current token and fetch the next (advance the  
> pointer). I have solved it by using the first lexer yielding tokens and  
> collecting them in an array. Afterwards I can fetch tokens from the  
> array. But could it be done in a smarter way?

You’re not providing much context. I am assuming that you want the  
current/next approach because your parser will pull tokens, presumably  
because you’re using recursive descent or another top-down parsing  
method.

If that’s what you are doing, and you want to stick with that (as  
opposed to switching to a bottom-up parser), then you’re dealing with a  
classic “inversion of control” problem.

I don’t really think making the first lexer yielding tokens buys you  
much over just making the parser call methods in the lexer to tokenize  
and return the tokens as a normal method call. In other words, if  
you’re using a top-down parsing method, you really want to consider  
making your parser pull tokens from the lexer, instead of having the  
lexer push tokens to the parser, which is what you are doing when you  
use yield.

However, if you want to stick to using yield, you can use “Generator”  
(see [http://ruby-doc.org/core/classes/Generator.html](http://ruby-doc.org/core/classes/Generator.html)) to invert the  
control and let you “pull” tokens from your yield’ing lexer without  
having to go via an array.

Vidar

---

<div class="post-metadata">

**Author:** ![Bjarke\_Walling](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e19b73/32.png) [@Bjarke\_Walling](https://www.ruby-forum.com/u/Bjarke_Walling)\
**Post date:** [November 29, 2006, 2:17pm UTC](https://www.ruby-forum.com/t/yield-and-old-way-iteration/77586/4 "2006-11-29T14:17:01Z")

</div>

Thank you for all your replies!

The real problem is that I am not that skilled in writing a parser, I  
think.

It might be the “inversion of control” problem I experience. The  
problem is that my FirstParse class push tokens using “yield” and in my  
SecondParse I want to pull tokens and decide upon them, and pull the  
next when I’m ready for it. I don’t know how to write the SecondParse  
another way without the code becomming too complex, but I have a book  
on grammers and languages. I read the part on regular expressions and  
on Turing machines, but missed out the grammers part. I have to read at  
least some of it 🙂

But for now I solve it by loading the tokens directly into an array in  
the first parse.

- Bjarke Walling

---

<div class="post-metadata">

**Author:** ![Bjarke\_Walling](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/e19b73/32.png) [@Bjarke\_Walling](https://www.ruby-forum.com/u/Bjarke_Walling)\
**Post date:** [November 29, 2006, 2:17am UTC](https://www.ruby-forum.com/t/yield-and-old-way-iteration/77586/5 "2006-11-29T02:17:29Z")

</div>

On Nov 28, 2006, at 1420 , Bjarke Walling wrote:

> collecting them in an array. Afterwards I can fetch tokens from the  
> array. But could it be done in a smarter way?
> 
> My code is like this:
> 
> class FirstParse

```
 include Enumerable

```

> def initialize  
> …  
> end  
> def each  
> … yield tokens …  
> end  
> end
> 
> class SecondParse  
> def initialize  
> def initialize(tokens)  
> @index = 0  
> @tokens = tokens  
> end  
> def read\_structure2  
> … read structures …  
> end  
> end

parser = SecondParse.new FirstParse.new.to\_a

Passing in an Array of tokens makes it easier to test, too.

–  
Eric H. - removed\_email\_address@domain.invalid - [http://blog.segment7.net](http://blog.segment7.net)

I LIT YOUR GEM ON FIRE!