Ruby needs continuations

Ruby
1

Warning: I don’t really know what I’m talking about so if I make any
mistakes in terminology, please try to correct me…

I've been doing a lot of Python programming and I've discovered that

it’s actually a very powerful language. The language, itself, lacks any
kind of elegance but it has all the power of Ruby and a little more
performance. It also has a richer set of libraries, although not in all
areas, surprisingly.
One thing that Ruby should take from Python are continuations.
Python
is moving away from list creation and version 3.0 functions will return
Python iterators, implemented with continuations, instead of actual
lists,
since they are not used nearly as much as you might think.
The reason why I say that Ruby needs continuations is because they
are
more versatile than Ruby iterators. The reason why I say that is
because
you can make Ruby iterators with continuations but you can’t make
continuations with Ruby iterators. This means that you can implement a
continuation iterator and Object class can automatically define a Ruby
iterator based on your continuation iterator. After all, Ruby iterators
are
nice. I’m surprised by how annoyed I am that Python for loops don’t
return
a value…
One weakness of Ruby iterators that continuations don’t have is
parallel
iteration. If you have two containers that represent different aspects
of
the same things, it’s difficult to iterate over both of them in Ruby.
In
Python, you can do this:

list1 = [1, 2, 3, 4]
list2 = [‘a’, ‘b’, ‘c’, ‘d’]

Okay, Python is more wordy than I’d like

I could have just used zip() but you rarely need the list!

for num, letter in itertools.izip(list1, list2):
# Do something with both numbers and letters
print num, letter
print

How would one do this in Ruby?  You can use .each_with_index and

index
the other list but that assumes that the other list is indexable, which
only
happens to be true of arrays but is not true in general. I’ve thought
about
implementing some Ruby equivalent of zip() (or preferably izip()) and
discovered that I can’t do so without continuations.
…Hence, my post. I’m actually at odds with Pythonic philosophy.
The
idea that there should only be one way to do things is ludicrous and a
constant up hill battle. One thing that Python does right is that it’s
not
afraid to “steal” from other languages and that’s the right attitude to
have. Adopt whatever is useful!
Python 3.0 looks like great language. I’m hoping that Ruby 2.0 will
be
even better 'cause, frankly, Ruby is more fun to program in…
Thank you…

2

On Nov 28, 2007 10:30 AM, Just Another Victim of the Ambient M. <
[email protected]> wrote:

How would one do this in Ruby? You can use .each_with_index and index
the other list but that assumes that the other list is indexable, which
only
happens to be true of arrays but is not true in general. I’ve thought
about
implementing some Ruby equivalent of zip() (or preferably izip()) and
discovered that I can’t do so without continuations.

I can’t comment much on the rest of your post, considering I lack
knowledge
both on the Python camp and the Ruby camp to make any meaningful
consideration.

Ruby does, however, have an implementation of zip in Enumerable. You can
do

list1 = [1, 2, 3, 4]
list2 = [‘a’, ‘b’, ‘c’, ‘d’]

list1.zip(list2).each {|num, letter| puts “#{num} #{letter}” }

just fine. You’ll actually get the same result you would with your
Python
example.

Best regards,

Vitor

3

[1,2,3].zip([4,5,6]).each{|a,b| p a => b}

also, ruby does have contiuations, check your facts (ri callcc) before
trying to complain, otherwise you will just appear as yet another
troll.

^ manveru

On 11/28/07, Just Another Victim of the Ambient M.

4

On 11/28/07, Vitor P. [email protected] wrote:

On Nov 28, 2007 10:30 AM, Just Another Victim of the Ambient M. <
[email protected]> wrote:

How would one do this in Ruby? You can use .each_with_index and index
the other list but that assumes that the other list is indexable, which
only happens to be true of arrays but is not true in general. I’ve thought
about implementing some Ruby equivalent of zip() (or preferably izip()) and
discovered that I can’t do so without continuations.

Actually, Ruby 1.9 is losing continuations (hard to do with native
threads, expensive for stack frame manipulation in JRuby). However,
you don’t need continuations to do parallel iteration, as Vitor
suggests:

Ruby does, however, have an implementation of zip in Enumerable. You can do

list1 = [1, 2, 3, 4]
list2 = [‘a’, ‘b’, ‘c’, ‘d’]

list1.zip(list2).each {|num, letter| puts “#{num} #{letter}” }

just fine. You’ll actually get the same result you would with your Python
example.

You’re also able to do:

a.zip(b, c, d, …)

to iterate multiple lists in parallel.

-austin

5

“Austin Z.” [email protected] wrote in message
news:[email protected]…

discovered that I can’t do so without continuations.
list2 = [‘a’, ‘b’, ‘c’, ‘d’]
to iterate multiple lists in parallel.
Thank you, both, for pointing this out!
Considering my failure to create such a method, I was about to ask
how
such a thing was implemented but then I just tried it out and I’ve
discovered how: arrays. I’ve already mentioned that not all containers
are
indexable: linked lists are not and trees are not efficiently.
Continuations allow even non-indexable containers to be iterated in
parallel…

6

On Nov 28, 2007 12:30 PM, Just Another Victim of the Ambient M.
[email protected] wrote:

Warning: I don’t really know what I’m talking about so if I make any
mistakes in terminology, please try to correct me…

I've been doing a lot of Python programming and I've discovered that

it’s actually a very powerful language. The language, itself, lacks any
kind of elegance but it has all the power of Ruby and a little more
performance. It also has a richer set of libraries, although not in all
areas, surprisingly.
One thing that Ruby should take from Python are continuations.

Ruby has continuations. Though in practice they are not used much.
They have caused serious pain in implementing JRuby for instance,
to the point that IIRC JRuby does not support them out of the box (and
this doesn’t diminish JRuby’s value really).

But there are a lot of really smart people looking into Ruby right now.
Don’t expect the dormant Lisp-like features to stay unused.

Python ‘lacks any kind of elegance’? Thats unfair. It is more correct to
say, that Python has a different value system for code expressiveness.

Python and Ruby are more or less peers on the Language Power Continuum.
Choice between either will depend more on specific task details,
specific
library availability or personal preference.

7

On Nov 28, 2007, at 7:00 AM, Austin Z. wrote:

about implementing some Ruby equivalent of zip() (or preferably
izip()) and
discovered that I can’t do so without continuations.

Actually, Ruby 1.9 is losing continuations (hard to do with native
threads, expensive for stack frame manipulation in JRuby).

That’s not entirely true. In Ruby 1.9, continuations have been moved
out of the core but are still available from a standard library.

James Edward G. II

8

“Michael F.” [email protected] wrote in message
news:[email protected]…

[1,2,3].zip([4,5,6]).each{|a,b| p a => b}

also, ruby does have contiuations, check your facts (ri callcc) before
trying to complain, otherwise you will just appear as yet another
troll.

Correct me if I'm wrong but aren't continuations being dropped for

Ruby
2.0?

9

On Nov 28, 6:29 am, “Just Another Victim of the Ambient M.”
[email protected] wrote:

Python iterators, implemented with continuations, instead of actual lists,
iteration. If you have two containers that represent different aspects of
print num, letter
afraid to “steal” from other languages and that’s the right attitude to
have. Adopt whatever is useful!
Python 3.0 looks like great language. I’m hoping that Ruby 2.0 will be
even better 'cause, frankly, Ruby is more fun to program in…
Thank you…

You actually need a generator, which can be implemented with
continuations, but doesn’t have to be (in 1.9 the generators are
implemented with threads – have a look at generator.rb in 1.8 and 1.9
to see the differences). Using generators, you can implement izip (and
all of itertools if you’d like, though most of it already has
equivalent ruby solutions). Here is a pretty close translation of the
python given in the itertools docs:

========

require ‘generator’

module Enumerable
def izip(*enumerables)
enumerables = [self] + enumerables
generators = enumerables.map { | enum |
Generator.new(enum)
}
while generators[0].next?
result = generators.map { | gen |
gen.next
}
yield result
end
end
end

[1,2,3].izip([4,5,6]) { | x, y |
puts x, y
}

========

NB. generator.rb says that generators are slow in 1.8.

Regards,
Jordan

10

On Nov 28, 9:19 am, MonkeeSage [email protected] wrote:

kind of elegance but it has all the power of Ruby and a little more
continuation iterator and Object class can automatically define a Ruby

implementing some Ruby equivalent of zip() (or preferably izip()) and
You actually need a generator, which can be implemented with

  yield result

NB. generator.rb says that generators are slow in 1.8.

Regards,
Jordan

Ps. The SyncEnumerator class from generator does the same thing as
izip:

SyncEnumerator.new([1,2,3], [4,5,6]).each { | x, y |
puts x, y
}

11

MonkeeSage wrote:

You actually need a generator, which can be implemented with
continuations, but doesn’t have to be (in 1.9 the generators are
implemented with threads – have a look at generator.rb in 1.8 and 1.9
to see the differences). Using generators, you can implement izip (and
all of itertools if you’d like, though most of it already has
equivalent ruby solutions). Here is a pretty close translation of the
python given in the itertools docs:

Ruby 1.9 also moves continuations out of core, but provides a form of
bounded continuations (coroutines) that’s possible to implement in
JRuby:

require ‘fiber’
f = Fiber.new { a = 1; while true; Fiber.yield a; a += 1; end }
5.times { puts f.resume }

$ jruby -J-Djruby.compat.version=ruby1_9 fiber_example.rb
1
2
3
4
5

Under JRuby, it’s using a native thread per live Fiber, so it’s heavier
than in Ruby 1.9 which uses green threads. However they’ll actually run
in parallel on JRuby, so that’s a benefit. JRuby also supports a thread
pool in 1.1 that helps blunt the cost of spinning up native threads.

  • Charlie
12

“Richard C.” [email protected] wrote in message
news:[email protected]…

areas, surprisingly.
Python ‘lacks any kind of elegance’? Thats unfair. It is more correct to
say, that Python has a different value system for code expressiveness.

Indeed, I wasn't trying to be fair, I was just expressing a personal

preference. While Python is clearly a powerful and useful language (I’m
programming in it, aren’t I?), its syntax choices irk me to no end.
len()
is a function that can take a list but .append() is a method of list’s?
Arg! Zero and empty strings evaluate to false? Arg! Both zip() and
dict.items() are built-in but izip() hides in the itertools module while
dict.itertimes() gets to be built-in? Arg! Considering how iterators
are
more useful than actual lists, double arg!
As it so happened, I did expand on some of the differences between
Python’s code values and mine, later on in my post…

Python and Ruby are more or less peers on the Language Power Continuum.
Choice between either will depend more on specific task details, specific
library availability or personal preference.

Of course.  I'm programming in Python, aren't I?
13

On Nov 28, 2007, at 6:00 AM, Austin Z. wrote:

a.zip(b, c, d, …)

this iterates a single freshly allocated array:

static VALUE
rb_ary_zip(argc, argv, ary)
int argc;
VALUE *argv;
VALUE ary;
{
int i, j;
long len;
VALUE result;

 for (i=0; i<argc; i++) {

argv[i] = to_ary(argv[i]);
}
if (rb_block_given_p()) {
for (i=0; i<RARRAY(ary)->len; i++) {
VALUE tmp = rb_ary_new2(argc+1);

  rb_ary_push(tmp, rb_ary_elt(ary, i));
  for (j=0; j<argc; j++) {
rb_ary_push(tmp, rb_ary_elt(argv[j], i));
  }
  rb_yield(tmp);

}
return Qnil;
}
len = RARRAY(ary)->len;
result = rb_ary_new2(len);
for (i=0; i<len; i++) {
VALUE tmp = rb_ary_new2(argc+1);

rb_ary_push(tmp, rb_ary_elt(ary, i));
for (j=0; j<argc; j++) {
rb_ary_push(tmp, rb_ary_elt(argv[j], i));
}
rb_ary_push(result, tmp);
}
return result;
}

when a, b, c, and d are large it’s most undesirable to use zip,
forcing one to use

lists = a,b,c,d
n = lists.map{|list| list.size}.max

(0 … n).each do |i|
ai, bi, ci, di = lists.map{|list| list[i]}

…

end

and, when b, c, and d are heterogeneous containers the above code is
most often impossible - you will first need to convert all the
containers to lists and then iterate. for me this is a major gripe
about ruby that i work around nearly every day as it turns out many
algorithms, such as those in image processing, revolve around
iterating multiple containers at once: often in non-linear ways. one
way around, without callcc, is just to use threads:

cfp:~ > cat a.rb
require ‘thread’
module Enumerable
def self.each *enumerables, &block
n = enumerables.size
done = Object.new
qs = Array.new(n){ SizedQueue.new 1 }
ts = Array.new(n){ |i|
Thread.new{
Thread.current.abort_on_exception = true
e, q = enumerables[i], qs[i]
e.each{|x| q.push x}
q.push done
}
}
dead = Array.new(n){ false }
loop do
values = []
finished = true
qs.each_with_index do |q,i|
if dead[i]
values << nil
next
end
value = q.pop
if value == done
dead[i] = true
values << nil
else
values << value
end
end
return enumerables if dead.all?
block.call *values
end
end
end

a = 0,1,2
b = 2,3,4
c = 5,6,7

lists = a,b,c

Enumerable.each(a,b,c){|x,y,z| p [x,y,z]}

cfp:~ > ruby a.rb
[0, 2, 5]
[1, 3, 6]
[2, 4, 7]

but this, i think we can all agree, is slow and fugly. esp compared
to something like this

http://kogs-www.informatik.uni-hamburg.de/~koethe/vigra/doc/vigra/
group__ImageIterators.html

external iterators are quite useful imho.

kind regards.

a @ http://codeforpeople.com/

14

“MonkeeSage” [email protected] wrote in message
news:[email protected]…

}

puts x, y
Ps. The SyncEnumerator class from generator does the same thing as
izip:

SyncEnumerator.new([1,2,3], [4,5,6]).each { | x, y |
puts x, y
}

Thank you, MonkeeSage, this is exactly what I'm looking for!
It's interesting that generators are slow enough to warn users about

its
lack of speed! Do you know if it would be any faster implemented with
continuations? I’m surprised it’s so slow considering Ruby employs
green
threads…

15

On Nov 28, 10:53 am, “Just Another Victim of the Ambient M.”
[email protected] wrote:

  yield result

}

Thank you, MonkeeSage, this is exactly what I'm looking for!
It's interesting that generators are slow enough to warn users about its

lack of speed! Do you know if it would be any faster implemented with
continuations? I’m surprised it’s so slow considering Ruby employs green
threads…

Other way around. :wink: The continuation version is slow. The threaded
version is fast.

16

“MonkeeSage” [email protected] wrote in message
news:[email protected]…

Thank you, MonkeeSage, this is exactly what I'm looking for!
It's interesting that generators are slow enough to warn users about

its
lack of speed! Do you know if it would be any faster implemented with
continuations? I’m surprised it’s so slow considering Ruby employs green
threads…

Other way around. :wink: The continuation version is slow. The threaded
version is fast.

Ah, that's good to hear!  I was confused there.  It looks like, as

long
as we have these green threads, we won’t really need continuations.
This is starting to get off topic but, if you happen to know, if
Ruby is
moving towards native threads, will we keep green threads around?
They’re
useful things, even with native threads available…

17

On Nov 28, 9:41 am, Charles Oliver N. [email protected]
wrote:

bounded continuations (coroutines) that’s possible to implement in JRuby:
5

Under JRuby, it’s using a native thread per live Fiber, so it’s heavier
than in Ruby 1.9 which uses green threads. However they’ll actually run
in parallel on JRuby, so that’s a benefit. JRuby also supports a thread
pool in 1.1 that helps blunt the cost of spinning up native threads.

  • Charlie

I just tried the above izip function in jruby, but it only gave me two
iterations…

$ jruby test.rb
1
4
2
5

…hmm?

$ jruby -v
ruby 1.8.5 (2007-11-28 rev 3841) [i386-jruby1.0]

Regards,
Jordan

18

On Nov 28, 7:29 am, “Just Another Victim of the Ambient M.”
[email protected] wrote:

for num, letter in itertools.izip(list1, list2):
# Do something with both numbers and letters
print num, letter
print

How would one do this in Ruby?

At RubyConf 2007 last month, Matz announced that 1.9 would have
external iterators. That would allow you to do this:

num_iter = list1.each
let_iter = list2.each
loop {
puts num_iter.next, let_iter.next
}

When you reach the end of either list (or any list, if we scale beyond
just two), the call to next would raise an exception, automatically
caught by the loop construct, causing the loop to simply exit. So
this construct works gracefully even if one of the lists is shorter
than the other(s).

Eric

P.S. I put together a very short presentation for the Southeast
Michigan Ruby U.’ Group summarizing what Matz said about 1.9. You
can see the slides at:

http://learnruby.com/ruby-1.9.html

====

Are you interested in on-site Ruby training that uses well-designed,
real-world, hands-on exercises? http://LearnRuby.com

19

On Nov 28, 12:08 pm, “Eric I.” [email protected] wrote:

list1 = [1, 2, 3, 4]

When you reach the end of either list (or any list, if we scale beyond

http://learnruby.com/ruby-1.9.html

====

Are you interested in on-site Ruby training that uses well-designed,
real-world, hands-on exercises?http://LearnRuby.com

Cool! I just tried that, and they’re already in there (1.9.0
patchlevel 0). :slight_smile:

Regards,
Jordan

20

ara.t.howard wrote:

…

end

zip doesn’t force you to do this, you can also pass a block to zip. In
Ruby 1.9. zip without a block won’t create and return an array anymore,
but it will return an Enumerator object as a promise instead.

and, when b, c, and d are heterogeneous containers the above code is
most often impossible - you will first need to convert all the
containers to lists and then iterate.
If you want zip to act smarter on it’s arguments you will have to use
Ruby 1.9.