Lazy.rb 0.9.5 -- transparent futures!


#1

I’d like to announce a new version of lazy.rb – this one offering
thread safety and transparent futures!

Here’s the web site, complete with gem and tarball downloads, plus a bit
of documentation:

http://moonbase.rydia.net/software/lazy.rb/

Bug reports would be very, very welcome.

== What is lazy.rb?

lazy.rb is a library providing transparent lazy evaluation and futures
for Ruby. It provides a bag of clever tricks to help you avoid doing
expensive computations up front.

= Lazy Evaluation

Lazy evaluation simply refers to computations which are run on an
as-needed basis. For example:

x = promise { 3 + 5 }

Means that the block – 3 + 5 – won’t actually be evaluated until
something tries to use x’s value.

p x # => #<Lazy::Promise computation=#Proc:...>

forces evaluation

p x * 3 # => 24

p x # => 8

You can also force evaluation using demand:

x = promise { 3 + 5 }

p x # => #<Lazy::Promise computation=#Proc:...>

forces evaluation

p demand( x ) # => 8

p x # => 8

It’s a bit silly for 3 + 5, but it’s handy for more intensive
calculations. You can unconditionally promise a computation, yet only
pay for it if and when its result is actually used.

= Futures

Futures are blocks of code that are evaluated immediately, but in a
background thread.

x = future { 3 + 5 }

p x # => #<Lazy::Future computation=#Proc:...>

You could do other stuff here while

the computation ran in the background

blocks until the background thread completes

p x * 3 # => 24

p x # => 8

Again, silly for 3 + 5 perhaps, but I’m sure you can see how this might
come in handy for more involved computations.

= Other stuff

lazy.rb also includes support for circular programming, where a
computation is passed its own result:

matryoshka = demand( promise { |result| [result] } )

p matryoshka # => [[…]]

p matryoshka.object_id # => -605506544

p matryoshka.first.object_id # => -605506544

p matryoshka.first.first.object_id # => -605506544

This works for both promises and futures, although it has the usual
limitations: if a computation tries to call methods on its own result,
it will diverge.

== What’s new in 0.9.5?

  • Optional support for multithreaded programs:

    require ‘lazy/threadsafe’ and you can safely use lazy evaluation in
    multithreaded programs.

  • Futures:

    With thread support, it turned out that futures were really
    easy to implement atop promises – just fire off a thread with the
    computation and return a promise to join the thread and grab its
    result. So I implemented that.

== What happened to lazy streams from 0.2?

I ditched the lazy streams API for now. It just wasn’t working out.

== What next?

Except perhaps for lazy streams (which might end up becoming a separate
library), I think we’re nearly feature-complete. Ideas and suggestions
are very welcome, though.

-mental


#2

On Feb 18, 2006, at 7:07 PM, MenTaLguY wrote:

the computation ran in the background

blocks until the background thread completes

p x * 3 # => 24

p x # => 8

Again, silly for 3 + 5 perhaps, but I’m sure you can see how this
might
come in handy for more involved computations.

Wow, that is just super studly!

== What happened to lazy streams from 0.2?

I ditched the lazy streams API for now. It just wasn’t working out.

Can you expand on what you mean by this? I’m working on a huge
article about infinite streams for my blog and lazy.rb 0.2 was a big
inspiration to me.

James Edward G. II


#3

MenTaLguY wrote:

I’d like to announce a new version of lazy.rb – this one offering
thread safety and transparent futures!

Nice.

forces evaluation

p x * 3 # => 24

What will it do for:
p 3 * x

Thanks for sharing.


#4

If you suspected “the right thing” you’d probably be right:

irb(main):001:0> require ‘lazy’
irb(main):002:0> x = promise { 5 + 3 }
=> #<Lazy::Promise computation=#Proc:0x0035788c@:2(irb)>
irb(main):003:0> p 3 * x
24
=> nil


#5

Quoting James Edward G. II removed_email_address@domain.invalid:

I ditched the lazy streams API for now. It just wasn’t working
out.

Can you expand on what you mean by this? I’m working on a huge
article about infinite streams for my blog and lazy.rb 0.2 was a
big inspiration to me.

Oh, mainly it was an aesthetic thing. It worked fine.

I’d been fighting with the streams API to make it more Ruby-esque
and easier to use properly, but I finally punted on it to get this
release out the door quickly.

In retrospect, I probably should have just kept the 0.2 API for now.
I definitely want something better for 1.0, though.

-mental


#6

Hah! Awesome mental, I was going to be implementing something like
this for an Actor library that I’m going to start work on Real Soon
™. And, of course, you did it way better than I would have.

One question: How hard would it be to modify the way that exceptions
are handled to hold off raising the exception until the result was
requested? I’m thinking specifically of some non-deterministic
situations where you may request a value but never end up needing to
use it.


#7

Quoting “removed_email_address@domain.invalid” removed_email_address@domain.invalid:

If you suspected “the right thing” you’d probably be right:

irb(main):001:0> require ‘lazy’
irb(main):002:0> x = promise { 5 + 3 }
=> #<Lazy::Promise computation=#Proc:0x0035788c@:2(irb)>
irb(main):003:0> p 3 * x
24
=> nil

It’s worth noting that this only works so well because of the very
nice coercion machinery Ruby has built around its numeric classes.

Ruby’s NUM2INT (for example) will call .to_int on the promise when
FIXNUM_P fails, so everything Just Works™.

For other cases (e.g. promises returning file handles), you may find
you need to unwrap them explicitly with demand() to pass them to
certain methods.

Of course this is an issue only when passing promises to methods
implemented in C; lazy.rb does a very good job of faking out Ruby
otherwise (the boolean issue notwithstanding).

-mental


#8

On Sun, 19 Feb 2006 10:07:12 +0900, MenTaLguY removed_email_address@domain.invalid wrote:

http://moonbase.rydia.net/software/lazy.rb/

Bug reports would be very, very welcome.

When I toyed with a proof-of-concept of Io’s asynchronous messages in
Ruby

http://www.siaris.net/index.cgi/Programming/LanguageBits/Ruby/Async.rdoc

Jim W. pointed out that methods later added to Object or Kernel
will
no longer be “missing” in the Async (or Future) class. One work around
is
to trap “method_added” up the chain, for example:

http://www.siaris.net/cgi-bin/siwiki.pl?FeedBack/AsyncMessages

But then I discovered that methods added by including a module weren’t
trapped by method_added – so in the end, a KernellessObject (from
evil.rb)
was needed (same link as above) as Async’s parent to keep the proxy
clean.

regards,
andrew


#9

Minkoo S. wrote:

Sorry for newbie question. I tried to install lazy only to fail:

[root@poseidon tmp]# ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[root@poseidon tmp]# ls -l lazy-0.9.5.gem
-rw-r–r-- 1 mkseo users 6656 2ì?? 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of ‘lazy-0.9.5.gem’
Successfully installed lazy, version 0.9.5
^^^^^^^^^^^^

The installatiof of the lazy software is OK …

Installing RDoc documentation for lazy-0.9.5…

lazy.rb:60:22: Couldn’t find DIVERGES. Assuming it’s a module

Its just that RDoc is complaining about something. I got the same error
on my system, but the RDoc looks ok, even with the error.


– Jim W.


#10

Sorry for newbie question. I tried to install lazy only to fail:

[root@poseidon tmp]# ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[root@poseidon tmp]# ls -l lazy-0.9.5.gem
-rw-r–r-- 1 mkseo users 6656 2ì?? 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of ‘lazy-0.9.5.gem’
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5…

lazy.rb:60:22: Couldn’t find DIVERGES. Assuming it’s a module
[root@poseidon tmp]#

Any idea?

Minkoo S.


#11

Still, no luck.

[root@poseidon tmp]# ll
í?©ê³? 12
-rw-r–r-- 1 mkseo users 6656 2ì?? 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of ‘lazy-0.9.5.gem’
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5…

lazy.rb:60:22: Couldn’t find DIVERGES. Assuming it’s a module
[root@poseidon tmp]# irb
irb(main):001:0> require ‘lazy’
LoadError: no such file to load – lazy
from (irb):1:in require' from (irb):1 irb(main):002:0> require 'lazy/future' LoadError: no such file to load -- lazy/future from (irb):2:inrequire’
from (irb):2
irb(main):003:0>

[root@poseidon tmp]# ls -l /usr/local/lib/ruby/gems/1.8/gems/
í?©ê³? 72
drwxr-xr-x 4 root root 4096 2ì?? 12 07:32 actionmailer-1.1.5/
drwxr-xr-x 5 root root 4096 2ì?? 12 07:32 actionpack-1.11.2/
drwxr-xr-x 5 root root 4096 2ì?? 12 07:32 actionwebservice-1.0.0/
drwxr-xr-x 5 root root 4096 2ì?? 12 07:31 activerecord-1.13.2/
drwxr-xr-x 3 root root 4096 2ì?? 12 07:31 activesupport-1.2.5/
drwxr-xr-x 3 root root 4096 2ì?? 22 00:42 lazy-0.9.5/
drwxr-xr-x 11 root root 4096 2ì?? 12 07:32 rails-1.0.0/
drwxr-xr-x 6 root root 4096 2ì?? 12 07:31 rake-0.7.0/
drwxr-xr-x 3 root root 4096 2ì?? 12 07:30 sources-0.0.1/

[root@poseidon tmp]# ls -l /usr/local/lib/ruby/1.8/ | grep lazy
[root@poseidon tmp]#

I’ve taken the liberty of posting installation problem to this post,
because this is the first time for me to install local *.gem file.
Can anybody tell me why this is happening? Isn’t the lazy.rb file
supposed to be installed in /usr/local/lib/ruby/1.8 ?

  • Minkoo S.

#12

On Wed, 2006-02-22 at 00:58 +0900, Minkoo S. wrote:

lazy.rb:60:22: Couldn’t find DIVERGES. Assuming it’s a module
[root@poseidon tmp]# irb
irb(main):001:0> require ‘lazy’
LoadError: no such file to load – lazy
from (irb):1:in require' from (irb):1 irb(main):002:0> require 'lazy/future' LoadError: no such file to load -- lazy/future from (irb):2:inrequire’
from (irb):2
irb(main):003:0>

Haven’t followed this thread all the way, but I’m assuming you must have
already tried:

$ irb
require 'rubygems'
require 'lazy'

Or alternatively passing -rubygems as an option to IRB.


#13

On Tue, 2006-02-21 at 16:00 +0900, Daniel N. wrote:

One question: How hard would it be to modify the way that exceptions
are handled to hold off raising the exception until the result was
requested? I’m thinking specifically of some non-deterministic
situations where you may request a value but never end up needing to
use it.

Hmm, if it doesn’t already do that, it’s a bug.

Can you give me a test case which demonstrates the problem?

-mental


#14

On Feb 21, 2006, at 10:58 AM, Minkoo S. wrote:

lazy.rb:60:22: Couldn’t find DIVERGES. Assuming it’s a module

drwxr-xr-x 3 root root 4096 2ì?? 12 07:30 sources-0.0.1/

  • Minkoo S.

require ‘rubygems’
require ‘lazy’


#15

On Tue, 2006-02-21 at 17:33 +0900, Andrew J. wrote:

But then I discovered that methods added by including a module weren’t
trapped by method_added – so in the end, a KernellessObject (from evil.rb)
was needed (same link as above) as Async’s parent to keep the proxy clean.

Hmm, good catch. Thanks!

I wonder if it’s worth introducing a dependency on evil.rb?

-mental


#16

On Tue, 2006-02-21 at 21:39 +0900, Jim W. wrote:

Its just that RDoc is complaining about something. I got the same error
on my system, but the RDoc looks ok, even with the error.

DIVERGES is a nodoc’d constant which is used internally. If anyone can
find a way to avoid the RDoc warning, I’d really appreciate it…

-mental


#17

On Feb 21, 2006, at 12:29 PM, MenTaLguY wrote:

no longer be “missing” in the Async (or Future) class. One work
proxy clean.

Hmm, good catch. Thanks!

I wonder if it’s worth introducing a dependency on evil.rb?

Oh, I don’t think so. It would cut off some users, like me. :wink:

James Edward G. II


#18

I must’ve misread the code (or misunderstood something)…

Yep. Thought the rescue clause raised the exception out of the thread
instead of just storing it.

In that case, what I want (I’m so damn greedy :D) is to be able to
raise the exception immediately (in a case where you damned well know
that you’re going to need the value and would prefer to fail early)
and I see that that’s trivial to implement (just have a conditional
toss the exception up in the thread if some @raise_immediately flag is
set to true).

There’s one other thing I was wondering about: What’s the prudency of
adding a method to Future to allow the Future to go back to sleep
(release the lock, pass the thread)? In this case, I’m considering
waiting for a resource that some other future/thread might be using,
or waiting for a port to get some data. (mostly implementation details
for that library I mentioned, I’ll deal with them, but I’m curious if
there’s any issue I’m forgetting).


#19

On Wed, 22 Feb 2006 03:35:03 +0900, James Edward G. II
removed_email_address@domain.invalid wrote:

On Feb 21, 2006, at 12:29 PM, MenTaLguY wrote:

I wonder if it’s worth introducing a dependency on evil.rb?

Oh, I don’t think so. It would cut off some users, like me. :wink:

At one point I was going to try to extract a minimum subset of evil
to just allow for KernellessObject, but never got to it. Would such
a small_evil.rb be less of a dependency concern?

andrew


#20

MenTaLguY wrote:

On Tue, 2006-02-21 at 21:39 +0900, Jim W. wrote:

Its just that RDoc is complaining about something. I got the same error
on my system, but the RDoc looks ok, even with the error.

DIVERGES is a nodoc’d constant which is used internally. If anyone can
find a way to avoid the RDoc warning, I’d really appreciate it…

Did you try # :nodoc: ?

-mental

E