Forum: Ruby Command Pipeline (pipes & filters) in Ruby

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
unknown (Guest)
on 2006-02-13 07:33
(Received via mailing list)
Anyone know of any generic implementation of a pipeline in ruby?  That
is, one which takes a bunch of GoF Command objects, and strings them
together in the classical pipeline configuration.

Any advice on that?

I have a process with a lot of sequential processing, and think that a
pipeline of Command objects (not just Proc's) might be the way to do
it.

The downsides of pipelines:
1) You can't do selections (if) or sequences (loops).  If you can
handle this, though, this becomes a virtue, as it keeps everything
crystal clear and simple.  Witness the ever useful Unix command line
pipelines.

2) They don't encapsulate very well, as each stage totally overwrites
the previous ones.  I'm not sure if I should just accept this, or try
to use some modification (append only pipeline, etc.).

Any experience or ideas on this appreciated.
James B. (Guest)
on 2006-02-13 07:48
(Received via mailing list)
removed_email_address@domain.invalid wrote:
> Anyone know of any generic implementation of a pipeline in ruby?  That
> is, one which takes a bunch of GoF Command objects, and strings them
> together in the classical pipeline configuration.
>
> Any advice on that?

Does Rake approach this?

Define each command as a task.  Let Rake handle the chaining and
dependency logic.




--
James B.

?Design depends largely on constraints.?
  ? Charles Eames
Robert K. (Guest)
on 2006-02-13 11:33
(Received via mailing list)
removed_email_address@domain.invalid wrote:
> The downsides of pipelines:
> 1) You can't do selections (if) or sequences (loops).  If you can
> handle this, though, this becomes a virtue, as it keeps everything
> crystal clear and simple.  Witness the ever useful Unix command line
> pipelines.
>
> 2) They don't encapsulate very well, as each stage totally overwrites
> the previous ones.  I'm not sure if I should just accept this, or try
> to use some modification (append only pipeline, etc.).
>
> Any experience or ideas on this appreciated.

There are several ways you can do pipelining.  Do you want concurrency
with that?  Does every stage have a single output the next stage wants
to
operate on?  etc.

Probably the simplest thing you can do is to use #inject:

commands = [
  lambda {|x| x * 2},
  lambda {|x| 0 - x},
  lambda {|x| x + 10},
]

>> commands.inject(0) {|x,cmd| cmd[x]}
=> 10
>> commands.inject(1) {|x,cmd| cmd[x]}
=> 8

Kind regards

    robert
unknown (Guest)
on 2006-02-13 18:15
(Received via mailing list)
On Mon, 13 Feb 2006 removed_email_address@domain.invalid wrote:

> The downsides of pipelines:
> 1) You can't do selections (if) or sequences (loops).  If you can
> handle this, though, this becomes a virtue, as it keeps everything
> crystal clear and simple.  Witness the ever useful Unix command line
> pipelines.
>
> 2) They don't encapsulate very well, as each stage totally overwrites
> the previous ones.  I'm not sure if I should just accept this, or try
> to use some modification (append only pipeline, etc.).
>
> Any experience or ideas on this appreciated.

this handles any dag.  i wrote it a long time ago

   http://codeforpeople.com/lib/ruby/flow/flow-2.0.0/

hth.

-a
Adam S. (Guest)
on 2006-02-13 18:43
(Received via mailing list)
That's neat, this might be handy too:

class Proc
  def |(other)
    proc{|*a| other.call(self.call(*a)) }
  end
end

I think I got the idea from something on Why's site, though I can't
find it now.  Anyways if you could now do:

f = lambda {|x| x * 2} | lambda {|x| 0 - x} | lambda {|x| x + 10}

or more verbosely:

a = lambda {|x| x * 2}
b = lambda {|x| 0 - x}
c = lambda {|x| x + 10}

f = (a|b|c)

Think of it as unix pipes :)
  .adam
Robert K. (Guest)
on 2006-02-13 19:03
(Received via mailing list)
Adam S. wrote:
>
> Think of it as unix pipes :)
>   .adam

Depending on the number of processors you might even run into stack size
limits which won't happen with an #inject based solution (see my other
posting).  (Ok, I know that I'm being obsessive about Enumerable#inject
-
it was love at first sight. :-)) )

Although this looks nice, when associating Unix pipes you might be
tempted
to expect those to execute concurrently which they don't.

We can combine both solutions:

class ProcChain
  def initialize(*procs) @chain = procs end
  def |(proc)
    @chain << proc
    self
  end
  def call(a)
    @chain.inject(a) {|x,pr| pr[x]}
  end
  alias [] call
end

class Proc
  def |(proc)
    ProcChain.new(self, proc)
  end
end

irb(main):039:0> f = lambda {|x| x * 2} | lambda {|x| 0 - x} | lambda
{|x|
x + 10}
=> #<ProcChain:0x4a6fed8 @chain=[#<Proc:0x04a70238@(irb):39>,
#<Proc:0x04a70148@(irb):39>, #<Proc:0x04a6ffc8@(irb):39>]>
irb(main):040:0> f[0]
=> 10
irb(main):041:0> f.call 1
=> 8

:-))

Kind regards

    robert
Christian N. (Guest)
on 2006-02-13 19:38
(Received via mailing list)
removed_email_address@domain.invalid writes:

> The downsides of pipelines:
> 1) You can't do selections (if) or sequences (loops).  If you can
> handle this, though, this becomes a virtue, as it keeps everything
> crystal clear and simple.  Witness the ever useful Unix command line
> pipelines.
>
> 2) They don't encapsulate very well, as each stage totally overwrites
> the previous ones.  I'm not sure if I should just accept this, or try
> to use some modification (append only pipeline, etc.).
>
> Any experience or ideas on this appreciated.

A long, long, time ago, I can still remember:

http://kronavita.de/chris/data/pipeline_rdoc.html
This topic is locked and can not be replied to.