Forum: Ruby Command Pipeline (pipes & filters) in Ruby

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
F3b786a23390451fa67108781b0c8fed?d=identicon&s=25 unknown (Guest)
on 2006-02-13 06:33
(Received via mailing list)
Anyone know of any generic implementation of a pipeline in ruby?  That
is, one which takes a bunch of GoF Command objects, and strings them
together in the classical pipeline configuration.

Any advice on that?

I have a process with a lot of sequential processing, and think that a
pipeline of Command objects (not just Proc's) might be the way to do
it.

The downsides of pipelines:
1) You can't do selections (if) or sequences (loops).  If you can
handle this, though, this becomes a virtue, as it keeps everything
crystal clear and simple.  Witness the ever useful Unix command line
pipelines.

2) They don't encapsulate very well, as each stage totally overwrites
the previous ones.  I'm not sure if I should just accept this, or try
to use some modification (append only pipeline, etc.).

Any experience or ideas on this appreciated.
Bc6d88907ce09158581fbb9b469a35a3?d=identicon&s=25 James Britt (Guest)
on 2006-02-13 06:48
(Received via mailing list)
eastcoastcoder@gmail.com wrote:
> Anyone know of any generic implementation of a pipeline in ruby?  That
> is, one which takes a bunch of GoF Command objects, and strings them
> together in the classical pipeline configuration.
>
> Any advice on that?

Does Rake approach this?

Define each command as a task.  Let Rake handle the chaining and
dependency logic.




--
James Britt

?Design depends largely on constraints.?
  ? Charles Eames
5befe95e6648daec3dd5728cd36602d0?d=identicon&s=25 Robert Klemme (Guest)
on 2006-02-13 10:33
(Received via mailing list)
eastcoastcoder@gmail.com wrote:
> The downsides of pipelines:
> 1) You can't do selections (if) or sequences (loops).  If you can
> handle this, though, this becomes a virtue, as it keeps everything
> crystal clear and simple.  Witness the ever useful Unix command line
> pipelines.
>
> 2) They don't encapsulate very well, as each stage totally overwrites
> the previous ones.  I'm not sure if I should just accept this, or try
> to use some modification (append only pipeline, etc.).
>
> Any experience or ideas on this appreciated.

There are several ways you can do pipelining.  Do you want concurrency
with that?  Does every stage have a single output the next stage wants
to
operate on?  etc.

Probably the simplest thing you can do is to use #inject:

commands = [
  lambda {|x| x * 2},
  lambda {|x| 0 - x},
  lambda {|x| x + 10},
]

>> commands.inject(0) {|x,cmd| cmd[x]}
=> 10
>> commands.inject(1) {|x,cmd| cmd[x]}
=> 8

Kind regards

    robert
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2006-02-13 17:15
(Received via mailing list)
On Mon, 13 Feb 2006 eastcoastcoder@gmail.com wrote:

> The downsides of pipelines:
> 1) You can't do selections (if) or sequences (loops).  If you can
> handle this, though, this becomes a virtue, as it keeps everything
> crystal clear and simple.  Witness the ever useful Unix command line
> pipelines.
>
> 2) They don't encapsulate very well, as each stage totally overwrites
> the previous ones.  I'm not sure if I should just accept this, or try
> to use some modification (append only pipeline, etc.).
>
> Any experience or ideas on this appreciated.

this handles any dag.  i wrote it a long time ago

   http://codeforpeople.com/lib/ruby/flow/flow-2.0.0/

hth.

-a
65bd9e4c5aebde25ebf16d599339d570?d=identicon&s=25 Adam Sanderson (Guest)
on 2006-02-13 17:43
(Received via mailing list)
That's neat, this might be handy too:

class Proc
  def |(other)
    proc{|*a| other.call(self.call(*a)) }
  end
end

I think I got the idea from something on Why's site, though I can't
find it now.  Anyways if you could now do:

f = lambda {|x| x * 2} | lambda {|x| 0 - x} | lambda {|x| x + 10}

or more verbosely:

a = lambda {|x| x * 2}
b = lambda {|x| 0 - x}
c = lambda {|x| x + 10}

f = (a|b|c)

Think of it as unix pipes :)
  .adam
5befe95e6648daec3dd5728cd36602d0?d=identicon&s=25 Robert Klemme (Guest)
on 2006-02-13 18:03
(Received via mailing list)
Adam Sanderson wrote:
>
> Think of it as unix pipes :)
>   .adam

Depending on the number of processors you might even run into stack size
limits which won't happen with an #inject based solution (see my other
posting).  (Ok, I know that I'm being obsessive about Enumerable#inject
-
it was love at first sight. :-)) )

Although this looks nice, when associating Unix pipes you might be
tempted
to expect those to execute concurrently which they don't.

We can combine both solutions:

class ProcChain
  def initialize(*procs) @chain = procs end
  def |(proc)
    @chain << proc
    self
  end
  def call(a)
    @chain.inject(a) {|x,pr| pr[x]}
  end
  alias [] call
end

class Proc
  def |(proc)
    ProcChain.new(self, proc)
  end
end

irb(main):039:0> f = lambda {|x| x * 2} | lambda {|x| 0 - x} | lambda
{|x|
x + 10}
=> #<ProcChain:0x4a6fed8 @chain=[#<Proc:0x04a70238@(irb):39>,
#<Proc:0x04a70148@(irb):39>, #<Proc:0x04a6ffc8@(irb):39>]>
irb(main):040:0> f[0]
=> 10
irb(main):041:0> f.call 1
=> 8

:-))

Kind regards

    robert
7264fb16beeea92b89bb42023738259d?d=identicon&s=25 Christian Neukirchen (Guest)
on 2006-02-13 18:38
(Received via mailing list)
eastcoastcoder@gmail.com writes:

> The downsides of pipelines:
> 1) You can't do selections (if) or sequences (loops).  If you can
> handle this, though, this becomes a virtue, as it keeps everything
> crystal clear and simple.  Witness the ever useful Unix command line
> pipelines.
>
> 2) They don't encapsulate very well, as each stage totally overwrites
> the previous ones.  I'm not sure if I should just accept this, or try
> to use some modification (append only pipeline, etc.).
>
> Any experience or ideas on this appreciated.

A long, long, time ago, I can still remember:

http://kronavita.de/chris/data/pipeline_rdoc.html
This topic is locked and can not be replied to.