Command Pipeline (pipes & filters) in Ruby

Ruby
1

Anyone know of any generic implementation of a pipeline in ruby? That
is, one which takes a bunch of GoF Command objects, and strings them
together in the classical pipeline configuration.

Any advice on that?

I have a process with a lot of sequential processing, and think that a
pipeline of Command objects (not just Proc’s) might be the way to do
it.

The downsides of pipelines:

  1. You can’t do selections (if) or sequences (loops). If you can
    handle this, though, this becomes a virtue, as it keeps everything
    crystal clear and simple. Witness the ever useful Unix command line
    pipelines.

  2. They don’t encapsulate very well, as each stage totally overwrites
    the previous ones. I’m not sure if I should just accept this, or try
    to use some modification (append only pipeline, etc.).

Any experience or ideas on this appreciated.

2

[email protected] wrote:

Anyone know of any generic implementation of a pipeline in ruby? That
is, one which takes a bunch of GoF Command objects, and strings them
together in the classical pipeline configuration.

Any advice on that?

Does Rake approach this?

Define each command as a task. Let Rake handle the chaining and
dependency logic.

–
James B.

?Design depends largely on constraints.?
? Charles Eames

3

[email protected] wrote:

The downsides of pipelines:

  1. You can’t do selections (if) or sequences (loops). If you can
    handle this, though, this becomes a virtue, as it keeps everything
    crystal clear and simple. Witness the ever useful Unix command line
    pipelines.

  2. They don’t encapsulate very well, as each stage totally overwrites
    the previous ones. I’m not sure if I should just accept this, or try
    to use some modification (append only pipeline, etc.).

Any experience or ideas on this appreciated.

There are several ways you can do pipelining. Do you want concurrency
with that? Does every stage have a single output the next stage wants
to
operate on? etc.

Probably the simplest thing you can do is to use #inject:

commands = [
lambda {|x| x * 2},
lambda {|x| 0 - x},
lambda {|x| x + 10},
]

commands.inject(0) {|x,cmd| cmd[x]}
=> 10

commands.inject(1) {|x,cmd| cmd[x]}
=> 8

Kind regards

robert
4

On Mon, 13 Feb 2006 [email protected] wrote:

The downsides of pipelines:

  1. You can’t do selections (if) or sequences (loops). If you can
    handle this, though, this becomes a virtue, as it keeps everything
    crystal clear and simple. Witness the ever useful Unix command line
    pipelines.

  2. They don’t encapsulate very well, as each stage totally overwrites
    the previous ones. I’m not sure if I should just accept this, or try
    to use some modification (append only pipeline, etc.).

Any experience or ideas on this appreciated.

this handles any dag. i wrote it a long time ago

http://codeforpeople.com/lib/ruby/flow/flow-2.0.0/

hth.

-a

5

That’s neat, this might be handy too:

class Proc
def |(other)
proc{|*a| other.call(self.call(*a)) }
end
end

I think I got the idea from something on Why’s site, though I can’t
find it now. Anyways if you could now do:

f = lambda {|x| x * 2} | lambda {|x| 0 - x} | lambda {|x| x + 10}

or more verbosely:

a = lambda {|x| x * 2}
b = lambda {|x| 0 - x}
c = lambda {|x| x + 10}

f = (a|b|c)

Think of it as unix pipes :slight_smile:
.adam

6

Adam S. wrote:

Think of it as unix pipes :slight_smile:
.adam

Depending on the number of processors you might even run into stack size
limits which won’t happen with an #inject based solution (see my other
posting). (Ok, I know that I’m being obsessive about Enumerable#inject

it was love at first sight. :-)) )

Although this looks nice, when associating Unix pipes you might be
tempted
to expect those to execute concurrently which they don’t.

We can combine both solutions:

class ProcChain
def initialize(*procs) @chain = procs end
def |(proc)
@chain << proc
self
end
def call(a)
@chain.inject(a) {|x,pr| pr[x]}
end
alias [] call
end

class Proc
def |(proc)
ProcChain.new(self, proc)
end
end

irb(main):039:0> f = lambda {|x| x * 2} | lambda {|x| 0 - x} | lambda
{|x|
x + 10}
=> #<ProcChain:0x4a6fed8 @chain=[#Proc:0x04a70238@:39(irb),
#Proc:0x04a70148@:39(irb), #Proc:0x04a6ffc8@:39(irb)]>
irb(main):040:0> f[0]
=> 10
irb(main):041:0> f.call 1
=> 8

:-))

Kind regards

robert
7

[email protected] writes:

The downsides of pipelines:

  1. You can’t do selections (if) or sequences (loops). If you can
    handle this, though, this becomes a virtue, as it keeps everything
    crystal clear and simple. Witness the ever useful Unix command line
    pipelines.

  2. They don’t encapsulate very well, as each stage totally overwrites
    the previous ones. I’m not sure if I should just accept this, or try
    to use some modification (append only pipeline, etc.).

Any experience or ideas on this appreciated.

A long, long, time ago, I can still remember:

http://kronavita.de/chris/data/pipeline_rdoc.html