# Ruby IPC In an OpenMosix Cluster

**URL:** <https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614>\
**Category:** Ruby\
**Created:** [March 22, 2006, 11:14pm UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614 "2006-03-22T23:14:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 22, 2006, 11:14pm UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/1 "2006-03-22T23:14:18Z")

</div>

I’m undertaking a project that will eventually become a processing  
pipeline application of sorts. It will receive data in a common  
format, transform it into one of _many_ other formats, compile and send  
it off to various endpoints (sounds spammish, but it’s all solicited,  
really :).

My team and I have tentatively decided on openMosix to provide an  
easily scalable cluster, and Ruby for the application itself. I’m very  
new to Ruby, and fairly new to IPC concepts – The Little Book of  
Semaphores and the many threads I’ve read here have helped me out a  
_lot_.

Our application will rely on 3 sets of consumer process pools; each  
pool is spawned by a daemon responsible for each basic operation (think  
MTA): receive, transform, compile/send. By using processes instead of  
threads we allow openMosix to migrate each process and make use of the  
entire cluster.

So we have the model down, but I need a bit of advise on how to most  
efficiently get these processes talking. What is the best form of IPC  
to use here? It seems there are tons of Ruby examples on concurrency  
and communication between threads, but I can’t seem to find anything  
definitive on IPC (to more than one child at least). I tried the sysv  
extension off RAA, but couldn’t get it to compile – though I didn’t  
try my best.

Things I’m considering:

- DRb
- UNIXSocket
- mkfifo
- SysV message queue (openMosix doesn’t support shmem segments)
- popen (though I can’t see how to do it without round robin  
producing)

If anyone has any advice, please shove me in the right direction.

Thanks in advance!

Also, thanks to matz for the great language (code blocks are uber  
bueno)!

Best,  
Dan

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 22, 2006, 11:27pm UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/2 "2006-03-22T23:27:24Z")

</div>

On Thu, 23 Mar 2006 removed\_email\_address@domain.invalid wrote:

> Our application will rely on 3 sets of consumer process pools; each pool is
> 
> Thanks in advance!
> 
> Also, thanks to matz for the great language (code blocks are uber  
> bueno)!
> 
> Best,  
> Dan

i recently built a system **exactly** like this for noaa. it’s built  
upon  
ruby queue (rq) for the clustering and dirwatch for the event driven  
components. both are on rubyforge and/or raa. the system uses a uniq  
library  
that allows classes to be parameterize, loaded, and run on input data in  
a few  
short lines. here is one class representing a processing flow

class Flo5 \< NRT::OLSSubscription::Geotiffed  
mode “production”

```
 roi 47,16,39,27
 satellites %w( F15 F16 )
 extensions %w( OIS )

 solarelevations -180, -12, 10.0

 hold 0

 username "flo"
 password "xxx"

 orbital_start_direction "descending"

```

end

this is one hundred percent of the coding needed to inject a new  
processing  
flow into the system, have incoming files spawn jobs for it, distribute  
processing to a cluster, and to package and deliver data.

i’d like to do a write up about it in the near future but i’ve just got  
alpha  
and am still pretty busy. in any case there are at least 6 or 7  
components  
that can be used from this system in any other such system. ping me on  
or off  
line and i can give you some more info… right now i’m late for a  
meeting…

[Linux Clustering with Ruby Queue: Small Is Beautiful | Linux Journal](http://www.linuxjournal.com/article/7922)  
[http://raa.ruby-lang.org/project/rq/](http://raa.ruby-lang.org/project/rq/)  
[http://raa.ruby-lang.org/project/dirwatch/](http://raa.ruby-lang.org/project/dirwatch/)

kind regards.

-a

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 23, 2006, 2:18am UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/3 "2006-03-23T02:18:57Z")

</div>

Right on, Ara. Thanks for your input!

I had looked at rq very briefly, but at first glance it seemed like it  
might be a hassle to maintain as we add nodes (having to  
update/kill/restart the application all nodes). What’s been your  
experience with regards to maintenance?

Thanks,  
Dan

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 23, 2006, 4:58am UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/4 "2006-03-23T04:58:55Z")

</div>

Ok, so I went back and actually read through the entire rq article this  
time (and noticed who wrote it – many props Ara :).

> From what I understood, you’re suggesting something like this:

1. Use dirwatch to wait for incoming data (files) on an NFS exported  
dir
2. Inject jobs into rq for each incoming file
3. rq executes commands on each node that read in each file from the  
NFS mount

How fast is a setup like this? I would think there would be a lot of  
overhead in forking processes for each job, and even more in the  
NFS/file IO. We’re shooting for 100 jobs/second, starting with a  
fairly small cluster and then scaling up. Each piece of data is 4-10k.

My thought was to spawn a pool of processes once, then start feeding  
them the data via [unknown IPC]. Seems like that would be a faster  
solution as long as openMosix is efficient in redirecting the IO across  
nodes. Of course, this may be a development nightmare (learning  
experience), since neither my team nor I have a lot of experience with  
multiprocessing.

If rq would satisfy our speed requirements, then I would love to avoid  
the extra development time. Perhaps we’ll just have to build a basic  
prototype and run some tests. 🙂

Best,  
Dan

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 23, 2006, 7:40am UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/5 "2006-03-23T07:40:00Z")

</div>

On Thu, 23 Mar 2006 removed\_email\_address@domain.invalid wrote:

> Right on, Ara. Thanks for your input!
> 
> I had looked at rq very briefly, but at first glance it seemed like it might  
> be a hassle to maintain as we add nodes (having to update/kill/restart the  
> application all nodes). What’s been your experience with regards to  
> maintenance?

hmmm. not quite clear on what you are asking - but we regularly add and  
remove nodes. you don’t need to stop all nodes to do this at all - to  
add a  
node simply start a feeder on it, to remove a node simply stop that  
nodes  
feeder.

is that what you are asking?

in summary we regularly use rq to put together ‘ad-hoc’ clusters of 3-10  
nodes  
and this generally takes less than 5 minutes or so.

regards.

-a

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 23, 2006, 8:08pm UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/6 "2006-03-23T20:08:58Z")

</div>

> hmmm. not quite clear on what you are asking - but we regularly add and  
> remove nodes. you don’t need to stop all nodes to do this at all - to add a  
> node simply start a feeder on it, to remove a node simply stop that nodes  
> feeder.

> is that what you are asking?

No, actually I really just spat out the wrong question before I read  
the entire article and understood the rq setup. 🙂

Dan

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 23, 2006, 7:55am UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/7 "2006-03-23T07:55:15Z")

</div>

On Thu, 23 Mar 2006 removed\_email\_address@domain.invalid wrote:

> How fast is a setup like this? I would think there would be a lot of  
> overhead in forking processes for each job, and even more in the  
> NFS/file IO. We’re shooting for 100 jobs/second, starting with a  
> fairly small cluster and then scaling up. Each piece of data is 4-10k.

yup this would defintely push the limits **unless** you can batch them.  
i’m  
actually working with a group now that will be injesting data at almost  
that  
exact same rate. in there case it’s sufficient to bundle jobs up -  
higher  
latency but also hight throughput. so basically a dirwatch would watch  
an  
incoming directory and, perhaps once per minute, scan the directory and  
submit  
bunches of 500 files for processing. for something really simply you  
might  
not even need dirwatch - just move files to another directory once  
they’ve  
been submitted - eg. sweep the directory as you submit. rq now supports  
providing stdin (and saving stdout and stderr) so submitting jobs that  
process  
500 files is really easy. anyhow, nfs scales pretty dang well on gigE  
with  
tuned tcp/ip stacks and fast disk, we certainly abuse ours with little  
problems. however, we also use vsftpd to access data and this is very  
easy to  
setup and extremely fast to use - about as fast as you can get. in any  
case  
you’ll have the same issue with any cluster: distributing jobs is easier  
than  
distributing data. for instance, many of our inputs are 3gb-600gb - one  
has  
to be careful with this sort of payload! 😉

> My thought was to spawn a pool of processes once, then start feeding them  
> the data via [unknown IPC]. Seems like that would be a faster solution as  
> long as openMosix is efficient in redirecting the IO across nodes. Of  
> course, this may be a development nightmare (learning experience), since  
> neither my team nor I have a lot of experience with multiprocessing.

i did look into this a bit - if i recall openmosix makes io transparent  
and  
can migrate process memory from node to node… in our case this would  
be a  
disaster : code must follow data and not the other way around. from  
what i  
know at the moment i can’t imagine that kernel level io network  
multiplexing  
would be faster than pulling data across and sending it back in huge  
chunks…  
the resident network expert here seems to think people need to move that  
way  
to optimize newer networks with things like jumbo frames. but i can’t  
say for  
certain i’m just a hacker!

> If rq would satisfy our speed requirements, then I would love to avoid the  
> extra development time. Perhaps we’ll just have to build a basic prototype  
> and run some tests. 🙂

yes. we ran a simulation here, just prducing false input and running a  
busy  
loop for a few seconds to appriximate the system that was similar to  
yours and  
it seemed fine. however they are not in production yet so i cannot say  
for  
sure. the good part is that all the bits are free and it should only  
take a  
few hours to mock something up - even on a stock linux machine with no  
root  
privs…

let me know if you go this route as i have upgrades to both dirwatch and  
rq  
that you’ll surely want.

regards.

-a

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [March 23, 2006, 8:19pm UTC](https://www.ruby-forum.com/t/ruby-ipc-in-an-openmosix-cluster/85614/8 "2006-03-23T20:19:00Z")

</div>

> yup this would defintely push the limits **unless** you can batch them.

Not sure if this is an option yet.

> let me know if you go this route as i have upgrades to both dirwatch and rq  
> that you’ll surely want.

I will definitely let you know. I’m going to experiment with a more  
generic drb setup first, and see how that goes. Do your updates  
contain the stdin/stdout support, or is that already in the newest  
release?

Thanks again for all your help.

Dan