Forum: Ruby rq-2.3.2

Announcement (2017-05-07): is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see and for other Rails- und Ruby-related community platforms.
Ara.T.Howard (Guest)
on 2006-04-29 01:38
(Received via mailing list)
   rq v2.3.2

   rq (queue | export RQ_Q=q) mode [mode_args]* [options]*


   ruby queue (rq) is a zero-admin zero-configuration tool used to
   instant linux clusters.  rq requires only a central nfs filesystem in
   to manage a simple sqlite database as a distributed priority work
   this simple design allows researchers to install and configure, in
only a
   few minutes and without root privileges, a robust linux cluster
capable of
   distributing processes to many nodes - bringing dozens of powerful
cpus to
   their knees with a single blow.  clearly this software should be kept
out of
   the hands of free radicals, seti enthusiasts, and one mr. j safran.

   the central concept of rq is that n nodes work in isolation to pull
   from an centrally mounted nfs priority work queue in a synchronized
   the nodes have absolutely no knowledge of each other and all
   is done via the queue meaning that, so long as the queue is available
   nfs and a single node is running jobs from it, the system will
continue to
   process jobs.  there is no centralized process whatsoever - all nodes
   to take jobs from the queue and run them as fast as possible.  this
   a system which load balances automatically and is robust in face of

   although the rq system is simple in it's design it features powerful
   functionality such as priority management, predicate and sql query ,
   streaming command-line processing, programmable api, hot-backup, and
   input/capture of the stdin/stdout/stderr io streams of remote jobs.
to date
   rq has had no reported runtime failures and is in operation at dozens
   research centers around the world.


Guillaume M. (Guest)
on 2006-05-03 21:08
(Received via mailing list)
I couldn't find the changes list. What is new in this version?


Le 28 avr. 06, à 17:36, Ara.T.Howard a écrit :
unknown (Guest)
on 2006-05-03 21:09
(Received via mailing list)
On Sat, 29 Apr 2006, Guillaume M. wrote:

> I couldn't find the changes list. What is new in this version?
> Guillaume.

sorry.  here you go

     - added 'rq q io' command
     - streamlined 'rq q status' output considerably
     - fixed few small bugs in util.rb
     - patched yaml inline behaviour for syck in 1.8.4.  backward
compatible with
       older ruby versions - at least through 1.8.2.
     - changed naming scheme for rotation and added automatic attempt to
       the rotation
     - made shutdown via ctrl-c when running from commandline (as
opposed to
       daemon) quieter
     - fixed a few bad exit codes, for example creating a queue exited
       $?==1, it is now correctly 0
     - added support for job stdin to be provided and stdout/stderr
       for instance

         rq queue submit cat --stdin=input_for_cat.txt

         cat input_for_cat.txt | rq queue submit cat --stdin=-

         rq queue submit cat --stdin=- < input_for_cat.txt

       stdout and stderr are stored in the queue directory in a file
named by
     - changed behaviour of reading jobs from stdin: ***IMPORTANT*** :
it's now
       **REQUIRED** to provide a '-' as the argument for any command
that should
       read from stdin.  for instance one would use

         rq queue submit - < joblist
         jobgenerator.rb | rq queue submit -

       vs the old

         rq queue submit < joblist
         jobgenerator.rb | rq queue submit

       the distiction was added to make running submissions from
       (where STDIN.tty? == false) easier and providing the job stdinput
on stdin
     - added a restart mode to restart a feeder : rq q restart
     - modes shutdown, stop, and restart now return a non-zero exit code
if they
       fail to signal the feeder, zero otherwise
     - fixed bug with restart mode whereby re-exec failed due to falsely
       pid was locked (it is - but only by the current process!)
     - added default bin dir underneath q dir.  this dir is added to the
       before excuting commands to it's an easy place to put things
where they'll
       be found

the biggest change is the ability to provide jobs stdin at submit time
and to
capture their stdout/stderr.  for instance you can now do this

   rq q submit cat --stdin=/etc/passwd


   find | grep *geo.tif | rq q submit geolocate.rb --stdin=-

to submit jobs that take stdin.  the stdin is squirled away in the q
later, when the job is run, the stdout and stderr are captured in the
way.  so, in the above example, if the submitted job had jid=42 you'd

   q/stdin/42  # the stdin provided

before the job was run and


once the job was set running.  this is extrememly cool since you can do

   tail -F q/stderr/42

to monitor a process on a remote node.


This topic is locked and can not be replied to.