Bj-1.0.0

Ruby
1

NAME
bj

SYNOPSIS
bj (migration_code|generate_migration|migrate|setup|plugin|run|
submit|list|set|config|pid) [options]+

HISTORY
1.0.0:
- use full path to ruby for plugin mode
- plugin correctly installs bin -->> script
- plugin install uses --force
- properly quote paths in windows (spaces)
- switch win signal to ABRT (was INT)
- background job regrestration now uses ppid to pin the
subprocess to a
parent
- use ppid to detect parent death and exit in event loop
- don’t use gem dependanices in plugin as they are broken when
loading from
muliple gem repos
- added a small amount of drb magic that allows signals to work
across
processes even on windows (see http://drawohara.com/post/
22540307)

DESCRIPTION

Overview

Backgroundjob (Bj) is a brain dead simple zero admin background
priority queue
for Rails. Bj is robust, platform independent (including
windows), and
supports internal or external manangement of the background runner
process.

Jobs can be submitted to the queue directly using the api or from
the command
line using the ./script/bj:

 api:
   Bj.submit 'cat /etc/password'

 command line:
   bj submit cat /etc/password

Bj’s priority queue lives in the database and is therefore durable

  • your jobs
    will live across an app crash or machine reboot. The job
    management is
    comprehensive capturing stdout, stderr, exit_status, and temporal
    statistics
    about each job:

    jobs = Bj.submit array_of_commands, :priority => 42

    …

    jobs.each do |job|
    if job.finished?
    p job.stdout
    p job.stderr
    p job.exit_status
    p job.started_at
    p job.finished_at
    end
    end

    In addition the background runner process logs all commands run
    and their
    exit_status to a log named using the following convention:

    rails_root/log/bj.#{ HOSTNAME }.#{ RAILS_ENV }.log

    Bj allows you to submit jobs to multiple databases; for instance,
    if your
    application is running in development mode you may do:

    Bj.in :production do
    Bj.submit ‘my_job.exe’
    end

    Bj manages the ever growing list of jobs ran by automatically
    archiving them
    into another table (by default jobs > 24 hrs old are archived) to
    prevent the
    jobs table from becoming bloated and huge.

    All Bj’s tables are namespaced and accessible via the Bj module:

    Bj.table.job.find(:all) # jobs table
    Bj.table.job_archive.find(:all) # archived jobs
    Bj.table.config.find(:all) # configuration and runner state

    Bj always arranges for submitted jobs to run with a current
    working directory
    of RAILS_ROOT and with the correct RAILS_ENV setting. For
    example, if you
    submit a job in production it will have ENV[‘RAILS_ENV’] ==
    ‘production’.

    When Bj manages the background runner it will never outlive the rails
    application - it is started and stopped on demand as the rails app
    is started
    and stopped. This is also true for ./script/console - Bj will
    automatically
    fire off the background runner to process jobs submitted using the
    console.

    Bj ensures that only one background process is running for your
    application -
    firing up three mongrels or fcgi processes will result in only one
    background
    runner being started. Note that the number of background runners
    does not
    determine throughput - that is determined primarily by the nature
    of the jobs
    themselves and how much work they perform per process.

    Architecture

    If one ignores platform specific details the design of Bj is quite
    simple: the
    main Rails application submits jobs to table, stored in the
    database. The act
    of submitting triggers exactly one of two things to occur:

    1. a new long running background runner to be started

    2. an existing background runner to be signaled

    The background runner refuses to run two copies of itself for a given
    hostname/rails_env combination. For example you may only have one
    background
    runner processing jobs on localhost in development mode.

    The background runner, under normal circumstances, is managed by
    Bj itself -
    you need do nothing to start, monitor, or stop it - it just
    works. However,
    some people will prefer manage their own background process, see
    ‘External
    Runner’ section below for more on this.

    The runner simply processes each job in a highest priority oldest-
    in fashion,
    capturing stdout, stderr, exit_status, etc. and storing the
    information back
    into the database while logging it’s actions. When there are no
    jobs to run
    the runner goes to sleep for 42 seconds; however this sleep is
    interuptable,
    such as when the runner is signaled that a new job has been
    submitted so,
    under normal circumstances there will be zero lag between job
    submission and
    job running for an empty queue.

    External Runner / Clustering

    For the paranoid control freaks out there (myself included) it is
    quite
    possible to manage and monitor the runner process manually. This
    can be
    desirable in production setups where monitoring software may kill
    leaking
    rails apps periodically.

    Recalling that Bj will only allow one copy of itself to process
    jobs per
    hostname/rails_env pair we can simply do something like this in cron

    cmd = bj run --forever
    –rails_env=development
    –rails_root=/Users/ahoward/rails_root

    */15 * * * * $cmd

    this will simply attempt the start the background runner every 15
    minutes if,
    and only if, it’s not already running.

    In addtion to this you’ll want to tell Bj not to manage the runner
    itself
    using

    Bj.config[“production.no_tickle”] = true

    Note that, for clusting setups, it’s as simple as adding a crontab
    and config
    entry like this for each host. Because Bj throttles background
    runners per
    hostname this will allow one runner per hostname - making it quite
    simple to
    cluster three nodes behind a besieged rails application.

    Designing Jobs

    Bj runs it’s jobs as command line applications. It ensures that
    all jobs run
    in RAILS_ROOT so it’s quite natural to apply a pattern such as

    mkdir ./jobs
    edit ./jobs/background_job_to_run

    …

    Bj.submit “./jobs/background_job_to_run”

    If you need to run you jobs under an entire rails environment
    you’ll need to
    do this:

    Bj.submit “./script/runner ./jobs/background_job_to_run”

    Obviously “./script/runner” loads the rails environment for you.
    It’s worth
    noting that this happens for each job and that this is by design:
    the reason
    is that most rails applications leak memory like a sieve so, if
    one were to
    spawn a long running process that used the application code base
    you’d have a
    lovely doubling of memory usage on you app servers. Although
    loading the
    rails environment for each background job requires a little time,
    a little
    cpu, and a lot less memory. A future version of Bj will provide a
    way to load
    the rails environment once and to process background jobs in this
    environment,
    but anyone wanting to use this in production will be required to
    duct tape
    their entire chest and have a team of oxen rip off the tape
    without screaming
    to prove steelyness of spirit and profound understanding of the
    other side.

    Don’t forget that you can submit jobs with command line arguments:

    Bj.submit “./jobs/a.rb 1 foobar --force”

    and that you can do powerful things by passing stdin to a job that
    powers
    through a list of work. For instance, assume a “./jobs/bulkmail” job
    resembling

    STDIN.each do |line|
    address = line.strip
    mail_message_to address
    end

    then you could

    stdin = [
    “[email protected]”,
    “[email protected]”,
    “[email protected]”,
    ]

    Bj.submit “./script/runner ./jobs/bulkmail”, :stdin => stdin

    and all those emails would be sent in the background.

    Bj’s power is putting jobs in the background in a simple and
    robust fashion.
    It’s your task to build intelligent jobs that leverage batch
    processing, and
    other, possibilities. The upshot of building tasks this way is
    that they are
    quite easy to test before submitting them from inside your
    application.

    Install

    Bj can be installed two ways: as a plugin or via rubygems

     plugin:
   1) ./script/plugin install http://

codeforpeople.rubyforge.org/svn/rails/plugins/bj
2) ./script/bj setup

   gem:
     1) $sudo gem install bj
     2) add "require 'bj'" to config/environment.rb
     3) bj setup

Api

submit jobs for background processing. ‘jobs’ can be a string or
array of
strings. options are applied to each job in the ‘jobs’, and the
list of
submitted jobs is always returned. options (string or symbol) can be

 :rails_env => production|development|key_in_database_yml
               when given this keyword causes bj to submit jobs

to the
specified database. default is RAILS_ENV.

 :priority => any number, including negative ones.  default is zero.

 :tag => a tag added to the job.  simply makes searching easier.

 :env => a hash specifying any additional environment vars the

background
process should have.

 :stdin => any stdin the background process should have.  must

respond_to
to_s

eg:

 jobs = Bj.submit 'echo foobar', :tag => 'simple job'

 jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority

=> 42

 jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env =>

‘production’

 jobs = Bj.submit './script/runner /dev/stdin',
                  :stdin => 'p RAILS_ENV',
                  :tag => 'dynamic ruby code'

 jobs Bj.submit array_of_commands, :priority => 451

when jobs are run, they are run in RAILS_ROOT. various attributes
are
available only once the job has finished. you can check whether
or not a
job is finished by using the #finished method, which simple does a
reload and
checks to see if the exit_status is non-nil.

 eg:

   jobs = Bj.submit list_of_jobs, :tag => 'important'
   ...

   jobs.each do |job|
     if job.finished?
       p job.exit_status
       p job.stdout
       p job.stderr
     end
   end

See lib/bj/api.rb for more details.

Sponsors

 http://www.engineyard.com/
 http://quintess.com/
 http://eparklabs.com/

 http://your_company.com/      <<-- (targeted marketing aimed at

you)

Version

1.0.0

PARAMETERS
–rails_root=rails_root, -R (0 ~> rails_root=)
the rails_root will be guessed unless you set this
–rails_env=rails_env, -E (0 ~> rails_env=development)
set the rails_env
–log=log, -l (0 ~> log=STDERR)
set the logfile
–help, -h

AUTHOR
[email protected]

URIS
http://codeforpeople.com/lib/ruby/
http://rubyforge.org/projects/codeforpeople/
http://codeforpeople.rubyforge.org/svn/rails/plugins/

enjoy.

a @ http://codeforpeople.com/