Bj and rails 2.1 - can't get bj to run jobs

dusty · July 22, 2008, 3:30am

I have been trying to setup bj with rails 2.1.0 and am having some
problems, was wondering if anyone could help. Here is what I’ve done
so far.

Basic rails install
$ rails tester
Install bj
$ cd tester
$ ./script/plugin install
http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
Setup bj
$ ./script/bj setup
Submit a job to the queue
$ ./script/console
$ Bj.submit ‘ls’

At that point, it puts the job in the database

Bj.table.job.all
=> [#<Bj::Table::Job bj_job_id: 1, command: “ls”, state: “pending”,
priority: 0, tag: “”, is_restartable: 1, submitter: “dustymini.local”,
runner: nil, pid: nil, submitted_at: “2008-07-22 01:15:46”,
started_at: nil, finished_at: nil, env: nil, stdin: nil, stdout: nil,
stderr: nil, exit_status: nil>]

The config setting shows this:

Bj.table.config.all
=> [#<Bj::Table::Config bj_config_id: 1, hostname: “dustymini.local”,
key: “development.23928.pid”, value: “23934”, cast: “to_i”>]

And, it seems to fire off the bj process to run that job

$ ps axwww | grep -i bj
23934 s000 R+ 0:01.99 /opt/local/bin/ruby /Users/dusty/tester/
script/bj run --forever --redirect=/Users/dusty/tester/log/
bj.dustymini.local.development.log --ppid=23928 –
rails_env=development --rails_root=/Users/dusty/tester

But, the job is never run, and I don’t have any errors. Here is some
snippets from my logs

$ cat log/bj.dustymini.local.development.log
I, [2008-07-21T21:15:48.828069 #23934] INFO – : STARTED

$ cat log/development.log
Bj::Table::Job Create (0.000862) INSERT INTO
“bj_job” (“is_restartable”, “pid”, “stdout”, “stdin”, “command”,
“submitter”, “finished_at”, “env”, “stderr”, “submitted_at”,
“priority”, “tag”, “runner”, “started_at”, “exit_status”, “state”)
VALUES(1, NULL, NULL, NULL, ‘ls’, ‘dustymini.local’, NULL, NULL, NULL,
‘2008-07-22 01:22:37’, 0, ‘’, NULL, NULL, NULL, ‘pending’)
Bj::Table::Config Load (0.000479) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.no_tickle’ AND
“bj_config”.“hostname” = ‘dustymini.local’) LIMIT 1
Bj::Table::Config Load (0.000446) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.no_tickle’ AND
“bj_config”.“hostname” = ‘') LIMIT 1
Bj::Table::Config Load (0.001018) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.23928.pid’ AND
“bj_config”.“hostname” = ‘dustymini.local’) LIMIT 1
Bj::Table::Config Load (0.001485) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘23934.uri’ AND “bj_config”.“hostname” =
‘dustymini.local’) LIMIT 1
Bj::Table::Config Load (0.000288) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘23934.uri’ AND “bj_config”.“hostname” = '’)
LIMIT 1

Any ideas on how I can get this to run my job queue?

Thanks

dusty · July 22, 2008, 3:35am

BTW - I just resubmitted a new job and then ran all the queries that
showed up in the logs manually to see what it was getting back here is
that output.

From development.log

Bj::Table::Job Create (0.001226) INSERT INTO
“bj_job” (“is_restartable”, “pid”, “stdout”, “stdin”, “command”,
“submitter”, “finished_at”, “env”, “stderr”, “submitted_at”,
“priority”, “tag”, “runner”, “started_at”, “exit_status”, “state”)
VALUES(1, NULL, NULL, NULL, ‘ls’, ‘dustymini.local’, NULL, NULL, NULL,
‘2008-07-22 01:30:09’, 0, ‘’, NULL, NULL, NULL, ‘pending’)
Bj::Table::Config Load (0.000502) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.no_tickle’ AND
“bj_config”.“hostname” = ‘dustymini.local’) LIMIT 1
Bj::Table::Config Load (0.000677) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.no_tickle’ AND
“bj_config”.“hostname” = ‘’) LIMIT 1
Bj::Table::Config Load (0.000361) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.23996.pid’ AND
“bj_config”.“hostname” = ‘dustymini.local’) LIMIT 1
Bj::Table::Config Load (0.000267) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.23996.pid’ AND
“bj_config”.“hostname” = '’) LIMIT 1
Bj::Table::Config Load (0.000276) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.0.pid’ AND “bj_config”.“hostname” =
‘dustymini.local’) LIMIT 1
Bj::Table::Config Load (0.000247) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘development.0.pid’ AND “bj_config”.“hostname” =
‘’) LIMIT 1
Bj::Table::Config Load (0.000254) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘.uri’ AND “bj_config”.“hostname” =
‘dustymini.local’) LIMIT 1
Bj::Table::Config Load (0.000244) SELECT * FROM “bj_config” WHERE
(“bj_config”.“key” = ‘.uri’ AND “bj_config”.“hostname” = '’) LIMIT 1

Running all those commands manually, one returned an answer (between
the **)

sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” =
‘development.no_tickle’ AND “bj_config”.“hostname” =
‘dustymini.local’) LIMIT 1;
sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” =
‘development.no_tickle’ AND “bj_config”.“hostname” = ‘*’) LIMIT 1;

**
sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” =
‘development.23996.pid’ AND “bj_config”.“hostname” =
‘dustymini.local’) LIMIT 1;
1|dustymini.local|development.23996.pid|24011|to_i
**

sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” =
‘development.23996.pid’ AND “bj_config”.“hostname” = ‘’) LIMIT 1;
sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” =
‘development.0.pid’ AND “bj_config”.“hostname” = ‘dustymini.local’)
LIMIT 1;
sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” =
‘development.0.pid’ AND “bj_config”.“hostname” = '’) LIMIT 1;
sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” = ‘.uri’
AND “bj_config”.“hostname” = ‘dustymini.local’) LIMIT 1;
sqlite> SELECT * FROM “bj_config” WHERE (“bj_config”.“key” = ‘.uri’
AND “bj_config”.“hostname” = ‘*’) LIMIT 1;

dusty · July 22, 2008, 4:20am

An update. I just ran the same setup as I had above, but ran it on
ubuntu and it worked fine. I was running on leopard previously. Any
ideas? Has anyone run bj on leopard?

dusty · July 22, 2008, 6:03am

On Jul 21, 2008, at 8:14 PM, dusty wrote:

An update. I just ran the same setup as I had above, but ran it on
ubuntu and it worked fine. I was running on leopard previously. Any
ideas? Has anyone run bj on leopard?

i developed it on leopard

frankly i didn’t see anything wrong with your setup. bj has a bug
with timezones where it’s unaware of the TZ configured in rails for AR

is the TZ for your app the same as for your machine?

a @ http://codeforpeople.com/

dusty · July 22, 2008, 2:30pm

It does appear to be time zone related.

My ec2 ubuntu instance has the same time zone for the machine and
rails
root@domU-12-31-39-00-7D-E5:~/tester# ./script/console
Loading development environment (Rails 2.1.0)

Time.zone
=> #<TimeZone:0x8f9bae8 @name=“UTC”, @tzinfo=nil, @utc_offset=0>

Time.zone.now
=> Tue, 22 Jul 2008 12:12:17 UTC +00:00

Time.now
=> Tue Jul 22 12:12:19 +0000 2008

My machine has EST local time, but Time.zone was set to UTC by
default.
[dusty@dustylaptop:~/tester] $ ./script/console
Loading development environment (Rails 2.1.0)

Time.zone
=> #<TimeZone:0x11d3548 @tzinfo=nil, @name=“UTC”, @utc_offset=0>

Time.zone.now
=> Tue, 22 Jul 2008 12:12:40 UTC +00:00

Time.now
=> Tue Jul 22 08:12:42 -0400 2008

However, changing rails to EST, doesn’t seem to help. I’d guess its
because rails stores it as UTC in the db.

Time.now
=> Tue Jul 22 08:17:35 -0400 2008

Time.zone
=> #<TimeZone:0x11d3a5c @tzinfo=#<TZInfo::DataTimezone: America/
New_York>, @name=“Eastern Time (US & Canada)”, @utc_offset=-18000>

Time.zone.now
=> Tue, 22 Jul 2008 08:20:14 EDT -04:00

Bj.table.job.last
=> #<Bj::Table::Job bj_job_id: 4, command: “ls”, state: “pending”,
priority: 0, tag: “”, is_restartable: 1, submitter:
“dustylaptop.local”, runner: nil, pid: nil, submitted_at: “2008-07-22
12:17:16”, started_at: nil, finished_at: nil, env: nil, stdin: nil,
stdout: nil, stderr: nil, exit_status: nil>

Setting my system clock to UTC, fixed it

Bj.table.job.last
=> #<Bj::Table::Job bj_job_id: 4, command: “ls”, state: “finished”,
priority: 0, tag: “”, is_restartable: 1, submitter:
“dustylaptop.local”, runner: “dustylaptop.local”, pid: 2453,
submitted_at: “2008-07-22 12:17:16”, started_at: “2008-07-22
12:22:53”, finished_at: “2008-07-22 12:22:53”, env: nil, stdin: nil,
stdout: “README\nRakefile\napp\nconfig\ndb\ndoc\nlib\nlog\npublic
\nsc…”, stderr: “”, exit_status: 0>

Time.now
=> Tue Jul 22 12:24:28 +0000 2008

I would prefer to be able to keep my laptop at my local timezone,
though. Is there a workaround yet? I could also try to come up with
something. Perhaps a hint at where in the code to look around at, and
I’ll see if I can come up with something.

Thanks for the response!

dusty · July 22, 2008, 2:50pm

One more thing. I also have problems submitting via command-line.

[dusty@dustylaptop:~/tester] $ ./script/bj submit cat /etc/password
F, [2008-07-22T12:38:34.993286 #2555] FATAL – : can’t convert nil
into String (TypeError)
./script/bj:434:in open' ./script/bj:434:inrun’
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:25:in call' /Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:25:inrun’
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:13:in catch' /Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:13:inrun’
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:436:in
mode_run!' /Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:10:inrun’
/Users/dusty/tester/vendor/plugins/bj/lib/main/factories.rb:11:in
run' /Users/dusty/tester/vendor/plugins/bj/lib/main/factories.rb:16:inMain’
./script/bj:6

Looks like that’s failing on this line:

      open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }

Any ideas on that one?

Also, is there a better place for this? I see there is a bug on this
in rubyforge, but its pretty old. Should I still submit things like
this to rubyforge instead of this mailing list?

Thanks

mode “submit” do
keyword(“file”){
argument :required
attr
}

def run
  joblist = Bj.joblist.for argv.join(' ')

  case file
    when "-"
      joblist.push(Bj.joblist.jobs_from_io(STDIN))
    when "--", "---"
      joblist.push(Bj.joblist.jobs_from_yaml(STDIN))
    else
      open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }
  end

  jobs = Bj.submit joblist, :no_tickle => true

  oh = lambda{|job| OrderedHash["id", job.id, "command",

job.command]}

  y jobs.map{|job| oh[job]}
end

end

dusty · July 22, 2008, 6:59pm

On Jul 22, 2008, at 6:44 AM, dusty wrote:

It appears to be working.

this plus fixes to the commad line interface (which was demo only btw)
will be in svn soon. thanks for the bug reports.

a @ http://codeforpeople.com/

dusty · July 22, 2008, 2:51pm

Answering my question.

http://rubyforge.org/tracker/index.php?func=detail&aid=19765&group_id=1024&atid=4025

They recommended to change all instances of Time.now to Time.now.utc
in

lib/bj/runner.rb
lib/bj/table.rb
lib/systemu.rb

It appears to be working.

dusty · August 8, 2008, 6:08pm

I am attempting to get the env hash from within the worker process.

Seems like it should be meshed into ENV but the documentation is
unclear.

I accomplished it this way. It works because we ensure only one job with
a certain tag is run at a time but it isn’t ideal.

job = Bj.table.job.find(:first, :conditions => { :tag =>
‘product_publisher’,
:state => ‘running’
})
env = YAML.load job.env

Is there a better way??

~)o
gustin

entryway software development
http://www.entryway.net

dusty · August 8, 2008, 7:13pm

On Aug 8, 2008, at 10:06 AM, Gustin Prudner wrote:

I am attempting to get the env hash from within the worker process.

Seems like it should be meshed into ENV but the documentation is
unclear.

did you simply try it?

you should be able to access it via ENV as systemu merges that in for
you

var = ENV[‘var’]

should be all you need, assuming you specified that var in the call
the bj submit…

env = YAML.load job.env

Is there a better way??

use ENV

~)o
gustin

entryway software development
http://www.entryway.net

Posted via http://www.ruby-forum.com/.

a @ http://codeforpeople.com/

dusty · April 24, 2009, 2:03am

you should be able to access it via ENV as systemu merges that in for
you

var = ENV[‘var’]

should be all you need, assuming you specified that var in the call
the bj submit…

use ENV

I’ve been struggling to get this to function… Any help would be
appreciated

While i have successfully created, by using bj, a text only PDF using
prawn/prawnto, i cannot successfully pass variables (ie an account ref)
through with :env , despite the earlier advice in this thread. Is it
possible, that in some circumstances, the full set of ENV would not be
passed to the bj background task ?

#-------------------------------------------------------
def bj_simple_prawn
Bj.submit “./script/runner ./script/simple_prawn_script.rb”, :env => {
:avar => @current_user.account_id, :bvar => 27 }
end

After submitting the job…

bj_job has the following in the :env field;

:bvar: 27
:avar: 1

I cannot seem to access :env field/variable from bj_job table in the
worker task

This line fails to extract the value passed in :env

@this_user = ENV[‘myvar’]

The following works ! - if use a fixed value, 1.
@this_user = 1

@myaccount = Account.find(:first, :conditions => [“accounts.id=?”,
@this_user])

Question : How does one access the values passed in the job :env ? I
suspect a syntax error on my part , but cannot pin it down
Like i say … any help /pointer would be appreciated…

Denis

dusty · April 24, 2009, 6:10pm

Update on my previous message/query…

I should clarify… The :env parameters are being passed in the ENV
and I can see
them being displayed in STDOUT in bj_job. The following line
generatess the STDOUT data

@all_of_ENV = ENV.each {|key, value| puts “#{key} is #{value}” }

It effectively outputs the complete ENV as [Key, Value] pairs in
STDOUT.
However the :env parameters I pass with the B.submit command seem to
be malformed and
not amenable to extraction from ENV.
They appears as follows as the 2nd last key/value output by the puts
function above ;

is
:avar: 1
is
:bvar: 42
is

followed by ;

RAILS_ENV is development

You can see the Key, Value formatting is different.
I would have expected them each to be on a single line ie ;

:avar is 1
:bvar is 2

Are the additional CR/LF elements intruding ?

Again , thanks for any help/pointers.
Regards

Denis

dusty · April 30, 2009, 3:15am

Update II *

I abandoned the attampt to use the Bj.submit :env switch to pass
values to the background task in bj.
I just could not figure how to extract/use the values passed as YAML
(?) in ENV to the task.
I wonder if anybody has successfuly used it ? Is it broken ?

I’m now using the command line parameters, in the foreground process .
viz,

Bj.submit “./script/runner ./script/simple_prawn_script.rb #
{@this_user}”

I can now access the passed value in the worker using this
expression:

@this_user = ARGV[0].

Bj rocks

Regards

Denis

Bj and rails 2.1 - can't get bj to run jobs

entryway software development http://www.entryway.net

This line fails to extract the value passed in :env

entryway software development
http://www.entryway.net