backgroundRB problem (newbie)

Rails
1

Hello,

I am trying to upload and process very large CSV files. For this I am
trying to use backgroundRB. The problem is that I cannot access a
variable inside my worker until the methods have finished even though
I am calling the method asynchronously.

My controller:

def authenticate_import
@dataset = Dataset.new params[:dataset]
@dataset.campaign = @campaign
#Check the normal info is valid
if !@dataset.valid?
render :update do |page|
page.replace_html ‘import’, :action => ‘import’
end
else
@key = Time.now.to_i
@worker = MiddleMan.worker(:dataset_worker, @key)
@worker.async_process_csv(:arg => @dataset)
render :update do |page|
page.replace_html ‘progress’, :partial =>
‘progress_meter’, :object => @key, :locals => {:progress =>
@worker.progress}
end
end
end

This should return the variable progress to the partial which at the
moment simply displays the local progress.

My worker:

class DatasetWorker < BackgrounDRb::MetaWorker
set_worker_name :dataset_worker

attr_accessor :progress

def create(args = nil)
logger.info “Dataset Worker setup”
@progress = 0
end

def process_csv(dataset)
logger.info “Process CSV method called”
logger.info “Dataset is a #{dataset.type}”

logger.info "Dataset: begin parsing file"
  dataset.process_file
  @progress = 30
logger.info "Dataset: parsing complete"
logger.info "Dataset: Creating table in db"
  if dataset.save
    table = dataset.table
    total = table.size
        begin
            #Create the fields first, otherwise we would have to

check whether they had been created
#for every row were inserting (alot of SQL calls for
large CSV files)
row_cnt = 0
table.headers.each do |field|
DatasetField.create({:dataset =>
dataset, :dataset_field_name => field})
end
#reload fields
dataset.dataset_fields(true)
table.by_row!.each do |row|
dataset.create_row row
row_cnt++
@progress = ((row_cnt/total)*70)+30
end
end
end
logger.info “Dataset: Finished creating table in db”
end

end

I am trying to access the variable progress so that I can display it.
Then I can periodically update the progress by using the job key. The
problem is the Ajax requests doesn’t finish until the method
process_csv has finished.

Any help would be greatly appreciated

2

On Wed, Aug 13, 2008 at 4:21 PM, Iain A.
[email protected] wrote:

Hello,

I am trying to upload and process very large CSV files. For this I am
trying to use backgroundRB. The problem is that I cannot access a
variable inside my worker until the methods have finished even though
I am calling the method asynchronously.

Problem is by default worker is synchronous and uses event driven
network programming, so next event is not picked until existing
handler returns. But, if you #defer process_csv inside a thread, next
call to @worker.progress should return as you expect (for using #defer
method inside worker, refer backgroundrb.rubyforge.org).

Also, if you are going to use threads, then you need to protect
@progress accessor with a mutex or something, so as concurrent
threads, update it synchronously.

Another alternative is to use, bakcgroundrb with memcache for caching
result objects (such as progress bar and stuff). From worker just do:

cache[:progress_percent] = 80

and from rails:

MiddleMan.worker(:foo_worker).ask_result(:progress_percent)

(refer official docs for more info).

   else

def create(args = nil)
@progress = 30
row_cnt = 0
end
process_csv has finished.

Any help would be greatly appreciated

–
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org