Rails csv import with progress bar

Rails
1

Hey guys, got a rails question:

I’ve finished creating a csv import ability in my web application that
parses a csv file and adds records into the MySQL db. I figuired it
would be nice to have a progress bar for this import process.
Unfortunately I have run into issues trying to implement this…

What I have tried right now is to call a javascript function on submit
of the form specifying the file to import. Every 5 seconds this
function creates an Ajax request to a “update_progress” method
inside my controller. What I’m seeing is that all the Ajax requests
are not called until AFTER my method “csv_import” that parses the csv
file completes (rails seems to be too busy parsing)… so all I get is
0% and then jumps to 100%. I’ve also tried adding a render statement
to the “update_progress” method inside the loop that parses the csv
file but this is a problem since rails only lets you render once in a
method (I’m already rendering the results at the end). Does any one
know a better way for me to display to the user the progress of my
“csv_import” method.

Basic outline of method “csv_import” and “update_progress”:

$total_lines_read = 0

def csv_import
open file
read & add record for each line to the db
increment $total_lines_read
render the results
end

def update_progress
render javascript function to update the bar sending it
$total_lines_read
end

2

Nabeel Mr. wrote:

Hey guys, got a rails question:

I’ve finished creating a csv import ability in my web application that
parses a csv file and adds records into the MySQL db. I figuired it
would be nice to have a progress bar for this import process.
Unfortunately I have run into issues trying to implement this…

Oh yeah. HTTP requires very narrow events. To fix this, you must stretch
one
logical event out over many physical events.

Try this:

  • the “Go” button starts a periodically_call_remote
  • that calls an action
  • the action opens the CSV,
    • reads one batch from it (say 20 lines)
    • stash the current record count in the session
    • returns nothing
  • the periodically_call_remote ticks the progress bar
  • the periodically_call_remote stops when the action
    • returns JavaScript assigning ‘false’ to a global variable
  • the periodically_call_remote’s condition is that variable

That sounds like a lot of work, but the alternatives are all worse!

And if you progress, you should have a cancel button. My system makes
your
reader function easier to interrupt.

–
Phlip
http://assert2.rubyforge.org/

3

Another option is to use backgroundrb to process the csv file outside
the current rails process. This has the advantage of not blocking web
requests to that process because your app is processing the file, as
well as allowing you to check in on the job periodically using the
worker id to see how it’s going and to update your progress bar.

then again, getting backgroundrb running for this particular problem
may be overkill :slight_smile:

hth, Matt

On 2/21/08, Phlip [email protected] wrote:

Unfortunately I have run into issues trying to implement this…
- stash the current record count in the session

–
Phlip
http://assert2.rubyforge.org/

–
Matt W.

Thermal Creative
http://blog.thermalcreative.com

4

Nabeel,

When you control the loop, a simple solution is “action chunking” :
cut your big task/action into smaller chunks, that chain/call themself
automatically, and display a little visual feedback between each
transition.

This process is documented here :

words :
http://docs.google.com/View?docid=ddmqwkxs_3972nphr5fk
video demos :
action_chunking_demo
http://screencast.com/t/ba5dD9OPv6

Alain R.

5

then again, getting backgroundrb running for this particular problem
may be overkill :slight_smile:

hth, Matt

That’s why I didn’t mention backgroundrb. (We abuse that all the time,
at my
day-job, to avoid action-chunking!)

If the OP learned they could install a fragile deployment-nightmare as
an excuse
not to split their inner loop up into action chunks, they might behave
the same
way Java programmers do when they hear that threading in Java is
“easy”…

–
Phlip
http://assert2.rubyforge.org/

6

I’ve finished creating a csv import ability in my web application that
parses a csv file and adds records into the MySQL db. I figuired it
would be nice to have a progress bar for this import process.
Unfortunately I have run into issues trying to implement this…

I had to do almost exactly this, I screen casted it and documented it
here:

Regards

Mikel