Forum: Ruby on Rails How to execute time consuming code

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
F50f5d582d76f98686da34917531fe56?d=identicon&s=25 Peter Szinek (Guest)
on 2006-05-22 15:20
(Received via mailing list)
Hello all,

I have a screen scraping application (go to a lots of sites, extract 10k
stuff, integrate the results, put them to DB etc). Now i want to use a
Rails application as a frontend to this: The user can push a button
which triggers the screen scraping app and view the results (preferably
asynchronously, but that does not really matter right now).

Questions:
- Should the screen scraping app reside in the Rails directory structure
or somewhere else (since it can be viewed as a standalone application-
Rails just displays data from a DB, does not necessarily need to know
who and how did generate the DB). OTOH, if it is easier to communicate
if the screen scraping dir is integrated into the Rails directory
structure, i don't have problem to do that either. In this case, where
should i put the screen scraping code? (We are talking about lets say 50
classes altogether)

- How to trigger the screen scraping in the background - i.e. the user
clicks a 'start' button, gets back a 'screen scraping started' message
and can work with the web page further, instead of waiting on the result
(since that can a last even several hours in extreme cases)

Thanks,
Peter
2f46d76f0e5db4dc318b03be07ebaac4?d=identicon&s=25 Tom Ward (Guest)
on 2006-05-22 18:28
(Received via mailing list)
We've solved similar problems the following way:

We have a table of long-running jobs, and a independent process that
polls and processes these jobs.  When our rails app needs to trigger a
job, it simply adds a row to the db.

For our job processing code, we still use the rails application
framework, but have new commands (in the script directory) to start
and stop the processor (independently of the server).  Both parts of
the application share as much code as possible (including for the most
part the environment.rb and database.yml config).

Tom
A0c079a7c3c9b2cf0bffebd84dc578b0?d=identicon&s=25 unknown (Guest)
on 2006-05-22 18:35
(Received via mailing list)
On May 22, 2006, at 8:17 AM, Peter Szinek wrote:

> Questions:
>
> - How to trigger the screen scraping in the background - i.e. the user
> clicks a 'start' button, gets back a 'screen scraping started' message
> and can work with the web page further, instead of waiting on the
> result
> (since that can a last even several hours in extreme cases)

Consider using BackgrounDRb [1] to handle long-running tasks. It
installs as a plugin and has all the code residing within your Rails
app. It can also communicate to remote servers via DRb so you can do
long-running tasks on another machine if you like.

cr

[1] http://brainspl.at/articles/2006/05/15/backgoundrb...
This topic is locked and can not be replied to.