# Update page with results as they are scraped **URL:** \ **Category:** Rails\ **Created:** [April 8, 2009, 7:17am UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005 "2009-04-08T07:17:31Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 8, 2009, 7:17am UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/1 "2009-04-08T07:17:31Z")
Hi, Im creating a screenscraper app that takes a users search term, scrapes several 3rd party sites and returns the aggregated results after a few seconds. Rather than wait for all the sites scraping to be completed before showing the results, I want to show the results as they are scraped, ie. when one sites results have been scraped and prepared, they are shown immediately whilst the next set are being done. Initially I thought id just make a scraper function like so def scrape\_sites sites.each do |a\_site| a\_sites\_results = scrape\_and\_parse\_this\_site(a\_site) yield a\_sites\_results end end Then in my controllers js response I could take the results and render them but then i found out htat you cant make multiple render calls in one go. How do i go about doing this? Does this behaviour have a name like say “live search” etc ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 8, 2009, 1:10pm UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/2 "2009-04-08T13:10:10Z")
ive found that this pattern is called multi stage download. Its used by [kayak.com](http://kayak.com) which is famous for its user interface. That site makes finding cheap flights actually enjoyable. Here are some general background links i found this discusses the pattern in general [http://ajaxpatterns.org/Multi-Stage\_Download#Code\_Example](http://ajaxpatterns.org/Multi-Stage_Download#Code_Example) and this is a review of the user interface for kayak [http://konigi.com/podcast/kayak-com](http://konigi.com/podcast/kayak-com) But i still dont know where to start looking for a rails implementation how to. Where should i start looking (ive googled for ages, cant find anything appropriate) ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 8, 2009, 1:47pm UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/3 "2009-04-08T13:47:10Z")
On Apr 8, 12:10 pm, Adam A. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > and this is a review of the user interface for kayak > > [http://konigi.com/podcast/kayak-com](http://konigi.com/podcast/kayak-com) > > But i still dont know where to start looking for a rails implementation > how to. > > Where should i start looking (ive googled for ages, cant find anything > appropriate) Given that this is all client side, what would a rails implementation be (maybe the odd helper function but the lack of them certainly shouldn’t stop you getting started). As far as your particular case goes all you need is something on the page that every however often pings your controller to say ‘do you have any more data for me’. It would be wise to push the actual scraping into a separate process Fred ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 8, 2009, 3:28pm UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/4 "2009-04-08T15:28:44Z")
pining the controller? should have read PINGING the controller. sorry ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 8, 2009, 3:28pm UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/5 "2009-04-08T15:28:06Z")
> Given that this is all client side, what would a rails implementation > be (maybe the odd helper function but the lack of them certainly > shouldn’t stop you getting started). > > As far as your particular case goes all you need is something on the > page that every however often pings your controller to say ‘do you > have any more data for me’. It would be wise to push the actual > scraping into a separate process Thanks Frederick, cna you (or anybody reading this) provide me with some helper names or subject names that i should be investigating re: pining the controller? is this a common rails thing? thanks so far for your help ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 8, 2009, 4:02pm UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/6 "2009-04-08T16:02:11Z")
On 8 Apr 2009, at 14:28, Adam A. wrote: > > To make things clear, pinging just means (in this context) make a request every so often. If you’re using prototype PeriodicalExecuter is a good way to go (periodically\_call\_remote is a helper for that) Fred ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 8, 2009, 5:35pm UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/7 "2009-04-08T17:35:03Z")
I do something similar in one of my apps. The way I handle it is using periodically\_call\_remote every 6 seconds and check it against a controller action which looks at the DB to see if any new records were added for that user since the last time it checked. If there are, do a render :update |page| and add an insert\_html with the new record on top. Naturally you can increase or decrease the time between checks, but I figure 5-10 seconds should be good for anyone. For my specific app, if I was scraping 5 items specifically, the periodically\_call\_remote function would turn itself off after I returned 5 records. ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [April 9, 2009, 12:24am UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/8 "2009-04-09T00:24:04Z")
Excellent, lll go away and look into those methods. Many thanks! ---
**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\ **Post date:** [May 25, 2009, 7:12am UTC](https://www.ruby-forum.com/t/update-page-with-results-as-they-are-scraped/165005/9 "2009-05-25T07:12:17Z")
Jack B. wrote: > I do something similar in one of my apps. The way I handle it is using > periodically\_call\_remote every 6 seconds and check it against a > controller action which looks at the DB to see if any new records were > added for that user since the last time it checked. If there are, do a > render :update |page| and add an insert\_html with the new record on top. > > Naturally you can increase or decrease the time between checks, but I > figure 5-10 seconds should be good for anyone. > > For my specific app, if I was scraping 5 items specifically, the > periodically\_call\_remote function would turn itself off after I returned > 5 records. Hi Jack, How do you turn off a periodically\_call\_remote? Also is it possible to call periodically\_call\_remote on some action? Thanks, Sudhindra