Forum: Ruby Help with Screen Scraper!

Announcement (2017-05-07): is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see and for other Rails- und Ruby-related community platforms.
Ta S. (Guest)
on 2008-11-19 21:52

I need some help with a simple screen scraper app. Here's how it works:

1) I open a connection to a url
2) I look for a specific string
3) When I get the string, I do something with it
4) Important: I'm polling a url looking for data changes.

Everything works fine, until the website is updated with new content. At
that point, my code throws an "private method `scan' called for
nil:NilClass (NoMethodError)"

I can get the code to work using

But, the problem is that it causes the connection to close and re-open.
This causes a further delay in grabbing the data.

Is there a way to handle this? Again, the polling works fine until the
server updates the new page that I'm scanning. At that time, the error
is thrown and I either have to re-open a connection or fail on error. My
Java application seems to be handling this more gracefully, but, Ruby is
faster on the polling. Thanks for any insight.


require 'open-uri'
  pval = "14"
  stop = false

    while stop == false
        #Create timestamp
        sts="S " <<"%m/%d/%y %H %M
        puts sts
      #Get Lines of data and find the SearchValue
      tdate=lines.detect {|line| line =~ /SearchValue/}.scan(/\d+/)
      #Compare SearchValue to pval
      if tdate[1] != pdate

        puts tdate[0]
        ts="F "<<"%m/%d/%y %H %M
        puts ".."
This topic is locked and can not be replied to.