Help with Screen Scraper!


#1

Hi,

I need some help with a simple screen scraper app. Here’s how it works:

  1. I open a connection to a url
  2. I look for a specific string
  3. When I get the string, I do something with it
  4. Important: I’m polling a url looking for data changes.

Problem:
Everything works fine, until the website is updated with new content. At
that point, my code throws an “private method `scan’ called for
nil:NilClass (NoMethodError)”

I can get the code to work using
begin

rescue
retry
end

But, the problem is that it causes the connection to close and re-open.
This causes a further delay in grabbing the data.

Is there a way to handle this? Again, the polling works fine until the
server updates the new page that I’m scanning. At that time, the error
is thrown and I either have to re-open a connection or fail on error. My
Java application seems to be handling this more gracefully, but, Ruby is
faster on the polling. Thanks for any insight.

Regards.


require ‘open-uri’
pval = “14”
url=“http://www.someurl.com
stop = false

while stop == false
    #Create timestamp
    sts="S " << Time.now.strftime("%m/%d/%y %H %M

%S.")<<Time.now.usec.to_s
puts sts
lines=open(url).read.split("\n")
#Get Lines of data and find the SearchValue
tdate=lines.detect {|line| line =~ /SearchValue/}.scan(/\d+/)
#Compare SearchValue to pval
if tdate[1] != pdate

    puts tdate[0]
    ts="F "<<Time.now.strftime("%m/%d/%y %H %M

%S.")<<Time.now.usec.to_s
else
puts “…”
end
end