Open uri buf read overflow

I’m using open-uri to open approx 3500 web pages to check to see if they
have active contect. I’m using open uri to do it, but after about 2346
pages have been open I get a buf_read error and the program crashes.
After looking in the open-uri documentation I couldn’t find anything
that closes the connection and dumps the memory after the open method is
called. Does anyone know how to dump the memory after I make each open()
call?
begin
while (line = f.readline)
line = line.split(/\n/)
url = “http://www.” + line[0] + “.com”
#puts url
begin
open(url) { #|page| page_content = page.read()
puts url + " exists"
nf.puts url
}
rescue
puts"URL does not exist"
#retry
end
end
rescue EOFError
f.close
nf.close
end

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs