SocketError when letting ruby open url from txt-file

[aris]

Hi all,

I have constructed a code that opens all urls in a textfile one by one,
parses through them and finally saves the results into an excel file.

When I run the code on a textfile with just a few urls, it works
perfectly.
When i run the code on a textfile with many thousands of urls, I get an
error (“in ‘intitialize’: getaddrinfo: Name or service not known
(SocketError)”). What might be causing the issue?

CODE:
require ‘nokogiri’
require ‘open-uri’
require ‘rubygems’
require ‘writeexcel’

workbook = WriteExcel.new(‘parseresult.xlsx’)
worksheet = workbook.add_worksheet
row = 0

File.foreach(“websites.txt”) do |line| #loop on basis urls textfile

searchablefile = Nokogiri::HTML(open(line)) #open each url

#creation of variables
referentieid = searchablefile.at_xpath("//td/strong[contains(text(),
‘Referentie’)]/parent::/following-sibling::")
status = searchablefile.at_xpath("//td/strong[contains(text(),
‘Status’)]/parent::/following-sibling::")

unless searchablefile.at_xpath("//td/strong[contains(text(),
‘Referentie’)]/parent::/following-sibling::").nil?
worksheet.write(row, 1, referentieid.content)
end
unless searchablefile.at_xpath("//td/strong[contains(text(),
‘Status’)]/parent::/following-sibling::").nil?
worksheet.write(row, 2, status.content)
end
row += 1 #next row for next url
end
workbook.close

ERROR:
[email protected]:~$ ruby directerubyparsewoningmarkt.rb
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:644:in
initialize': getaddrinfo: Name or service not known (SocketError) from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:644:inopen’
from
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:644:in
block in connect' from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/timeout.rb:44:intimeout’
from
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/timeout.rb:89:in
timeout' from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:644:inconnect’
from
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:637:in
do_start' from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/net/http.rb:626:instart’
from
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:306:in
open_http' from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:769:inbuffer_open’
from
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:203:in
block in open_loop' from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:201:incatch’
from
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:201:in
open_loop' from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:146:inopen_uri’
from
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:671:in
open' from /home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:33:inopen’
from directerubyparsewoningmarkt.rb:12:in block in <main>' from directerubyparsewoningmarkt.rb:10:inforeach’
from directerubyparsewoningmarkt.rb:10:in `

Thanks a bunch.

[sybrenkooistra]

On Tue, Oct 9, 2012 at 6:45 PM, Sybren K. [email protected]
wrote:

File.foreach(“websites.txt”) do |line| #loop on basis urls textfile
‘Referentie’)]/parent::/following-sibling::").nil?
ERROR:
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/timeout.rb:44:in
from
`block in open_loop’
/home/wadiem/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/open-uri.rb:671:in

Posted via http://www.ruby-forum.com/.

1.9.2p290 :001 > require ‘open-uri’
=> true
1.9.2p290 :002 > open(“http://ldfmldmflasfmkdfm”)
SocketError: getaddrinfo: Name or service not known

There’s probably a wrong URL in that file. Can you print it before
opening it?

Jesus.

[sybrenkooistra]

It’s half a million files… So I don’t know if printing the file first
would help (or what exactly do you mean)?

Is there a way to workaround possible bad-urls

[sybrenkooistra]

It works, perfect!

Jesus, thanks for all the help!

[sybrenkooistra]

On Tue, Oct 9, 2012 at 7:58 PM, Sybren K. [email protected]
wrote:

It’s half a million files… So I don’t know if printing the file first
would help (or what exactly do you mean)?

Yes, I meant that there’s probably some bad (invalid) uris. Or maybe
garbage in the file.

Is there a way to workaround possible bad-urls

Catch the exception and log the line, for example:

File.foreach(“websites.txt”) do |line| #loop on basis urls textfile
begin
searchablefile = Nokogiri::HTML(open(line)) #open each url

the rest of the logic

rescue
puts “there was an error opening #{line}”
end
end

Jesus.