Parallel::ForkManager, Net::HTTP and catching Timeout::Error

Hello,

I’m about to release Parallel::ForkManager 1.1 and while generating
new examples (that use Net::HTTP) for PFM 1.1 features, I catch the
following error sometimes (maybe 10-15% of the time) when I try to
connect to a URL where the host part doesn’t exist or isn’t reachable:

/usr/lib/ruby/1.8/timeout.rb:60:in rbuf_fill': execution expired (Timeout::Error) from /usr/lib/ruby/1.8/net/protocol.rb:134:inrbuf_fill’
from /usr/lib/ruby/1.8/net/protocol.rb:86:in read' from /usr/lib/ruby/1.8/net/http.rb:2212:inread_body_0’
from /usr/lib/ruby/1.8/net/http.rb:2173:in read_body' from /usr/lib/ruby/1.8/net/http.rb:773:inget’
from /usr/lib/ruby/1.8/net/http.rb:1053:in request' from /usr/lib/ruby/1.8/net/http.rb:2136:inreading_body’
from /usr/lib/ruby/1.8/net/http.rb:1052:in request' from /usr/lib/ruby/1.8/net/http.rb:1037:inrequest’
from /usr/lib/ruby/1.8/net/http.rb:543:in start' from /usr/lib/ruby/1.8/net/http.rb:1035:inrequest’
from /usr/lib/ruby/1.8/net/http.rb:772:in get' from ./parallel_http_get2.rb:36 from ./lib/parallel/forkmanager.rb:232:incall’
from ./lib/parallel/forkmanager.rb:232:in start' from ./lib/parallel/forkmanager.rb:232:infork’
from ./lib/parallel/forkmanager.rb:232:in start' from ./parallel_http_get2.rb:30 from ./parallel_http_get2.rb:26:ineach’
from ./parallel_http_get2.rb:26

Based on my review of the error message, it would seem to be a simple
case that I should catch (rescue) ‘Exception’ because Timeout::Error
is sending to stderr or such, but when I try to catch ‘Exception’ in
my test program, nothing happens! I never catch the error!

Here’s my test code:

#!/usr/bin/env ruby

require ‘net/http’
require ‘lib/parallel/forkmanager’

save_dir = ‘/tmp’

my_urls = [
http://www.cnn.com/index.html’,
http://oreilly.com/index.html’,
http://www.cakewalk.com/index.html’,
http://www.asdfsemicolonl.kj/index.htm
]
my_timeout = 5 # seconds

max_proc = 20
pfm = Parallel::ForkManager.new(max_proc)

pfm.run_on_finish(
lambda {
|pid,exit_code,ident|
print “** PID (#{pid}) for #{ident} exited with code
#{exit_code}!\n”
}
)

my_urls.each {
|my_url|

begin
    pfm.start(my_url)  {
        url = URI.parse(my_url)
        out_file = save_dir + '/' + url.host + '.txt';

        http = Net::HTTP.new(url.host, url.port)
        http.open_timeout = http.read_timeout = my_timeout
        res = http.get(url.path)
        status = res.code

        if status.to_i == 200
            f = File.open(out_file, 'w')
            f.print res.body
            f.close()
            exit 0
        else
            exit 255
        end

        http = Net::HTTP.new(url.host, url.port)
        http.open_timeout = http.read_timeout = my_timeout
        res = http.get(url.path)
        status = res.code

        if status.to_i == 200
            f = File.open(out_file, 'w')
            f.print res.body
            f.close()
            exit 0 # start() with a block means that we exit with

status or else it’s 0 all the time.
else
exit 255
end
} # end pfm.start { … }
rescue Exception => e
print "Arggh, exception: ", e, “\n”
exit 255
end
}

pfm.wait_all_children()

end

Would somebody be able to shed some light on why I’m unable to handle
the exceptions that Net::HTTP is throwing? Am I just trying to catch
the wrong exception or is there something else? Is it because I’m
trying to handle the exception in the child?

Note that this is: ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux]
under Ubuntu 9.10 (Karmic).

nvp [email protected] wrote:

Hello,

I’m about to release Parallel::ForkManager 1.1 and while generating
new examples (that use Net::HTTP) for PFM 1.1 features, I catch the
following error sometimes (maybe 10-15% of the time) when I try to
connect to a URL where the host part doesn’t exist or isn’t reachable:

/usr/lib/ruby/1.8/timeout.rb:60:in `rbuf_fill’: execution expired
(Timeout::Error)

> Based on my review of the error message, it would seem to be a simple > case that I should catch (rescue) 'Exception' because Timeout::Error > is sending to stderr or such, but when I try to catch 'Exception' in > my test program, nothing happens! I never catch the error! > > Here's my test code:
begin
    pfm.start(my_url)  {
        url = URI.parse(my_url)
        out_file = save_dir + '/' + url.host + '.txt';

        http = Net::HTTP.new(url.host, url.port)
        http.open_timeout = http.read_timeout = my_timeout
        res = http.get(url.path)
    } # end pfm.start { ... }
rescue Exception => e
    print "Arggh, exception: ", e, "\n"
    exit 255
end

}

Would somebody be able to shed some light on why I’m unable to handle
the exceptions that Net::HTTP is throwing? Am I just trying to catch
the wrong exception or is there something else? Is it because I’m
trying to handle the exception in the child?

It looks like the parent is trying to rescue exception in a child, which
isn’t possible. Exceptions are Ruby-level objects that normally don’t
get transferred/communicated between Unix processes. The parent can
only normally get the exit status of a child process, otherwise it needs
to resort to explicit IPC mechanisms (pipes/sockets/files/shared
memory/…).

On 31 déc. 2009, at 00:50, nvp wrote:

/usr/lib/ruby/1.8/timeout.rb:60:in `rbuf_fill’: execution expired
(Timeout::Error)

FWIW, I’m getting the exact same error - at roughly the same rate - in a
regular mongrel-based application which talks to a separate HTTP backend
and I would love to get rid of it too :slight_smile: