# Too many open files causes Mechanize/Net::HTTP to timeout?

**URL:** <https://www.ruby-forum.com/t/too-many-open-files-causes-mechanize-net-http-to-timeout/122328>\
**Category:** Ruby\
**Created:** [December 6, 2007, 5:29pm UTC](https://www.ruby-forum.com/t/too-many-open-files-causes-mechanize-net-http-to-timeout/122328 "2007-12-06T17:29:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gabetax](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/dfb087/32.png) [@gabetax](https://www.ruby-forum.com/u/gabetax)\
**Post date:** [December 6, 2007, 5:29pm UTC](https://www.ruby-forum.com/t/too-many-open-files-causes-mechanize-net-http-to-timeout/122328/1 "2007-12-06T17:29:54Z")

</div>

I’m experiencing an issue where my get/submit calls to the Mechanize  
agent are leading to timeout exceptions when the ruby script has too  
many open file descriptors ( \> 1000). However, I’m not seeing anything  
about an overstep violation sent anywhere to syslog, and no error  
message sent to stdout. The process has the large amount of open files  
because it’s executed by PHP from an Apache install with \> 500 VHosts  
(with two logs for each vhost), and all of the open FD’s get inherited  
into the ruby process (as reported by lsof). I can get the script  
operating properly by commenting out several of the logs in my Apache  
install, but I’d prefer to get to the heart of what wall  
Ruby/Mechanize/Net::HTTP is running into (and not logging/reporting).

Since the cause of this is too many open files, I’ve tried the  
following:

- Setting the hard and soft ulimits in /etc/limits and  
/etc/security/limits.conf (however if there was an overstep, grsecurity  
should have logged this to the syslog/dmesg)
- 
# cat /proc/sys/fs/file-max

205126

- Recompiled ruby (ruby 1.8.6 (2007-09-24 patchlevel 111) [i686-linux])  
after updating the ulimits (this was something recommended in a similar  
situation with Squid proxy, although I didn’t see anything mentioned  
about FD\_SETSIZE or similar)

What I’m imagining is that the default FD\_SETSIZE defined in the linux  
headers is what’s getting hit, although I’ve heard that modifying those  
headers to increase the value can lead to instability in some services.

I’m hoping that someone can tell me:

- Which limit the process is hitting, and how to extend that limit
- Is this potentially a bug in Ruby that should go upstream?
- If there’s any sane way to discard all the inherited open FD’s that it  
inherits
- (and a long-shot non-ruby PHP question) if there’s any way I can have  
PHP execute the ruby script and have it not inherit the FD’s

== References ==

Sample driver of the .php script that executes the ruby script:

\<?php $id = 108; // hard-coded test $command = 'ruby ' . escapeshellarg('/www/CLIENT/htdocs/include/script/nysif\_scrape/run.rb') . ' ' . escapeshellarg($id) . ' \>/dev/null &'; // /dev/null redir is needed to keep program in the background system($command); ?\>

Here’s the backtrace on the exception that gets thrown on the timeout:  
/usr/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill' /usr/lib/ruby/1.8/timeout.rb:56:in`timeout’  
/usr/lib/ruby/1.8/timeout.rb:76:in `timeout' /usr/lib/ruby/1.8/net/protocol.rb:132:in`rbuf\_fill’  
/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil' /usr/lib/ruby/1.8/net/protocol.rb:126:in`readline’  
/usr/lib/ruby/1.8/net/http.rb:2029:in `read_status_line' /usr/lib/ruby/1.8/net/http.rb:2018:in`read\_new’  
/usr/lib/ruby/1.8/net/http.rb:1059:in `request' /usr/lib/ruby/gems/1.8/gems/mechanize-0.6.10/lib/mechanize.rb:514:in`fetch\_page’  
/usr/lib/ruby/gems/1.8/gems/mechanize-0.6.10/lib/mechanize.rb:185:in  
`get' /www/CLIENT/htdocs/include/script/nysif_scrape/lib/AuthedAgent.rb:18:in`initialize’  
/usr/lib/ruby/1.8/singleton.rb:95:in `new' /usr/lib/ruby/1.8/singleton.rb:95:in`instance’  
www/CLIENT/htdocs/include/script/nysif\_scrape/run.rb:20

For reference, here’s my AuthedAgent class definition. It’s a Singleton  
providing access to an Agent that has already logged into the Login form  
on the site, and $import is just an ActiveRecord row to allow for ajax-y  
progress bar updates on the web-side. It times out on either the ‘get’  
or the ‘submit’ method calls.

class AuthedAgent \< WWW::Mechanize  
include Singleton

def initialize  
super

```
   $import.status = 'connecting';
   $import.save!
   login_page = get($config[:urls][:login])
   raise SiteChangedException, "Login page is 404" if

```

(!login\_page.title.nil? && login\_page.title.include?(“The page cannot be  
found”))

```
   $import.status = 'authenticating';
   $import.save!
   form = login_page.form('frmLogin0')
   raise SiteChangedException, "Could not locate form 'frmLogin0' on

```

login page" if form.nil?  
form.fields.name(“LOGIN”).value = $config[:username]  
form.fields.name(“PWD”).value = $config[:password]  
post\_login = submit(form)  
raise SiteChangedException, “Username and password do not appear  
valid. Check credentials, or possible site change.” if !authenticated?  
end

def authenticated?  
cookie = cookies.find { |c| c.name == “LOGIN” }  
raise SiteChangedException, “‘LOGIN’ cookie is no longer being  
set” if cookie.nil?  
cookie.value.include? “TAG3=” + $config[:username]  
end  
end