# Too many open files on OpenBSD

**URL:** <https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100>\
**Category:** Ruby\
**Created:** [March 6, 2008, 1:33am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100 "2008-03-06T01:33:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 1:33am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/1 "2008-03-06T01:33:06Z")

</div>

Hi,

I am the maintainer of the TeXLive typesetting suite for OpenBSD. Each  
year I must parse a database in order to calculate subsets of the  
TeXLive system for packaging purposes.

The database format changed this year and I have thrown away my python  
script and had a go at it with ruby. I am quite new to ruby, but not to  
programming (previously C, C++, Java, BASIC… )

This is the error I am stuck with:

—8\<—  
./tlpsrcnode.rb:19:in `initialize': Too many open files - /home/tl/tl/Master/tlpkg/tlpsrc/wordcount.tlpsrc (Errno::EMFILE) from ./tlpsrcnode.rb:19:in`new’  
from ./tlpsrcnode.rb:19:in `parse' from ./tlpsrcnode.rb:30:in`parse’  
from ./tlpsrcnode.rb:20:in `each' from ./tlpsrcnode.rb:20:in`parse’  
from ./tlpsrcnode.rb:30:in `parse' from ./tlpsrcnode.rb:20:in`each’  
from ./tlpsrcnode.rb:20:in `parse' ... 3055 levels... from ./tlpsrcnode.rb:20:in`each’  
from ./tlpsrcnode.rb:20:in `parse' from ./roottlpsrcnode.rb:53:in`startParse’  
from ./rbmfsplit:54  
—8\<—

This appears to happen after 297 files are concurrently opened.

Immediately I dump myself into the staff login class and uncap the file  
descriptor count in login.conf. Log out then back in.

—8\<—  
puff% ulimit -a  
-t: cpu time (seconds) unlimited  
-f: file size (blocks) unlimited  
-d: data seg size (kbytes) 524288  
-s: stack size (kbytes) 4096  
-c: core file size (blocks) unlimited  
-m: resident set size (kbytes) 1747436  
-l: locked-in-memory size (kb) 582993  
-u: processes 128  
-n: file descriptors 1024  
—8\<—

Uncapped in this case appears to be 1024. Plenty.

The error still occurs after 297 files are open.

Is there a workaround for this or is ruby not suitable for this nature  
of processing?

Thanks

Edd

---

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 1:37am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/2 "2008-03-06T01:37:45Z")

</div>

Edd B. wrote:

> —8\<—  
> ./tlpsrcnode.rb:19:in `initialize': Too many open files - /home/tl/tl/Master/tlpkg/tlpsrc/wordcount.tlpsrc (Errno::EMFILE) from ./tlpsrcnode.rb:19:in`new’  
> from ./tlpsrcnode.rb:19:in `parse' from ./tlpsrcnode.rb:30:in`parse’  
> from ./tlpsrcnode.rb:20:in `each' from ./tlpsrcnode.rb:20:in`parse’  
> from ./tlpsrcnode.rb:30:in `parse' from ./tlpsrcnode.rb:20:in`each’  
> from ./tlpsrcnode.rb:20:in `parse' ... 3055 levels... from ./tlpsrcnode.rb:20:in`each’  
> from ./tlpsrcnode.rb:20:in `parse' from ./roottlpsrcnode.rb:53:in`startParse’  
> from ./rbmfsplit:54  
> —8\<—
> 
> This appears to happen after 297 files are concurrently opened.

Sorry this statement is inaccurate.

297 files have previously been opened and after each one has been parsed  
is immediately closed.

The call stack should be an indication of concurrently opened files? In  
this case 5 files. (Its recursive).

i suppose the src of the parse function helps (probably disobeying all  
coding conventions, sorry):  
—8\<—  
def parse()  
fileHandle = File.new @path  
for line in fileHandle do

```
  line =~ /^(.*?) (.*)/
  cmd = $~[1]
  arg = $~[2]

  if cmd =~ /^#.*/ then
    nil # comment
  elsif cmd == "depend" then
    newNode = TlpSrcNode.new @basePath + arg + ".tlpsrc", @rootNode
    newNode.parse # and so recursion starts
    @rootNode.incrementDeps
  elsif cmd == "runpattern" then
    @rootNode.addRunFile arg
  elsif cmd == "docpattern" then
    @rootNode.addDocFile arg
  elsif cmd == "srcpattern" then
    @rootNode.addSrcFile arg
  elsif @nothingCmds.detect { |i| i == cmd } then
    nil # OK
  elsif @todoCmds.detect { |i| i == cmd } then
    nil # XXX work out what these are
  else
    puts "*error: unknown cmd: '#{cmd}' arg='#{arg}'"
    exit 1
  end
end

fileHandle.close

```

end  
—8\<—

---

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 2:04am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/3 "2008-03-06T02:04:43Z")

</div>

Hello Edd, an OpenBSD user here too.

The error messages speaks of more than 3055 recursion levels. Not  
every level is a call to parse, but if you take out the #new and #each  
methods I’m sure there are a lot more than just 5 files opened at the  
same time.

Are there that many dependencies? I don’t think so. I haven’t seen  
those .tlpsrc files yet. I’ll take a look.

---

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 2:09am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/4 "2008-03-06T02:09:24Z")

</div>

Hi,

In message “Re: Too many open files on OpenBSD”  
on Thu, 6 Mar 2008 09:32:58 +0900, Edd B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
writes:

|This is the error I am stuck with:  
|  
|—8\<—  
|./tlpsrcnode.rb:19:in `initialize': Too many open files - |/home/tl/tl/Master/tlpkg/tlpsrc/wordcount.tlpsrc (Errno::EMFILE) | from ./tlpsrcnode.rb:19:in `new’  
| from ./tlpsrcnode.rb:19:in `parse' | from ./tlpsrcnode.rb:30:in `parse’  
| from ./tlpsrcnode.rb:20:in `each' | from ./tlpsrcnode.rb:20:in `parse’  
| from ./tlpsrcnode.rb:30:in `parse' | from ./tlpsrcnode.rb:20:in `each’  
| from ./tlpsrcnode.rb:20:in `parse' | ... 3055 levels... | from ./tlpsrcnode.rb:20:in `each’  
| from ./tlpsrcnode.rb:20:in `parse' | from ./roottlpsrcnode.rb:53:in `startParse’  
| from ./rbmfsplit:54  
|—8\<—

It seems you recursively call parse, which keep files opened. How  
about reading whole file content in a string by File.read, then call  
each\_line on the string? E.g.

def parse()  
File.read(@path).each\_line do |line|  
line =~ /^(._?) (._)/

```
  ....
end

```

end

---

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 8:54am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/5 "2008-03-06T08:54:07Z")

</div>

On Behalf Of Damjan R.:

# Yukihiro M. wrote:

# \> def parse()

# \> File.read(@path).each\_line do |line|

# \> line =~ /^(._?) (._)/

# \>

# \> …

# \> end

# \> end

# 

# Or even better:

# 

# def parse()

# lines = File.open(@path) {|f| f.read}

# lines.each\_line do |line|

# line =~ /^(._?) (._)/

# 

# …

# end

# end

# 

# With File.open file gets closed automaticly after block is processed.

File.read will also auto close, in fact it does not need a block 🙂  
and you can specify length and offset, too.

## botp@pc4all:/root$ qri IO.read --------------------------------------------------------------- IO::read IO.read(name, [length [, offset]] ) =\> string

```
 Opens the file, optionally seeks to the given offset, then returns
 length bytes (defaulting to the rest of the file). read ensures
 the file is closed before returning.
 ....

```

so basically,

```
File.read(@path)

```

and  
lines = File.open(@path) {|f| f.read}

do same thing, but i prefer the former in this case, obviously…

kind regards -botp

---

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 8:24am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/6 "2008-03-06T08:24:11Z")

</div>

Yukihiro M. wrote:

> Hi,
> 
> It seems you recursively call parse, which keep files opened. How  
> about reading whole file content in a string by File.read, then call  
> each\_line on the string? E.g.
> 
> def parse()  
> File.read(@path).each\_line do |line|  
> line =~ /^(._?) (._)/
> 
> ```
> ....
> end
> 
> ```
> 
> end

Or even better:

def parse()  
lines = File.open(@path) {|f| f.read}  
lines.each\_line do |line|  
line =~ /^(._?) (._)/

```
   ....
 end

```

end

With File.open file gets closed automaticly after block is processed.

by  
TheR

---

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 5:01pm UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/7 "2008-03-06T17:01:10Z")

</div>

Sorry for top posting, Gmail doesn’t support my browser.

Indeed, I’m waiting for that OPENBSD\_4\_3\_BASE tag 😉

What I meant is that you are opening a file in every call to parse. If  
you have many recursive calls to parse, you’ll end up with many files  
open.

As some others have suggested, you could transform recursion into  
iteration.

---

<div class="post-metadata">

**Author:** ![vext01](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/87869e/32.png) [@vext01](https://www.ruby-forum.com/u/vext01)\
**Post date:** [March 6, 2008, 10:43am UTC](https://www.ruby-forum.com/t/too-many-open-files-on-openbsd/131100/8 "2008-03-06T10:43:09Z")

</div>

Hello,

Thanks for all the replies so far. I will later try the suggestion to  
read the file into memory straight away then close the file.

Gerardo S. Gómez Garrido wrote:

> Hello Edd, an OpenBSD user here too.

Great 🙂 Ready for 4.3?

> The error messages speaks of more than 3055 recursion levels. Not  
> every level is a call to parse, but if you take out the #new and #each  
> methods I’m sure there are a lot more than just 5 files opened at the  
> same time.

Ah i see. So each call frame is a “file” in ruby? I probably  
misunderstood.

> Are there that many dependencies? I don’t think so. I haven’t seen  
> those .tlpsrc files yet. I’ll take a look.

[http://tug.org/svn/texlive/trunk/Master/tlpkg/tlpsrc/](http://tug.org/svn/texlive/trunk/Master/tlpkg/tlpsrc/)

More than 1000 last time I checked.

This means I will be soring a lot of strings. Is the GC in ruby  
effecient, or should I be encouraging it Java style. System.gc() 😛

Thanks

Edd