Errno::ENOMEM reading a device in Ruby, not in Java though


#1

Question on how to avoid an Errno::ENOMEM

Currently if I open a tape device from my machine [/dev/st2]
in ruby reading the second file from it results in:

read.rb:2:in `read’: Cannot allocate memory - /dev/st2 (Errno::ENOMEM)
from read.rb:2

Java to do the same works, however.

the “cat” command [i.e. cat </dev/st2] also fails with “Not enough
memory”

Here is an strace of the two:

ruby:

open("/dev/st2", O_RDONLY) = 3
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), …}) = 0
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), …}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
(Invalid argument)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b5b53902000
read(3, 0x2b5b53902000, 4096) = -1 ENOMEM (Cannot allocate
memory)

java:

[pid 25476] open("/dev/nst2", O_RDONLY) = 4
[pid 25476] fstat(4, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 130),
…}) = 0
[pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
[pid 25476] read(4, “\n<type name”…, 65536) = 116

Anybody know if the difference between mmap and mprotect might be
causing this?
Thanks!
-=r


#2

Hi,

In message “Re: Errno::ENOMEM reading a device in Ruby, not in Java
though”
on Thu, 29 Jan 2009 05:27:13 +0900, Roger P.
removed_email_address@domain.invalid writes:

|ruby:
|
|open("/dev/st2", O_RDONLY) = 3
|fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), …}) = 0
|fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), …}) = 0
|ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
|(Invalid argument)
|mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
|= 0x2b5b53902000
|read(3, 0x2b5b53902000, 4096) = -1 ENOMEM (Cannot allocate
|memory)

Strange. ENOMEM is not listed among errors that read(2) can raise.

          matz.

#3

2009/1/29 Roger P. removed_email_address@domain.invalid:

the “cat” command [i.e. cat </dev/st2] also fails with "Not enough
(Invalid argument)
…}) = 0
[pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
[pid 25476] read(4, “\n<type name”…, 65536) = 116

Anybody know if the difference between mmap and mprotect might be
causing this?
The another difference between ruby and java is read buffer size.
Ruby’s buffer size is 4096 and java’s buffer size is 65536.

The man page of st says:

RETURN VALUE

   ENOMEM        The byte count in read() is smaller than the next 

physi-
cal block on the tape. (Before 2.2.18 and
2.4.0-test6 the
extra bytes have been silently ignored.)

Refer to http://fts.ifac.cnr.it/cgi-bin/dwww?type=runman&location=st/4

Regards,

Park H.


#4

Hi,

In message “Re: Errno::ENOMEM reading a device in Ruby, not in Java
though”
on Thu, 29 Jan 2009 14:47:11 +0900, Heesob P. removed_email_address@domain.invalid
writes:

|The another difference between ruby and java is read buffer size.
|Ruby’s buffer size is 4096 and java’s buffer size is 65536.
|
|The man page of st says:
|
|RETURN VALUE
|
| ENOMEM The byte count in read() is smaller than the next physi-
| cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
| extra bytes have been silently ignored.)

I didn’t know that. In that case, you have to pre-allocate reading
buffer (string) and specify it to read method.

          matz.

#5

ge of st says:

|
|RETURN VALUE
|
| ENOMEM The byte count in read() is smaller than the next physi-
| cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
| extra bytes have been silently ignored.)

I didn’t know that. In that case, you have to pre-allocate reading
buffer (string) and specify it to read method.

Interesting.
Perhaps somebody with brighter eyes can help me out with this?
[how to?]

a = File.open ‘/dev/nst2’

b = ‘a’*129000
a.read b
TypeError: can’t convert String into Integer

a.read 65000
Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Also thinking out loud I wonder why java has 64kB blocks and ruby 4k? is
64 k any other benefit?

Thanks!
-=r


#6

Hi,

In message “Re: Errno::ENOMEM reading a device in Ruby, not in Java
thou”
on Fri, 30 Jan 2009 05:40:45 +0900, Roger P.
removed_email_address@domain.invalid writes:

|> I didn’t know that. In that case, you have to pre-allocate reading
|> buffer (string) and specify it to read method.
|
|Interesting.
|Perhaps somebody with brighter eyes can help me out with this?
|[how to?]
|
|>> b = ‘a’*129000
|>> a.read b

You have to specify the optional second argument:

---------------------------------------------------------------- IO#read
ios.read([length [, buffer]]) => string, buffer, or nil

 Reads at most _length_ bytes from the I/O stream, or to the end of
 file if _length_ is omitted or is +nil+. _length_ must be a
 non-negative integer or nil. If the optional _buffer_ argument is
 present, it must reference a String, which will receive the data.

b = " "*65000
a.read(b.size, b)

|>> a.read 65000
|Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Hmm, since read with number pre-allocate specified sized buffer, it
should work. Perhaps you have to use sysread instead of mere read,
since read method retries until exact number of bytes specified read.

          matz.

#7

Interesting.
with 1.9, it will read all 64K at once (not with 1.8 though–it reads 4K
blocks until it reached 64K).

sysread did work with 1.8, though

As a followup gotcha (in case anybody runs into this).

irb(main):001:0> a = File.open(’/dev/nst2’, ‘rb’)
=> #<File:/dev/nst2>
irb(main):002:0> a.eof?
irb(main):005:0> a.sysread 64*1024
IOError: sysread for buffered IO
from (irb):5:in `sysread’
from (irb):5

meant “when you call eof? it actually converts the file descriptor
internally into buffered mode, and read a byte from it to see if it
reads zero bytes (implying EOF), thus you cannot call sysread after a
buffered read call like eof?”

-r


#8

b = " "*65000
a.read(b.size, b)

|>> a.read 65000
|Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Hmm, since read with number pre-allocate specified sized buffer, it
should work. Perhaps you have to use sysread instead of mere read,
since read method retries until exact number of bytes specified read.

          matz.

Interesting.
with 1.9, it will read all 64K at once (not with 1.8 though–it reads 4K
blocks until it reached 64K).

sysread did work with 1.8, though

for some reason I had to use exactly 64K block size.

file.sysread(64*1024)

guess something to remember if you have to read from a tape drive.

I did notice a few of these when using 1.9

ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd172a8) = -1 ENOTTY
(Inappropriate ioctl for device)

dunno if that’s cause for concern or not.
Cheers!
-=r


#9

Hi,

In message “Re: Errno::ENOMEM reading a device in Ruby, not in Java
thou”
on Tue, 17 Nov 2009 02:40:50 +0900, Roger P.
removed_email_address@domain.invalid writes:

|As a followup gotcha (in case anybody runs into this).
|
|irb(main):001:0> a = File.open(’/dev/nst2’, ‘rb’)
|=> #<File:/dev/nst2>
|irb(main):002:0> a.eof?
|irb(main):005:0> a.sysread 64*1024
|IOError: sysread for buffered IO
| from (irb):5:in `sysread’
| from (irb):5
|
|meant “when you call eof? it actually converts the file descriptor
|internally into buffered mode, and read a byte from it to see if it
|reads zero bytes (implying EOF), thus you cannot call sysread after a
|buffered read call like eof?”

rewind it first to clear buffering.

          matz.

#10

|irb(main):005:0> a.sysread 64*1024
|IOError: sysread for buffered IO

rewind it first to clear buffering.

Interesting. It seems to indeed allow for it.

I was unable to actually test it on a tape devicesince it appears that
eof? calls a read of size 8192:

read(3, 0x6c9ba90, 8192) = -1 ENOMEM (Cannot allocate
memory)

which would be smaller than what I would need [which is ok I can live
without eof].

Also as a note, it ends up that, with tape drives the trick is to read
from a device into a buffer that’s at least as large as the largest
“block” of data written onto the tape, or else you’ll get ENOMEM.

ex:
write 1000 bytes
File.open(‘tape_device like /dev/nst2’, ‘w’) do |f| f.syswrite 'a*1000;
end
now read
File.open(‘tape_device’, ‘r’) do |f| f.sysread 100; end # fails
File.open(‘tape_device’, ‘r’) do |f| f.sysread 1000; end # succeeds

Nothing to do with ruby itself, but thought I’d mention it for
followers.
-r