Forum: Ruby Errno::ENOMEM reading a device in Ruby, not in Java though

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Roger P. (Guest)
on 2009-01-28 22:29
Question on how to avoid an Errno::ENOMEM

Currently if I open a tape device from my machine [/dev/st2]
in ruby reading the second file from it results in:

read.rb:2:in `read': Cannot allocate memory - /dev/st2 (Errno::ENOMEM)
        from read.rb:2

Java to do the same works, however.

the "cat" command [i.e. cat </dev/st2] also fails with "Not enough
memory"

Here is an strace of the two:

ruby:

open("/dev/st2", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
(Invalid argument)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b5b53902000
read(3, 0x2b5b53902000, 4096)           = -1 ENOMEM (Cannot allocate
memory)


java:

[pid 25476] open("/dev/nst2", O_RDONLY) = 4
[pid 25476] fstat(4, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 130),
...}) = 0
[pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
[pid 25476] read(4, "<familysearch-format>\n<type name"..., 65536) = 116

Anybody know if the difference between mmap and mprotect might be
causing this?
Thanks!
-=r
Yukihiro M. (Guest)
on 2009-01-29 07:05
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
though"
    on Thu, 29 Jan 2009 05:27:13 +0900, Roger P.
<removed_email_address@domain.invalid> writes:

|ruby:
|
|open("/dev/st2", O_RDONLY)              = 3
|fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
|fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
|ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
|(Invalid argument)
|mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
|= 0x2b5b53902000
|read(3, 0x2b5b53902000, 4096)           = -1 ENOMEM (Cannot allocate
|memory)

Strange.  ENOMEM is not listed among errors that read(2) can raise.

              matz.
Heesob P. (Guest)
on 2009-01-29 07:50
(Received via mailing list)
2009/1/29 Roger P. <removed_email_address@domain.invalid>:
> the "cat" command [i.e. cat </dev/st2] also fails with "Not enough
> (Invalid argument)
> ...}) = 0
> [pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
> [pid 25476] read(4, "<familysearch-format>\n<type name"..., 65536) = 116
>
> Anybody know if the difference between mmap and mprotect might be
> causing this?
The another difference between ruby and java is read buffer size.
Ruby's buffer size is 4096 and java's buffer size is 65536.

The man page of st says:

RETURN VALUE

       ENOMEM        The byte count in read() is smaller than the next
physi-
                     cal block on the tape. (Before 2.2.18 and
2.4.0-test6 the
                     extra bytes have been silently ignored.)

Refer to http://fts.ifac.cnr.it/cgi-bin/dwww?type=runman&lo...

Regards,

Park H.
Yukihiro M. (Guest)
on 2009-01-29 19:54
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
though"
    on Thu, 29 Jan 2009 14:47:11 +0900, Heesob P. 
<removed_email_address@domain.invalid>
writes:

|The another difference between ruby and java is read buffer size.
|Ruby's buffer size is 4096 and java's buffer size is 65536.
|
|The man page of st says:
|
|RETURN VALUE
|
|       ENOMEM        The byte count in read() is smaller than the next  physi-
|                     cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
|                     extra bytes have been silently ignored.)

I didn't know that.  In that case, you have to pre-allocate reading
buffer (string) and specify it to read method.

              matz.
Roger P. (Guest)
on 2009-01-29 22:43
ge of st says:
> |
> |RETURN VALUE
> |
> |       ENOMEM        The byte count in read() is smaller than the next  physi-
> |                     cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
> |                     extra bytes have been silently ignored.)
>
> I didn't know that.  In that case, you have to pre-allocate reading
> buffer (string) and specify it to read method.

Interesting.
Perhaps somebody with brighter eyes can help me out with this?
[how to?]

a = File.open '/dev/nst2'
>> b = 'a'*129000
>> a.read b
TypeError: can't convert String into Integer
>> a.read 65000
Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Also thinking out loud I wonder why java has 64kB blocks and ruby 4k? is
64 k any other benefit?

Thanks!
-=r
Yukihiro M. (Guest)
on 2009-01-29 23:09
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
thou"
    on Fri, 30 Jan 2009 05:40:45 +0900, Roger P.
<removed_email_address@domain.invalid> writes:

|> I didn't know that.  In that case, you have to pre-allocate reading
|> buffer (string) and specify it to read method.
|
|Interesting.
|Perhaps somebody with brighter eyes can help me out with this?
|[how to?]
|
|>> b = 'a'*129000
|>> a.read b

You have to specify the optional second argument:

---------------------------------------------------------------- IO#read
     ios.read([length [, buffer]])    => string, buffer, or nil
------------------------------------------------------------------------
     Reads at most _length_ bytes from the I/O stream, or to the end of
     file if _length_ is omitted or is +nil+. _length_ must be a
     non-negative integer or nil. If the optional _buffer_ argument is
     present, it must reference a String, which will receive the data.

b = " "*65000
a.read(b.size, b)

|>> a.read 65000
|Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Hmm, since read with number pre-allocate specified sized buffer, it
should work.  Perhaps you have to use sysread instead of mere read,
since read method retries until exact number of bytes specified read.

              matz.
Roger P. (Guest)
on 2009-06-10 02:07
> b = " "*65000
> a.read(b.size, b)
>
> |>> a.read 65000
> |Errno::ENOMEM: Cannot allocate memory - /dev/nst2
>
> Hmm, since read with number pre-allocate specified sized buffer, it
> should work.  Perhaps you have to use sysread instead of mere read,
> since read method retries until exact number of bytes specified read.
>
>               matz.

Interesting.
with 1.9, it will read all 64K at once (not with 1.8 though--it reads 4K
blocks until it reached 64K).

sysread did work with 1.8, though

for some reason I had to use exactly 64K block size.

 file.sysread(64*1024)

guess something to remember if you have to read from a tape drive.

I did notice a few of these when using 1.9

ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd172a8) = -1 ENOTTY
(Inappropriate ioctl for device)

dunno if that's cause for concern or not.
Cheers!
-=r
Roger P. (Guest)
on 2009-11-16 19:40
> Interesting.
> with 1.9, it will read all 64K at once (not with 1.8 though--it reads 4K
> blocks until it reached 64K).
>
> sysread did work with 1.8, though

As a followup gotcha (in case anybody runs into this).

irb(main):001:0> a = File.open('/dev/nst2', 'rb')
=> #<File:/dev/nst2>
irb(main):002:0> a.eof?
irb(main):005:0> a.sysread 64*1024
IOError: sysread for buffered IO
        from (irb):5:in `sysread'
        from (irb):5

meant "when you call eof? it actually converts the file descriptor
internally into buffered mode, and read a byte from it to see if it
reads zero bytes (implying EOF), thus you cannot call sysread after a
buffered read call like eof?"

-r
Yukihiro M. (Guest)
on 2009-11-16 20:16
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
thou"
    on Tue, 17 Nov 2009 02:40:50 +0900, Roger P.
<removed_email_address@domain.invalid> writes:

|As a followup gotcha (in case anybody runs into this).
|
|irb(main):001:0> a = File.open('/dev/nst2', 'rb')
|=> #<File:/dev/nst2>
|irb(main):002:0> a.eof?
|irb(main):005:0> a.sysread 64*1024
|IOError: sysread for buffered IO
|        from (irb):5:in `sysread'
|        from (irb):5
|
|meant "when you call eof? it actually converts the file descriptor
|internally into buffered mode, and read a byte from it to see if it
|reads zero bytes (implying EOF), thus you cannot call sysread after a
|buffered read call like eof?"

rewind it first to clear buffering.

              matz.
Roger P. (Guest)
on 2009-11-16 22:04
> |irb(main):005:0> a.sysread 64*1024
> |IOError: sysread for buffered IO
...
> rewind it first to clear buffering.

Interesting.  It seems to indeed allow for it.

I was unable to actually test it on a tape devicesince it appears that
eof? calls a read of size 8192:

read(3, 0x6c9ba90, 8192)                = -1 ENOMEM (Cannot allocate
memory)

which would be smaller than what I would need [which is ok I can live
without eof].

Also as a note, it ends up that, with tape drives the trick is to read
from a device into a buffer that's at least as large as the largest
"block" of data written onto the tape, or else you'll get ENOMEM.

ex:
write 1000 bytes
File.open('tape_device like /dev/nst2', 'w') do |f| f.syswrite 'a*1000;
end
now read
File.open('tape_device', 'r') do |f| f.sysread 100; end # fails
File.open('tape_device', 'r') do |f| f.sysread 1000; end # succeeds

Nothing to do with ruby itself, but thought I'd mention it for
followers.
-r
This topic is locked and can not be replied to.