Forum: Ruby Errno::ENOMEM reading a device in Ruby, not in Java though

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2009-01-28 21:29
Question on how to avoid an Errno::ENOMEM

Currently if I open a tape device from my machine [/dev/st2]
in ruby reading the second file from it results in:

read.rb:2:in `read': Cannot allocate memory - /dev/st2 (Errno::ENOMEM)
        from read.rb:2

Java to do the same works, however.

the "cat" command [i.e. cat </dev/st2] also fails with "Not enough
memory"

Here is an strace of the two:

ruby:

open("/dev/st2", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
(Invalid argument)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b5b53902000
read(3, 0x2b5b53902000, 4096)           = -1 ENOMEM (Cannot allocate
memory)


java:

[pid 25476] open("/dev/nst2", O_RDONLY) = 4
[pid 25476] fstat(4, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 130),
...}) = 0
[pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
[pid 25476] read(4, "<familysearch-format>\n<type name"..., 65536) = 116

Anybody know if the difference between mmap and mprotect might be
causing this?
Thanks!
-=r
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2009-01-29 06:05
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
though"
    on Thu, 29 Jan 2009 05:27:13 +0900, Roger Pack
<rogerpack2005@gmail.com> writes:

|ruby:
|
|open("/dev/st2", O_RDONLY)              = 3
|fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
|fstat(3, {st_mode=S_IFCHR|0660, st_rdev=makedev(9, 2), ...}) = 0
|ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff5721d900) = -1 EINVAL
|(Invalid argument)
|mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
|= 0x2b5b53902000
|read(3, 0x2b5b53902000, 4096)           = -1 ENOMEM (Cannot allocate
|memory)

Strange.  ENOMEM is not listed among errors that read(2) can raise.

              matz.
666b4e17b4bb0e2d999037a25f65a7cb?d=identicon&s=25 Heesob Park (phasis)
on 2009-01-29 06:50
(Received via mailing list)
2009/1/29 Roger Pack <rogerpack2005@gmail.com>:
> the "cat" command [i.e. cat </dev/st2] also fails with "Not enough
> (Invalid argument)
> ...}) = 0
> [pid 25476] mprotect(0x2aab3409b000, 65536, PROT_READ|PROT_WRITE) = 0
> [pid 25476] read(4, "<familysearch-format>\n<type name"..., 65536) = 116
>
> Anybody know if the difference between mmap and mprotect might be
> causing this?
The another difference between ruby and java is read buffer size.
Ruby's buffer size is 4096 and java's buffer size is 65536.

The man page of st says:

RETURN VALUE

       ENOMEM        The byte count in read() is smaller than the next
physi-
                     cal block on the tape. (Before 2.2.18 and
2.4.0-test6 the
                     extra bytes have been silently ignored.)

Refer to http://fts.ifac.cnr.it/cgi-bin/dwww?type=runman&lo...

Regards,

Park Heesob
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2009-01-29 18:54
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
though"
    on Thu, 29 Jan 2009 14:47:11 +0900, Heesob Park <phasis@gmail.com>
writes:

|The another difference between ruby and java is read buffer size.
|Ruby's buffer size is 4096 and java's buffer size is 65536.
|
|The man page of st says:
|
|RETURN VALUE
|
|       ENOMEM        The byte count in read() is smaller than the next  physi-
|                     cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
|                     extra bytes have been silently ignored.)

I didn't know that.  In that case, you have to pre-allocate reading
buffer (string) and specify it to read method.

              matz.
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2009-01-29 21:43
ge of st says:
> |
> |RETURN VALUE
> |
> |       ENOMEM        The byte count in read() is smaller than the next  physi-
> |                     cal block on the tape. (Before 2.2.18 and 2.4.0-test6 the
> |                     extra bytes have been silently ignored.)
>
> I didn't know that.  In that case, you have to pre-allocate reading
> buffer (string) and specify it to read method.

Interesting.
Perhaps somebody with brighter eyes can help me out with this?
[how to?]

a = File.open '/dev/nst2'
>> b = 'a'*129000
>> a.read b
TypeError: can't convert String into Integer
>> a.read 65000
Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Also thinking out loud I wonder why java has 64kB blocks and ruby 4k? is
64 k any other benefit?

Thanks!
-=r
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2009-01-29 22:09
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
thou"
    on Fri, 30 Jan 2009 05:40:45 +0900, Roger Pack
<rogerpack2005@gmail.com> writes:

|> I didn't know that.  In that case, you have to pre-allocate reading
|> buffer (string) and specify it to read method.
|
|Interesting.
|Perhaps somebody with brighter eyes can help me out with this?
|[how to?]
|
|>> b = 'a'*129000
|>> a.read b

You have to specify the optional second argument:

---------------------------------------------------------------- IO#read
     ios.read([length [, buffer]])    => string, buffer, or nil
------------------------------------------------------------------------
     Reads at most _length_ bytes from the I/O stream, or to the end of
     file if _length_ is omitted or is +nil+. _length_ must be a
     non-negative integer or nil. If the optional _buffer_ argument is
     present, it must reference a String, which will receive the data.

b = " "*65000
a.read(b.size, b)

|>> a.read 65000
|Errno::ENOMEM: Cannot allocate memory - /dev/nst2

Hmm, since read with number pre-allocate specified sized buffer, it
should work.  Perhaps you have to use sysread instead of mere read,
since read method retries until exact number of bytes specified read.

              matz.
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2009-06-10 00:07
> b = " "*65000
> a.read(b.size, b)
>
> |>> a.read 65000
> |Errno::ENOMEM: Cannot allocate memory - /dev/nst2
>
> Hmm, since read with number pre-allocate specified sized buffer, it
> should work.  Perhaps you have to use sysread instead of mere read,
> since read method retries until exact number of bytes specified read.
>
>               matz.

Interesting.
with 1.9, it will read all 64K at once (not with 1.8 though--it reads 4K
blocks until it reached 64K).

sysread did work with 1.8, though

for some reason I had to use exactly 64K block size.

 file.sysread(64*1024)

guess something to remember if you have to read from a tape drive.

I did notice a few of these when using 1.9

ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd172a8) = -1 ENOTTY
(Inappropriate ioctl for device)

dunno if that's cause for concern or not.
Cheers!
-=r
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2009-11-16 18:40
> Interesting.
> with 1.9, it will read all 64K at once (not with 1.8 though--it reads 4K
> blocks until it reached 64K).
>
> sysread did work with 1.8, though

As a followup gotcha (in case anybody runs into this).

irb(main):001:0> a = File.open('/dev/nst2', 'rb')
=> #<File:/dev/nst2>
irb(main):002:0> a.eof?
irb(main):005:0> a.sysread 64*1024
IOError: sysread for buffered IO
        from (irb):5:in `sysread'
        from (irb):5

meant "when you call eof? it actually converts the file descriptor
internally into buffered mode, and read a byte from it to see if it
reads zero bytes (implying EOF), thus you cannot call sysread after a
buffered read call like eof?"

-r
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2009-11-16 19:16
(Received via mailing list)
Hi,

In message "Re: Errno::ENOMEM reading a device in Ruby, not in Java
thou"
    on Tue, 17 Nov 2009 02:40:50 +0900, Roger Pack
<rogerpack2005@gmail.com> writes:

|As a followup gotcha (in case anybody runs into this).
|
|irb(main):001:0> a = File.open('/dev/nst2', 'rb')
|=> #<File:/dev/nst2>
|irb(main):002:0> a.eof?
|irb(main):005:0> a.sysread 64*1024
|IOError: sysread for buffered IO
|        from (irb):5:in `sysread'
|        from (irb):5
|
|meant "when you call eof? it actually converts the file descriptor
|internally into buffered mode, and read a byte from it to see if it
|reads zero bytes (implying EOF), thus you cannot call sysread after a
|buffered read call like eof?"

rewind it first to clear buffering.

              matz.
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2009-11-16 21:04
> |irb(main):005:0> a.sysread 64*1024
> |IOError: sysread for buffered IO
...
> rewind it first to clear buffering.

Interesting.  It seems to indeed allow for it.

I was unable to actually test it on a tape devicesince it appears that
eof? calls a read of size 8192:

read(3, 0x6c9ba90, 8192)                = -1 ENOMEM (Cannot allocate
memory)

which would be smaller than what I would need [which is ok I can live
without eof].

Also as a note, it ends up that, with tape drives the trick is to read
from a device into a buffer that's at least as large as the largest
"block" of data written onto the tape, or else you'll get ENOMEM.

ex:
write 1000 bytes
File.open('tape_device like /dev/nst2', 'w') do |f| f.syswrite 'a*1000;
end
now read
File.open('tape_device', 'r') do |f| f.sysread 100; end # fails
File.open('tape_device', 'r') do |f| f.sysread 1000; end # succeeds

Nothing to do with ruby itself, but thought I'd mention it for
followers.
-r
This topic is locked and can not be replied to.