Kenneth K. has brought up a HighLine issue and I’m trying to look
into it. Oddly, it seems to happen when interacting with the
Net::HTTP library. I’ve narrowed it done to the following example on
my box:
$ cat stdin_closed_issue.rb
require ‘net/http’
require ‘io/wait’
Net::HTTP.start(‘www.ruby-lang.org’, 80) do |http|
body = http.get(‘/en/license.txt’).body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true
Can anyone explain why $stdin is closed after the page read?
James Edward G. II
On Jan 27, 2007, at 17:13, James Edward G. II wrote:
body = http.get(‘/en/license.txt’).body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true
Can anyone explain why $stdin is closed after the page read?
$stdin isn’t closed, its at the end of file. Use #closed? to test if
an IO has been closed or not.
$ cat test.rb
require ‘net/http’
require ‘io/wait’
Net::HTTP.start(‘www.ruby-lang.org’, 80) do |http|
body = http.get(‘/en/license.txt’).body
end
p $stdin.closed?
p $stdin.eof?
$ ruby test.rb
false # hit ^D to close $stdin on the sending side.
true
$
$ echo hi | ruby test.rb
false
false
–
Eric H. - [email protected] - http://blog.segment7.net
I LIT YOUR GEM ON FIRE!
On Jan 27, 2007, at 18:38, James Edward G. II wrote:
On Jan 27, 2007, at 7:39 PM, Eric H. wrote:
$stdin isn’t closed, its at the end of file. Use #closed? to test
if an IO has been closed or not.
Right, good point.
However, when I call eof?() before the Net::HTTP call it behaves
differently (stalls and prints false). Why does it not behave the
same after that page read?
$ cat test.rb
puts “closed? %p” % $stdin.closed?
puts “eof? %p” % $stdin.eof?
require ‘net/http’
require ‘io/wait’
Net::HTTP.start(‘localhost’, 80) do |http|
body = http.get(‘/’).body
end
puts “closed? %p” % $stdin.closed?
puts “eof? %p” % $stdin.eof?
$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$
For the first #eof? no data written, so Ruby waits until something’s
been flushed. I typed some text and hit return to flush the
terminal’s stdout (Ruby’s $stdin). At the second #eof? no input on
$stdin has been consumed, so Ruby doesn’t need to check for #eof?
again by waiting.
$ cat test.rb
puts “eof? %p” % $stdin.eof?
gets
puts “eof? %p” % $stdin.eof?
$ ruby test.rb
aoeu
eof? false
aoeu
eof? false
–
Eric H. - [email protected] - http://blog.segment7.net
I LIT YOUR GEM ON FIRE!
On Jan 27, 2007, at 7:39 PM, Eric H. wrote:
$stdin isn’t closed, its at the end of file. Use #closed? to test
if an IO has been closed or not.
Right, good point.
However, when I call eof?() before the Net::HTTP call it behaves
differently (stalls and prints false). Why does it not behave the
same after that page read?
James Edward G. II
On Jan 27, 2007, at 8:48 PM, Eric H. wrote:
same after that page read?
end
$
For the first #eof? no data written, so Ruby waits until
something’s been flushed.
OK, that makes sense. However, why does moving that eof?() check
below a Net::HTTP page fetch change this? In that case, why does
Ruby not need for something to be flushed in that case and why is eof?
() then +true+?
James Edward G. II
On Jan 27, 2007, at 22:01, James Edward G. II wrote:
differently (stalls and prints false). Why does it not behave
For the first #eof? no data written, so Ruby waits until
something’s been flushed.
OK, that makes sense. However, why does moving that eof?() check
below a Net::HTTP page fetch change this?
I don’t see this behavior. With your original example I have to hit
^D to get a prompt back.
$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]
In that case, why does Ruby not need for something to be flushed
Because #eof? reads a character if feof(3) is not true and there’s no
data pending:
rb_io_eof(io)
VALUE io;
{
OpenFile *fptr;
int ch;
GetOpenFile(io, fptr);
rb_io_check_readable(fptr);
if (feof(fptr->f)) return Qtrue;
if (READ_DATA_PENDING(fptr->f)) return Qfalse;
READ_CHECK(fptr->f);
clearerr(fptr->f);
TRAP_BEG;
ch = getc(fptr->f);
^^^^^^^^^^^^^^^^^^^
Analysis of sampling pid 15640 every 10.000000 milliseconds
Call graph:
[…]
300 rb_io_eof
300 getc
[…]
in that case and why is eof?() then +true+?
#eof? will only be true when there’s no more data to read.
–
Eric H. - [email protected] - http://blog.segment7.net
I LIT YOUR GEM ON FIRE!
On Jan 29, 2007, at 12:20 PM, James Edward G. II wrote:
I still don’t get prompted for input and it still tells me $stdin
is closed:
I’m sorry, I meant it still claims $stdin is at eof?().
James Edward G. II
On Jan 29, 2007, at 3:54 AM, Eric H. wrote:
On Jan 27, 2007, at 22:01, James Edward G. II wrote:
OK, that makes sense. However, why does moving that eof?() check
below a Net::HTTP page fetch change this?
I don’t see this behavior. With your original example I have to
hit ^D to get a prompt back.
$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]
I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don’t get prompted for input and it still tells me $stdin
is closed:
Firefly:~/Desktop$ ruby -v
ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.1]
Firefly:~/Desktop$ cat stdin_eof_issue.rb
require ‘net/http’
require ‘io/wait’
Net::HTTP.start(‘www.ruby-lang.org’, 80) do |http|
body = http.get(‘/en/license.txt’).body
end
p $stdin.eof?
Firefly:~/Desktop$ ruby stdin_eof_issue.rb
true
James Edward G. II
On Jan 29, 2007, at 10:32, Jacob F. wrote:
On 1/29/07, James Edward G. II [email protected] wrote:
I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don’t get prompted for input and it still tells me $stdin
is [eof]:
I can confirm on my machine:
Then your getc may do something different than mine (BSD-ish).
On Jan 30, 2007, at 12:38 PM, Eric H. wrote:
On Jan 29, 2007, at 10:32, Jacob F. wrote:
On 1/29/07, James Edward G. II [email protected] wrote:
I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don’t get prompted for input and it still tells me
$stdin
is [eof]:
I can confirm on my machine:
Then your getc may do something different than mine (BSD-ish).
In other words, you don’t feel this is a Ruby issue, right?
James Edward G. II
On 1/29/07, James Edward G. II [email protected] wrote:
I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don’t get prompted for input and it still tells me $stdin
is [eof]:
I can confirm on my machine:
lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ cat test.rb
require ‘net/http’
require ‘io/wait’
Net::HTTP.start(‘www.ruby-lang.org’, 80) do |http|
body = http.get(‘/en/license.txt’).body
end
p $stdin.eof?
lukfugl@hephaestus:~$ ruby test.rb
true
One note: the first time I ran it, I thought it was waiting for input
too, but it was really just a few second delay while fetching
http://www.ruby-lang.org/en/license.txt.
Jacob F.
On Jan 27, 2007, at 8:13 PM, James Edward G. II wrote:
Kenneth K. has brought up a HighLine issue and I’m trying to
look into it. Oddly, it seems to happen when interacting with the
Net::HTTP library. I’ve narrowed it done to the following example
on my box:
Here is a simpler example that illustrates the issue:
Thread.new { }
p $stdin.eof?
As Eric H. pointed out, EOF on a terminal device is detected by
trying to read from
the device. It appears that prior to the creation of a thread, a
read on a terminal
device will simply block waiting for some activity on the device;
either actual data
or indication of an eof condition (someone typing control-d).
After a thread has been created, Ruby’s green thread implementation
kicks in. This
means that the read caused by IO#eof? on the terminal device is not
allowed to block.
Instead it fails as an interrupted system call, which is interpreted
by IO#eof? as an
end of file condition. This is true even if the main thread is the
only running thread.
James’ original example used Net::HTTP, which uses Timeout, which
creates a thread to
gain control of code that may block (i.e., an HTTP connection to a
remote host).
I don’t know enough about Ruby’s thread implementation to know if
this is a bug or
a feature. My hunch is that IO#eof? needs to be fixed so that an
interrupted system
call doesn’t get interpreted as an end of file condition. Another
change would be to
turn off the software interrupt that is used to multiplex IO activity
by threads when
only the main thread is running.
Gary W.
On 1/30/07, [email protected] [email protected] wrote:
ruby 1.8.4 (2005-12-01) [i686-linux]
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
so looks to be a bsd/mac issue?
On my box:
lukfugl@hephaestus:~$ cat a.rb
Thread.new { }
p $stdin.eof?
lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
true
lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ uname -srm
Linux 2.6.17-2-686 i686
So it’s present for me with ruby 1.8.5 on linux 2.6.17. I did just
notice that my ruby is compiled i486-linux – that may have
something to do with it, but I don’t think so.
Jacob F.
On Jan 30, 2007, at 5:35 PM, [email protected] wrote:
p $stdin.eof?
so looks to be a bsd/mac issue?
My post was based on ruby 1.8.5 on Darwin 8.8.0 Power Macintosh
The anomaly does not occur with ruby-1.9 on the same OS/hardware.
Gary W.
On Wed, 31 Jan 2007 [email protected] wrote:
p $stdin.eof?
is interpreted by IO#eof? as an end of file condition. This is true even if
multiplex IO activity by threads when only the main thread is running.
Gary W.
sounds quite reasonable. however, on my box:
harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?
harp:~ > ruby a.rb # and then type ctrl-d
true
harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
so looks to be a bsd/mac issue?
-a
On Wed, 31 Jan 2007, Jacob F. wrote:
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ uname -srm
Linux 2.6.17-2-686 i686
So it’s present for me with ruby 1.8.5 on linux 2.6.17. I did just notice
that my ruby is compiled i486-linux – that may have something to do with
it, but I don’t think so.
strange:
[nrt@anchor ~]$ ruby -v -e’ Thread.new{} and p STDIN.eof? ’ #
ctrl-d
ruby 1.8.5 (2006-12-11 patchlevel 5000) [i686-linux]
true
[nrt@anchor ~]$ uname -srm
Linux 2.6.9-42.0.3.ELsmp i686
and:
harp:~ > ruby19 -v -e’ Thread.new{} and p STDIN.eof? ’ # ctrl-d
ruby 1.9.0 (2006-07-26) [i686-linux]
true
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
this could get tricky to debug! none of my linux versions or ruby
versions
show this…
just to be clear (to everyone): have you all compiled ruby from source?
all
my installs are from source using nothing but ‘–prefix’ given to
./configure.
if anyone has a package (cough) manager or vendor (cough cough) install
i’d
reccomend compiling from source without any options and testing again.
kind regards.
-a
On Jan 30, 2007, at 13:31, [email protected] wrote:
As Eric H. pointed out, EOF on a terminal device is detected by
trying to read from the device. It appears that prior to the
creation of a thread, a read on a terminal device will simply block
waiting for some activity on the device; either actual data or
indication of an eof condition (someone typing control-d).
After a thread has been created, Ruby’s green thread
implementation kicks in. This means that the read caused by
IO#eof? on the terminal device is not allowed to block.
If there are other threads and the IO isn’t readable, ruby will run
those until they complete. Your thread immediately stopped.
Since the thread in your example has completed there’s no threads to
switch to, so the main thread blocks waiting for a character (on a
BSD-ish libc). If there was a thread running, the main thread would
remain blocked (as if it were blocked on getc(3)) and the other
threads were running.
To figure out what your platform is doing use a tool that gives
system-call information like ktrace.
On Jan 30, 2007, at 5:28 PM, [email protected] wrote:
this could get tricky to debug!
Bingo. It’s actually a pretty significant problem for HighLine
because after you do something like fetching a web page, HighLine
believes $stdin to e closed and won’t allow you to ask the user any
more questions. I fear we will need to disable HighLine’s eof?()
check the work around this.
just to be clear (to everyone): have you all compiled ruby from
source?
I compiled from source. I did --enable-pthread as well.
James Edward G. II
On Jan 30, 2007, at 5:40 PM, [email protected] wrote:
On Jan 30, 2007, at 6:17 PM, Eric H. wrote:
Since the thread in your example has completed there’s no threads
to switch to, so the main thread blocks waiting for a character
(on a BSD-ish libc). If there was a thread running, the main
thread would remain blocked (as if it were blocked on getc(3)) and
the other threads were running.
But my point (and James’ point) was that on my (his) platform it
doesn’t block when it logically should.
Thank you. I was beginning to think I was going crazy. 
James Edward G. II
On Jan 30, 2007, at 6:17 PM, Eric H. wrote:
If there are other threads and the IO isn’t readable, ruby will run
those until they complete. Your thread immediately stopped.
I coded it that way on purpose to see if the issue depended on
whether another thread was running. On my platform it doesn’t
matter. It also doesn’t matter if the main thread joins the now dead
thread. It is the initiation of
another thread that changes the behavior of IO#eof? on certain
platforms. The two platforms that have been reported
are:
ruby 1.8.5 Linux 2.6.17-2-686 i686
ruby 1.8.5 Darwin 8.8.0 Power Macintosh (Mac OS X 10.4.8)
Since the thread in your example has completed there’s no threads
to switch to, so the main thread blocks waiting for a character (on
a BSD-ish libc). If there was a thread running, the main thread
would remain blocked (as if it were blocked on getc(3)) and the
other threads were running.
But my point (and James’ point) was that on my (his) platform it
doesn’t block when it logically should.
To figure out what your platform is doing use a tool that gives
system-call information like ktrace.
I did that and came up with the simpler example in the process. If
you run just:
p $stdin.eof?
Ruby blocks waiting for input. If you first create a thread
which immediately terminates (my simple example) then Ruby doesn’t
block (ruby 1.8.4 on Mac OS X) instead the read call is interrupted
with a software interrupt, which I interpreted as being party of Ruby’s
thread implementation.
Gary W.