Forum: Ruby Net::HTTP Closes STDIN

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
James G. (Guest)
on 2007-01-28 03:14
(Received via mailing list)
Kenneth K. has brought up a HighLine issue and I'm trying to look
into it.  Oddly, it seems to happen when interacting with the
Net::HTTP library.  I've narrowed it done to the following example on
my box:

$ cat stdin_closed_issue.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
   body = http.get('/en/license.txt').body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true

Can anyone explain why $stdin is closed after the page read?

James Edward G. II
Eric H. (Guest)
on 2007-01-28 03:40
(Received via mailing list)
On Jan 27, 2007, at 17:13, James Edward G. II wrote:
>   body = http.get('/en/license.txt').body
> end
> p $stdin.eof?
> $ ruby stdin_closed_issue.rb
> true
>
> Can anyone explain why $stdin is closed after the page read?

$stdin isn't closed, its at the end of file.  Use #closed? to test if
an IO has been closed or not.

$ cat test.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
     body = http.get('/en/license.txt').body
end
p $stdin.closed?
p $stdin.eof?

$ ruby test.rb
false # hit ^D to close $stdin on the sending side.
true
$
$ echo hi | ruby test.rb
false
false


--
Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net

I LIT YOUR GEM ON FIRE!
James G. (Guest)
on 2007-01-28 04:39
(Received via mailing list)
On Jan 27, 2007, at 7:39 PM, Eric H. wrote:

> $stdin isn't closed, its at the end of file.  Use #closed? to test
> if an IO has been closed or not.

Right, good point.

However, when I call eof?() before the Net::HTTP call it behaves
differently (stalls and prints false).  Why does it not behave the
same after that page read?

James Edward G. II
Eric H. (Guest)
on 2007-01-28 04:49
(Received via mailing list)
On Jan 27, 2007, at 18:38, James Edward G. II wrote:
> On Jan 27, 2007, at 7:39 PM, Eric H. wrote:
>
>> $stdin isn't closed, its at the end of file.  Use #closed? to test
>> if an IO has been closed or not.
>
> Right, good point.
>
> However, when I call eof?() before the Net::HTTP call it behaves
> differently (stalls and prints false).  Why does it not behave the
> same after that page read?

$ cat test.rb
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?

require 'net/http'
require 'io/wait'

Net::HTTP.start('localhost', 80) do |http|
     body = http.get('/').body
end

puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?

$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$

For the first #eof? no data written, so Ruby waits until something's
been flushed.  I typed some text and hit return to flush the
terminal's stdout (Ruby's $stdin).  At the second #eof? no input on
$stdin has been consumed, so Ruby doesn't need to check for #eof?
again by waiting.

$ cat test.rb
puts "eof? %p" % $stdin.eof?
gets
puts "eof? %p" % $stdin.eof?
$ ruby test.rb
aoeu
eof? false
aoeu
eof? false

--
Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net

I LIT YOUR GEM ON FIRE!
James G. (Guest)
on 2007-01-28 08:01
(Received via mailing list)
On Jan 27, 2007, at 8:48 PM, Eric H. wrote:

>> same after that page read?
> end
> $
>
> For the first #eof? no data written, so Ruby waits until
> something's been flushed.

OK, that makes sense.  However, why does moving that eof?() check
below a Net::HTTP page fetch change this?  In that case, why does
Ruby not need for something to be flushed in that case and why is eof?
() then +true+?

James Edward G. II
Eric H. (Guest)
on 2007-01-29 11:55
(Received via mailing list)
On Jan 27, 2007, at 22:01, James Edward G. II wrote:
>>> differently (stalls and prints false).  Why does it not behave
>> For the first #eof? no data written, so Ruby waits until
>> something's been flushed.
>
> OK, that makes sense.  However, why does moving that eof?() check
> below a Net::HTTP page fetch change this?

I don't see this behavior.  With your original example I have to hit
^D to get a prompt back.

$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]

> In that case, why does Ruby not need for something to be flushed

Because #eof? reads a character if feof(3) is not true and there's no
data pending:

rb_io_eof(io)
     VALUE io;
{
     OpenFile *fptr;
     int ch;

     GetOpenFile(io, fptr);
     rb_io_check_readable(fptr);

     if (feof(fptr->f)) return Qtrue;
     if (READ_DATA_PENDING(fptr->f)) return Qfalse;
     READ_CHECK(fptr->f);
     clearerr(fptr->f);
     TRAP_BEG;
     ch = getc(fptr->f);
     ^^^^^^^^^^^^^^^^^^^

Analysis of sampling pid 15640 every 10.000000 milliseconds
Call graph:
[...]
  300 rb_io_eof
    300 getc
[...]

> in that case and why is eof?() then +true+?

#eof? will only be true when there's no more data to read.

--
Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net

I LIT YOUR GEM ON FIRE!
James G. (Guest)
on 2007-01-29 20:21
(Received via mailing list)
On Jan 29, 2007, at 3:54 AM, Eric H. wrote:

> On Jan 27, 2007, at 22:01, James Edward G. II wrote:
>> OK, that makes sense.  However, why does moving that eof?() check
>> below a Net::HTTP page fetch change this?
>
> I don't see this behavior.  With your original example I have to
> hit ^D to get a prompt back.
>
> $ ruby -v
> ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]

I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don't get prompted for input and it still tells me $stdin
is closed:

Firefly:~/Desktop$ ruby -v
ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.1]
Firefly:~/Desktop$ cat stdin_eof_issue.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
   body = http.get('/en/license.txt').body
end
p $stdin.eof?
Firefly:~/Desktop$ ruby stdin_eof_issue.rb
true

James Edward G. II
James G. (Guest)
on 2007-01-29 20:22
(Received via mailing list)
On Jan 29, 2007, at 12:20 PM, James Edward G. II wrote:

> I still don't get prompted for input and it still tells me $stdin
> is closed:

I'm sorry, I meant it still claims $stdin is at eof?().

James Edward G. II
Jacob F. (Guest)
on 2007-01-29 20:33
(Received via mailing list)
On 1/29/07, James Edward G. II <removed_email_address@domain.invalid> wrote:
> I just upgraded to Ruby 1.8.5 to see if it would make a difference,
> but I still don't get prompted for input and it still tells me $stdin
> is [eof]:

I can confirm on my machine:

lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ cat test.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
  body = http.get('/en/license.txt').body
end
p $stdin.eof?
lukfugl@hephaestus:~$ ruby test.rb
true

One note: the first time I ran it, I thought it was waiting for input
too, but it was really just a few second delay while fetching
http://www.ruby-lang.org/en/license.txt.

Jacob F.
Eric H. (Guest)
on 2007-01-30 20:39
(Received via mailing list)
On Jan 29, 2007, at 10:32, Jacob F. wrote:
> On 1/29/07, James Edward G. II <removed_email_address@domain.invalid> wrote:
>> I just upgraded to Ruby 1.8.5 to see if it would make a difference,
>> but I still don't get prompted for input and it still tells me $stdin
>> is [eof]:
>
> I can confirm on my machine:

Then your getc may do something different than mine (BSD-ish).
James G. (Guest)
on 2007-01-30 20:42
(Received via mailing list)
On Jan 30, 2007, at 12:38 PM, Eric H. wrote:

> On Jan 29, 2007, at 10:32, Jacob F. wrote:
>> On 1/29/07, James Edward G. II <removed_email_address@domain.invalid> wrote:
>>> I just upgraded to Ruby 1.8.5 to see if it would make a difference,
>>> but I still don't get prompted for input and it still tells me
>>> $stdin
>>> is [eof]:
>>
>> I can confirm on my machine:
>
> Then your getc may do something different than mine (BSD-ish).

In other words, you don't feel this is a Ruby issue, right?

James Edward G. II
unknown (Guest)
on 2007-01-30 23:32
(Received via mailing list)
On Jan 27, 2007, at 8:13 PM, James Edward G. II wrote:

> Kenneth K. has brought up a HighLine issue and I'm trying to
> look into it.  Oddly, it seems to happen when interacting with the
> Net::HTTP library.  I've narrowed it done to the following example
> on my box:

Here is a simpler example that illustrates the issue:

   Thread.new { }
   p $stdin.eof?

As Eric H. pointed out, EOF on a terminal device is detected by
trying to read from
the device.  It appears that *prior* to the creation of a thread, a
read on a terminal
device will simply block waiting for some activity on the device;
either actual data
or indication of an eof condition (someone typing control-d).

*After* a thread has been created, Ruby's green thread implementation
kicks in.  This
means that the read caused by IO#eof? on the terminal device is not
allowed to block.
Instead it fails as an interrupted system call, which is interpreted
by IO#eof? as an
end of file condition.  This is true even if the main thread is the
only running thread.

James' original example used Net::HTTP, which uses Timeout, which
creates a thread to
gain control of code that may block (i.e., an HTTP connection to a
remote host).

I don't know enough about Ruby's thread implementation to know if
this is a bug or
a feature.  My hunch is that IO#eof? needs to be fixed so that an
interrupted system
call doesn't get interpreted as an end of file condition.  Another
change would be to
turn off the software interrupt that is used to multiplex IO activity
by threads when
only the main thread is running.

Gary W.
unknown (Guest)
on 2007-01-31 00:36
(Received via mailing list)
On Wed, 31 Jan 2007 removed_email_address@domain.invalid wrote:

>  p $stdin.eof?
> is interpreted by IO#eof? as an end of file condition.  This is true even if
> multiplex IO activity by threads when only the main thread is running.
>
> Gary W.
>

sounds quite reasonable.  however, on my box:

   harp:~ > cat a.rb
   Thread.new { }
   p $stdin.eof?

   harp:~ > ruby a.rb  # and then type ctrl-d
   true

   harp:~ > ruby -v
   ruby 1.8.4 (2005-12-01) [i686-linux]

   harp:~ > uname -srm
   Linux 2.4.21-47.0.1.EL i686

so looks to be a bsd/mac issue?

-a
Jacob F. (Guest)
on 2007-01-31 00:47
(Received via mailing list)
On 1/30/07, removed_email_address@domain.invalid 
<removed_email_address@domain.invalid> wrote:
>    ruby 1.8.4 (2005-12-01) [i686-linux]
>
>    harp:~ > uname -srm
>    Linux 2.4.21-47.0.1.EL i686
>
> so looks to be a bsd/mac issue?

On my box:

  lukfugl@hephaestus:~$ cat a.rb
  Thread.new { }
  p $stdin.eof?

  lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
  true

  lukfugl@hephaestus:~$ ruby -v
  ruby 1.8.5 (2006-08-25) [i486-linux]

  lukfugl@hephaestus:~$ uname -srm
  Linux 2.6.17-2-686 i686

So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just
notice that my ruby is compiled i486-linux -- that *may* have
something to do with it, but I don't think so.

Jacob F.
unknown (Guest)
on 2007-01-31 01:08
(Received via mailing list)
On Jan 30, 2007, at 5:35 PM, removed_email_address@domain.invalid wrote:
>   p $stdin.eof?
> so looks to be a bsd/mac issue?
My post was based on ruby 1.8.5 on Darwin 8.8.0 Power Macintosh

The anomaly does not occur with ruby-1.9 on the same OS/hardware.

Gary W.
Eric H. (Guest)
on 2007-01-31 01:18
(Received via mailing list)
On Jan 30, 2007, at 13:31, removed_email_address@domain.invalid wrote:
>
> As Eric H. pointed out, EOF on a terminal device is detected by
> trying to read from the device.  It appears that *prior* to the
> creation of a thread, a read on a terminal device will simply block
> waiting for some activity on the device; either actual data or
> indication of an eof condition (someone typing control-d).
>
> *After* a thread has been created, Ruby's green thread
> implementation kicks in.  This means that the read caused by
> IO#eof? on the terminal device is not allowed to block.

If there are other threads and the IO isn't readable, ruby will run
those until they complete.  Your thread immediately stopped.

Since the thread in your example has completed there's no threads to
switch to, so the main thread blocks waiting for a character (on a
BSD-ish libc).  If there was a thread running, the main thread would
remain blocked (as if it were blocked on getc(3)) and the other
threads were running.

To figure out what your platform is doing use a tool that gives
system-call information like ktrace.
unknown (Guest)
on 2007-01-31 01:29
(Received via mailing list)
On Wed, 31 Jan 2007, Jacob F. wrote:

> ruby 1.8.5 (2006-08-25) [i486-linux]
>
> lukfugl@hephaestus:~$ uname -srm
> Linux 2.6.17-2-686 i686
>
> So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just notice
> that my ruby is compiled i486-linux -- that *may* have something to do with
> it, but I don't think so.

strange:

   [nrt@anchor ~]$ ruby -v -e'  Thread.new{} and p STDIN.eof?  '   #
ctrl-d
   ruby 1.8.5 (2006-12-11 patchlevel 5000) [i686-linux]
   true

   [nrt@anchor ~]$ uname -srm
   Linux 2.6.9-42.0.3.ELsmp i686

and:

   harp:~ > ruby19 -v -e'  Thread.new{} and p STDIN.eof?  '   # ctrl-d
   ruby 1.9.0 (2006-07-26) [i686-linux]
   true

   harp:~ > uname -srm
   Linux 2.4.21-47.0.1.EL i686

this could get tricky to debug!  none of my linux versions or ruby
versions
show this...

just to be clear (to everyone): have you all compiled ruby from source?
all
my installs are from source using nothing but '--prefix' given to
./configure.
if anyone has a package (cough) manager or vendor (cough cough) install
i'd
reccomend compiling from source without any options and testing again.

kind regards.

-a
unknown (Guest)
on 2007-01-31 01:41
(Received via mailing list)
On Jan 30, 2007, at 6:17 PM, Eric H. wrote:
> If there are other threads and the IO isn't readable, ruby will run
> those until they complete.  Your thread immediately stopped.

I coded it that way on purpose to see if the issue depended on
whether another thread was running.  On my platform it doesn't
matter.  It also doesn't matter if the main thread joins the now dead
thread.  It is the initiation of
another thread that changes the behavior of IO#eof? on certain
platforms.  The two platforms that have been reported
are:
  ruby 1.8.5 Linux 2.6.17-2-686 i686
  ruby 1.8.5 Darwin 8.8.0 Power Macintosh  (Mac OS X 10.4.8)

> Since the thread in your example has completed there's no threads
> to switch to, so the main thread blocks waiting for a character (on
> a BSD-ish libc).  If there was a thread running, the main thread
> would remain blocked (as if it were blocked on getc(3)) and the
> other threads were running.

But my point (and James' point) was that on my (his) platform it
*doesn't* block when it logically should.

> To figure out what your platform is doing use a tool that gives
> system-call information like ktrace.

I did that and came up with the simpler example in the process.  If
you run just:

   p $stdin.eof?

Ruby blocks waiting for input.  If you first create a thread
which immediately terminates (my simple example) then Ruby doesn't
block (ruby 1.8.4 on Mac OS X) instead the read call is interrupted
with a software interrupt, which I interpreted as being party of Ruby's
thread implementation.

Gary W.
James G. (Guest)
on 2007-01-31 05:09
(Received via mailing list)
On Jan 30, 2007, at 5:28 PM, removed_email_address@domain.invalid wrote:

> this could get tricky to debug!

Bingo.  It's actually a pretty significant problem for HighLine
because after you do something like fetching a web page, HighLine
believes $stdin to e closed and won't allow you to ask the user any
more questions.  I fear we will need to disable HighLine's eof?()
check the work around this.

> just to be clear (to everyone): have you all compiled ruby from
> source?

I compiled from source.  I did --enable-pthread as well.

James Edward G. II
James G. (Guest)
on 2007-01-31 05:10
(Received via mailing list)
On Jan 30, 2007, at 5:40 PM, removed_email_address@domain.invalid wrote:

>
> On Jan 30, 2007, at 6:17 PM, Eric H. wrote:
>> Since the thread in your example has completed there's no threads
>> to switch to, so the main thread blocks waiting for a character
>> (on a BSD-ish libc).  If there was a thread running, the main
>> thread would remain blocked (as if it were blocked on getc(3)) and
>> the other threads were running.
>
> But my point (and James' point) was that on my (his) platform it
> *doesn't* block when it logically should.

Thank you.  I was beginning to think I was going crazy.  ;)

James Edward G. II
Eric H. (Guest)
on 2007-01-31 08:31
(Received via mailing list)
On Jan 30, 2007, at 19:08, James Edward G. II wrote:
> On Jan 30, 2007, at 5:28 PM, removed_email_address@domain.invalid wrote:
>> this could get tricky to debug!
>
> Bingo.  It's actually a pretty significant problem for HighLine
> because after you do something like fetching a web page, HighLine
> believes $stdin to e closed and won't allow you to ask the user any
> more questions.  I fear we will need to disable HighLine's eof?()
> check the work around this.

Then it sounds like HighLine needs to check #closed? or continue
reading until IOError.  If #eof? returns true you've read everything
you could up to now.  Nothing prevents more data from appearing on
the socket after you've checked for #eof?.
James G. (Guest)
on 2007-01-31 21:24
(Received via mailing list)
On Jan 31, 2007, at 12:30 AM, Eric H. wrote:

> Then it sounds like HighLine needs to check #closed? or continue
> reading until IOError.

But, as you pointed out, closed?() is not the same as eof?().

Watching for an IOError also isn't very reliable since HighLine's
input stream may be anything from STDIN, to a Socket, with IO and
StringIO in between.

For now, I've added a workaround so user code can disable HighLine's
EOF checks as needed.

Thank to all for helping me to understand this issue.

James Edward G. II
unknown (Guest)
on 2007-01-31 21:33
(Received via mailing list)
On Thu, 1 Feb 2007, James Edward G. II wrote:

>
> Thank to all for helping me to understand this issue.
>

great - that make one of us!  ;-)

-a
unknown (Guest)
on 2007-02-01 02:06
(Received via mailing list)
On Jan 31, 2007, at 2:32 PM, removed_email_address@domain.invalid wrote:
> On Thu, 1 Feb 2007, James Edward G. II wrote:
>>
>> Thank to all for helping me to understand this issue.
>>
>
> great - that make one of us!  ;-)

I spent a bit of time looking at the code in io.c.  This
file has changed radically from the 1.8 branch to the
1.9 (trunk) branch.  But...

On my platform (Darwin 8.8.0 Power Macintosh):
1.9     from sources    works
1.8 (head)   from source     works
1.8.5     from darwin ports   broken
1.8.3    from source     works

So perhaps there is something to Ara's suggestion that
the packaged version of ruby for Mac OS X is broken. I
don't have access to linux.

I'm rebuilding 1.8.5 from source now...


Gary W.
unknown (Guest)
on 2007-02-01 22:36
(Received via mailing list)
On Jan 31, 2007, at 7:05 PM, removed_email_address@domain.invalid wrote:
> I'm rebuilding 1.8.5 from source now...

Is the Darwin Ports (macports) maintainer for Ruby on this list?

The problem that James originally reported only seems
to exist in the Darwin Ports version of 1.8.5:

$ ruby -v
ruby 1.8.5 (2006-08-25) [powerpc-darwin8.7.0]

I just built from the current 1.8.5 branch and the problem doesn't
exist:

$ ./ruby -v
ruby 1.8.5 (2007-01-27 patchlevel 17) [powerpc-darwin8.8.0]

$ ./ruby -e 'Thread.new{} and p $stdin.eof?'   # pause until ctrl-D
entered
true

Gary W.
James G. (Guest)
on 2007-02-01 22:40
(Received via mailing list)
On Feb 1, 2007, at 2:35 PM, removed_email_address@domain.invalid wrote:

> The problem that James originally reported only seems
> to exist in the Darwin Ports version of 1.8.5:

Na.  I don't use MacPorts.  I built Ruby from source.

James Edward G. II
James G. (Guest)
on 2007-02-01 22:47
(Received via mailing list)
On Feb 1, 2007, at 2:35 PM, removed_email_address@domain.invalid wrote:

> I just built from the current 1.8.5 branch and the problem doesn't
> exist:
>
> $ ./ruby -v
> ruby 1.8.5 (2007-01-27 patchlevel 17) [powerpc-darwin8.8.0]

I wonder if this is related to the --enable-pthread option.  I used
it and wouldn't be surprised to hear that MacPorts did too.  Did you
use this option when you did this test?

James Edward G. II
unknown (Guest)
on 2007-02-01 23:00
(Received via mailing list)
On Feb 1, 2007, at 3:46 PM, James Edward G. II wrote:

> you use this option when you did this test?
I just typed:

autoconf; ./configure; make

I'll try:

./configure --enable-pthread; make

and report back...

Gary W.
unknown (Guest)
on 2007-02-02 01:52
(Received via mailing list)
On Feb 1, 2007, at 3:46 PM, James Edward G. II wrote:
>
> I wonder if this is related to the --enable-pthread option.  I used
> it and wouldn't be surprised to hear that MacPorts did too.  Did
> you use this option when you did this test?

Sure enough.  When I compiled 1.8.5 with --enable-pthread the test
failed.

Ruby 1.9 wouldn't compile with --enable-pthread but works without.


Gary W.
James G. (Guest)
on 2007-02-02 03:42
(Received via mailing list)
On Feb 1, 2007, at 5:51 PM, removed_email_address@domain.invalid wrote:

>
> On Feb 1, 2007, at 3:46 PM, James Edward G. II wrote:
>>
>> I wonder if this is related to the --enable-pthread option.  I
>> used it and wouldn't be surprised to hear that MacPorts did too.
>> Did you use this option when you did this test?
>
> Sure enough.  When I compiled 1.8.5 with --enable-pthread the test
> failed.

Ah, so it's likely a pthread issue.  Most of the instruction articles
floating around the web for installing Ruby on Mac OS X recommend
adding that option, which is probably why we see this on that platform.

What is the advantage of pthread?  I don't know.

James Edward G. II
This topic is locked and can not be replied to.