# Potential bug in Net::HTTP, and tentative patch **URL:** \ **Category:** Ruby\ **Created:** [April 13, 2009, 5:41am UTC](https://www.ruby-forum.com/t/potential-bug-in-net-http-and-tentative-patch/165361 "2009-04-13T05:41:25Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Yves-Eric\_Martin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/f9ae1b/32.png) [@Yves-Eric\_Martin](https://www.ruby-forum.com/u/Yves-Eric_Martin)\ **Post date:** [April 13, 2009, 5:41am UTC](https://www.ruby-forum.com/t/potential-bug-in-net-http-and-tentative-patch/165361/1 "2009-04-13T05:41:25Z")
Hi all, I am building a nice little system using Ruby (on Rails), one part of which uses Net::HTTP to retrieve some data over HTTP. Everything seems to work fine, but on some requests, I get an EOFError. As I found out, this problem has already been reported, but without any answer. See: [http://rubyforge.org/forum/forum.php?thread\_id=28826&forum\_id=6052](http://rubyforge.org/forum/forum.php?thread_id=28826&forum_id=6052) I think I may have traced the problem back to a bug in Net::HTTP. Here is a two-liner to reproduce the error: $ irb > > require ‘net/http’ > > =\> true > > res = > > Net::HTTP.get\_response(URI.parse(‘[http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com](http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com)’)) > > EOFError: end of file reached > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:133:in > > `sysread' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:133:in `rbuf\_fill’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:56:in > > `timeout' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:76:in `timeout’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:132:in > > `rbuf_fill' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:126:in > > `readline' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2236:in `read\_chunked’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2216:in > > `read_body_0' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2182:in `read\_body’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2207:in > > `body' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2146:in `reading\_body’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1061:in > > `request' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:957:in `request\_get’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:380:in > > `get_response' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:547:in `start’ > > from > > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:379:in > > `get\_response’ > > from (irb):2\>\> My quick analysis: In http.rb, line 2236 the “read\_chunked” function calls @socket.readline. This “readline” function, in protocol.rb line 126, calls readuntil(“\n”). This works fine if the data chunk is “\n-terminated”, but throws an EOFError if it is not. I am not sure about the underlying standards of chunked http, maybe the data chunk is supposed to always been \n-terminated, and it may be a mis-behaving server, but the fact is: I can get the example image fine with any browser, but not with Net::HTTP. As a tentative fix, I wrote a patch that catches the EOFError in read\_chunked. You will the patch file attached. With the patch, things work fine: $ irb > > require ‘net/http’ > > =\> true > > res = > > Net::HTTP.get\_response(URI.parse(‘[http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com](http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com)’)) > > =\> #\ > > File.open(‘test.jpg’,‘w’).write res.body > > =\> 2881 With the patch, the above gives me a perfectly fine JPEG file. However, I am afraid my current patch, with a big begin…rescue around most of the body of the read\_chunked function, catches the EOFError at level higher than necessary, which is not good practice… Anyway, before continuing any further, could someone involved in the development of Ruby take a look at this, confirm the existence of the bug, and maybe even come up with a better fix? PS: please let me know if I posted this in the wrong list, or if I should open a bug report on some bug tracking system. Thank you for your help. ---
**Author:** ![Yves-Eric\_Martin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/f9ae1b/32.png) [@Yves-Eric\_Martin](https://www.ruby-forum.com/u/Yves-Eric_Martin)\ **Post date:** [April 13, 2009, 1:20pm UTC](https://www.ruby-forum.com/t/potential-bug-in-net-http-and-tentative-patch/165361/2 "2009-04-13T13:20:10Z")
Hi, At Mon, 13 Apr 2009 12:40:32 +0900, Yves-Eric Martin wrote in [ruby-talk:333704]: > In http.rb, line 2236 the “read\_chunked” function calls > @socket.readline. This “readline” function, in protocol.rb line 126, > calls readuntil("\n"). This works fine if the data chunk is > “\n-terminated”, but throws an EOFError if it is not. Not “\n-terminated”. According to RFC2616 and RFC2068, chunks consist from chunk-size and chunk-body, and the chunked-body is terminated by “0” size chunk. That is, the response doesn’t seem to follow the RFCs. ---
**Author:** ![Yves-Eric\_Martin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/f9ae1b/32.png) [@Yves-Eric\_Martin](https://www.ruby-forum.com/u/Yves-Eric_Martin)\ **Post date:** [April 14, 2009, 5:33am UTC](https://www.ruby-forum.com/t/potential-bug-in-net-http-and-tentative-patch/165361/3 "2009-04-14T05:33:42Z")
Thank you for pointing me to the RFC. Indeed, the response does not seem RFC-compliant… Other than my quick and dirty patch, is there a way to tell Net::HTTP to ignore the EOFError and accept non-compliant input? Again, the point is that an image, which displays fine in Internet Explorer, Firefox and Safari, cannot be downloaded with Net::HTTP. While I understand the “not RFC-compliant” argument, for practical reasons, it does seem a bit limiting… Thank you, PS: I will also contact the administrator of the problem site regarding this RFC compliance issue. – Yves-Eric ---
**Author:** ![Yves-Eric\_Martin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/f9ae1b/32.png) [@Yves-Eric\_Martin](https://www.ruby-forum.com/u/Yves-Eric_Martin)\ **Post date:** [April 14, 2009, 10:33am UTC](https://www.ruby-forum.com/t/potential-bug-in-net-http-and-tentative-patch/165361/4 "2009-04-14T10:33:49Z")
Works like a charm! Thank you Nobu for your great help. I owe you a beer. – Yves-Eric ---
**Author:** ![Yves-Eric\_Martin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/f9ae1b/32.png) [@Yves-Eric\_Martin](https://www.ruby-forum.com/u/Yves-Eric_Martin)\ **Post date:** [April 14, 2009, 6:26am UTC](https://www.ruby-forum.com/t/potential-bug-in-net-http-and-tentative-patch/165361/5 "2009-04-14T06:26:51Z")
Hi, At Tue, 14 Apr 2009 12:33:01 +0900, Yves-Eric Martin wrote in [ruby-talk:333820]: > Other than my quick and dirty patch, is there a way to tell Net::HTTP > to ignore the EOFError and accept non-compliant input? Again, the point > is that an image, which displays fine in Internet Explorer, Firefox and > Safari, cannot be downloaded with Net::HTTP. While I understand the > “not RFC-compliant” argument, for practical reasons, it does seem a bit > limiting… See rdoc of Net::HTTPResponse#read\_body and Net::HTTP#request\_get. out = “” # or open(destfile, “wb”) begin Net::HTTP.get\_response(uri) do |res| res.read\_body {|s| out \<\< s} end rescue EOFError end