Forum: Ruby Feeding data to be PUT with Net::HTTP?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
50adc37a6c4d5923e3c34250f0694c70?d=identicon&s=25 Chris Markle (cmarkle)
on 2009-02-23 18:25
Since I am HTTP PUT'ing large amounts of data (ideally using Net::HTTP)
I would like to feed data in some way to an HTTP PUT so that I don't
have to have the entire amount of data in memory before using methods
like request or send_request to initiate the PUT. I saw one post that
suggests it can't be done but I thought I'd check again.

If it can't be done with Net::HTTP, are there other HTTP client
libraries that I might be able to do this with, and maybe even some
examples of how to do it?

Thanks in advance...

Chris
54185df1d348bbd34587fcd4f8e4779b?d=identicon&s=25 Louis-Philippe (Guest)
on 2009-02-24 16:56
(Received via mailing list)
have you tried 'httpclient'?there seems to be a streaming post method
available:
http://dev.ctor.org/doc/httpclient/

L-P

2009/2/23 Chris Markle <cmarkle@comcast.net>
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2009-02-25 14:21
Chris Markle wrote:
> Since I am HTTP PUT'ing large amounts of data (ideally using Net::HTTP)
> I would like to feed data in some way to an HTTP PUT so that I don't
> have to have the entire amount of data in memory before using methods
> like request or send_request to initiate the PUT. I saw one post that
> suggests it can't be done but I thought I'd check again.

Best way to understand this is to go to the source code - probably
somewhere like /usr/lib/ruby/1.8/net/http.rb on your system.

    def put(path, data, initheader = nil)   #:nodoc:
      res = request(Put.new(path, initheader), data)
      res.value unless @newimpl
      res
    end

Trace it through, this calls req.exec to send the request.

    def exec(sock, ver, path)   #:nodoc: internal use only
      if @body
        send_request_with_body sock, ver, path, @body
      elsif @body_stream
        send_request_with_body_stream sock, ver, path, @body_stream
      else
        write_header sock, ver, path
      end
    end

Looking at send_request_with_body_stream, you can pass any object which
responds to read(n), such as an open IO stream. If you set
"Tranfer-Encoding: chunked" then you'll get it posted in chunks too,
which simplifies buffering at the receiving side (if it also supports
chunking)

But it looks like you may need to create your Request object by hand and
call #body_stream=, because set_body_internal doesn't understand about
streaming objects. Perhaps something like this (untested):

  req = Net::HTTP::Put.new(path)
  req.body_stream = File.open("/path/to/large/file")
  res = Net::HTTP.start(url.host, url.port) {|http|
      http.request(req)
  }

HTH,

Brian.
50adc37a6c4d5923e3c34250f0694c70?d=identicon&s=25 Chris Markle (cmarkle)
on 2009-02-25 16:34
Brian,

> [...] Looking at send_request_with_body_stream, you can pass any object
> which responds to read(n), such as an open IO stream. If you set
> "Tranfer-Encoding: chunked" then you'll get it posted in chunks too,
> which simplifies buffering at the receiving side (if it also supports
> chunking). But it looks like you may need to create your Request object
> by hand and call #body_stream=, because set_body_internal doesn't
> understand about streaming objects. [...]

Thank you very much for such a detailed answer, and for including some
code snippets. I was able to stream using this suggestion. And your
comments about "you can pass any object that responds to read(n)" gives
me other food for thought about this.

Thanks again...

Chris
This topic is locked and can not be replied to.