# Reading web service resposes in chunks?

**URL:** <https://www.ruby-forum.com/t/reading-web-service-resposes-in-chunks/87067>\
**Category:** Ruby\
**Created:** [February 27, 2007, 11:08pm UTC](https://www.ruby-forum.com/t/reading-web-service-resposes-in-chunks/87067 "2007-02-27T23:08:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![benjohnson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/3ec8ea/32.png) [@benjohnson](https://www.ruby-forum.com/u/benjohnson)\
**Post date:** [February 27, 2007, 11:08pm UTC](https://www.ruby-forum.com/t/reading-web-service-resposes-in-chunks/87067/1 "2007-02-27T23:08:30Z")

</div>

I am making a web service call and getting back very large responses  
(sometimes 5gb). When I get this response it eats all of my RAM. I need  
to read the response in chunks so I can store it in a file and I have no  
idea how to do this. Any help is greatly appreciated.

Here is my code:

require ‘soap/wsdlDriver’

soap = SOAP::WSDLDriverFactory.new(“some url”).create\_rpc\_driver  
soap.wiredump\_file\_base = “soapfile”

response = soap.GetWhatever(:whatever =\> "whatever)

Ironically, when reading the response it doesn’t dump it into the file  
until it gets the entire response into memory, this is what’s killing my  
server. Is there a more efficient way of doing this?

Thanks for your help and time.

---

<div class="post-metadata">

**Author:** ![benjohnson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/3ec8ea/32.png) [@benjohnson](https://www.ruby-forum.com/u/benjohnson)\
**Post date:** [February 28, 2007, 9:57pm UTC](https://www.ruby-forum.com/t/reading-web-service-resposes-in-chunks/87067/2 "2007-02-28T21:57:18Z")

</div>

On Wed, Feb 28, 2007 at 07:08:32AM +0900, Ben J. wrote:

> soap = SOAP::WSDLDriverFactory.new(“some url”).create\_rpc\_driver  
> soap.wiredump\_file\_base = “soapfile”
> 
> response = soap.GetWhatever(:whatever =\> "whatever)
> 
> Ironically, when reading the response it doesn’t dump it into the file  
> until it gets the entire response into memory, this is what’s killing my  
> server. Is there a more efficient way of doing this?

You just want to get the whole response into a file? Then I’d suggest:

1. build the SOAP XML request as a string

2. connect to the server using HTTP

3. post the XML you built in step 1

4. read the response as a stream and write it to a file.

To get the response as a stream, you can probably still use Net::HTTP  
for  
this. If the response from the server is chunked (use tcpdump to check  
this), you can call HTTPResponse#read\_body with a block, and you will  
get  
the chunks passed to you in turn. The following example is given in the  
documentation:

```
 # using block
 http.request_post('/cgi-bin/nice.rb', 'datadatadata...') 

```

{|response|  
p response.status  
p response[‘content-type’]  
response.read\_body do |str| # read body now  
print str  
end  
}

If the response is not chunked, then just pull out the @socket from the  
object and read(65536) it in a loop.

If you want to _parse_ the response on the fly, then you could use rexml  
in  
stream parsing mode: see  
[http://www.germane-software.com/software/XML/rexml/docs/tutorial.html](http://www.germane-software.com/software/XML/rexml/docs/tutorial.html)  
and  
scroll down to “Stream Parsing”

You then may need an IO.pipe or similar object which accepts the HTTP  
chunks  
on one side and gives a readable stream on the other.

But this may still be a problem if your 5GB response consists mainly of  
a  
single element, …5GB of data…. I’m not sure if  
REXML will call text() with blocks, or will try to slurp the whole 5GB  
in  
before calling text() once.

HTH,

Brian.