Reading a file in chunks, to a byte array

Hi everyone,

I’m pretty new to Ruby, I’m taking courses and reading as much info as I
can, but I’m stuck on this one problem that may/may not have a simple
answer.

A quick summary of what I’m trying to accomplish:

I’ve written a simple SOAP client that interfaces with a web app used
for file transfer… It works great, however, if a file is over a
certain size it just hangs - and I know the reason why…
The webapp receives the file as a chunk, in the form of a byte array,
the great thing about Ruby is that strings are basically an arrangement
of arbitrary bytes, so I can just pass a string with the file object
into the web service and upload small files.

But I’d like to split a file into “chunks” … I know that I can do this:


def get_chunk(bytearray)
bytearray.read(61440) #read a 60kb chunk of the byte array
end

localFile = File.open(‘test.dat’)

get_chunk(localFile)


But I need to keep track of more information … The web app is looking
for an offset integer , which basically works like this:

(with a 60k chunksize)

the first request will be:
offset = 0

second request:
offset = 61440

third request:
offset = 122880

Does anyone know how I could iterate over a bytearray, keeping track of
the offset, and read 60k chunks all at the same time?
in pseudo code:

get_chunk(bytearray)

bytearray.read(61440)
offset = 0
puts bytearray.pos
=> 61440

bytearray.read(61440)
offset += 61440
puts offset
=> 61440
puts bytearray.pos
=> 122880

After I have the proper offset, and chunks of a bytearray - I need to
keep calling the web service method with that data until the file = eof.

I’m horrible at code blocks and iteration, if anyone has any
suggestions, it would be greatly appreciated.

Thanks,

Brian

On 29.01.2009 22:23, [email protected] wrote:

get_chunk(localFile)
second request:
bytearray.read(61440)

After I have the proper offset, and chunks of a bytearray - I need to keep calling the web service method with that data until the file = eof.

I’m horrible at code blocks and iteration, if anyone has any suggestions, it would be greatly appreciated.

It’s not fully clear to me which one is the driving application here.
If it is your client application, it’s easy

File.open “foo”, “rb” do |io|
pos = io.tell
while buffer = io.read(CHUNK_SIZE)
send_to_web_app(pos, buffer)
pos = io.tell
end
end

Kind regards

robert

Robert,

 That did it! Thank you so much, very simple solution - I just need to
understand iteration/blocks a bit better for me to figure that out on my
own.

 thanks,

Brian

----- Original Message -----
From: “Robert K.” [email protected]
To: “ruby-talk ML” [email protected]
Sent: Thursday, January 29, 2009 4:57:37 PM GMT -05:00 US/Canada Eastern
Subject: Re: Reading a file in chunks, to a byte array

On 29.01.2009 22:23, [email protected] wrote:

get_chunk(localFile)
second request:
bytearray.read(61440)

After I have the proper offset, and chunks of a bytearray - I need to keep calling the web service method with that data until the file = eof.

I’m horrible at code blocks and iteration, if anyone has any suggestions, it would be greatly appreciated.

It’s not fully clear to me which one is the driving application here.
If it is your client application, it’s easy

File.open “foo”, “rb” do |io|
  pos = io.tell
  while buffer = io.read(CHUNK_SIZE)
   send_to_web_app(pos, buffer)
   pos = io.tell
  end
end

Kind regards

        robert