Forum: Ruby Reading a file in chunks, to a byte array

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
unknown (Guest)
on 2009-01-29 23:26
(Received via mailing list)
Hi everyone,

I'm pretty new to Ruby, I'm taking courses and reading as much info as I
can, but I'm stuck on this one problem that may/may not have a simple
answer.

A quick summary of what I'm trying to accomplish:

I've written a simple SOAP client that interfaces with a web app used
for file transfer... It works great, however, if a file is over a
certain size it just hangs - and I know the reason why...
The webapp receives the file as a chunk, in the form of a byte array,
the great thing about Ruby is that strings are basically an arrangement
of arbitrary bytes, so I can just pass a string with the file object
into the web service and upload small files.

But I'd like to split a file into "chunks" .. I know that I can do this:

---

def get_chunk(bytearray)
bytearray.read(61440) #read a 60kb chunk of the byte array
end

localFile = File.open('test.dat')

get_chunk(localFile)

---

But I need to keep track of more information .. The web app is looking
for an offset integer , which basically works like this:

(with a 60k chunksize)

the first request will be:
offset = 0

second request:
offset = 61440

third request:
offset = 122880

Does anyone know how I could iterate over a bytearray, keeping track of
the offset, and read 60k chunks all at the same time?
in pseudo code:

get_chunk(bytearray)

bytearray.read(61440)
offset = 0
puts bytearray.pos
=> 61440

bytearray.read(61440)
offset += 61440
puts offset
=> 61440
puts bytearray.pos
=> 122880
....

After I have the proper offset, and chunks of a bytearray - I need to
keep calling the web service method with that data until the file = eof.

I'm horrible at code blocks and iteration, if anyone has any
suggestions, it would be greatly appreciated.

Thanks,

Brian
Robert K. (Guest)
on 2009-01-30 00:03
(Received via mailing list)
On 29.01.2009 22:23, removed_email_address@domain.invalid wrote:
>
> get_chunk(localFile)
> second request:
> bytearray.read(61440)
> ...
>
> After I have the proper offset, and chunks of a bytearray - I need to keep calling the 
web service method with that data until the file = eof.
>
> I'm horrible at code blocks and iteration, if anyone has any suggestions, it would be 
greatly appreciated.

It's not fully clear to me which one is the driving application here.
If it is your client application, it's easy

File.open "foo", "rb" do |io|
   pos = io.tell
   while buffer = io.read(CHUNK_SIZE)
     send_to_web_app(pos, buffer)
     pos = io.tell
   end
end

Kind regards

  robert
unknown (Guest)
on 2009-01-30 03:26
(Received via mailing list)
Robert,



 That did it! Thank you so much, very simple solution - I just need to
understand iteration/blocks a bit better for me to figure that out on my
own.



 thanks,



Brian


----- Original Message -----
From: "Robert K." <removed_email_address@domain.invalid>
To: "ruby-talk ML" <removed_email_address@domain.invalid>
Sent: Thursday, January 29, 2009 4:57:37 PM GMT -05:00 US/Canada Eastern
Subject: Re: Reading a file in chunks, to a byte array

On 29.01.2009 22:23, removed_email_address@domain.invalid wrote:
>
> get_chunk(localFile)
> second request:
> bytearray.read(61440)
> ...
>
> After I have the proper offset, and chunks of a bytearray - I need to keep calling the 
web service method with that data until the file = eof.
>
> I'm horrible at code blocks and iteration, if anyone has any suggestions, it would be 
greatly appreciated.

It's not fully clear to me which one is the driving application here.
If it is your client application, it's easy

File.open "foo", "rb" do |io|
   pos = io.tell
   while buffer = io.read(CHUNK_SIZE)
     send_to_web_app(pos, buffer)
     pos = io.tell
   end
end

Kind regards

        robert
This topic is locked and can not be replied to.