Forum: Ruby how to know if completely donwloaded

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Dy D. (Guest)
on 2009-04-14 02:33
i have tried combining all what i research...

#code starts here:

listofFile = [ ... ]   #<-- internet files like PDF, etc...

listofFile.each do |each_file|
  fileBaseName = File.basename(each_file)
  filePath = Dir.pwd + '/' + fileBaseName
  begin
    if File.exist?(fileBaseName) != true
      openFileRead = open(each_file)
    else
      puts "message: File already exist = #{fileBaseName}"
    end
  rescue
    retry
  else
    begin
      writeOut = open(filePath, 'wb+')
#<-- if internet cuts here, how do i know if its completely downloaded
      writeOut.write(openFileRead.read)
    rescue => err
      puts err.message
      puts "error:  #{each_file}  -- download error."
    else
      writeOut.close
    end
  end
end

# code ends here

actually i have internet connection problem here... if the cuts the file
will be corrupted...

please advice me.. or do i miss something...

thanks
Dy D. (Guest)
on 2009-04-14 04:21
guy please help me... i cant detect if internet connection cuts the file
will be corrupted...
Luke C. (Guest)
on 2009-04-14 19:01
Dy Dymy wrote:
> guy please help me... i cant detect if internet connection cuts the file
> will be corrupted...

It appears that you have already downloaded the files and are attempting
to validate them after the download process is complete.

I'd suggest a couple of different strategies: one would be to have the
download process raise an error if the download does not complete and
the other would be to check the files against the server you are
downloading from. If possible use an md5 checksum and otherwise you
could check the size.

If you were downloading zip, gzip files or some other compressed format
they will usually have built in checksumming, so you could verify the
file that way. Other file formats might have ways of being validated
too.

Luke
Dy D. (Guest)
on 2009-04-16 03:49
Luke C. wrote:
> Dy Dymy wrote:
>> guy please help me... i cant detect if internet connection cuts the file
>> will be corrupted...
>
> It appears that you have already downloaded the files and are attempting
> to validate them after the download process is complete.
>
> I'd suggest a couple of different strategies: one would be to have the
> download process raise an error if the download does not complete and
> the other would be to check the files against the server you are
> downloading from. If possible use an md5 checksum and otherwise you
> could check the size.
>
> If you were downloading zip, gzip files or some other compressed format
> they will usually have built in checksumming, so you could verify the
> file that way. Other file formats might have ways of being validated
> too.
>
> Luke

hi Luke,

thanks for the reply...

firstly its correct that i am validating files after i download the
file.

"raise an error if the download does not complete"
 question: Where i will put my raise error in my code?... i put a "begin
- rescue - else - end" in my code... hardest part is ".read" has no
error exemption to check if it reads all throughout the file.

"check the files against the server you are downloading from."
 question: how do i check the files against the server?

"md5 checksum"
 question: what is md5 checksum?

"check the size"
 comment: after i made this thread, i added my code to check the size.
Before i download the file i check the size using "uri meta .length" and
compare it after i download it using "File.size"... but still no use
because if it break at ".read" the size that i will get is still same as
what i downloaded... meaning uri.length = File.size

thank ahead...
Luke C. (Guest)
on 2009-04-16 18:58
>
> "raise an error if the download does not complete"
>  question: Where i will put my raise error in my code?... i put a "begin
> - rescue - else - end" in my code... hardest part is ".read" has no
> error exemption to check if it reads all throughout the file.
>

I'm talking about adding an exception into the code that does the
downloading, not the code that reads off the disk. If you can detect
that a download has ended unexpectedly, you can queue the download to
start again.

> "check the files against the server you are downloading from."
>  question: how do i check the files against the server?
>

By md5 checksum or the file size.

> "md5 checksum"
>  question: what is md5 checksum?
>

Do some googling my friend.

> "check the size"
>  comment: after i made this thread, i added my code to check the size.
> Before i download the file i check the size using "uri meta .length" and
> compare it after i download it using "File.size"... but still no use
> because if it break at ".read" the size that i will get is still same as
> what i downloaded... meaning uri.length = File.size
>

If you know the length the content is supposed to be doing something
like this:

Net::HTTP.start("www.example.com", 80) {|res| h = res.head("/"); puts
h.to_yaml }

You can check that against the size of the downloaded file.

I think I might be able to help you more if you posted your download
code.

> thank ahead...

No problem.

Luke
This topic is locked and can not be replied to.