Forum: Ruby FileUtils.cp_r is slow?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Paul L. (Guest)
on 2009-01-14 19:47
(Received via mailing list)
I wrote a ruby script to backup files from my hard drive to my flash
drive.   For some reason, with one of my flash drives, copying files
with FileUtils.cp_r is about 250 times slower than if I drag and drop
the files from one drive to the other in Windows.  I'm suspecting a
buffer size problem.  Is there a way to control the buffer size cp_r
selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
am about to try writing my own, but someone must already have a
solution....)
Robert K. (Guest)
on 2009-01-14 22:55
(Received via mailing list)
On 14.01.2009 18:45, Paul L. wrote:
> I wrote a ruby script to backup files from my hard drive to my flash
> drive.   For some reason, with one of my flash drives, copying files
> with FileUtils.cp_r is about 250 times slower than if I drag and drop
> the files from one drive to the other in Windows.  I'm suspecting a
> buffer size problem.  Is there a way to control the buffer size cp_r
> selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
> am about to try writing my own, but someone must already have a
> solution....)

Could it be that Windows is doing the copy in the background while cp_r
doesn't?  After all, there is a reason why you must not simply plug off
an USB drive...

Kind regards

  robert
Robert K. (Guest)
on 2009-01-14 22:56
(Received via mailing list)
On 14.01.2009 18:45, Paul L. wrote:
> I wrote a ruby script to backup files from my hard drive to my flash
> drive.   For some reason, with one of my flash drives, copying files
> with FileUtils.cp_r is about 250 times slower than if I drag and drop
> the files from one drive to the other in Windows.  I'm suspecting a
> buffer size problem.  Is there a way to control the buffer size cp_r
> selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
> am about to try writing my own, but someone must already have a
> solution....)

Could it be that Windows is doing the copy in the background while cp_r
doesn't?  After all, there is a reason why you must not simply plug off
an USB drive...

Kind regards

  robert
Paul L. (Guest)
on 2009-01-17 18:13
(Received via mailing list)
I did some more testing on this.  I found that if I ran FileUtils.cp_r
or
FileUtils.cp from within irb, the speed was reasonable (maybe 30s to
copy
25MB to my flash drive.)  But, if I double-clicked my script to run it,
it
took about 30 minutes (roughly-- I didn't wait that long.)  I wrote my
own
copy method that opens the files in binary mode and uses a 10 MB buffer
size
(with the read and write methods) and speed dropped back down to 26s.
This
is still 6x slower than Windows drag and drop, but it is tolerable.

Here's my copy code, in case anyone else has this problem:
  def self.copy_file(from_file, to_file)
    buffer_size = 10*MB
    begin
      to_s = File.open(to_file, 'wb')
      File.open(from_file, 'rb') do |from_s|
        while(data = from_s.read(buffer_size))
          log("read #{data.length}")
          to_s.write(data)
        end
      end
    rescue
      log("Caught error: "+$!)
      to_s.close if to_s
      raise
    end
  end
Robert K. (Guest)
on 2009-01-19 09:35
(Received via mailing list)
2009/1/17 Paul L. <removed_email_address@domain.invalid>:

> On Wed, Jan 14, 2009 at 12:45 PM, Paul L. <removed_email_address@domain.invalid> wrote:
>
>> I wrote a ruby script to backup files from my hard drive to my flash
>> drive.   For some reason, with one of my flash drives, copying files
>> with FileUtils.cp_r is about 250 times slower than if I drag and drop
>> the files from one drive to the other in Windows.  I'm suspecting a
>> buffer size problem.  Is there a way to control the buffer size cp_r
>> selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
>> am about to try writing my own, but someone must already have a
>> solution....)

Just another idea: if your files reside on two different physical
disks you might get a benefit from separating reading and writing into
two different threads.

Kind regards

robert
Luc E. (Guest)
on 2009-01-19 10:42
(Received via mailing list)
For file backup I'm using rsync on a Linux system, its possible to start
this tool from Ruby.
   Rsync copy the difference between the files and is very fast.

   I don't now rsync exist in Windows (maybe Powershell), but its
possible
to run on windows via Cygwin.


   Luc.

On Mon, Jan 19, 2009 at 8:33 AM, Robert K.
This topic is locked and can not be replied to.