Forum: Ruby FileUtils.cp_r is slow?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
7b74ccf93dde7dd2ac5d2980d14fdc7b?d=identicon&s=25 Paul Lynch (plynchnlm)
on 2009-01-14 18:47
(Received via mailing list)
I wrote a ruby script to backup files from my hard drive to my flash
drive.   For some reason, with one of my flash drives, copying files
with FileUtils.cp_r is about 250 times slower than if I drag and drop
the files from one drive to the other in Windows.  I'm suspecting a
buffer size problem.  Is there a way to control the buffer size cp_r
selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
am about to try writing my own, but someone must already have a
solution....)
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2009-01-14 21:55
(Received via mailing list)
On 14.01.2009 18:45, Paul Lynch wrote:
> I wrote a ruby script to backup files from my hard drive to my flash
> drive.   For some reason, with one of my flash drives, copying files
> with FileUtils.cp_r is about 250 times slower than if I drag and drop
> the files from one drive to the other in Windows.  I'm suspecting a
> buffer size problem.  Is there a way to control the buffer size cp_r
> selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
> am about to try writing my own, but someone must already have a
> solution....)

Could it be that Windows is doing the copy in the background while cp_r
doesn't?  After all, there is a reason why you must not simply plug off
an USB drive...

Kind regards

  robert
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2009-01-14 21:56
(Received via mailing list)
On 14.01.2009 18:45, Paul Lynch wrote:
> I wrote a ruby script to backup files from my hard drive to my flash
> drive.   For some reason, with one of my flash drives, copying files
> with FileUtils.cp_r is about 250 times slower than if I drag and drop
> the files from one drive to the other in Windows.  I'm suspecting a
> buffer size problem.  Is there a way to control the buffer size cp_r
> selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
> am about to try writing my own, but someone must already have a
> solution....)

Could it be that Windows is doing the copy in the background while cp_r
doesn't?  After all, there is a reason why you must not simply plug off
an USB drive...

Kind regards

  robert
7b74ccf93dde7dd2ac5d2980d14fdc7b?d=identicon&s=25 Paul Lynch (plynchnlm)
on 2009-01-17 17:13
(Received via mailing list)
I did some more testing on this.  I found that if I ran FileUtils.cp_r
or
FileUtils.cp from within irb, the speed was reasonable (maybe 30s to
copy
25MB to my flash drive.)  But, if I double-clicked my script to run it,
it
took about 30 minutes (roughly-- I didn't wait that long.)  I wrote my
own
copy method that opens the files in binary mode and uses a 10 MB buffer
size
(with the read and write methods) and speed dropped back down to 26s.
This
is still 6x slower than Windows drag and drop, but it is tolerable.

Here's my copy code, in case anyone else has this problem:
  def self.copy_file(from_file, to_file)
    buffer_size = 10*MB
    begin
      to_s = File.open(to_file, 'wb')
      File.open(from_file, 'rb') do |from_s|
        while(data = from_s.read(buffer_size))
          log("read #{data.length}")
          to_s.write(data)
        end
      end
    rescue
      log("Caught error: "+$!)
      to_s.close if to_s
      raise
    end
  end
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2009-01-19 08:35
(Received via mailing list)
2009/1/17 Paul Lynch <plynchnlm@gmail.com>:

> On Wed, Jan 14, 2009 at 12:45 PM, Paul Lynch <plynchnlm@gmail.com> wrote:
>
>> I wrote a ruby script to backup files from my hard drive to my flash
>> drive.   For some reason, with one of my flash drives, copying files
>> with FileUtils.cp_r is about 250 times slower than if I drag and drop
>> the files from one drive to the other in Windows.  I'm suspecting a
>> buffer size problem.  Is there a way to control the buffer size cp_r
>> selects?  If not, what alternatives are there to FileUtils.cp_r?  (I
>> am about to try writing my own, but someone must already have a
>> solution....)

Just another idea: if your files reside on two different physical
disks you might get a benefit from separating reading and writing into
two different threads.

Kind regards

robert
F7d41a4824622680a50f8ac6eac8336f?d=identicon&s=25 Luc Evers (lucevers)
on 2009-01-19 09:42
(Received via mailing list)
For file backup I'm using rsync on a Linux system, its possible to start
this tool from Ruby.
   Rsync copy the difference between the files and is very fast.

   I don't now rsync exist in Windows (maybe Powershell), but its
possible
to run on windows via Cygwin.


   Luc.

On Mon, Jan 19, 2009 at 8:33 AM, Robert Klemme
This topic is locked and can not be replied to.