Forum: Ruby slow file access with mapped drives ?!

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Rebhan, Gilbert (Guest)
on 2007-05-14 13:39
(Received via mailing list)
Hi,

running  ruby 1.8.4 (2006-04-14) [i386-mswin32]
on Windows2000

i have a script that scans some directories for given filename patterns
and does string replacement in those found files.

it works fine, but when it comes to work with files on a mapped drive
the script hangs for about 20secs, afterwards does his work and all OK.

i tried with =

V:/path/to/files
and also with
//servername/c$/path/to/files

and it's both the same, no difference.

Any ideas how to speed up the file access ?
Are there general problems with unc path and ruby ?


Gilbert
Rebhan, Gilbert (Guest)
on 2007-05-14 14:05
(Received via mailing list)
Hi,

after some irb it shows the bottleneck lies in
the Dir.glob method

i do something like =

config['targetdirs'].each do |dir|

   Dir.chdir("//servername/c$/targetdir")
   Dir.glob("**/"<<config['targetfilepattern']).each do |file|
   ...
  end
...
end


any alternatives to Dir.glob ?


Regards, Gilbert
Robert K. (Guest)
on 2007-05-14 14:20
(Received via mailing list)
On 14.05.2007 12:05, Rebhan, Gilbert wrote:
>    Dir.chdir("//servername/c$/targetdir")
>    Dir.glob("**/"<<config['targetfilepattern']).each do |file|

Why don't you just do

Dir.glob("//servername/c$/targetdir/**/"<<config['targetfilepattern']).each
do |file|

Btw, I don't see any reference to block variable "dir" is this on
purpose?

>    ...
>   end
> ..
> end
>
>
> any alternatives to Dir.glob ?

Find.find - but don't expect that it's faster because the slowness lies
in file system accesses.  Network drives are inherently slower than
local disks.

Kind regards

  robert
Rebhan, Gilbert (Guest)
on 2007-05-14 16:19
(Received via mailing list)
Hi, Robert
Robert K. (Guest)
on 2007-05-14 16:46
(Received via mailing list)
On 14.05.2007 14:18, Rebhan, Gilbert wrote:
> Why don't you just do
> - Y:/tempwork
> - T:/rubytest
> - //wvp10175/c$/SCM_Server

???  The code you posted had no dynamic piece there.  Did you maybe mean
to write

Dir.chdir("//servername/c$/#{dir}")

> ..end
That was not in the code you posted.

>> any alternatives to Dir.glob ?
>
> /*
> Find.find - but don't expect that it's faster because the slowness lies
> in file system accesses.  Network drives are inherently slower than
> local disks.
> */
>
> it runs in almost half the time, much quicker :-)

... which might be due to the fact that your new code doesn't do the
chdir!  You changed multiple parameters at the same time so it's hard to
attribute the perceived performance change to any one parameter.

> thanks for the pointer !! , now i have =
>
> config['targetdirs'].each do |dir|
>     Find.find(dir) do |f|
>       if f =~ /#{config['targetfilepattern']}/
>       puts "... processing "<<f
>       filesed!(config['replacefrom'],s,f)
>       end
>     end
>   end

Please make sure you post the code you are actually testing - otherwise
everybody else will have a hard time understanding what's going on let
alone come up with helpful replies.

Regards

  robert
Rebhan, Gilbert (Guest)
on 2007-05-14 16:57
(Received via mailing list)
Hi,
Robert K. (Guest)
on 2007-05-14 17:55
(Received via mailing list)
On 14.05.2007 14:56, Rebhan, Gilbert wrote:
  > -----Original Message-----
>
> i don't want to bother with bad code, but here is the whole thing,
> for you to understand what i try to achieve =

I was referring to the fact that you present one bit of code and in your
answer to my posting you present another bit of code as if *that* was
the original code.

However, there are quite a few things to say about this bit of code.

> q.default = "#{ENV["USERNAME"]}"
> q.echo = true}
> cvspass = ask("Enter password:       ") { |q| q.echo = '*' }
> puts "\n\n"
>
> config['cvsrepos'].each {|x|
> puts "Login CVS Repository >> #{x} ..."
> IO.popen("#{config['CVSEXE']} -d
> :pserver:#{cvsuser}:#{cvspass}@cvsprod:d:/cvsrepos/#{x} login")
> }

Why do you use popen if you do not read the pipe?  *If* you use popen
you should make sure the pipe is read from - even if you ignore what you
find because otherwise the other process might get blocked.

>       }
Rather use the non block form of gsub unless you want to make sure that
there is no metacharacter interpretation going on.  (Even in that case
I'd probably prefer the non block form with a modified replaceString.)

>   end
>   tmpFile.close
>   srcFile .close

Rather use the block form of File.open which is much safer.

>
>   s=config['replaceto1']<<"#{@cvsreg}"<<config['replaceto2']

Why do you use "#{@cvsreg}" instead of plain @cvsreg or @cvsreg.to_s?

>   config['targetdirs'].each do |dir|
>     puts "\n\n"
>     Find.find(dir) do |f|
>       if f =~ /#{config['targetfilepattern']}/

You should pull out this regexp compilation from both loops for more
efficiency.

>       puts "... processing "<<f
>       filesed!(config['replacefrom'],s,f)

Indentation.

>       end
>     end
>   end
>   puts "\n\nDone !!"
>   sleep 1

Why the sleep?

>   else
>   puts "\n\nDone !!"
>     sleep 1

Why the sleep?

> - foo
>
> targetfilepattern: Scm\w+\.xml$
>
> replacefrom: "<Passwort>.*</Passwort>"

This pattern will kill you if there are two sections with
<Passwort></Passwort> in the file.  You should at least use the
reluctanct qualifier.

> replaceto1: "<Passwort><![CDATA["
> replaceto2: "]></Passwort>"

This cries for using a regexp group.  Also filesed is pretty inflexible.
  I'd rather do something like this:

def file_replace(file, tmp = file + Time.now.usec.to_s)
   File.open(tmp, "w") do |out|
     File.open(file) do |inf|
       inf.each_line {|line| out.puts(yield(line)) }
     end
   end
   File.mv(tmp, file, :force => true)
end

Now you can do arbitrary line based replacements like

file_replace "foo.txt" do |line|
   "# " << line
end

> ----yaml----
>
> the filesed method is not by me, found it somewhere, don't remember
>
> Regards, Gilbert

Regards

  robert
Rebhan, Gilbert (Guest)
on 2007-05-15 16:49
(Received via mailing list)
Hi,
ruby carillo (Guest)
on 2007-05-16 23:32
(Received via mailing list)
wala ba kayong tulog?ang galing
This topic is locked and can not be replied to.