Do arrays use up alot of memory space?

I just started ruby not too long ago and I’m really new to the language.
But I came out with something; to copy files from path names to path
names using each respecive array elements.


$options=
[“2008*”, “2008*”, “700*”, “2008*”, “2008*”]

$source=
%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

$dest=
%w[U:/test_1/ U:/dest1/ U:/dest2/ U:/dest3/ U:/dest4/]

while i<=j && i1<=j1 && i2<=j2

Dir.chdir($source[i])
print "\nSource: " + Dir.getwd + “\t\n”
print “Dest: " + $dest[i1] + “\n”
print “Options: " + $options[i2] +”\n”
FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
print “File Mov Test:Success”
i+=1
i1+=1
i2+=1
end


Apparently, when moving large files (i.e file size 50mb) it takes
relatively long as when i use ROBOCOPY (a robust copying software by
MSServer), but I cant use robocopy because it has it’s limitations,hence
i used ruby. As I will be running the ruby program on a server, I cant
afford to use too much memory while transferring files from one folder
to another as the server is used for more impt tasks like running impt
applications in the office.

So is there anything that i could do to quicken the copying process
without compromising on the ability to have different path names and
options for copying files? (i understand that arrays do take up more
memory)
Thanks in advance. =)

Clement Ow wrote:


$options=
[“2008*”, “2008*”, “700*”, “2008*”, “2008*”]

$source=
%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

$dest=
%w[U:/test_1/ U:/dest1/ U:/dest2/ U:/dest3/ U:/dest4/]

You have 3 arrays with 5 short strings in them. A character in a string
takes up 1 byte of memory. I don’t know exactly what the overhead for
an array is, but it is comparatively tiny–although you never know with
ruby. I would guess all your arrays would take up less than 1,000
bytes or 1 kB in memory. That’s tiny.

while i<=j && i1<=j1 && i2<=j2

Dir.chdir($source[i])
print "\nSource: " + Dir.getwd + “\t\n”
print “Dest: " + $dest[i1] + “\n”
print “Options: " + $options[i2] +”\n”
FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
print “File Mov Test:Success”
i+=1
i1+=1
i2+=1
end


Apparently, when moving large files (i.e file size 50mb) it takes
relatively long as when i use ROBOCOPY (a robust copying software by
MSServer), but I cant use robocopy because it has it’s limitations,hence
i used ruby. As I will be running the ruby program on a server, I cant
afford to use too much memory while transferring files from one folder
to another as the server is used for more impt tasks like running impt
applications in the office.

So is there anything that i could do to quicken the copying process
without compromising on the ability to have different path names and
options for copying files? (i understand that arrays do take up more
memory)
Thanks in advance. =)

On Mon, Apr 7, 2008 at 6:29 AM, 7stud – [email protected] wrote:

U:/movtest/new1]
takes up 1 byte of memory. I don’t know exactly what the overhead for
an array is, but it is comparatively tiny–although you never know with
ruby.
A well judged statement again…

For Ruby 1.9 the overhead of an array is one long and 4 VALUES, I am
not a VM Guru but it would surprise me a lot if a VALUE used more than
4 bytes given their semantic meanings shared flags and klass, the
fourth being a pointer to the content.
I am sure that someone will correct me if I am wrong but I estimate
the overhead of less than 21 bytes.

I would guess all your arrays would take up less than 1,000

yes that seems indeed a save guess…
HTH
Robert

http://ruby-smalltalk.blogspot.com/


Whereof one cannot speak, thereof one must be silent.
Ludwig Wittgenstein

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

|
| Thanks for the prompt reply, 7student and Robert =) But while
| transferring files, it does take a significant amount time though, so
| the primary reason does not lie in the arrray? If so, is there any way I
| can quicken the process of transferring files?

It probably isn’t the array. You probably are IO bound (i.e. speed of
the hard drive(s)/network connection).

Are you copying the files on the same disk, or from disk to disk, and on
different IDE/SATA channels?

If you are copying across a network, how good is the connection?


Phillip G.
Twitter: twitter.com/cynicalryan

:zorkmid: /zork’mid/ n. The canonical unit of currency in
~ hacker-written games. This originated in {zork} but has spread
~ to {nethack} and is referred to in several other games.
~ – The AI Hackers Dictionary
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf63s8ACgkQbtAgaoJTgL8IUQCdEx+pcVfXapfAMdCszaz8D/bx
7zQAnjL4OJxQ+/Z3MMQ4tym7AipPLKfs
=dnhS
-----END PGP SIGNATURE-----

Robert D. wrote:

On Mon, Apr 7, 2008 at 6:29 AM, 7stud – [email protected] wrote:

U:/movtest/new1]
takes up 1 byte of memory. I don’t know exactly what the overhead for
an array is, but it is comparatively tiny–although you never know with
ruby.
A well judged statement again…

For Ruby 1.9 the overhead of an array is one long and 4 VALUES, I am
not a VM Guru but it would surprise me a lot if a VALUE used more than
4 bytes given their semantic meanings shared flags and klass, the
fourth being a pointer to the content.
I am sure that someone will correct me if I am wrong but I estimate
the overhead of less than 21 bytes.

I would guess all your arrays would take up less than 1,000

yes that seems indeed a save guess…
HTH
Robert

http://ruby-smalltalk.blogspot.com/


Whereof one cannot speak, thereof one must be silent.
Ludwig Wittgenstein

Thanks for the prompt reply, 7student and Robert =) But while
transferring files, it does take a significant amount time though, so
the primary reason does not lie in the arrray? If so, is there any way I
can quicken the process of transferring files?

From: Clement Ow [mailto:[email protected]]

$options=

you may remove the “$” sign :wink:

[“2008*”, “2008*”, “700*”, “2008*”, “2008*”]

and i don’t think these are options but really file/dir
patterns/selections in your case…

$source=

%w[C:/movtest/testing

C:/movtest/testing/new

when you recurse /new will be re-traversed by /testing and
/testing/new. You can remove /testing/new since it has the same option
with /testing or, it would be fine if you do not need to recourse at
all.

U:/movtest/source

U:/movtest/new

U:/movtest/new1]

$dest=

%w[U:/test_1/

U:/dest1/

U:/dest2/

U:/dest3/

U:/dest4/]

while i<=j && i1<=j1 && i2<=j2

hmmm, you just created 3 parallel arrays. Either you can create one
multi-dimensional or a hash.

Dir.chdir($source[i])

you may not need this

print "\nSource: " + Dir.getwd + “\t\n”

print "Dest: " + $dest[i1] + “\n”

print “Options: " + $options[i2] +”\n"

FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]

print “File Mov Test:Success”

i+=1

i1+=1

i2+=1

if you modify your code, you may not need those increments :wink:

end

------------------

Apparently, when moving large files (i.e file size 50mb) it takes

relatively long as when i use ROBOCOPY (a robust copying software by

MSServer),

robocopy seems to do stat on the filetransfer speed to get the optimum
chunk to transfer. You can do something like that in ruby, but you’ll
have to work harder (since your code/logic will get a bit longer)

but I cant use robocopy because it has it’s limitations

it’s not perfect, and w ruby, you can beat it :slight_smile:

i used ruby. As I will be running the ruby program on a server, I cant

afford to use too much memory while transferring files from one folder

to another as the server is used for more impt tasks like running impt

applications in the office.

So is there anything that i could do to quicken the copying process

without compromising on the ability to have different path names and

options for copying files? (i understand that arrays do take up more

memory)

obviously, no problems with array here since it’s very small.

So, if i were to tackle this, i’d first do, (note, this is ruby
pseudocode, ergo untested since i do not have time and space to test it
now…)

source=%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

dest=%w[U:/test_1/ U:/dest1/ U:/dest2/ U:/dest3/ U:/dest4/]

selections=[“2008*”, “2008*”, “700*”, “2008*”, “2008*”]

here i combine those 3 arrays into 1 multi-dim array

sd_a=source.zip(dest,options)

sd_a.each do |sd|
source, destination, selections = sd
src = File.join source,selections
puts “Source: #{src}”
puts “Dest: #{destination}”

i comment the 2 lines below for you to choose bw recurse or not

again, if you recurse, check for directory overlaps

FileUtils.cp src, destination

FileUtils.cp_r src, destination

puts “File Copy Test:Success”
end

END

note the absence of the increments wc makes the code cleaner and
clearer. And if we have a clear code, it makes it easier for us to
monitor/debug and optimize if needed :slight_smile:

kind regards -botp

From: [email protected]

> # here i combine those 3 arrays into 1 multi-dim array

> sd_a=source.zip(dest,options)

hmmm, would combining the arrays be feasible if i were to add on more

values to the either of the arrays?

i zipped it because,

  1. i was too lazy to retype it again
  2. i knew it would work since your arrays are one-to-one on each other

> sd_a.each do |sd|

> source, destination, selections = sd

> src = File.join source,selections

> puts “Source: #{src}”

> puts “Dest: #{destination}”

>

> # i comment the 2 lines below for you to choose bw recurse or not

> # again, if you recurse, check for directory overlaps

>

> # FileUtils.cp src, destination

> # FileUtils.cp_r src, destination

This command runs an error because the value of src is

“C:/movtest/test/2008*” and it doesnt recognise this wildcard but

instead looks for the file which has the name 2008*.

so i tried to chang it alil:

no problem, just do

FileUtils.cp Dir.glob(src), destination

that should work because i just tested it here :slight_smile:

kind regards -botp

Peña, Botp wrote:

From: [email protected]

> # here i combine those 3 arrays into 1 multi-dim array

> sd_a=source.zip(dest,options)

hmmm, would combining the arrays be feasible if i were to add on more

values to the either of the arrays?

i zipped it because,

  1. i was too lazy to retype it again
  2. i knew it would work since your arrays are one-to-one on each other

> sd_a.each do |sd|

> source, destination, selections = sd

> src = File.join source,selections

> puts “Source: #{src}”

> puts “Dest: #{destination}”

>

> # i comment the 2 lines below for you to choose bw recurse or not

> # again, if you recurse, check for directory overlaps

>

> # FileUtils.cp src, destination

> # FileUtils.cp_r src, destination

This command runs an error because the value of src is

“C:/movtest/test/2008*” and it doesnt recognise this wildcard but

instead looks for the file which has the name 2008*.

so i tried to chang it alil:

no problem, just do

FileUtils.cp Dir.glob(src), destination

that should work because i just tested it here :slight_smile:

kind regards -botp

it works :wink:
But one issue, when i changed to FileUtils.mv Dir.glob(src), destination
because i needed to move the files, it shows a big error msg:
c:/ruby/lib/ruby/1.8/fileutils.rb:505:in rename': Permission denied - C:/movtes t/testing/20080321 or U:/test_1/20080321 (Errno::EACCES) from c:/ruby/lib/ruby/1.8/fileutils.rb:505:inmove’
from c:/ruby/lib/ruby/1.8/fileutils.rb:1395:in
fu_each_src_dest' from c:/ruby/lib/ruby/1.8/fileutils.rb:1404:infu_each_src_dest0’
from c:/ruby/lib/ruby/1.8/fileutils.rb:1402:in each' from c:/ruby/lib/ruby/1.8/fileutils.rb:1402:infu_each_src_dest0’
from c:/ruby/lib/ruby/1.8/fileutils.rb:1393:in
fu_each_src_dest' from c:/ruby/lib/ruby/1.8/fileutils.rb:494:inmove’
from testing.rb:42
from testing.rb:32:in `each’
from testing.rb:32

It seems like some problem with the access rights… Any advice?
I’m still quite new to ruby so ya, but I’m really thankful for the help
so far though. :wink:

Thanks Botp and Philip for the help! :wink: anyway, ans philip, the network
i’m running is quite optimal though not really sure about the speeds, as
I’m running this program in a company with alot of servers dealing with
loads of jobs a day :slight_smile:

Peña, Botp wrote:

From: Clement Ow [mailto:[email protected]]

$options=

you may remove the “$” sign :wink:

i use the “$” sign because I have a config file that holds all my
values, but I didnt include it for the purpose of this example :wink:

U:/movtest/source

U:/movtest/new

U:/movtest/new1]

$dest=

%w[U:/test_1/

U:/dest1/

U:/dest2/

U:/dest3/

U:/dest4/]

while i<=j && i1<=j1 && i2<=j2

hmmm, you just created 3 parallel arrays. Either you can create one
multi-dimensional or a hash.

Dir.chdir($source[i])

you may not need this

print "\nSource: " + Dir.getwd + “\t\n”

print "Dest: " + $dest[i1] + “\n”

print “Options: " + $options[i2] +”\n"

FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]

print “File Mov Test:Success”

i+=1

i1+=1

i2+=1

if you modify your code, you may not need those increments :wink:

end

------------------

Apparently, when moving large files (i.e file size 50mb) it takes

relatively long as when i use ROBOCOPY (a robust copying software by

MSServer),

robocopy seems to do stat on the filetransfer speed to get the optimum
chunk to transfer. You can do something like that in ruby, but you’ll
have to work harder (since your code/logic will get a bit longer)

but I cant use robocopy because it has it’s limitations

it’s not perfect, and w ruby, you can beat it :slight_smile:
yea, limitations being that I cant have exceptions while selecting a
serious of folders. EG. I wanna select files and folders that start with
2008 but I want to keep the last day of the month untouched.(still yet
to figure out this part though)

So, if i were to tackle this, i’d first do, (note, this is ruby
pseudocode, ergo untested since i do not have time and space to test it
now…)

source=%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

dest=%w[U:/test_1/ U:/dest1/ U:/dest2/ U:/dest3/ U:/dest4/]

selections=[“2008*”, “2008*”, “700*”, “2008*”, “2008*”]

here i combine those 3 arrays into 1 multi-dim array

sd_a=source.zip(dest,options)

hmmm, would combining the arrays be feasible if i were to add on more
values to the either of the arrays?

sd_a.each do |sd|
source, destination, selections = sd
src = File.join source,selections
puts “Source: #{src}”
puts “Dest: #{destination}”

i comment the 2 lines below for you to choose bw recurse or not

again, if you recurse, check for directory overlaps

FileUtils.cp src, destination

FileUtils.cp_r src, destination

This command runs an error because the value of src is
“C:/movtest/test/2008*” and it doesnt recognise this wildcard but
instead looks for the file which has the name 2008*.
so i tried to chang it alil:

Dir.chdir(source)
FileUtils.cp_r Dir.glob(‘2008*’), source, destination

But still runs an error here showing :
Source: C:/movtest/testing/2008*
Dest: U:/test_1/
C:/movtest/testing
c:/ruby/lib/ruby/1.8/fileutils.rb:1438:in delete': can't convert Symbol into St ring (TypeError) from c:/ruby/lib/ruby/1.8/fileutils.rb:1438:infu_check_options’
from c:/ruby/lib/ruby/1.8/fileutils.rb:1437:in each' from c:/ruby/lib/ruby/1.8/fileutils.rb:1437:infu_check_options’
from c:/ruby/lib/ruby/1.8/fileutils.rb:418:in cp_r' from testing.rb:44 from testing.rb:32:ineach’
from testing.rb:32

I have tried ways and means to try correct it but to no avail. Is there
anything majorly wrong with my code?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

|
| It seems like some problem with the access rights… Any advice?
| I’m still quite new to ruby so ya, but I’m really thankful for the help
| so far though. :wink:

Yeah, contact your sysadmin, if you have write permission. :stuck_out_tongue:

If you are the admin (and executing the script with admin rights), make
sure no other programs are using the directory/file (file locks).

That means, you can’t have the script in the directory that you move. :wink:

(Sounds obvious, but it happens to the best…)


Phillip G.
Twitter: twitter.com/cynicalryan

You were ever good at sudden commendations.
~ – William Shakespeare (1564-1616), King Henry VIII
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf8NYIACgkQbtAgaoJTgL+/8wCfbBnRMgzG/oV0pkaAa67EmTLX
eHcAn2db9lJ0bNi6Iti/NUQU71lIn3w6
=SjNH
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

| The script is residing in C:\rubytest which is not in any of the path
| names above.
| And I’m quite sure it’s not due to the options i.e Verbose, force or
| noop.
| Tried it with the force option (returns no error) but no file move was
| done.
| hmmm, which makes it more puzzling…

Are you sure there are no open file handles? FileMon should reveal this:
http://technet.microsoft.com/en-us/sysinternals/bb896642.aspx

If there are no file handles shown, check if you can move the
directories/files via explorer.


Phillip G.
Twitter: twitter.com/cynicalryan

[Abstract art is] a product of the untalented, sold by the unprincipled
~ to the utterly bewildered.
~ – Al Capp
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf8OPUACgkQbtAgaoJTgL9anQCbBvBvvBkraRKzy5flCuycYOeS
D3sAn16d9VsMJzZx6wm8TkVfC4BiIS3k
=24fF
-----END PGP SIGNATURE-----

Phillip G. wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

|
| It seems like some problem with the access rights… Any advice?
| I’m still quite new to ruby so ya, but I’m really thankful for the help
| so far though. :wink:

Yeah, contact your sysadmin, if you have write permission. :stuck_out_tongue:

If you are the admin (and executing the script with admin rights), make
sure no other programs are using the directory/file (file locks).
Im running the script with admin rights.

That means, you can’t have the script in the directory that you move. :wink:
The script is residing in C:\rubytest which is not in any of the path
names above.
And I’m quite sure it’s not due to the options i.e Verbose, force or
noop.
Tried it with the force option (returns no error) but no file move was
done.
hmmm, which makes it more puzzling…

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:
| Phillip G. wrote:
|
|> Are you sure there are no open file handles? FileMon should reveal this:
|> http://technet.microsoft.com/en-us/sysinternals/bb896642.aspx
|>
|> If there are no file handles shown, check if you can move the
|> directories/files via explorer.
|
| Hmmm, i dont quite understand about the open file handles…
| Do you mean if the folder is being in use without the knowledge of the
| user which in this case is myslelf? If so, I dont see the files being
| used by other programs whatsoever. I even closed explorer and run the
| script, but to no avail…

There still could be something using the folder or files, without your
knowledge.

In the following (broad) scenarios, Windows blocks access to files:

A) You don’t have permission to change these files. Not even the
Administrator account can change all files in Windows, usually deep
within Windows.

However, access control can change that (a normal user, for example,
doesn’t have write access to %programfiles%, where applications install
themselves by default, or even using EFS.

B) The file is still in use by a program. Any program. Either locally or
via a network, Windows cannot delete that file.

You can use FileMon to investigate which files are currently open. Just
closing Explorer doesn’t help (a shell can still be pointing there, a
Word document could still be open, a backup tool or anti virus
application is using it…)


Phillip G.
Twitter: twitter.com/cynicalryan

~ “Until you stalk and overrun, you can’t devour anyone.”
~ — Hobbes
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf8RmYACgkQbtAgaoJTgL+/QQCZAXynHuARN160ZnnJFfY8vKUb
Fo0AnjgNFH4wjtzr8kJFFDzV+soYe3fT
=n/qR
-----END PGP SIGNATURE-----

On Mon, Apr 7, 2008 at 8:36 PM, Clement Ow
[email protected] wrote:

Thanks for the prompt reply, 7student and Robert =) But while
transferring files, it does take a significant amount time though, so
the primary reason does not lie in the arrray?

Yes, that’s correct.

If so, is there any way I
can quicken the process of transferring files?

I don’t think this is a Ruby problem. Possibly network and/or program?

Though it has little, if anything, to do with your original post; is
there a good reason to include a sub-directory in the same recursive
listing (I’m looking at C:/movtest/testing and
C:/movtest/testing/new)? Just a side question.

Todd

Todd B. wrote:

On Mon, Apr 7, 2008 at 8:36 PM, Clement Ow
[email protected] wrote:

Thanks for the prompt reply, 7student and Robert =) But while
transferring files, it does take a significant amount time though, so
the primary reason does not lie in the arrray?

Yes, that’s correct.

If so, is there any way I
can quicken the process of transferring files?

I don’t think this is a Ruby problem. Possibly network and/or program?

Though it has little, if anything, to do with your original post; is
there a good reason to include a sub-directory in the same recursive
listing (I’m looking at C:/movtest/testing and
C:/movtest/testing/new)? Just a side question.
This is just for testing purposes.

Apparently, the script fails to run only when I try to move files.
Copying files using the cp_r command seems to be working fine though.
Reckon it has got to do with the ‘Read Only’ option when i click on the
folder’s properties. But then again, when i tried to check randomly any
folder on my local disk, it all shows that it’s ‘Read Only’ and that I
have access permission to modify anything. (i cant uncheck the ‘Read
Only’ option though)hmmm, does this mean i cant delete folders and files
if this still happens? Pretty wierd since im the administrator here…

Phillip G. wrote:

Are you sure there are no open file handles? FileMon should reveal this:
http://technet.microsoft.com/en-us/sysinternals/bb896642.aspx

If there are no file handles shown, check if you can move the
directories/files via explorer.

Hmmm, i dont quite understand about the open file handles…
Do you mean if the folder is being in use without the knowledge of the
user which in this case is myslelf? If so, I dont see the files being
used by other programs whatsoever. I even closed explorer and run the
script, but to no avail…

-in desperate need for help

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs