Hi,
I’ve been slowly hacking on my parallel recursive readdir()
implementation using a limited number of parallel processes, since as
you all know, Ruby threads are not truly parallel.
Anyway, I’m now getting wierd errors like the following:
Starting DRb on server
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 Exception DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/stoffj-test_17986
Exception `DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/stoffj-test_17986
counter.count=1
And it’s not really clear to my why I’m getting these errors, or how I
can trap them so I’m not bothered by them. The basic code uses both DRb
and Slave libraries to make the interprocess communication simpler, and
to have a central counting server to regulate the number of
sub-processes I’ll be using at any one point in time.
So, first off, here’s a full set of output of my code on a test
directory.
Thanks for any hints or suggestions. I think I need to use begin …
rescue … end blocks possibly, but it’s not clear where. Oh yeah, I’m
running this all on a CentOS 5.2 (RHEL5.2) Final system, against a small
350Mb test directory tree.
Thanks,
John
------------------------------- log ----------------------------------
$ ./readdir-drb.rb tmp
Starting DRb on server
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 Exception DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/stoffj-test_17986
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 counter.count=1 Threaded! Exception DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537506070_17986_17988_0_0.125356081811465
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537506070_17986_17988_0_0.125356081811465 Exception DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537506070_17986_17988_0_0.125356081811465
Threaded!
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 Exception DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 size = 372104960 size = 373477128 Exception NameError’ at /usr/lib/ruby/site_ruby/1.8/slave.rb:409 -
uninitialized constant Slave::DBb
Exception RuntimeError' at /usr/lib/ruby/site_ruby/1.8/slave.rb:519 - already shutdown exit (SystemExit) /usr/lib/ruby/site_ruby/1.8/slave.rb:265:in exit’
/usr/lib/ruby/site_ruby/1.8/slave.rb:265:in cling' /usr/lib/ruby/site_ruby/1.8/slave.rb:259:in call’
/usr/lib/ruby/site_ruby/1.8/slave.rb:259:in on_cut' /usr/lib/ruby/site_ruby/1.8/slave.rb:251:in initialize’
/usr/lib/ruby/site_ruby/1.8/slave.rb:251:in new' /usr/lib/ruby/site_ruby/1.8/slave.rb:251:in on_cut’
/usr/lib/ruby/site_ruby/1.8/slave.rb:265:in cling' /usr/lib/ruby/site_ruby/1.8/slave.rb:415:in initialize’
/usr/lib/ruby/site_ruby/1.8/slave.rb:593:in new' /usr/lib/ruby/site_ruby/1.8/slave.rb:593:in object’
./readdir-drb.rb:114:in readdir' ./readdir-drb.rb:86:in foreach’
./readdir-drb.rb:86:in `readdir’
./readdir-drb.rb:207
Total size: 373477128 B
Total size: 364723 KB
Total size: 356 MB
Total size: 0 GB
Exception Errno::ENOENT' at /usr/lib/ruby/1.8/fileutils.rb:1281 - No such file or directory - /tmp/slave_proc_537506070_17986_17988_0_0.125356081811465 Exception RuntimeError’ at /usr/lib/ruby/site_ruby/1.8/slave.rb:519 -
already shutdown
rt3.taec.com:~/src/Tools/philesight-20081120$ Exception NameError' at /usr/lib/ruby/site_ruby/1.8/slave.rb:409 - uninitialized constant Slave::DBb Exception Errno::ENOENT’ at /usr/lib/ruby/1.8/fileutils.rb:1281 - No
such file or directory -
/tmp/slave_proc_537512250_17988_17989_0_0.20115187038118
Exception IOError' at /usr/lib/ruby/1.8/drb/unix.rb:92 - stream closed Exception Errno::EBADF’ at /usr/lib/ruby/1.8/drb/unix.rb:92 - Bad file
descriptor - ///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118
/usr/lib/ruby/1.8/drb/unix.rb:92:in close': Bad file descriptor - ///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 (Errno::EBADF) from /usr/lib/ruby/1.8/drb/unix.rb:92:in close’
from /usr/lib/ruby/1.8/drb/drb.rb:1433:in run' from /usr/lib/ruby/1.8/drb/drb.rb:1427:in start’
from /usr/lib/ruby/1.8/drb/drb.rb:1427:in run' from /usr/lib/ruby/1.8/drb/drb.rb:1347:in initialize’
from /usr/lib/ruby/1.8/drb/drb.rb:1627:in new' from /usr/lib/ruby/1.8/drb/drb.rb:1627:in start_service’
from /usr/lib/ruby/site_ruby/1.8/slave.rb:396:in initialize' ... 28 levels... from ./readdir-drb.rb:114:in readdir’
from ./readdir-drb.rb:86:in foreach' from ./readdir-drb.rb:86:in readdir’
from ./readdir-drb.rb:207
--------------------------- source
And here’s my source code, excuse my lack of Ruby knowledge, I’m a newb
to Ruby, though not to programming. Hopefully that is shown. :]
#!/usr/bin/ruby
require ‘getoptlong’
require ‘thread’
require ‘slave’
require ‘drb’
require ‘drb/unix’
$VERSION = “v1.0”;
$max_slaves = 3
$count = 50
Local Socket - need to worry about multiple runs of this script, so
add on
unique PID.
$URI = “drbunix:///tmp/stoffj-test_” + Process.pid.to_s
slave_cnt_mutex = Mutex.new
opts = GetoptLong.new(
[ “–help”, “-h”, GetoptLong::NO_ARGUMENT],
[ “–kids”, “-k”, GetoptLong::REQUIRED_ARGUMENT]
)
#---------------------------------------------------------------------
class Counter
def initialize(max=1)
@slaves = []
@max = max
@count = 1
@count_mutex = Mutex.new
end
def count
@count
end
def max
@max
end
Increment the count of slaves, returning 1 if incremented, 0 if not.
def increment
ok = nil
@count_mutex.synchronize do
if (@count < @max) then
@count += 1
ok = 1
end
end
ok
end
Decrement the count of slaves
def decrement
@count_mutex.synchronize do
if (@count > 1) then
@count -= 1
end
end
@count
end
end
#---------------------------------------------------------------------
class ReadDir
def initialize(server)
@server = server
# Slave pool for this level of readdir() recursion.
@kids = []
end
def readdir(dir)
#puts "readdir(#{dir})"
size_file = {}
size_dir = {}
size_total = 0
# Traverse the directory and collect the size of all files and
# directories
begin
Dir.foreach(dir) do |f|
#print " #{f},"
if(f != "." && f != "..") then
f_full = addpath(dir, f)
stat = File.lstat(f_full)
if(!stat.symlink?) then
if(stat.file?) then
#puts " File: #{f}"
size = File.size(f_full)
size_file[f] = size
size_total += size
end
if(stat.directory?) then
#puts "DIR= #{f}"
if (@server.max <= 1) then
puts " no threads."
size = readdir(f_full)
if (size > 0) then
size_dir[f] = size
size_total += size
end
else
ok = @server.increment
if (ok)
puts " Threaded!"
@kids << Slave.object(:async => true) {
size = readdir(f_full)
puts "size = #{size}"
# Duh... return the size from the slave properly
size
}
else
#puts " no free threads, do anyway"
size = readdir(f_full)
if(size > 0) then
size_dir[f] = size
size_total += size
end
end
end
end
end
end
end
end
@kids.each { |kid|
size_total += kid.value
}
#Puts "Dir: #{dir} = #{size_total}"
return size_total
end
end
#---------------------------------------------------------------------
Read a directory and add to the database; this function is recursive
for sub-directories
#---------------------------------------------------------------------
def usage
puts
puts “usage: readdir-drb [–kids NUM] ”
puts " defaults to #{$max_kids} children"
puts
puts " version: #{$version}"
puts
end
#---------------------------------------------------------------------
def addpath(a, b)
return a + b if(a =~ //$/)
return a + “/” + b
end
#---------------------------------------------------------------------
Main
#---------------------------------------------------------------------
$DEBUG = true
opts.each do |opt,arg|
case opt
when “–kids”
$max_slaves = arg.to_i
else
usage
exit
end
end
if ARGV.length != 1
puts “Missing dir argument (try --help)”
exit 0
end
dir = ARGV.shift
Start the DRb service.
puts “Starting DRb on server”
DRb.start_service $URI, Counter.new($max_slaves)
Child
Fire up the first slave process which will do the work of readdir()
DRb.start_service
counter = DRbObject.new_with_uri $URI
puts “counter.count=#{counter.count}”
Fire up a new Kid Class readdir.
kid = ReadDir.new(counter)
Now let’s try to do a recursive readdir() algorith with threads.
size = kid.readdir(dir)
sizekb = size / 1024;
sizemb = sizekb / 1024;
sizegb = sizemb / 1024;
puts “”
puts “Total size: #{size} B”
puts “Total size: #{sizekb} KB”
puts “Total size: #{sizemb} MB”
puts “Total size: #{sizegb} GB”