Quieting DRb messages in fork() code

Hi,

I’ve got some code which seems to work well, but the warning messages
are driving me insane! I can’t use this code if it’s always going to
spew warnings like these on a CentOS 5.2 system with ruby 1.8.5
(2006-08-25) [i386-linux] as the environment. I’m also using
slave-1.2.1 from http://www.codeforpeople.com
as well.

$ ./readdir-slave-only.rb --kids 4 tmp
Starting Slave Counter
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375 ExceptionDRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375 ExceptionDRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
Exception `DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
counter.count=0
counter.max=4

@count incremented to 1
Threaded!
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537509010_11835_11839_0_0.449993964192234 ExceptionDRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537509010_11835_11839_0_0.449993964192234
Exception `DRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537509010_11835_11839_0_0.449993964192234
#kids = 1
Reaped a kid…
@count incremented to 2

I’ve tried to capture these exceptions using ‘begin … rescue … end’
blocks, but I’m obviously not doing something right. What’s frustrating
is that when I pull the code out to just use ‘sleep’ calls, it seems to
work just fine, without all the warnings.

I know my code is quite long, but even so it’s not complete since I need
to insert my results into a DB at some point as well, so just getting
the basics working will be a big help!

Here’s my code:

#!/usr/bin/ruby

require ‘getoptlong’
#require ‘thread’
require ‘slave’

$VERSION = “v1.1”;
$max_slaves = 1
$count = 50

MYURI = “drbunix:/tmp/stoffj-test_” + Process.pid.to_s

opts = GetoptLong.new(
[ “–help”, “-h”, GetoptLong::NO_ARGUMENT],
[ “–kids”, “-k”, GetoptLong::REQUIRED_ARGUMENT]
)

#---------------------------------------------------------------------
class Counter
def initialize(max)
@slaves = []
@max = max
@count = 0
@count_mutex = Mutex.new
end

def count
@count
end

def max
@max
end

Increment the count of slaves, returning 1 if incremented, 0 if not.

def increment
ok = nil
@count_mutex.synchronize do
if (@count < @max) then
@count += 1
puts “@count incremented to #{@count}”
ok = 1
end
end
ok
end

Decrement the count of slaves

def decrement
@count_mutex.synchronize do
if (@count > 1) then
@count -= 1
puts “@count decremented to #{@count}”
end
end
@count
end
end

#---------------------------------------------------------------------
class ReadDir
def initialize(server)
@server = server
end

def readdir(dir)

#puts "readdir(#{dir})"
#sleep 1

size_file = {}
size_dir = {}
size_total = 0

kids = []

# Traverse the directory and collect the size of all files and
# directories

begin
  Dir.foreach(dir) do |f|
    #print " #{f},"
    if(f != "." && f != "..") then
      f_full = addpath(dir, f)
      stat = File.lstat(f_full)

      if(!stat.symlink?) then

        if(stat.file?) then
          #puts "  File: #{f}"
          size = File.size(f_full)
          size_file[f] = size
          size_total += size
        end

        if(stat.directory?) then
          #puts "DIR= #{f}"
          if (@server.max < 2) then
            #puts " no threads."
            size = readdir(f_full)
            if (size > 0) then
              size_dir[f] = size
              size_total += size
            end
          else
            ok = @server.increment
            if (ok)
              puts " Threaded!"
              kids << Slave.object(:async => true, :threadsage =>

true) {
size = readdir(f_full)
@server.decrement
#puts “size = #{size}”
# return the size from the slave properly
size
}
puts “#kids = #{kids.length}”
else
size = readdir(f_full)
if(size > 0) then
size_dir[f] = size
size_total += size
end
end
end
end
end
end
end
end

kids.each { |kid|
  puts "Reaped a kid..."
  size_total += kid.value
  @server.decrement
}

#Puts "Dir: #{dir} = #{size_total}"
return size_total

end

end

#---------------------------------------------------------------------

Read a directory and add to the database; this function is recursive

for sub-directories

#---------------------------------------------------------------------
def usage
puts
puts “usage: readdir-drb [–kids NUM] "
puts " defaults to #{$max_kids} children”
puts
puts " version: #{$version}"
puts
end

#---------------------------------------------------------------------
def addpath(a, b)
return a + b if(a =~ //$/)
return a + “/” + b
end

#---------------------------------------------------------------------

Main

#---------------------------------------------------------------------
$DEBUG = true

opts.each do |opt,arg|
case opt
when “–kids”
$max_slaves = arg.to_i
else
usage
exit
end
end

if ARGV.length != 1
puts “Missing dir argument (try --help)”
exit 0
end

dir = ARGV.shift

Start the Slave for Counting…

puts “Starting Slave Counter”
slave = Slave.new :object => Counter.new($max_slaves), :async => true

counter = slave.object
puts “counter.count=#{counter.count}”
puts “counter.max=#{counter.max}”
puts

Fire up a new Kid Class readdir.

kid = ReadDir.new(counter)

Now let’s try to do a recursive readdir() algorith with threads.

size = kid.readdir(dir)

sizekb = size / 1024;
sizemb = sizekb / 1024;
sizegb = sizemb / 1024;

puts “”
printf “Total size: %d Bytes\n”, size
printf “Total size: %.2f Kb\n”, sizekb
printf “Total size: %.2f Mb\n”, sizemb
if (sizegb > 1.5)
printf “Total size: %.2f Gb\n”, sizegb
end

John Stoffel wrote:

Hi,

I’ve got some code which seems to work well, but the warning messages
are driving me insane! I can’t use this code if it’s always going to
spew warnings like these on a CentOS 5.2 system with ruby 1.8.5
(2006-08-25) [i386-linux] as the environment. I’m also using
slave-1.2.1 from http://www.codeforpeople.com
as well.

$ ./readdir-slave-only.rb --kids 4 tmp
Starting Slave Counter
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375 ExceptionDRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375

The exception is “normal” insofar as being expected behavior from DRb.
It tries each loaded protocol until it finds a match, rescuing the
non-matches.

You are seeing these messages because you have $DEBUG=true. If you are
determined to avoid them in the presence of $DEBUG then you could put a
preferred protocol at the beginning of the search list.

module DRb::DRbProtocol
module_function
def prefer_protocol(proto)
@protocol.delete(proto)
@protocol.unshift(proto)
end
end

DRb::DRbProtocol.prefer_protocol(DRb::DRbUNIXSocket)

James M. Lawrence wrote:

John Stoffel wrote:

Hi,

I’ve got some code which seems to work well, but the warning messages
are driving me insane! I can’t use this code if it’s always going to
spew warnings like these on a CentOS 5.2 system with ruby 1.8.5
(2006-08-25) [i386-linux] as the environment. I’m also using
slave-1.2.1 from http://www.codeforpeople.com
as well.

$ ./readdir-slave-only.rb --kids 4 tmp
Starting Slave Counter
Exception DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375 ExceptionDRb::DRbBadScheme’ at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375

The exception is “normal” insofar as being expected behavior from DRb.
It tries each loaded protocol until it finds a match, rescuing the
non-matches.

You are seeing these messages because you have $DEBUG=true. If you are
determined to avoid them in the presence of $DEBUG then you could put a
preferred protocol at the beginning of the search list.

module DRb::DRbProtocol
module_function
def prefer_protocol(proto)
@protocol.delete(proto)
@protocol.unshift(proto)
end
end

DRb::DRbProtocol.prefer_protocol(DRb::DRbUNIXSocket)

Duh, I feel like a fool now. I had forgotten about the $DEBUG setting
(my mistake for burying it down low…) and now that I’ve turned it off,
it’s much better! Now I can address my remaining issues with the code
and final cleanup.

I would have hoped that DRb would have a DRb::DEBUG value instead, or at
least something in the documentation which talks about this more since I
was going nuts trying to figure out why it was so darn noisy.

This is a huge hug help, thanks!

John

John Stoffel wrote:

This is a huge hug help, thanks!

Sometimes we all need hug help :slight_smile:

Joel VanderWerf wrote:

John Stoffel wrote:

This is a huge hug help, thanks!

Sometimes we all need hug help :slight_smile:

laugh That’s very true. Now I’m actually getting useful work out of
my script, though I’ve got a logic bug somewhere which is causing
problems.

More questions as I figure out what I’m doing wrong.

John