Forum: Ruby Quieting DRb messages in fork() code

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Db524516a64318b0e937d41357c729eb?d=identicon&s=25 John Stoffel (johnstoffel)
on 2009-02-18 18:39
Hi,

I've got some code which seems to work well, but the warning messages
are driving me insane!  I can't use this code if it's always going to
spew warnings like these on a CentOS 5.2 system with ruby 1.8.5
(2006-08-25) [i386-linux] as the environment.  I'm also using
slave-1.2.1 from http://www.codeforpeople.com
as well.


$ ./readdir-slave-only.rb --kids 4 tmp
Starting Slave Counter
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
counter.count=0
counter.max=4

@count incremented to 1
 Threaded!
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537509010_11835_11839_0_0.449993964192234
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537509010_11835_11839_0_0.449993964192234
Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
drbunix:///tmp/slave_proc_537509010_11835_11839_0_0.449993964192234
#kids = 1
Reaped a kid...
@count incremented to 2


I've tried to capture these exceptions using 'begin ... rescue ... end'
blocks, but I'm obviously not doing something right.  What's frustrating
is that when I pull the code out to just use 'sleep' calls, it seems to
work just fine, without all the warnings.

I know my code is quite long, but even so it's not complete since I need
to insert my results into a DB at some point as well, so just getting
the basics working will be a big help!

Here's my code:


#!/usr/bin/ruby

require 'getoptlong'
#require 'thread'
require 'slave'

$VERSION = "v1.1";
$max_slaves = 1
$count = 50

MYURI = "drbunix:/tmp/stoffj-test_" + Process.pid.to_s

opts = GetoptLong.new(
                      [ "--help", "-h", GetoptLong::NO_ARGUMENT],
                      [ "--kids", "-k", GetoptLong::REQUIRED_ARGUMENT]
                      )

#---------------------------------------------------------------------
class Counter
  def initialize(max)
    @slaves = []
    @max = max
    @count = 0
    @count_mutex = Mutex.new
  end

  def count
    @count
  end

  def max
    @max
  end

  # Increment the count of slaves, returning 1 if incremented, 0 if not.
  def increment
    ok = nil
    @count_mutex.synchronize do
      if (@count < @max) then
        @count += 1
        puts "@count incremented to #{@count}"
        ok = 1
      end
    end
    ok
  end

  # Decrement the count of slaves
  def decrement
    @count_mutex.synchronize do
      if (@count > 1) then
        @count -= 1
        puts "@count decremented to #{@count}"
      end
    end
    @count
  end
end

#---------------------------------------------------------------------
class ReadDir
  def initialize(server)
    @server = server
  end

  def readdir(dir)

    #puts "readdir(#{dir})"
    #sleep 1

    size_file = {}
    size_dir = {}
    size_total = 0

    kids = []

    # Traverse the directory and collect the size of all files and
    # directories

    begin
      Dir.foreach(dir) do |f|
        #print " #{f},"
        if(f != "." && f != "..") then
          f_full = addpath(dir, f)
          stat = File.lstat(f_full)

          if(!stat.symlink?) then

            if(stat.file?) then
              #puts "  File: #{f}"
              size = File.size(f_full)
              size_file[f] = size
              size_total += size
            end

            if(stat.directory?) then
              #puts "DIR= #{f}"
              if (@server.max < 2) then
                #puts " no threads."
                size = readdir(f_full)
                if (size > 0) then
                  size_dir[f] = size
                  size_total += size
                end
              else
                ok = @server.increment
                if (ok)
                  puts " Threaded!"
                  kids << Slave.object(:async => true, :threadsage =>
true) {
                    size = readdir(f_full)
                    @server.decrement
                    #puts "size = #{size}"
                    # return the size from the slave properly
                    size
                  }
                  puts "#kids = #{kids.length}"
                else
                  size = readdir(f_full)
                  if(size > 0) then
                    size_dir[f] = size
                    size_total += size
                  end
                end
              end
            end
          end
        end
      end
    end

    kids.each { |kid|
      puts "Reaped a kid..."
      size_total += kid.value
      @server.decrement
    }

    #Puts "Dir: #{dir} = #{size_total}"
    return size_total
  end

end

#---------------------------------------------------------------------
# Read a directory and add to the database; this function is recursive
# for sub-directories


#---------------------------------------------------------------------
def usage
  puts
  puts "usage: readdir-drb [--kids NUM] <dir>"
  puts "  defaults to #{$max_kids} children"
  puts
  puts "  version: #{$version}"
  puts
end

#---------------------------------------------------------------------
def addpath(a, b)
  return a + b if(a =~ /\/$/)
  return a + "/" + b
end



#---------------------------------------------------------------------
# Main
#---------------------------------------------------------------------
$DEBUG = true

opts.each do |opt,arg|
  case opt
  when "--kids"
    $max_slaves = arg.to_i
  else
    usage
    exit
  end
end

if ARGV.length != 1
  puts "Missing dir argument (try --help)"
  exit 0
end

dir = ARGV.shift

# Start the Slave for Counting...
puts "Starting Slave Counter"
slave = Slave.new :object => Counter.new($max_slaves), :async => true

counter = slave.object
puts "counter.count=#{counter.count}"
puts "counter.max=#{counter.max}"
puts

# Fire up a new Kid Class readdir.
kid = ReadDir.new(counter)

# Now let's try to do a recursive readdir() algorith with threads.
size = kid.readdir(dir)

sizekb = size / 1024;
sizemb = sizekb / 1024;
sizegb = sizemb / 1024;

puts ""
printf "Total size: %d Bytes\n", size
printf "Total size: %.2f Kb\n", sizekb
printf "Total size: %.2f Mb\n", sizemb
if (sizegb > 1.5)
  printf "Total size: %.2f Gb\n", sizegb
end
Efdb04c4dac4a962d86e25361f8fd450?d=identicon&s=25 James M. Lawrence (quix)
on 2009-02-19 05:17
John Stoffel wrote:
> Hi,
>
> I've got some code which seems to work well, but the warning messages
> are driving me insane!  I can't use this code if it's always going to
> spew warnings like these on a CentOS 5.2 system with ruby 1.8.5
> (2006-08-25) [i386-linux] as the environment.  I'm also using
> slave-1.2.1 from http://www.codeforpeople.com
> as well.
>
>
> $ ./readdir-slave-only.rb --kids 4 tmp
> Starting Slave Counter
> Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
> drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
> Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
> drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375

The exception is "normal" insofar as being expected behavior from DRb.
It tries each loaded protocol until it finds a match, rescuing the
non-matches.

You are seeing these messages because you have $DEBUG=true.  If you are
determined to avoid them in the presence of $DEBUG then you could put a
preferred protocol at the beginning of the search list.

module DRb::DRbProtocol
  module_function
  def prefer_protocol(proto)
    @protocol.delete(proto)
    @protocol.unshift(proto)
  end
end

DRb::DRbProtocol.prefer_protocol(DRb::DRbUNIXSocket)
Db524516a64318b0e937d41357c729eb?d=identicon&s=25 John Stoffel (johnstoffel)
on 2009-02-19 15:33
James M. Lawrence wrote:
> John Stoffel wrote:
>> Hi,
>>
>> I've got some code which seems to work well, but the warning messages
>> are driving me insane!  I can't use this code if it's always going to
>> spew warnings like these on a CentOS 5.2 system with ruby 1.8.5
>> (2006-08-25) [i386-linux] as the environment.  I'm also using
>> slave-1.2.1 from http://www.codeforpeople.com
>> as well.
>>
>>
>> $ ./readdir-slave-only.rb --kids 4 tmp
>> Starting Slave Counter
>> Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
>> drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
>> Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 -
>> drbunix:///tmp/slave_counter_537520870_11835_11836_0_0.697410256497375
>
> The exception is "normal" insofar as being expected behavior from DRb.
> It tries each loaded protocol until it finds a match, rescuing the
> non-matches.
>
> You are seeing these messages because you have $DEBUG=true.  If you are
> determined to avoid them in the presence of $DEBUG then you could put a
> preferred protocol at the beginning of the search list.
>
> module DRb::DRbProtocol
>   module_function
>   def prefer_protocol(proto)
>     @protocol.delete(proto)
>     @protocol.unshift(proto)
>   end
> end
>
> DRb::DRbProtocol.prefer_protocol(DRb::DRbUNIXSocket)

Duh, I feel like a fool now.  I had forgotten about the $DEBUG setting
(my mistake for burying it down low...) and now that I've turned it off,
it's much better!  Now I can address my remaining issues with the code
and final cleanup.

I would have hoped that DRb would have a DRb::DEBUG value instead, or at
least something in the documentation which talks about this more since I
was going nuts trying to figure out why it was so darn noisy.

This is a huge hug help, thanks!

John
47b1910084592eb77a032bc7d8d1a84e?d=identicon&s=25 Joel VanderWerf (Guest)
on 2009-02-19 17:34
(Received via mailing list)
John Stoffel wrote:
> This is a huge hug help, thanks!

Sometimes we all need hug help :)
Db524516a64318b0e937d41357c729eb?d=identicon&s=25 John Stoffel (johnstoffel)
on 2009-02-19 21:33
Joel VanderWerf wrote:
> John Stoffel wrote:
>> This is a huge hug help, thanks!
>
> Sometimes we all need hug help :)

*laugh* That's very true.  Now I'm actually getting useful work out of
my script, though I've got a logic bug somewhere which is causing
problems.

More questions as I figure out what I'm doing wrong.

John
This topic is locked and can not be replied to.