Difference in behaviour using 'require' within threads


#1

Hi list,

Following on from my previous question regarding threading I have
observed what appears to be a difference in behaviour of ‘require’ when
running code within a number of different threads. I’m not sure whether
this is expected behaviour, or a consequence of JRuby’s native threading
when running on the JVM.

I have one file with two methods defined:

main.rb

def mainmethod(work)
require “dbmethods.rb”
while (work)
threadcount += 1
threads.push thread.new(threadcount) { |threadid|
anothermethod(threadid)
}
end
end
end

def anothermethod(id)
require “moremethods.rb”
puts “called by” + id
externalmethod()
end

Another file, has a third method defined:

moremethods.rb

def externalmethod
stuff()
end

The code works and it executes anothermethod() and externalmethod() is
then called, but I always get a “No method” error relating to the
function externalmethod part way through the execution of the code of
anothermethod(). This can vary from the very first thread that calls
externalmethod, or it can run a couple of times and return a valid
result until the 3rd, 4th or 5th time.

If I change mainfunc() to use something along the lines of:

while (work)
anothermethod
end

ie. without the threads, then the require of moremethods.rb works
every time and I never get a “No method” error.
In addition, if I instead change the require to be in the main body of
mainfunc(), rather than anothermethod() I also never see the error.

Could anyone explain why I see this behaviour? My naivety says that if I
‘require’ another library or file, then that dependency is loaded before
execution continues.

The actual code spawning the threads from the main body of the
application looks like this:

while (threadcount < maxthreads)
workerthreads.push Thread.new(threadcount) { |threadid|
userconn = sqlConn()
ldapconn = ldapConn()
if (verbose)
puts “thread #{threadid} started”
end
until ((accountqueue.length < 1) && (accounts.length <1))
login = accountqueue.deq
if (verbose)
puts “thread #{threadid} Got some work : #{login}”
end
status = ldapAccountBuildStaff(verbose, userconn, threadid, login)
if (status)
added.enq(login)
else
failed.enq(login)
end
end
if (verbose)
puts “thread #{threadid} exiting”
end
userconn.close
ldapconn.unbind
}
threadcount += 1
end

… and the ldapAccountBuildStaff method is rather simple:

def ldapAccountBuildStaff(verbose, sqlconn, threadid, login)

Builds a Staff LDAP object

Load locally defined libraries

require “libs/ldapstaffaccount.rb”
if (verbose >1)
puts “thread #{threadid} ldapAccountBuildStaff : #{login}”
end
personattrs = ldapStaffAttrs(verbose, sqlconn, login)
return personattrs
end

The “No method” error relating to ldapStaffAttrs (which is defined in
libs/ldapstaffaccount.rb) is what I keep getting.

As I say, I can resolve the issue either by running from a simple loop,
or by moving the require to the main body, rather than the
ldapAccountBuildStaff method. But I would appreciate any insight on why
the error could be occurring!

John


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#2

See:
http://jira.codehaus.org/browse/JRUBY-3078

We also have an older thread about a week or two old in our mailing
list about thread safety in require. It is not a simple problem, but
we think we can fix it.

This is really an effect of JRuby having native threads. With Java
native threads all require/loads are happening concurrently. We need
to synchronize on registration, but we cannot naively synchronize or
we can deadlock in several scenarios (one of which may not be that
uncommon).

-Tom

On Sat, Oct 25, 2008 at 8:55 AM, John S.
removed_email_address@domain.invalid wrote:

while (work)
puts “called by” + id

           end
                   end

… and the ldapAccountBuildStaff method is rather simple:
end
http://xircles.codehaus.org/manage_email


Blog: http://www.bloglines.com/blog/ThomasEEnebo
Email: removed_email_address@domain.invalid , removed_email_address@domain.invalid


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#3

Thanks Tom, that certainly describes the symptoms I have seen.

Moving the relevant require statements to the body of the main function
has solved the issue as expected. It’s not a show stopper for me as I’ve
only got 6 or so additional libraries that I require throughout the life
of the thread.

I’ll keep an eye on 3078 and 3057.

Cheers

John

Unix & Web Infrastructure Management
Faculty of Medical Sciences Computing
University of Newcastle

Email : removed_email_address@domain.invalid
Web: http://www.ncl.ac.uk/medev

We also have an older thread about a week or two old in our mailing

I have one file with two methods defined:
end

externalmethod, or it can run a couple of times and return a valid

In addition, if I instead change the require to be in the main body of
workerthreads.push Thread.new(threadcount) { |threadid|
: #{login}"
puts “thread #{threadid} exiting”
# Builds a Staff LDAP object
The “No method” error relating to ldapStaffAttrs (which is defined in
To unsubscribe from this list, please visit:
Email: removed_email_address@domain.invalid , removed_email_address@domain.invalid


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email