JRuby threading - thread safety of database connection objects


#1

Hi guys,

I’m trying to make the code I have written that populates an LDAP server
from a list of user accounts a little more efficient.

The logic behind the application is fairly straightforward:

For every username in an array
Check whether the user is already in ldap
If the user is not in ldap then
Check the type of user (staff/student/invalid)
Collect all relevant attributes for the user type from database
Return user attributes
If user attributes are not null then build them into an LDAP:MOD
object
Return LDAP:MOD object
Add user to LDAP
Append successful accounts to list of added accounts
Append failed accounts to list of remaining accounts
Return to main loop

Coded up, this looks like:

accounts.each do |login|
if (ldapAdd(ldapconn, userconn, login))
added.push login
else
failed.push login
end
end

The application works fine as a sequential process, and when run in AOT
and server vm mode the speed is comparable to some legacy Python code
that we have been using - however I’m eager to see if I can get some
extra speed from JRuby’s native threading (I’m also doing it for my own
curiosity :slight_smile:

The logic for the threaded version of the application is very similar:

until (accounts.empty?) do
if (threadcount < maxthreads)
threadlock.synchronize do
threadcount += 1
end
login = accounts.pop

workerthreads.push Thread.new(login) { |loginthread|
  status = ldapAdd(ldapconn, sqlconn, loginthread)
  threadlock.synchronize do
    if (status)
      added.push loginthread
    else
      failed.push loginthread
    end
      threadcount -= 1
  end
}

end
end

Testing, retrieving and building LDAP attributes for each person should
be able to be done in parallel for at least a small number of threads at
any given time. The code above works ok and I can indeed see the
interleaved output from ldapAdd and its subsequent children (check
account, retrieve attribs, assemble ldap object, add ldap object) for a
number of accounts simulatenously - however I’m not getting the speed or
concurrency that I think I ought to.

One query I had was relating to the ldap and sql connection objects that
are passed to each worker thread; I create a single connection object
for both LDAP (a jruby-ldap LDAP::SSLConn) and JDBC
(Java::com.mysql.jdbc.Driver) outside the main loop and pass them to the
thread (as ldapconn & sqlconn). Are these connection objects thread
safe, or will workers using these objects block? I guess that could be
the cause…

Having written all of this out I now figure I can spawn a number of long
running threads that consume data from the account array, each with
their own ldap and sql connection objects and not have the overhead of
creating and destroying threads each time around the main loop. However
it would still be interesting to know if those connection objects are
thread safe, or if there are alternatives, such as using Java connection
pooling technology within JRuby?

Cheers

John

Unix & Web Infrastructure Management
Faculty of Medical Sciences Computing
University of Newcastle

Email : removed_email_address@domain.invalid
Web: http://www.ncl.ac.uk/medev


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email