Forum: JRuby JRuby threading - thread safety of database connection objects

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
0513c92286df6a35d3de502676a2e58d?d=identicon&s=25 John Snowdon (Guest)
on 2008-10-23 11:30
(Received via mailing list)
Hi guys,

I'm trying to make the code I have written that populates an LDAP server
from a list of user accounts a little more efficient.

The logic behind the application is fairly straightforward:

For every username in an array
  Check whether the user is already in ldap
  If the user is not in ldap then
    Check the type of user (staff/student/invalid)
      Collect all relevant attributes for the user type from database
      Return user attributes
    If user attributes are not null then build them into an LDAP:MOD
object
    Return LDAP:MOD object
  Add user to LDAP
  Append successful accounts to list of added accounts
  Append failed accounts to list of remaining accounts
Return to main loop

Coded up, this looks like:

accounts.each do |login|
  if (ldapAdd(ldapconn, userconn, login))
    added.push login
  else
    failed.push login
  end
end

The application works fine as a sequential process, and when run in AOT
and server vm mode the speed is comparable to some legacy Python code
that we have been using - however I'm eager to see if I can get some
extra speed from JRuby's native threading (I'm also doing it for my own
curiosity :-)

The logic for the threaded version of the application is very similar:

until (accounts.empty?) do
  if (threadcount < maxthreads)
    threadlock.synchronize do
      threadcount += 1
    end
    login = accounts.pop

    workerthreads.push Thread.new(login) { |loginthread|
      status = ldapAdd(ldapconn, sqlconn, loginthread)
      threadlock.synchronize do
        if (status)
          added.push loginthread
        else
          failed.push loginthread
        end
          threadcount -= 1
      end
    }
  end
end

Testing, retrieving and building LDAP attributes for each person should
be able to be done in parallel for at least a small number of threads at
any given time. The code above works ok and I can indeed see the
interleaved output from ldapAdd and its subsequent children (check
account, retrieve attribs, assemble ldap object, add ldap object) for a
number of accounts simulatenously - however I'm not getting the speed or
concurrency that I think I ought to.

One query I had was relating to the ldap and sql connection objects that
are passed to each worker thread; I create a single connection object
for both LDAP (a jruby-ldap LDAP::SSLConn) and JDBC
(Java::com.mysql.jdbc.Driver) outside the main loop and pass them to the
thread (as ldapconn & sqlconn). Are these connection objects thread
safe, or will workers using these objects block? I guess that could be
the cause....

Having written all of this out I now figure I can spawn a number of long
running threads that consume data from the account array, each with
their own ldap and sql connection objects and not have the overhead of
creating and destroying threads each time around the main loop. However
it would still be interesting to know if those connection objects are
thread safe, or if there are alternatives, such as using Java connection
pooling technology within JRuby?

Cheers

John
---
 Unix & Web Infrastructure Management
 Faculty of Medical Sciences Computing
 University of Newcastle

 Email : j.p.snowdon@ncl.ac.uk
 Web: http://www.ncl.ac.uk/medev


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.