JRuby complete, multi-threading, and java.lang.NoClassDefFoundError

addis_a · February 4, 2015, 5:27pm

I am having problems running a JRuby scripting container accessed by
multiple threads. Here’s my declaration:

protected static ScriptingContainer container = new
ScriptingContainer(LocalContextScope.THREADSAFE);

When I run my application which has multiple threads utilizing this
container, I often get java.lang.NoClassDefFoundError or other similar
exceptions soon after starting up. Not always, but often. And not always
the same class. Here are four different stack traces that I have seen:

==> stack-1.txt <==
org.jruby.embed.InvokeFailedException: (NameError) uninitialized
constant
Krypt::Provider::JDK
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:317)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1508)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.JRubyScraper.init(JRubyScraper.java:43)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.Scraper.scrape(Scraper.java:90)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
com.needco.scrapers.ScraperProcessor.run(ScraperProcessor.java:108)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[?:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: org.jruby.exceptions.RaiseException: (NameError)
uninitialized
constant Krypt::Provider::JDK
at
org.jruby.RubyModule.const_missing(org/jruby/RubyModule.java:2726)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
RUBY.(root)(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/krypt/core.rb:41)
~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1071)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
Kernel.require(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:73)
~[?:?]

==> stack-2.txt <==
org.jruby.embed.InvokeFailedException: (LoadError) load error: kryptcore

java.lang.NoClassDefFoundError: org/jruby/ext/krypt/KryptCoreService
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:317)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1508)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.JRubyScraper.init(JRubyScraper.java:43)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.Scraper.scrape(Scraper.java:90)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
com.needco.scrapers.ScraperProcessor.run(ScraperProcessor.java:108)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[?:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error:
kryptcore – java.lang.NoClassDefFoundError:
org/jruby/ext/krypt/KryptCoreService
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1071)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
Kernel.require(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55)
~[?:?]
at
Kernel.require(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:53)
~[?:?]
at
RUBY.(root)(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/krypt/core.rb:37)
~[?:?]

==> stack-3.txt <==
org.jruby.embed.InvokeFailedException: (LoadError) load error: kryptcore

java.lang.NoClassDefFoundError: org/jruby/ext/krypt/helper/RubyHelper
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:317)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1508)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.JRubyScraper.init(JRubyScraper.java:43)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.Scraper.scrape(Scraper.java:90)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
com.needco.scrapers.ScraperProcessor.run(ScraperProcessor.java:108)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[?:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error:
kryptcore – java.lang.NoClassDefFoundError:
org/jruby/ext/krypt/helper/RubyHelper
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1071)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
Kernel.require(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:126)
~[?:?]
at
Kernel.require(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:124)
~[?:?]
at
RUBY.(root)(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/krypt/core.rb:37)
~[?:?]

==> stack-4.txt <==
org.jruby.embed.InvokeFailedException: (LoadError) load error:
jopenssl/load – java.lang.NoClassDefFoundError:
org/bouncycastle/asn1/DERBoolean
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:317)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1508)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.JRubyScraper.init(JRubyScraper.java:43)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at com.needco.scrapers.Scraper.scrape(Scraper.java:90)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
com.needco.scrapers.ScraperProcessor.run(ScraperProcessor.java:108)
[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[?:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error:
jopenssl/load – java.lang.NoClassDefFoundError:
org/bouncycastle/asn1/DERBoolean
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1071)
~[nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
Kernel.require(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:126)
~[?:?]
at
Kernel.require(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:124)
~[?:?]
at
RUBY.(root)(/home/scraper/scrapers/nc-scrapers-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/openssl.rb:1)
~[?:?]

I have tried setting global.require.lock=true and
.compile.invokedynamic=false as they had been suggested to others that
seemed to have similar issues, but that has not helped.

I am using jruby-complete 1.7.18 and Java 1.7.0_71.

Can anyone help?

Thanks!

Greg_Allen · February 4, 2015, 5:41pm

could you set

System.setProperty(“jruby.debug.loadService”, “true” );

just before you instantiate the ScriptingContainer. it produces a lot of
output but could help to understand what is going on.

all those errors are with one or the other default gems and their jars
from
those default gems.

Greg_Allen · February 4, 2015, 6:44pm

I passed that option in with -Djruby.debug.loadService=true and got some
output. I’m not sure what to do with the output thought. I’ve
compressed
and attached it here.

Does it give you any insight?

Thanks.

Greg_Allen · February 4, 2015, 7:17pm

well, it shows what was almost clear. you start kind of the jruby
runtimes
at the “same” time and with loading the jar into jruby-classloader of
the
runtime something goes wrong. LocalContextScope.THREADSAFE should do the
trick but apparently it does NOT

Greg_Allen · February 4, 2015, 8:20pm

So any suggestions? Is my only option to use
LocalContextScope.SINGLETON
and have each thread use it’s own ruby runtime?

Greg_Allen · February 4, 2015, 9:57pm

sorry can not help more here but I filed an issue:

github.com/jruby/jruby

ScriptingContainer with LocalContextScope.THREADSAFE share JRubyClassLoader with all runtimes

opened 08:55PM - 04 Feb 15 UTC

closed 05:09PM - 05 Jan 16 UTC

mkristian

JRuby 1.7.x embed

ScriptingContainer with LocalContextScope.THREADSAFE share one instance of JRuby…ClassLoader with all runtimes. since each runtime loads it own set of jar files but they get shared by the shared JRubyClassLoader. for example each runtime can load its own version of bouncycastle jars which will result one classloader having all those BC jars loaded. the real world problem is when a lot of thread startup with `require 'openssl'` conditions can arise where some classes can not be loaded from the JRubyClassLoader: http://ruby.11.x6.nabble.com/JRuby-complete-multi-threading-and-java-lang-NoClassDefFoundError-td5004808.html IMO each runtime should have its own JRubyClassloader to separate the set if required jars per runtime.

Greg_Allen · February 4, 2015, 9:47pm

I looked into the code and yes those runtimes do share the
JRubyClassLoader
which is a bug IMO

somehow you need to serialize the startup of the runtime. the crucial
thing
is “require ‘openssl’” which loads a few jars. once these jars are
loading
things probably work OK.

not sure how to do this.

Greg_Allen · February 6, 2015, 10:23pm

Thanks, Christian.

I worked around this by using some static code in my Java class to force
the Ruby runtime to load first before any of the thread start. Ugly, but
it
appears to be effective.

I’m surprised that nobody else has hit this yet.

JRuby complete, multi-threading, and java.lang.NoClassDefFoundError

==> stack-2.txt <== org.jruby.embed.InvokeFailedException: (LoadError) load error: kryptcore

==> stack-3.txt <== org.jruby.embed.InvokeFailedException: (LoadError) load error: kryptcore

==> stack-2.txt <==
org.jruby.embed.InvokeFailedException: (LoadError) load error: kryptcore

==> stack-3.txt <==
org.jruby.embed.InvokeFailedException: (LoadError) load error: kryptcore