Forum: JRuby Bundler super-slow: can't use dependency API

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
2abdb50caf0dc5b510330f68b02db8e4?d=identicon&s=25 Rodrigo Rosenfeld Rosas (Guest)
on 2015-01-12 20:23
(Received via mailing list)
Hi, I tried to run "bundle" with jruby-1.7.18 installed by rbenv for an
existing project running on MRI and this is what I get:

$ bundle
Fetching gem metadata from https://rails-assets.org/..
Fetching gem metadata from https://rubygems.org/.Retrying dependency api
due to error (2/3): Bundler::MarshalError ArgumentError: marshal data
too short
Retrying dependency api due to error (3/3): Bundler::MarshalError
ArgumentError: marshal data too short

It takes forever (several hours so far and still counting).

This doesn't happen with bundler on MRI 2.2.0, same bundler version. Any
ideas on why this might be happening for JRuby only or how to fix it?

Thanks in advance,
Rodrigo.
2c0c4cf3ccc8da22f7c3b9586ce1cd70?d=identicon&s=25 Christian MICHON (Guest)
on 2015-01-12 21:09
(Received via mailing list)
Is jruby-openssl installed?
On Jan 12, 2015 8:22 PM, "Rodrigo Rosenfeld Rosas" <rr.rosas@gmail.com>
2abdb50caf0dc5b510330f68b02db8e4?d=identicon&s=25 Rodrigo Rosenfeld Rosas (Guest)
on 2015-01-12 21:10
(Received via mailing list)
Yes.
2c0c4cf3ccc8da22f7c3b9586ce1cd70?d=identicon&s=25 Christian MICHON (Guest)
on 2015-01-12 21:27
(Received via mailing list)
OK next verification. Are you going thru a proxy?
On 12 Jan, 2015 9:09 PM, "Rodrigo Rosenfeld Rosas" <rr.rosas@gmail.com>
2abdb50caf0dc5b510330f68b02db8e4?d=identicon&s=25 Rodrigo Rosenfeld Rosas (Guest)
on 2015-01-12 22:10
(Received via mailing list)
No :-) also it works fine on MRI :-)
Em 12/01/2015 18:26, "Christian MICHON" <christian.michon@gmail.com>
escreveu:
B05d3cbc64b0031a24c2887fb6ddc173?d=identicon&s=25 christian (Guest)
on 2015-01-13 10:35
(Received via mailing list)
which bundler version (probably the latest) ? is it possible to share
the
Gemfile ? would give it a trial on my machine ;)

- christian


On Mon, Jan 12, 2015 at 9:07 PM, Rodrigo Rosenfeld Rosas
<rr.rosas@gmail.com
2abdb50caf0dc5b510330f68b02db8e4?d=identicon&s=25 Rodrigo Rosenfeld Rosas (Guest)
on 2015-01-13 11:24
(Received via mailing list)
Attachment: Gemfile (2 KB)
Bundler 1.7.12.

Here's the attached Gemfile (I removed a few lines from the original one
in order to make it possible for you to try it since it contained some
references to some of our custom gems but I tested this one and it
behaves the same).

I believe this is the main problem:

"could not fetch from the dependency API, trying the full index"

I'd like to understand why it can't use the dependency API on JRuby...

Thanks for trying :)
B05d3cbc64b0031a24c2887fb6ddc173?d=identicon&s=25 christian (Guest)
on 2015-01-13 14:31
(Received via mailing list)
yes, once the full index is used things take longer (or much longer) BUT
my
output look like this:
$ bundle install --verbose
Updating git://github.com/Compass/compass-rails.git
Updating git://github.com/rosenfeld/rspec_around_all.git
Updating git://github.com/rosenfeld/letter_opener_web.git
HTTP GET https://rails-assets.org/api/v1/dependencies
HTTP 200 OK
HTTP GET https://bundler.rubygems.org/api/v1/dependencies
HTTP 200 OK
Fetching gem metadata from https://rails-assets.org/
Query List: ["railties", "actionmailer", "sprockets-rails", "sprockets",
"puma", "foreman", "byebug", "pg", "bson_ext", "ruby-prof",
"rack-mini-profiler", "flamegraph", "stackprof", "mongo",
"jruby-openssl",
"jdbc-postgres", "sequel", "devise", "devise-encryptable",
"sequel-devise",
"orm_adapter-sequel", "redis", "hiredis", "connection_pool", "rsolr",
"rest-client", "rails-web-console", "writeexcel", "axlsx", "remotipart",
"sidekiq", "sidekiq-limit_fetch", "sinatra", "ruby-progressbar",
"redcarpet", "sass-rails", "coffee-rails", "compass", "therubyracer",
"uglifier", "jquery-rails", "rails-assets-jquery", "jquery-ui-rails",
"common-dialogs", "eco", "js-routes", "jquery-layout-rails",
"rails-assets-knockout", "rails-assets-jquery.inputmask",
"rails-assets-jquery-file-upload", "rails-assets-numeral",
"rails-assets-es5-shim", "rails-assets-jquery-form",
"rails-assets-jqtree",
"rails-assets-underscore", "rails-assets-jquery.scrollTo",
"rails-assets-noty", "rails-assets-syn", "rspec", "rspec-rails", "oojs",
"pry", "capybara", "selenium-webdriver", "database_cleaner",
"better_errors", "binding_of_caller", "spring", "spring-commands-rspec",
"factory_girl_rails", "launchy", "guard-rspec", "newrelic_rpm",
"letter_opener", "shoulda-matchers", "combustion"]
Query Gemcutter Dependency Endpoint API:
railties,actionmailer,sprockets-rails,sprockets,puma,foreman,byebug,pg,bson_ext,ruby-prof,rack-mini-profiler,flamegraph,stackprof,mongo,jruby-openssl,jdbc-postgres,sequel,devise,devise-encryptable,sequel-devise,orm_adapter-sequel,redis,hiredis,connection_pool,rsolr,rest-client,rails-web-console,writeexcel,axlsx,remotipart,sidekiq,sidekiq-limit_fetch,sinatra,ruby-progressbar,redcarpet,sass-rails,coffee-rails,compass,therubyracer,uglifier,jquery-rails,rails-assets-jquery,jquery-ui-rails,common-dialogs,eco,js-routes,jquery-layout-rails,rails-assets-knockout,rails-assets-jquery.inputmask,rails-assets-jquery-file-upload,rails-assets-numeral,rails-assets-es5-shim,rails-assets-jquery-form,rails-assets-jqtree,rails-assets-underscore,rails-assets-jquery.scrollTo,rails-assets-noty,rails-assets-syn,rspec,rspec-rails,oojs,pry,capybara,selenium-webdriver,database_cleaner,better_errors,binding_of_caller,spring,spring-commands-rspec,factory_girl_rails,launchy,guard-rspec,newrelic_rpm,letter_opener,shoulda-matchers,combustion
HTTP GET
https://rails-assets.org/api/v1/dependencies?gems=...
HTTP 200 OK
Query List: ["rails-assets-blueimp-tmpl",
"rails-assets-blueimp-load-image",
"rails-assets-blueimp-canvas-to-blob"]
Query Gemcutter Dependency Endpoint API:
rails-assets-blueimp-tmpl,rails-assets-blueimp-load-image,rails-assets-blueimp-canvas-to-blob
HTTP GET
https://rails-assets.org/api/v1/dependencies?gems=...
HTTP 200 OK
Query List: []
Fetching gem metadata from https://rubygems.org/
Query List: ["railties", "actionmailer", "sprockets-rails", "sprockets",
"puma", "foreman", "byebug", "pg", "bson_ext", "ruby-prof",
"rack-mini-profiler", "flamegraph", "stackprof", "mongo",
"jruby-openssl",
"jdbc-postgres", "sequel", "devise", "devise-encryptable",
"sequel-devise",
"orm_adapter-sequel", "redis", "hiredis", "connection_pool", "rsolr",
"rest-client", "rails-web-console", "writeexcel", "axlsx", "remotipart",
"sidekiq", "sidekiq-limit_fetch", "sinatra", "ruby-progressbar",
"redcarpet", "sass-rails", "coffee-rails", "compass", "therubyracer",
"uglifier", "jquery-rails", "rails-assets-jquery", "jquery-ui-rails",
"common-dialogs", "eco", "js-routes", "jquery-layout-rails",
"rails-assets-knockout", "rails-assets-jquery.inputmask",
"rails-assets-jquery-file-upload", "rails-assets-numeral",
"rails-assets-es5-shim", "rails-assets-jquery-form",
"rails-assets-jqtree",
"rails-assets-underscore", "rails-assets-jquery.scrollTo",
"rails-assets-noty", "rails-assets-syn", "rspec", "rspec-rails", "oojs",
"pry", "capybara", "selenium-webdriver", "database_cleaner",
"better_errors", "binding_of_caller", "spring", "spring-commands-rspec",
"factory_girl_rails", "launchy", "guard-rspec", "newrelic_rpm",
"letter_opener", "shoulda-matchers", "combustion"]
Query Gemcutter Dependency Endpoint API:
railties,actionmailer,sprockets-rails,sprockets,puma,foreman,byebug,pg,bson_ext,ruby-prof,rack-mini-profiler,flamegraph,stackprof,mongo,jruby-openssl,jdbc-postgres,sequel,devise,devise-encryptable,sequel-devise,orm_adapter-sequel,redis,hiredis,connection_pool,rsolr,rest-client,rails-web-console,writeexcel,axlsx,remotipart,sidekiq,sidekiq-limit_fetch,sinatra,ruby-progressbar,redcarpet,sass-rails,coffee-rails,compass,therubyracer,uglifier,jquery-rails,rails-assets-jquery,jquery-ui-rails,common-dialogs,eco,js-routes,jquery-layout-rails,rails-assets-knockout,rails-assets-jquery.inputmask,rails-assets-jquery-file-upload,rails-assets-numeral,rails-assets-es5-shim,rails-assets-jquery-form,rails-assets-jqtree,rails-assets-underscore,rails-assets-jquery.scrollTo,rails-assets-noty,rails-assets-syn,rspec,rspec-rails,oojs,pry,capybara,selenium-webdriver,database_cleaner,better_errors,binding_of_caller,spring,spring-commands-rspec,factory_girl_rails,launchy,guard-rspec,newrelic_rpm,letter_opener,shoulda-matchers,combustion
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: ["mail", "actionview", "actionpack", "rails-dom-testing",
"activejob", "text-format", "nokogiri", "rubyzip", "htmlentities",
"rake",
"active_support", "rmagick", "activesupport", "i18n", "zip", "rack",
"coderay", "erubis", "debug_inspector", "bson", "debugger-linecache",
"slop", "columnize", "xpath", "rack-test", "mime-types", "culerity",
"celerity", "coffee-script", "thor", "rails", "rb-inotify",
"rb-fsevent",
"chunky_png", "compass-import-once", "compass-core", "sass", "fssm",
"haml", "listen", "json", "responders", "thread_safe", "bcrypt",
"orm_adapter", "warden", "bcrypt-ruby", "eco-source", "execjs",
"factory_girl", "fast_stack", "dotenv", "win32console",
"term-ansicolor",
"posix-spawn", "guard-compat", "guard", "rake-compiler",
"bouncy-castle-java", "spoon", "addressable", "configuration", "ffi",
"echoe", "rails_sandbox_jasmine", "fake-ajax-server", "oojspec",
"activemodel", "method_source", "ruby_parser", "yard", "jeweler",
"rdoc",
"rack-ssl", "netrc", "builder", "rspec-mocks", "rspec-expectations",
"rspec-core", "cucumber", "hoe", "diff-lcs", "syntax", "spicycode-rcov",
"rspec-support", "webrat", "rspec-collection_matchers", "tilt",
"websocket", "childprocess", "multi_json", "json_pure", "libwebsocket",
"metaid", "ParseTree", "ruby2ruby", "sequel_core", "sequel_model",
"assistance", "RubyInline", "bourne", "celluloid", "redis-namespace",
"sprockets-sass", "slim", "mongrel", "shotgun", "rack-protection",
"backports", "hike", "libv8", "ref", "ruby-ole"]
Query Gemcutter Dependency Endpoint API:
mail,actionview,actionpack,rails-dom-testing,activejob,text-format,nokogiri,rubyzip,htmlentities,rake,active_support,rmagick,activesupport,i18n,zip,rack,coderay,erubis,debug_inspector,bson,debugger-linecache,slop,columnize,xpath,rack-test,mime-types,culerity,celerity,coffee-script,thor,rails,rb-inotify,rb-fsevent,chunky_png,compass-import-once,compass-core,sass,fssm,haml,listen,json,responders,thread_safe,bcrypt,orm_adapter,warden,bcrypt-ruby,eco-source,execjs,factory_girl,fast_stack,dotenv,win32console,term-ansicolor,posix-spawn,guard-compat,guard,rake-compiler,bouncy-castle-java,spoon,addressable,configuration,ffi,echoe,rails_sandbox_jasmine,fake-ajax-server,oojspec,activemodel,method_source,ruby_parser,yard,jeweler,rdoc,rack-ssl,netrc,builder,rspec-mocks,rspec-expectations,rspec-core,cucumber,hoe,diff-lcs,syntax,spicycode-rcov,rspec-support,webrat,rspec-collection_matchers,tilt,websocket,childprocess,multi_json,json_pure,libwebsocket,metaid,ParseTree,ruby2ruby,sequel_core,sequel_model,assistance,RubyInline,bourne,celluloid,redis-namespace,sprockets-sass,slim,mongrel,shotgun,rack-protection,backports,hike,libv8,ref,ruby-ole
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: ["rails-html-sanitizer", "journey", "rack-cache",
"rails-deprecated_sanitizer", "rack-mount", "tzinfo", "globalid",
"activemodel-globalid", "minitest", "memcache-client", "mocha",
"timers",
"facter", "coffee-script-source", "multi_test", "gherkin",
"cucumber-core",
"treetop", "polyglot", "debugger-ruby_core_source", "ruby_core_source",
"dotenv-deployment", "rubyforge", "highline", "gemcutter", "allison",
"rcov", "abstract", "sinon-rails", "bones", "notiffany", "shellany",
"nenv", "formatador", "lumberjack", "sys-uname", "growl", "libnotify",
"bundler", "open_gem", "terminal-table", "maruku", "git",
"thoughtbot-shoulda", "mhennemeyer-output_catcher", "rr", "redgreen",
"peterwald-git", "schacon-git", "github_api", "spruz", "rb-fchange",
"rb-kqueue", "celluloid-io", "tlsmail", "archive-tar-minitar",
"daemons",
"fastthread", "gem_plugin", "cgi_multipart_eof_fix", "mini_portile",
"racc", "tenderlove-frex", "rexical", "weakling",
"rails-sandbox-assets",
"sexp_processor", "SexpProcessor", "test-spec", "camping", "fcgi",
"ruby-openid", "thin", "escape_utils", "activerecord", "activeresource",
"actionwebservice", "ZenTest", "validatable", "temple",
"sprockets-helpers", "tins", "text-hyphen", "atomic", "hpricot"]
Query Gemcutter Dependency Endpoint API:
rails-html-sanitizer,journey,rack-cache,rails-deprecated_sanitizer,rack-mount,tzinfo,globalid,activemodel-globalid,minitest,memcache-client,mocha,timers,facter,coffee-script-source,multi_test,gherkin,cucumber-core,treetop,polyglot,debugger-ruby_core_source,ruby_core_source,dotenv-deployment,rubyforge,highline,gemcutter,allison,rcov,abstract,sinon-rails,bones,notiffany,shellany,nenv,formatador,lumberjack,sys-uname,growl,libnotify,bundler,open_gem,terminal-table,maruku,git,thoughtbot-shoulda,mhennemeyer-output_catcher,rr,redgreen,peterwald-git,schacon-git,github_api,spruz,rb-fchange,rb-kqueue,celluloid-io,tlsmail,archive-tar-minitar,daemons,fastthread,gem_plugin,cgi_multipart_eof_fix,mini_portile,racc,tenderlove-frex,rexical,weakling,rails-sandbox-assets,sexp_processor,SexpProcessor,test-spec,camping,fcgi,ruby-openid,thin,escape_utils,activerecord,activeresource,actionwebservice,ZenTest,validatable,temple,sprockets-helpers,tins,text-hyphen,atomic,hpricot
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: ["activerecord-deprecated_finders", "arel",
"rails-observers",
"little-plugger", "loquacious", "bones-git", "bones-extras", "markaby",
"mab", "nio4r", "win32-security", "win32-dir", "CFPropertyList",
"sys-admin", "win32-api", "windows-api", "windows-pr", "net-scp",
"trollop", "descendants_tracker", "oauth2", "faraday", "hashie",
"termios",
"metaclass", "multimap", "loofah", "ruby-yadis", "flexmock",
"eventmachine", "eventmachine-le", "http_parser.rb", "preforker",
"hitimes", "facets"]
Query Gemcutter Dependency Endpoint API:
activerecord-deprecated_finders,arel,rails-observers,little-plugger,loquacious,bones-git,bones-extras,markaby,mab,nio4r,win32-security,win32-dir,CFPropertyList,sys-admin,win32-api,windows-api,windows-pr,net-scp,trollop,descendants_tracker,oauth2,faraday,hashie,termios,metaclass,multimap,loofah,ruby-yadis,flexmock,eventmachine,eventmachine-le,http_parser.rb,preforker,hitimes,facets
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: ["bones-rcov", "bones-rubyforge", "bones-rspec",
"bones-zentest", "libxml-ruby", "multipart-post", "mkrf", "net-ssh",
"httpauth", "jwt", "faraday-middleware", "multi_xml", "test-unit",
"curses"]
Query Gemcutter Dependency Endpoint API:
bones-rcov,bones-rubyforge,bones-rspec,bones-zentest,libxml-ruby,multipart-post,mkrf,net-ssh,httpauth,jwt,faraday-middleware,multi_xml,test-unit,curses
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: ["needle", "jruby-pageant", "power_assert"]
Query Gemcutter Dependency Endpoint API:
needle,jruby-pageant,power_assert
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: ["pattern-match"]
Query Gemcutter Dependency Endpoint API: pattern-match
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: []
Fetching additional metadata from https://rails-assets.org/
Query List: ["faraday-middleware", "active_support"]
Query Gemcutter Dependency Endpoint API:
faraday-middleware,active_support
HTTP GET
https://rails-assets.org/api/v1/dependencies?gems=...
HTTP 200 OK
Query List: []
Fetching additional metadata from https://rubygems.org/
Query List: ["faraday-middleware", "active_support"]
Query Gemcutter Dependency Endpoint API:
faraday-middleware,active_support
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
Query List: []
Resolving dependencies.......................................


and I needed to make sure bundler does not run over any proxy (which I
usually do). my setup rbenv, jruby-1.7.18, bundler-1.7.12 on ubuntu with
java 1.7.0_72

how far does bundler comes with you before switching to the "full
index",
usually bundler gives three trials but has a rather short timeout on
those
trials

- christian


On Tue, Jan 13, 2015 at 10:22 AM, Rodrigo Rosenfeld Rosas <
2abdb50caf0dc5b510330f68b02db8e4?d=identicon&s=25 Rodrigo Rosenfeld Rosas (Guest)
on 2015-01-13 19:28
(Received via mailing list)
In my case, this is what happens after some debugging:

bundle --retry=0 --jobs=1 --verbose
...
Query Gemcutter Dependency Endpoint API:
railties,actionmailer,sprockets-rails,sprockets,puma,foreman,jruby-openssl,rsolr,activesupport,rake,rspec,guard-rspec,nokogiri,tidy_ffi,fakefs,pry-byebug
HTTP GET
https://bundler.rubygems.org/api/v1/dependencies?g...
HTTP 200 OK
could not fetch from the dependency API, trying the full index

What happens is that even though the response is OK the body is empty
for some reason. The exception then happens here with data an empty
string:

# lib/bundler.rb
def load_marshal(data)
    Marshal.load(data)
rescue => e
    ...
end

Now, I have no idea why the response body is empry since obviously this
URL returns quite some data.

Basically in #fetch here's what happens:

response = request(uri)
case response
   ...
   when Net::HTTPSuccess # this is the selected option
         response.body # this is empty for the URI above

I gave up on debugging because it's too much complex to wire up things
in net/http/persistent in order for me to simulate a NET::HTTP request
in a pure JRuby irb session.

Ah, I forgot to mention I'm using the 2.0 mode in case that makes any
difference.

But it seems like a bug in JRuby net/http:

irb -r net/http
 > Net::HTTP.get 'bundler.rubygems.org',
'/api/v1/dependencies?gems=railties,actionmailer,sprockets-rails,sprockets,puma,foreman,jruby-openssl,rsolr,activesupport,rake,rspec,guard-rspec,nokogiri,tidy_ffi,fakefs,pry-byebug'
=> ""

This works on MRI 2.2.0.

This is the complete output of ruby -v for my JRuby:

jruby 1.7.18 (2.0.0p598) 2014-12-22 625381c on Java HotSpot(TM) 64-Bit
Server VM 1.8.0_25-b17 +jit [linux-amd64]

I tried to force the 1.9 mode and it worked for my surprise:

export JRUBY_OPTS=--1.9
bundle

In 1.9 mode it worked but in 2.0 mode it doesn't. Any ideas why?

Best,
Rodrigo.
B05d3cbc64b0031a24c2887fb6ddc173?d=identicon&s=25 christian (Guest)
on 2015-01-13 19:58
(Received via mailing list)
now I can reproduce it with 2.0 mode :(

no idea why - only that 2.0 is not fully supported with jruby-1.7.x
2abdb50caf0dc5b510330f68b02db8e4?d=identicon&s=25 Rodrigo Rosenfeld Rosas (Guest)
on 2015-01-13 20:18
(Received via mailing list)
Thanks for confirming. I've created a ticket for this issue:

https://github.com/jruby/jruby/issues/2457
This topic is locked and can not be replied to.