Removing maven support from RubyGems?

Ruby JRuby
1

Well folks, it was a great experiment, but I’m thinking maybe we
should bail on the idea of integrating maven directly into RubyGems.
There’s a number of reasons I think we should ditch it:

  • It would bring us closer to running stock RubyGems, so you can all
    just use update --system to get up-to-date RubyGems as versions come
    out.
  • It has never worked perfectly; there are dependency graphs that
    can’t be represented well, Maven versions that don’t map at all to
    RubyGems versions, and Maven and RubyGems have different ways of
    specifying “soft” version dependencies.
  • It has never worked on Windows; we chose poorly when we went with :
    as a group:artifact separator.
  • RubyGems folks have never really been interested in incorporating
    our changes or supporting Maven directly.
  • Bundler folks have never really been interested in supporting Maven
    directly, and would have run into the same issues as above.

It was a really nice feature that we just never were able to implement
well, given RubyGems versioning and dependency logic.

Is anyone using RubyGems Maven support in JRuby? Would you miss it?

  • Charlie
2

On Wed, Sep 12, 2012 at 2:17 AM, Charles Oliver N.
[email protected]wrote:

Well folks, it was a great experiment, but I’m thinking maybe we
should bail on the idea of integrating maven directly into RubyGems.

Maybe, it would be better with a tool for generating a gem from jar
files.
For example, if a-0.1.jar depends on b-0.2.jar, and this depends on
c-0.3.jar, we will have 3 gems (a-0.1gem, b-0.2.gem, c-0.3.gem). Any
other
jar which depends on c-0.3.jar could reuse the c-0.3.gem

This requires a some manual work for JRuby users, but it will work in
any
environment where RubyGems/Bundler is ready.

3

I want to add one more thought about the rubygems maven support. when
we create a gem out of a jar we mapped the jar version somehow to
rubygems version - given it some version range. since maven version do
resolve totally differently from rubygems, the need arose to make the
mapped version range bigger and bigger.

to accommodate jar like joda-time where version 2.1 is backward
compatible with version 1.6.x then the mapping would basically means
“>0” or at least [<3, >=1] which means to almost omit version
constraints altogether.

the version mapping between jar and rubygems does not work. the
conclusion from that is to use the dependency resolution libraries
which do the right job. but that needs to lives outside of rubygems
and following the idea of bundler the project jbundler tries to fill
that hole.

(in case you really want to have a look:
GitHub - mkristian/jbundler: bundler support for jars for jruby - but you do not need to, there
are (java) projects which resolves their deps manually)

  • Kristian
4

This is coming from someone who has only used JRuby with rails -

I can imagine a possibility where maven could be used with a project and
asked to dump the jars in a particular directory from where the
application
does a blind ‘require everything’. Maven appears to be very
configurable.
All the directory locations are configurable, it can be made to point to
some empty directory for any directory that is required. There will need
to
exist a full fledged pom.xml at the application root.

It already has a ‘copy-dependencies’ task where it dumps all
dependencies
at a configurable location. It can also be made to write out a file
which
contains the classpath, including all the jar names in the format where
it
can be called from command line. All the other lifecycles can be made to
do
nothing.

People can use whatever they want from RubyGems and maven stuff can be
loaded from another place. There might be some gotchas for sure but
overall
it all doesn’t sound too out of the ordinary to me right now.

5

What if gems were OSGI bundles?

6

great idea Pradeep.

but we also need to find a solution to have gems depending on jars
instead of vendoring them as t done currently. here my proposal for
solving the “gem depends on a jar” problem:

here maven would need something more.

-Kristian

On Wed, Sep 12, 2012 at 10:47 AM, Pradeep S.

7

Maven has a lot of cruft that is not useful for a JRuby project. What we
really need is the transitive dependencies resolution provided by Maven.
That way a JRuby can just defined a handful of Jars and have all their
transitive Jars automatically installed.

This is the goal of LockJar -
https://github.com/mguymon/lock_jarhttp://mguymon.github.com/lock_jar/

Similar to Bundler, the flow is to create a Jarfile that is used to
generated a Jarfile.lock . The Jarfile.lock can be used to automatically
install Jars to the local system and populate the classpath. The Jar
dependencies from the Jarfile.lock can even be installed when a Gem is
installed.

Once Bundler gets plugins (
Bundler should have a plugin system · Issue #1945 · rubygems/bundler · GitHub ) I can add a plugin
that will sync up LockJar with Bundler. Then the Jarfile.lock will not
have to managed separately.

8

Kristian and I are starting to collaborate on a broader solution for
Gems with Jars, which should cover the goals of Kristian’s proposal. We
are still kicking around ideas and use cases that need covered.

Btw, LockJar will handle Jar dependencies for libraries. Even install
the Jars when a Gem installs. The only thing a lib needs to do is call
the LockJar.lock when it is required:

#get jarfile relative the gem dir
lockfile = File.expand_path( "../Jarfile.lock", __FILE__ )

LockJar.load( :lockfile => lockfile )

–M

9

hi Theo,

I just want to pick on rubygems. it loads gems lazy, whenever you
require ‘something’ it will looks in its gems to find ‘something.rb’
and “activate” any gem needed on the way. if you require
‘something_else’ then it will do the same for something else. in case
it needs a gem with a different version of an “active” gem, it bails
off with an error “already activate gems . . . .” (or so). in can be
that things are just fine when you first require ‘something_else’ and
then require ‘something’

rubygems gems resolution is also changing over time, i.e. it first
tries to load the latest version of a needed gem. so one day you use
gem A with version 2.0 and the next you will use version 2.1 because
on of your new gems pulled in the newer version. I can not even be
sure if someone else uses the gems as I since meanwhile there is even
newer version available.

so gems resolution with rubygems depends on the order at which gems
gets activated, i.e. also the execution order within your code can
matter. it is changing over time. IMO not really you want to rely on.

this would be also true if rubygems could resolve jar dependencies.

this lead to a lot of headache and then first rails (2.x) made their
own gem-dependeny-management and later bundler replaced it with rails
(3.x). the reason why bundler works much better is that look at a set
of dependencies and resolve them all together, it guarantees
reproducibility (over time).

I do see for gem projects following Gemfile

source :rubygems
gemspec

so just manage you gems with bundler for that project.

for me rubygems is a convenient tool for trying out things with or
without irb. it is not useful for manage the deps of a project,
library or application.

regards,
Kristian

PS
@Theo can I add you to our little discussion with Michael - more view
points are appreciated ?
@Micheal there is already an email in the pipeline but I am not so
fast right now :frowning:

On Mon, Sep 17, 2012 at 10:45 AM, Michel G.

10

On Mon, Sep 17, 2012 at 2:12 PM, Theo [email protected] wrote:

Hi,

(sorry if you get this twice, I had problems with my list subscription)

that’s why I did not receive your first email - I just saw today the
reply of Micheal :slight_smile:

OK, I see there are people around who just want to rubygems with rvm
or by setting GEM_HOME/GEM_PATH per project, or similar :wink:

since the maven_gem support never made it into rubygems the question
is what can be done to provide the desired functionality.

the only way see right now is something like

$ gem install_with_jars mygem

this could at least make sure there are some versions of the required
jars installed. the thing is to “activate” those jars. the need of
custom code depending on a tools really ties the tool (which should be
external IMO)

any ideas ?

  • Kristan
11

Hi,

(sorry if you get this twice, I had problems with my list subscription)

I’d love to be more deeply involved in this discussion. It’s something
I’ve been struggling with a lot since most of the gems I build work on
top of Java libraries.

About Rubygems: don’t get me wrong, I’m not saying I want Rubygems to
handle gem activation, I’m saying that for any solution to succeed
Rubygems needs to be involved in the install-time dependency resolution.
Everyone uses Bundler for version management, and many use RVM for
sandboxing, but those work on top of Rubygems dependency resolution
(whether or not Bundler is there managing exactly which versions gets
installed). I may not have gotten the exact technical definitions
correct there, but I hope you can parse out what I’m getting at.

In my opinion JRuby must be able to to the following, out of the box:
install all dependencies for a gem, including any JAR/Maven/whatever
dependencies of gems that the gem depends on.

This means that if gem A depends on JAR X, and gem B depends on gem A,
then installing B will install X, without the user having to specify
that X is needed.

Adding Bundler, or some other tool, into the mix, it can help making
sure no incompatible versions end up getting installed, or required in a
bad order – but the important thing is that the JAR got installed, and
that the user didn’t have to specify that a dependency of A has to be
installed when she installs B.

Consider a similar situation: Bundler has an awesome feature that lets
you specify a git repository as the source of a gem. However, the gem in
that repository cannot in turn depend on another git repository, because
Rubygems doesn’t support the feature, and Bundler doesn’t replace
Rubygems (for example, Bundler could look for a Gemfile in the
repository and discover transitive dependencies that way, but it
doesn’t). This means that if you want to live on the bleeding edge and
install a gem from a repository, and have all its dependencies installed
from their respective repositories, you have to specify all of the
dependencies in your Gemfile, and all the dependencies of all of the
dependencies (and so on) too.

Focusing on Bundler, JBundler, or LockJar or any other tool like that
does not solve that problem, unless we want to invent a parallel
dependency management environment to Rubygems. Being able to do what
Bundler can do with git repositories is trivial, what is needed is
something that handles the transitivity of installation, not just the
transitivity of version resolution.

The proposal to use the “requirements” property in gemspecs is nice,
because it works with Rubygems, but in my opinion it needs to go into
Rubygems itself, or at least into JRuby so that it works out of the box.
LockJar and JBundler are nice, but this problem needs to be solved at a
lower level.

T#

12

just thinking aloud :wink:

a gem plugin could be a way to hook something into the “gem install”.
whenever you call a gem command all plugins get loaded. maybe it is
possible to do so in way which survives a few updates of rubygems.

  • Kristian
13

On 09/17/2012 05:22 AM, kristian wrote:

or by setting GEM_HOME/GEM_PATH per project, or similar :wink:

since the maven_gem support never made it into rubygems the question
is what can be done to provide the desired functionality.

the only way see right now is something like

$ gem install_with_jars mygem
What about using the Gemspec’s external to handle downloading the Jar
dependencies when the Gem install? The external can be set to a
Rakefile, the only juggling act is that Rakefile’s default must be used
for the Gemspec external call. While not ideal (native support would be
nice), it does at least give the ability to package Gems and have them
install all dependencies.

A gem plugin is a good idea. To be able to see what jars a Gem is
dependent on, re download Jars, etc.

14

On Mon, Sep 17, 2012 at 10:58 PM, kristian [email protected] wrote:

just thinking aloud :wink:

and led to some discussions but a proof of concept:

a little demo is here:
$ jruby -S gem install rubyjams
$ jruby -S gem install nokogiri-maven
$ jruby -rubygems -e ‘require “rubyjams”;gem “nokogiri-maven”; require
“nokogiri”’`

so the install of nokogiri-maven downloads and installs dependent
jars. the gem ‘nokogiri-maven’ chooses the right gem (and not just
nokogiri) and the last require will add the dependent jars to the
classloader.

so the idea is to mimic rubygems for jars. keeping things as
symmetrically as possible to people find themselves on familiar
ground. of course it is not at all perfect and might fail rather soon.
but if you need more control over the jars you need to use a tool like
jbundler.

regards,
Kristian