A Heartfelt Plea for a Universal Open Source Package Managem

Ruby
1

Paulus E. wrote:

and “suggests” dependencies. (after all, apt is probably the mother of

FWIW, I’d much appreciate adding such a notion to gem.

Paulus

Which brings up a very interesting question/comment/pet peeve of mine.
Every Linux distro has a package management system, but there really is
no “universal” one. Debian has “apt”, Red Hat and quite a few others
have something based on RPM, Gentoo has Portage, etc. OK … that’s
fine, maybe. Now the languages: Perl has CPAN and tools to make use of
it, plus ActiveState has their own Perl Package Manager and
repositories. Ruby has RubyGems, R has CRAN and its own package
management system. I don’t do much with PHP, Python or Tcl, but I know
they have their own way of doing things, and that in the case ot Tcl/Tk
and Python, there’s also an ActiveState way. That’s decidedly not fine
– I spent two days at my day job recently wrestling with just the CPAN,
Cygwin and ActiveState differences and that’s just Perl! Nobody is going
to take me seriously if I suggest that Ruby is a better way if they have
to learn another package management system and I have to spend my time
teaching people how to use it. :slight_smile:

So – why isn’t there a universal open source package management system?
The space is large, but finite, and I don’t think the growth rate, at
least in non-testing-level packages, is so high that it couldn’t be
managed with open source volunteers. After all, that’s the way the dozen
or so major ones are managed now. How much of this precious resource –
open source volunteer time-- is being wasted because:

a. there are three major Linux dependency trees plus the BSD Ports
system,
b. every open source package has to be re-packaged and re-tested in
source form and in binary form for a number of architectures in all the
package management systems,
c. all of the major languages (Perl, Python, PHP, Tcl/Tk, Java, R) have
not only their internal dependency trees to worry about but underlying
OS dependencies as well?

Red Hat has proved in spades that this is something people will pay cold
hard cash for. No matter what anyone thinks about the superiority of
“apt” or Portage or anything else, RPM, up2date and yum are as close to
a “universal” open source package management system as I’ve been able to
find so far. You can load an RPM on a Debian or Gentoo box and tell the
local package manager that you’ve thus satisfied all the dependencies
that package satisfies. So – why can’t a Gem in fact be an RPM? Why
can’t a Perl package, an R package, etc. be an RPM? Why does everyone
have to re-invent the package management system, especially when Red Hat
placed RPM and most of its infrastructure under GPL years ago?

–
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.

2

On 1/13/07, M. Edward (Ed) Borasky [email protected] wrote:

I suggest looking at Debian’s way of doing it. They have “recommends”
two concepts should do the trick.
no “universal” one. Debian has “apt”, Red Hat and quite a few others
to learn another package management system and I have to spend my time
b. every open source package has to be re-packaged and re-tested in
find so far. You can load an RPM on a Debian or Gentoo box and tell the
If God had meant for carrots to be eaten cooked, He would have given rabbits fire.

Under the same question, why can’t a gem be a DEB, or why can’t a perl
package be a GEM, or a gem a perl package, or an R package, or a port?

3

M. Edward (Ed) Borasky wrote:

Red Hat has proved in spades that this is something people will pay cold
hard cash for. No matter what anyone thinks about the superiority of
“apt” or Portage or anything else, RPM, up2date and yum are as close to
a “universal” open source package management system as I’ve been able to
find so far. You can load an RPM on a Debian or Gentoo box and tell the
local package manager that you’ve thus satisfied all the dependencies
that package satisfies. So – why can’t a Gem in fact be an RPM? Why
can’t a Perl package, an R package, etc. be an RPM? Why does everyone
have to re-invent the package management system, especially when Red Hat
placed RPM and most of its infrastructure under GPL years ago?

I understand what you mean but there is need for a much better
infrastructure, then. Gems allow install to personal directories, for
instance, which RPM and DEBs obviously can’t provide. Moreover, where
would you put manpages for a personal installation ?

Then, there is the so called ‘maintainer scripts’, that is the ones
which are dealing with updating /etc/… configuration files, setting up
rc.* bootup links, registering fonts, making stuff available for
menus… That’s not a simple thing.

BTW, apt is also GPLed… Like many other package managing systems
;-)… I’m every time sceptical about one-for-all solutions: either they
don’t work for everyone, either they’re too complex no one will ever
choose to move from their current way system to the ultimate solution…

Cheers,

Vince

4

Under the same question, why can’t a gem be a DEB, or why can’t a perl
package be a GEM, or a gem a perl package, or an R package, or a port?

–
Chris C.
concentrationstudios.com
brynmawrcs.com

The point was that there be a universal packaging system. I don’t think
anyone would mind using Baltar’s QQQ packing system if it was universal,
open source, extensible, and worked as advertised.

I’m all for a universal packaging system. But I’ll settle just for a
universal reusable Ruby system.
Gems, Plugins, Engines, Components … Whiskey Tango Foxtrot ?

Matt

5

Chris C. wrote:

Under the same question, why can’t a gem be a DEB, or why can’t a perl
package be a GEM, or a gem a perl package, or an R package, or a port?
There’s no reason it can’t … all I am saying is that there are some
very brilliant people spending significant amounts of time packaging
valuable open source software in multiple formats, and I think that’s
wrong. Most of the common gems, for example, are wrapped in Gentoo
ebuilds. A Gentoo developer, most likely a volunteer, does that, so on
my system, I can either type “gem install --remote rails” and add a line
for rails to “/etc/portage/package.provided”, or “emerge -v rails”.
There’s probably someone else that has to do this for Debian, and for
Red Hat/Fedora/CentOS, etc. And when a new Ruby package comes out,
there’s a lag as it makes its way through the gem system to a .deb, a
.rpm, a .ebuild, etc. In addition, because of the labors of one
individual, a large number of R packages are packaged for Debian but
nobody has volunteered to do the same for Red Hat or Gentoo formats.

In most cases, the dependencies, both inside the Ruby infrastructure and
in the interactions between Ruby and the compilers and operating
systems, are identical – the only thing that differs is the physical
format of the package and whether or not compilable language components
are pre-compiled or not. And at least for Linux, the same is true for
Perl, Python, PHP, Tcl/Tk, R, LaTeX and probably lots of others. A –
perhaps “standardized” is a better word than “universal” – package
management system would free up developers and users to focus on
domain-specific value creation and not worry about packaging. It’s all
automated anyhow. That’s part of my complaint – the numerous package
systems also consume developer resources in developing and maintaining
the automation infrastructure.

–
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.

6

On Sun, Jan 14, 2007 at 05:52:19AM +0900, M. Edward (Ed) Borasky wrote:

Chris C. wrote:

Under the same question, why can’t a gem be a DEB, or why can’t a perl
package be a GEM, or a gem a perl package, or an R package, or a port?

There’s no reason it can’t … all I am saying is that there are some
very brilliant people spending significant amounts of time packaging
valuable open source software in multiple formats, and I think that’s
wrong.

When we speak of packaging systems we expect several features, among
them:

  1. anything that’s installable comes in a single, downloadable file
    (e.g.
    .gem, .deb, .rpm)
  2. there is a way of discovering which package(s) are needed for
    specific
    functionality
  3. the package installation process, once initiated, results in fully
    installed and ready to use functionality
  4. package files identify other packages on which their functionality
    depends
  5. the installation process therefore retrieves the package we request
    and
    automatically retrieves all packages on which it depends, and
    installs
    them all

It’s really easy to get #1 working, and to keep #1 working when
converting
between packaging systems. There is the aforementioned gem2spec, which
will
turn a RubyGem into an RPM. There is alien, which will convert between
RPM,
.deb, Stampede (no idea), and tarball (Slackware). The resulting package
will be installable on whatever target system, but that doesn’t mean it
will fulfill any of the other features mentioned above.

Those of us who have been using Linux for enough years remember RPM
Hell,
which has been largely (though not completely) solved with the advent of
apt-rpm, yum, YaST, and similar. The problem wasn’t that the RPM format
was
inherently bad as a package format, the problem was that there was no
clearly defined and rigorously adhered to naming convention for
dependencies, nor significant quality control around third-party RPMs.

Debian solved this problem by laying out strict naming guidelines, by
making it much easier for packages that would otherwise be third-party
to
be included in the Debian repository, and by running automated tests on
the
packages in the repository to enforce quality control. (I’m not claiming
it’s perfect, but I don’t believe any other packaging
system/distribution
has anywhere near as many packages or developers working together such
that, in the stable distribution at least, installing a package will
never
step on the toes of another package and will always be installed with
all
of its dependencies.)

There’s a lot of overhead in all that, and generally special-purpose
tools
(or dedicated packagers) are involved in repackaging foreign packages
for
use on a Debian system. One example of this is java-pakcage, which
provides
the make-jpkg tool that will convert a binary installer downloaded from
Sun
(or, on PPC, IBM) into an installable .deb with an appropriate name,
appropriate dependencies, and appropriate “provides” (as in, the package
is
named jdk-XXX-whatever but it provides java-compiler and java-runtime,
on
which other packages depend). I believe there is ongoing work to do the
same for gems. The escape hatch, of course, is to install it in
/usr/local
and manage it separately, which is what I do and will continue to do
until
a gem-package is created to make them play nice with the rest of Debian.

Most of the common gems, for example, are wrapped in Gentoo
ebuilds. A Gentoo developer, most likely a volunteer, does that, so on
my system, I can either type “gem install --remote rails” and add a line
for rails to “/etc/portage/package.provided”, or “emerge -v rails”.
There’s probably someone else that has to do this for Debian, and for
Red Hat/Fedora/CentOS, etc. And when a new Ruby package comes out,
there’s a lag as it makes its way through the gem system to a .deb, a
.rpm, a .ebuild, etc. In addition, because of the labors of one
individual, a large number of R packages are packaged for Debian but
nobody has volunteered to do the same for Red Hat or Gentoo formats.
[…]

The point I’m making above is twofold: first, a universal packaging
system
requires a whole lot more than just a file format, it requires buy-in
from
everyone involved on naming conventions and file placement and a variety
of
other things; second, the best solution is not to have packagers doing
the
repackaging, but to have someone create a tool for the target package
management system to convert packages from the foreign package
management
system.

M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC§
–Greg

7

Hi –

On Sun, 14 Jan 2007, matt wrote:

anyone would mind using Baltar’s QQQ packing system if it was universal,
open source, extensible, and worked as advertised.

I’m all for a universal packaging system. But I’ll settle just for a
universal reusable Ruby system.
Gems, Plugins, Engines, Components … Whiskey Tango Foxtrot ?

Plugins, engines, and components are Rails add-ons. They don’t
pertain to the matter of packaging and distributing Ruby libraries and
applications in general.

David

8

Vincent F. wrote:

I understand what you mean but there is need for a much better
infrastructure, then. Gems allow install to personal directories, for
instance, which RPM and DEBs obviously can’t provide. Moreover, where
would you put manpages for a personal installation ?

I don’t remember how this is done in Debian and Red Hat/Fedora, but
Gentoo has “overlays” for this purpose – directories outside the
“master” Portage tree where one can manage “local” or “personal”
packages, or do the same within a community.

Then, there is the so called ‘maintainer scripts’, that is the ones
which are dealing with updating /etc/… configuration files, setting up
rc.* bootup links, registering fonts, making stuff available for
menus… That’s not a simple thing.

There is – or was – an effort to create a “Linux Standard Base” that
is essentially the “common underbelly” of Red Hat, SuSE and Debian
standards. I haven’t heard much about that recently, so it may have
disintegrated because Red Hat is a corporation and Debian is a
not-for-profit and there wasn’t any clear financial benefit to both. :slight_smile:
Gentoo, on the other hand, is an entirely different kettle of fish. I’ve
been using Gentoo so long I am nearly incapable of administering a Red
Hat or Debian system at the config-file level. :slight_smile: Yet another example of
resources wasted because of lack of standardization.

BTW, apt is also GPLed… Like many other package managing systems
;-)… I’m every time sceptical about one-for-all solutions: either they
don’t work for everyone, either they’re too complex no one will ever
choose to move from their current way system to the ultimate solution…

Yeah, in complex, adaptive systems like “open source software”,
ultimately things change only when there’s some kind of “crash” or
“market shakeout”. So at the moment, we have the “local optimum” of
RubyGems for Ruby, CRAN for R, RPM, apt and Gentoo for Linux, CPAN and
ActiveState for Perl, and nobody rich enough to say, “I will only
support open source software if it’s 100 percent Red Hat Enterprise
Linux compatible, packaged as RPMs, etc.” In the past, though, a number
of things have been standardized because the government of the USA did
just that – said, “if it doesn’t meet this standard, you can’t sell it
to us.” Will that happen to Linux? It’s quite possible. Will there be an
ANSI Standard for Perl, Python, PHP and Ruby? It’s quite possible.

–
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.

9

On Sun, 2007-01-14 at 04:43 +0900, [email protected] wrote:

brynmawrcs.com
Plugins, engines, and components are Rails add-ons. They don’t
pertain to the matter of packaging and distributing Ruby libraries and
applications in general.

David

That was my point exactly! Rails is nothing more than an extension of
Ruby via Gems packaging, but now there are more choices (or should I say
confusions), that break away from universality. I know this is a Ruby
list and not Rails, so I’ll break with this: The use of a universal
packaging system would flow out to any extensions of the language as
well, so a if Gems was the way to go, then, Rails would have to as well
(in a perfect, utopian, no more advancement, socialistic kind of way),
but one that would still fit the bill of the original topic.

Matt

10

M. Edward (Ed) Borasky wrote:

systems also consume developer resources in developing and maintaining
the automation infrastructure.

You seem to forget one thing: language-specific packagings are in
principle cross-platform (including macos,*nix,win…), which is
obviously not the case of distribution-specific packaging. You’ll never
get a global packaging solution, just like no language will be the
ultimate solution for all programming.

Maybe what we’re missing are good conversion tools ? Say, a gem2deb
program that would get most of the debian developer’s job done before
the final touch ?

Vince

11

Gregory S. wrote:

The point I’m making above is twofold: first, a universal packaging system
requires a whole lot more than just a file format, it requires buy-in from
everyone involved on naming conventions and file placement and a variety of
other things;
Agreed … it requires a lengthy standardization process – a committee
of stakeholders, a project management infrastructure, funding, etc. At
a certain level, I thought that’s what the Linux Standard Base was all
about. Maybe that’s still going on and I haven’t heard anything about it
recently. In any event it only initially requires “buy-in” from
dissatisfied “customers”, or someone like the US Government to get the
ball rolling. Then it doesn’t require “buy-in”, it requires “stay-in”,
which is much more difficult to acquire. :slight_smile:
second, the best solution is not to have packagers doing the
repackaging, but to have someone create a tool for the target package
management system to convert packages from the foreign package management
system.

Yes, automation is the best way to do it. To minimize the amount of
effort in building the tools, a standardized target needs to exist and
the tools to convert from the dozens of open source package systems
to the standard are the only tools that should be built. I suspect
when all the smoke clears, it could play out like the ASCII-EBCDIC war
in the 1960s. A large vendor, IBM, invented EBCDIC and made it a “de
facto” standard, but the US government mandated that all the computers
it bought had to support ASCII. So IBM and the clones used EBCDIC and
offered tools to work with ASCII, and everyone else supported ASCII
only.

By analogy, I can imagine Red Hat and the clones using RPM and some
humongous RPM source repository, but the US government mandating the
Debian format and repository for all the Linux systems it buys, etc.,
because I think the Debian way is technologically superior to the RPM
way. And Debian supports more architectures in binary format than Red
Hat.

–
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.

12

On 1/13/07, matt [email protected] wrote:

That was my point exactly! Rails is nothing more than an extension of
Ruby via Gems packaging, but now there are more choices (or should I say
confusions), that break away from universality. I know this is a Ruby
list and not Rails, so I’ll break with this: The use of a universal
packaging system would flow out to any extensions of the language as
well, so a if Gems was the way to go, then, Rails would have to as well
(in a perfect, utopian, no more advancement, socialistic kind of way),
but one that would still fit the bill of the original topic.

There are folks who are working on a system for enabling gems as
plug-ins to Rails, IIRC. They wanted to depend on certain things
(e.g., autorequire) that are going away, but they can enable such
things with convention and scaffolding better than they can with
misfeatures.

-austin

13

Chris C. wrote:

One of the big problems, and the reasons it may take so long for every
package/OS distro to have every package is because every distro is
different. Debian may install packages in /usr/local, BSD may choose
/opt, and every distro has their own requirements for packages.
Yes, exactly my plea – standardize all of this so all the distros and
all of the open source languages do it the same way. That frees a lot of
people to add other value. Red Hat can add enterprise stability value,
Debian can add “free software rigor” value, R can add open source
graphical and statistical value, Ruby can add object-oriented scripting
value, etc. And for the rest of the world, Microsoft and Apple can go on
building home entertainment centers. :slight_smile:

–
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.

14

Chris C. wrote:

One of the big problems, and the reasons it may take so long for every
package/OS distro to have every package is because every distro is
different. Debian may install packages in /usr/local, BSD may choose
/opt, and every distro has their own requirements for packages.

Asking for the One True Package Manager seems similar to asking for the
One True Programming Language.

Be careful what you wish for.

–
James B.

“Every object obscures another object.”
- Luis Bunuel

15

M. Edward (Ed) Borasky wrote:

Agreed … it requires a lengthy standardization process – a
committee of stakeholders, a project management infrastructure,
funding, etc. At a certain level, I thought that’s what the Linux
Standard Base was all about. Maybe that’s still going on and I haven’t
heard anything about it recently. In any event it only initially
requires “buy-in” from dissatisfied “customers”, or someone like the
US Government to get the ball rolling. Then it doesn’t require
“buy-in”, it requires “stay-in”, which is much more difficult to
acquire. :slight_smile:
The good news is: Linux Standard Base (LSB) is very much alive and well.
And there is more good news – they just (December 2006) established a
working group on packaging. See
http://www.freestandards.org/en/Packaging and
http://ianmurdock.com/?p=388 for some links.

The bad news: Red Hat, SuSE and Debian are represented, Gentoo is not.
The other bad news: Perl and Python are represented, Ruby and R are not.

I’ll go bug the Gentoo and R folks – somebody here want to to represent
Ruby? :slight_smile:

–
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.

16

On 1/13/07, M. Edward (Ed) Borasky [email protected] wrote:

There’s probably someone else that has to do this for Debian, and for
are pre-compiled or not. And at least for Linux, the same is true for
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given rabbits fire.

One of the big problems, and the reasons it may take so long for every
package/OS distro to have every package is because every distro is
different. Debian may install packages in /usr/local, BSD may choose
/opt, and every distro has their own requirements for packages.

17

On 1/13/07, James B. [email protected] wrote:

One True Programming Language.

Be careful what you wish for.

–
James B.

“Every object obscures another object.”
- Luis Bunuel

Don’t worry, I like both gems and deb and I wouldn’t want to be stuck
with just one.

18

On 1/13/07, Vincent F. [email protected] wrote:

Maybe what we’re missing are good conversion tools ? Say, a gem2deb
program that would get most of the debian developer’s job done before
the final touch ?

I don’t know about debian, but David Lutterkort produced a nice
solution for gem to rpm conversion:
http://people.redhat.com/dlutter/gem2spec.html.