Anyone interested in a Package Manager in Ruby?

Ruby
1

Hi,

About five years ago I started writing a few ruby scripts to help me
compile software from source, typically for GNU autoconfigure based
systems.

Basically the idea was quite simple. For instance, do:

compile htop

And with this you would compile the program called htop.

compile htop /usr/local

To compile with /usr/local prefix. (The --prefix options and so
on are also available.)

compile htop non_traditional
compile htop ntrad

Compile into standalone directory (Default scheme for this right
now is via the $PROGRAMS variable, which for me points at /Programs.
This would compile Htop into /Programs/Htop/0.9.4 for instance but
can be easily changed to use other prefixes like /pkg and other
convention schematas as well))

compile htop with_dependencies
compile htop wdeps

Would compile htop with the dependencies (the long name
“with_dependencies” is the real “action” and “wdeps” is
just an alias to this “action”, for convenience.)

compile htop url

Would feedback the main url used for compilation (colourized on a
terminal) like this:

=> URL for htop points at:
=> http://dfn.dl.sourceforge.net/sourceforge/htop/htop-0.9.tar.gz
=> htop download | SourceForge.net

Would use the URL
http://dfn.dl.sourceforge.net/sourceforge/htop/htop-0.9.tar.gz

If the program htop would not be available locally, these scripts
would try to download it, extract it, compile it.

Over the years this has grown a bit.

For example, one idea was to generate valid standalone shell scripts for
compilation (“compile htop generate_shellscript”) that would write a
valid shell script which can be used to compile and extract something.

Generation of valid ebuilds/PKGBUILD/RPMs etc… would also be possible
(partially buggy or incomplete right now though.)

And use binaries too, when available.

It should also work with cmake, scons and so forth.

Compiled files will be tracked and stored in a yaml file right now (this
I will have to change soon, my current yaml file has 5 MB in size and
78600 file entries and this seems to become a bit too slow for my daily
use.)

Now the thing is, my scripts work quite ok for my own use cases.

There are a few bugs here and there, especially when I change behaviour
or feature set. Plus, the style of these scripts is a bit strange … 5
years ago my Ruby knowledge wasn’t really that great. It still isn’t
great, but I got somewhat better. If I were to start from scratch, I
would rewrite this all very, very differently.

But the main point of this here is:

  • Is there anyone else out there who may have a use case for something
    like this, even if there are bugs and countless things that could be
    improved?

If there is some interest, I will try to improve things in a way that
others can use these scripts too. For my own use, it is ok. I can live
with some bugs and fix them every once in a while when it is fun. But
for a general release, it would require more time invested into it and
before I do this I would like to know if there is anyone else who could
actually have a use for this too.

Otherwise, I’ll stay lazy, which isn’t so bad either. :slight_smile:

2

Marc H. wrote in post #1009838:

  • Is there anyone else out there who may have a use case for something
    like this, even if there are bugs and countless things that could be
    improved?

Have you compared your scripts against homebrew?

http://mxcl.github.com/homebrew/

3

Hmm.

Homebrew is the easiest and most flexible way to install the
UNIX tools Apple didn’t include with OS X.

Haven’t yet, but what Homebrew does seems super trivial. Just like
stow, right, but written in ruby? And constrained to Apple.

Here the link to stow:
ftp://mirrors.usc.edu/pub/gnu/stow/stow-1.3.3.tar.gz

Perhaps I am missing something.

Packages are installed into their own isolated prefixes and
then symlinked into /usr/local.

Well, my scripts do that since 5 years … it’s what Gobolinux does
too, except that it symlinks into /System/Links/* … even in a
chrooted compile environment, unless I misremember. The place for
/usr/local can be freely chosen.

Homebrew formula are simple Ruby scripts:

There is where my scripts would differ probably. My dataset is in yaml
files right now. But you can generate ruby standalone classes or
standalone shell scripts too. (And eventually valid formats for ebuild,
PKGBUILD etc… as well, but it is all not 100% complete)

Last but not least, the ruby scripts here should work anywhere, no
matter the OS. If ruby runs, the scripts should work too.

I used the word “Package Manager” but the scripts are really more like a
set of tools. I.e. it should be possible to eventually compile a LFS
(Linux from Scratch) with them too (once ruby works on the system, that
is, without ruby apparently the scripts can not work)

It’s only a hobby … I am working on it when it is fun and do so fairly
slowly - when I need something, I add it eventually. So it is mostly a
one person project currently. I am trying to find out if anyone else
might need it, if not this is also fine as it would save me work, lazy
as I am. :slight_smile:

4

Hi, thanks Marc W…

nixpkgs (www.nixos.org) should also be known by you.

Yup, nixos is a GREAT idea. Always was much more sophisticated than
Gobolinux (I am still sad that Gobolinux is more or less dead :frowning: )

How I understood it though, NixOS does not retain the elegance and
simplicity of Gobolinux? Upgrades of programs are simple on Gobolinux
too, as is keeping settings. As said though, it is not as advanced
as NixOS, I think the ideal solution would be a mix of NixOS and
Gobolinux :wink:

a) ruby is supported pretty well :slight_smile:
b) it has many nice features such as atomic upgrades,
distributed building etc.

Interesting, especially the atomic upgrades while retaining 100% of
the old behaviour set.

I think in my current workflow, I’d prefer to use nixos on top of
an existing linux installation though. I have become very lazy
and reluctant to install something new, even if it may be better,
it kind of breaks my workflow, and I don’t like that.

I mean that if I would have to install NixOS as a complete
distribution.

Thus before you write a new tool check it out and give it a
try. Don’t think there are that many OSX users yet - but
the backbone of the system is pretty stable.

Yes, probably. My current idea here is that whoever has an itch
to scratch when compiling or installing something, a tool to
write or perhaps already has (in Ruby), he can find a place
to contribute without too much fuzz. Like an “umbrella” project
to store and gather useful scripts. (They need to integrate
though, hence perhaps may have to be adjusted. There would be
nothing worse than use 100 different tools that behave completely
inconsistent… that would only confuse users and developers.)

marc

5

Excerpts from Marc H.'s message of Sat Jul 16 04:10:44 +0200 2011:

Yup, nixos is a GREAT idea. Always was much more sophisticated than
Gobolinux (I am still sad that Gobolinux is more or less dead :frowning: )
It depends on what you want. nixpkgs was designed to generate results
you can regenerate on a different machine (if possible).
That’s a unique feature. It was also designed to offer atomic upgrades.
From this point it follows that it should support multiple versions.
Because you may still be using

How I understood it though, NixOS does not retain the elegance and
simplicity of Gobolinux?
Well. Don’t use a computer but a pencile. You can’t be more simple :slight_smile:
I mean if you want the features described above you can’t make it
simpler.

Interesting, especially the atomic upgrades while retaining 100% of
the old behaviour set.
Well - its not quite true. Application state (eg gimp’s configuration)
is stored in ~/.gimp. Unfortunately it also has some pointers to
/nix/store/xxxxx-this-gimp-build. Thus if the hash xxx changes your
~/.gimp is broken. This results in gimp no longer being able to open
jpeg files. However this kind of problem is rare and it can easily be
fixed.

I think in my current workflow, I’d prefer to use nixos on top of
an existing linux installation though. I have become very lazy
and reluctant to install something new, even if it may be better,
it kind of breaks my workflow, and I don’t like that.
nix: the package manager
nixos: the linux distribution based on nix (and the nixpkgs build
descriptions).

The nice thing: because everything is in /nix/store it does not
interfere with your existing system much.

Do whatever you want :slight_smile: I wanted to make sure you don’t miss nixpkgs
cause I think it the best system available. There are some drawbacks
though. Eg if you rebuild bash everything has to be rebuild because you
want deterministic results :confused:

yours
Marc W.

6

Marc H. wrote in post #1009838:

Hi,

About five years ago I started writing a few ruby scripts to help me
compile software from source, typically for GNU autoconfigure based
systems.

…

Hi, May I ask how this post was made? Email? Google Group? Ruby-Forum?
or Other?

I ask because it did not show up in the Google Group, nor in the
ruby-talk archives.

Thanks.

(Posted this via Ruby-Forum)

7

nixpkgs (www.nixos.org) should also be known by you.
a) ruby is supported pretty well :slight_smile:
b) it has many nice features such as atomic upgrades, distributed
building etc.

Thus before you write a new tool check it out and give it a try. Don’t
think there are that many OSX users yet - but the backbone of the
system is pretty stable.

Marc W.

8

Hi, May I ask how this post was made? Email? Google Group?
Ruby-Forum? or Other?

I ask because it did not show up in the Google Group, nor in
the ruby-talk archives.

It was made via http://www.ruby-forum.com/ and the login there.

I have the “edit” functionality here so it feels more like a forum
than an email. I think this may be the reason why some people
do not recieve it by email perhaps.

The edit functionality is nice, I can edit my message before
“sending” it. But perhaps the “edit” functionality should be
removed after a day or two, and a real email-message should be
sent so that everyone recieves it. (Just offering a suggestion
here)

9

Excerpts from Thomas S.'s message of Sat Jul 16 05:38:13 +0200 2011:

I ask because it did not show up in the Google Group, nor in the
ruby-talk archives.
If you use mutt, sup or the like you can edit all header messages.
Then you can keep “In-reply-to” fields by accident when replying to
an arbitrary message. My sup showed “one unreceived message”.

Not sure whether this helps you.

Marc W.

10

Hi Marc W.,

It depends on what you want. nixpkgs was designed to generate
results you can regenerate on a different machine (if possible).

That’s a unique feature.

I agree with you. :slight_smile:

It was also designed to offer atomic upgrades.
From this point it follows that it should support
multiple versions.

Ok, I see. NixOS comes from a slightly different angle here.

Versioned directories are very simple and primitive compared to
what NixOS is doing. But both are much better than the FHS
solution to spread everything out directly without any structure
at all.

Don’t use a computer but a pencile.

But I like the computer because I make less errors.

When I write with a pencil, I sometimes can’t even read my own
writing. :slight_smile:

But I can always read computer-text.

You can’t be more simple :slight_smile:

Well. I guess the FHS is simpler than Gobolinux (and much simpler
than NixOS) because you do not have ANY real structure with the
FHS other than an arbitrary convention of where things should
go - but this convention was derived 99% because it already was
there. Reasons why /usr or /etc exist, because the old C hackers
were lazy to type longer words. You will never see /system or
/System, you will only see /sys because they HATE to type long
names. Or uppercased words as well. This also means they will
NEVER change to ANYTHING that makes them type more … even if
the resulting structure is more logical.

I think /Users is a lot more logical than /usr. Yet, /usr is
short for “users”, and not for the backronym “unix system resource”
which would not make a lot of sense on linux either. We could
use a /lsr if usr really meant “unix system resource”.

I mean if you want the features described above you can’t
make it simpler.

I am not disputing this, I can see that. However I am so far
using versioned directories and it is working beautifully and
is very simple. This is the only thing I disliked about NixPS
so far, which is that directories like:

/nix/store/5rnfzla9kcx4mj5zdc7nlnv8na1najvg-firefox-3.5.4/

Aren’t friendly to my eyes. And if things are needlessly
complex to my eyes, then my eyes complain to my brain and
my brain starts to enter the “not want to” mood. I’d even
suggest for NixOS to overlay these directories into a
/Programs structure (or anywhere else the user wants to)
with a clear versioned structure. Well, actually … Gobolinux
could be a subpart of NixOS where NixOS would manage
everything Gobolinux does - but also gives the user the
ability to entirely stay within the simplicity and beauty of
/Programs/ProgramName/VersionNumber :wink:

This is probably not completely possible as NixOS seems to do
more than that under the hood, but still, I myself am too
much in love with the abovementioned scheme to abandon it.

But as said before, the NixOS model rocks.

Well - its not quite true. Application state (eg gimp’s
configuration) is stored in ~/.gimp. Unfortunately it also
has some pointers to /nix/store/xxxxx-this-gimp-build.
Thus if the hash xxx changes your ~/.gimp is broken.
This results in gimp no longer being able to open jpeg
files. However this kind of problem is rare and it can
easily be fixed.

Interesting.

nix: the package manager
nixos: the linux distribution based on nix (and the nixpkgs
build descriptions).

The nice thing: because everything is in /nix/store it does not
interfere with your existing system much.

Do whatever you want :slight_smile: I wanted to make sure you don’t miss
nixpkgs cause I think it the best system available. There are
some drawbacks though. Eg if you rebuild bash everything has
to be rebuild because youwant deterministic results :confused:

Well, it sounds very interesting. So I could skip NixOS as a
distribution and solely use Nix. This is making it a lot more
interesting. :slight_smile:

Because if I can use nix without necessarily having to use NixOS
at the same time, then I can try it out, try things with it, and
so on and so forth.

I’ve become too much of a tinkerer.

Thanks for your writing by the way, appreciated!

11

To Brian,

Have you compared your scripts against homebrew?

http://mxcl.github.com/homebrew/

I had some time to look at homebrew the ~ last 2 days.

First, at #homebrew. Well … 42 people … no reply to a question I
asked.

:frowning:

So I went to the site and noticed they used Github. This is GREAT.

They have a structure with about 1600 formulas.

Basically, a formula provides program information.

A formula is a ruby file which looks like this:

https://github.com/mxcl/homebrew/blob/master/Library/Formula/simh.rb

It uses methods to convey the information:

url ‘http://simh.trailing-edge.com/sources/simhv38-1.zip’
version ‘3.8-1’
homepage ‘http://simh.trailing-edge.com/’
md5 ‘e15f65a82e21ea49e14b438326d93d5c’

url method probably points to the URL of the program where you can
download it … homepage provides some information … md5 the md5sum I
suppose… version may be something internal, or whatever. The -1 part
at the version is probably a revision number.

I am unhappy with the choice to store the data in pure ruby though.

I feel the data should be ideally decoupled from any language.

Take
https://github.com/mxcl/homebrew/blob/master/Library/Formula/a2ps.rb

You have boilerplate code there:

require ‘formula’

class A2ps < Formula

Perhaps I am too stringent on purity but to me it feels like a
suboptimal solution. The same could be achieved by a bit of
metaprogramming.

The class name derives from the file name.rb anyway …

The main script seems to be at:

https://github.com/mxcl/homebrew/blob/master/Library/Homebrew/formula.rb

There is another thing I don’t understand.

This could work on *nix systems just as easily. Just compile into a
standalone directory, register the files there, and symlink or copy them
into the FHS directory try. Why does homebrew want to be known as only
Mac OS usable? I don’t quite understand that.

There are some interesting strategies used in formula.rb though.

For instance, they bundle different download strategies together:

VCS_SYMBOLS = {
:bzr => BazaarDownloadStrategy,
:curl => CurlDownloadStrategy,
:cvs => CVSDownloadStrategy,
:git => GitDownloadStrategy,
:hg => MercurialDownloadStrategy,
:nounzip => NoUnzipCurlDownloadStrategy,
:post => CurlPostDownloadStrategy,
:svn => SubversionDownloadStrategy,
}

def initialize(url, specs = nil)
unless specs.nil?
# Get download strategy hint, if any
@using = specs.delete :using

detected = VCS_SYMBOLS[@using]

I found this nice. Never thought of a fairly complete download strategy
yet, especially not :post or :nounzip haha :smiley:

12

Hi,

On 16.07.2011 20:11, Marc H. wrote:

Hi, May I ask how this post was made? Email? Google Group?
Ruby-Forum? or Other?

I ask because it did not show up in the Google Group, nor in
the ruby-talk archives.

It was made via http://www.ruby-forum.com/ and the login there.

The problem of missing messages currently has it heights for me again
unfortunately. See this example from Luis L. with his kind
RubyInstaller contribution: http://i.imgur.com/jUmnC.png .

It almost looks like he’s talking to himself :slight_smile: when in fact he’s
replying to someone who’s messages I don’t get at all :frowning:

This media cross-over is a great idea. I haven’t seen this anywhere else
yet, but given these troubles, maybe that’s the reason …

  • Markus
13

Excerpts from Marc H.'s message of Sat Jul 16 20:24:10 +0200 2011:

/nix/store/5rnfzla9kcx4mj5zdc7nlnv8na1najvg-firefox-3.5.4/
Don’t know what to say. Usually you don’t work with those paths. So
there is no reason to worry about them. That’s why I’ve a hard time
understanding what you’re doing. On the one hand you seem to know the
basics of nixos - on the other hand you’re worrying about such small
details which don’t matter in practice cause implementation is hidden
from user. You use KDE or use PATH in shell. And there you only type
“firefox” or click on an icon. So it looks like you never really tried
using it. Well - mainstream distros (ubuntu,debian,…) still work better
for some cases such as mono development. However the most important
applications are working well. If you want to be little less mainstream
nixos easily wins. From my point of view we can close this
discussion - cause I sayd what I had to say.

Marc W.

14

I use gentoo prefix

also use macports( formerly darwin ports)

before that fink.

I on occasion will compile by hand.

I hear brew is nice. how much control does it give you? Can I compile
zsh without perl pcre support or ruby without emacs etc.

15

Marc H. wrote in post #1011032:

Homebrew is the easiest and most flexible way to install the
UNIX tools Apple didn’t include with OS X.

Haven’t yet, but what Homebrew does seems super trivial. Just like
stow, right, but written in ruby? And constrained to Apple.

My understanding (not having tried it myself) is that Homebrew scripts
build the application using configure / make / make install, as your
scripts do. Since OSX is BSD-derived, I’d expect such scripts to work on
other platforms with minimal changes - or at least for you to be able to
write homebrew scripts to build applications on other platforms.

If you want something truly cross-platform, then look at pkgsrc (the
NetBSD package system). Stuff in pkgsrc builds on all sorts of things,
including Linux. However, it’s not written in ruby - it’s
Makefile-based.

16

Heretix was a linux distribution which aimed at using Ruby instead of
traditional sh/dash scripts or other GNU tools.

I don’t know how far along the package management was, but that was a
goal along with a rewrite of init scripts.

The package management was to be inspired by GoboLinux, with packages in
individual subdirectories and with automated symlink maintenance.

http://www.h-e-r-e-t-i-x.org
http://wayback.archive.org/web/*/www.h-e-r-e-t-i-x.org

The project died in early 2007. If there is enough interest, I could
dig up my contact information for the authors.

17

On Jul 23, 2012, at 04:10 , Thomas S. wrote:

I worked on such a project back 2003/2004 called Baker. I still have the
code laying around here, but I abandoned the project mainly b/c its a
hell of a lot work and I had too many other things to deal with. Little
known fact, “Recipes” was a term I coined that was picked up by
GoboLinux Compile system, which was new then. (Get it? “Baker” used
“Recipes”). Like the suggested Ruby-Build-Tools, Baker used YAML to
store package information.

Well known fact: “Recipes” was used in the Perl Cookbook (get it?) long
before Baker came around. According to Amazon, the 1st Edition was
published in August 1, 1998 and I bought my first copy on September 4,
1998.

18

On Monday, July 23, 2012 5:09:43 PM UTC-4, Ryan D. wrote:

Well known fact: “Recipes” was used in the Perl Cookbook (get it?) long
before Baker came around. According to Amazon, the 1st Edition was
published in August 1, 1998 and I bought my first copy on September 4,
1998.

And gee wiz, in the 1743 it was first used to mean “cooking
instructions”.
And in 1580s it meant “medical prescription”. And long before that it
just
meant “take” in Latin.

Damn those Romans, they get all the credit! :-p

19

I just came across this thread, as old as it is, I’d like to contribute.

I worked on such a project back 2003/2004 called Baker. I still have the
code laying around here, but I abandoned the project mainly b/c its a
hell of a lot work and I had too many other things to deal with. Little
known fact, “Recipes” was a term I coined that was picked up by
GoboLinux Compile system, which was new then. (Get it? “Baker” used
“Recipes”). Like the suggested Ruby-Build-Tools, Baker used YAML to
store package information.

While I think there is merit in a language independent system, it would
have to justify the effort when GoboLinux’s shell-based scripts are
already so extensive. However, that doesn’t mean a new system can’t
support a new way and not support the old “legacy” formats too.

Also, I had no idea that this is what homebrew was all about. Not being
a MacOS user I never bothered to have a look. I am surprised. Now I
think maybe Baker had more potential then I realized when I gave it up.
But then, that was well before EC2, GitHub and all that jazz.

20

On Thu, Jul 26, 2012 at 1:41 PM, Intransition [email protected]
wrote:

And in 1580s it meant “medical prescription”. And long before that it just
meant “take” in Latin.

Damn those Romans, they get all the credit! :-p

+1

–

Saji N Hameed,
ARC-ENV, Center for Advanced Information Science and Technology,
University of Aizu, Tsuruga, Ikki-machi,
Aizuwakamatsu-shi, Fukushima 965-8580,
Japan

Tel: +81242 37-2736
Fax:+81242 37-2760
email: [email protected]
url: http://www.u-aizu.ac.jp
bib: Web of Science