Ruby Forum Ruby > Get your hands dirty: Help bootstrap Ruby on Windows.

This was asked, and asked, and asked too many times, so here we go.

Since I don't want to hold anyone to try it, I'll like to share some
progress I've made in this area.
I'll like to make a few points clear before:

1) VC6 vs. VC8 vs. MinGW have no room in this discussion. I took the
MinGW path since was the only one that could be bootstrapped from
scratch.

2) This is a work in progress. Even that I'm 110% busy with work,
interest on this seems it deserve be shared, no mater how "alpha" the
whole process is.

3) Don't blame me if isn't work or your Rails application don't work
either. Isn't complete!. Instead: grab the code, hack it and send me
your patches.

4) Make code talk for you. I've heard this too many times on other
projects, so please move from simple comments to share your ruby-fu.

5) Part of this project is bring a new build procedure to One-Click
Installer, if we succeed, everybody will enjoy the benefits.

For those who aren't aware, I'm the new maintainer of OCI (One-Click
Ruby Installer for Windows). Curt give me the kingdom keys a few
months back and the latest release was completely build in my office
computer.

OCI, in its current state is a 'Package of Ruby Tools and libraries',
not a Installer. There are plans to change that.

So, now the code:

http://code.mmediasys.com/installer3/latest.zip

This is a pure-ruby solution that download and setup a mingw+msys
environment to get Ruby build.

It uses some rake recipes to download all the needed packages
At this time Ruby 1.8.6 p111 is targeted, but targeting Ruby 1.9 will
no have mayor issues.

Dependencies are not compiled, just downloaded for the time being.
Tested all the scenarios (pre-built and compile from source) and there
are issues with them which I reported previously.

To bootstrap and use this project, you need:
- Ruby 1.8.5 or greater
- Rake 0.7.3 or greater
- Tar [1] and Unzip [2] binary packages from GnuWin32

If someone can provide me a way to unpack 4MB zip, tgz and tar.bz2
files without external dependencies (just ruby) -- you're welcome.

The rake tasks available:
rake check      # Run tests for the interpreter in the sandbox.
rake clean      # Remove any temporary products.
rake clobber    # Remove any generated file.
rake compile    # Compile the interpreter.
rake configure  # Run the configure process for the interpreter.
rake download   # Download all components.
rake extract    # Extract all downloaded components.
rake install    # Install the interpreter in the sandbox.
rake prepare    # Prepare the freshly extracted components.

clean will remove the sandbox and clobber will remove all the
downloaded files.

The step sequence is:

download, extract, prepare, configure, compile, install, check

you will find the installed ruby in sandbox/ruby_mingw

You need to install before check, so the check process use the sandbox
interpreter instead of the one you're using to build it.

check will fail in the readline tests, since I cannot get them working
properly.

I suggest to those who are interested, help me get this working.

====

Checkout code:

I'm using Bazaar (bzr) as distributed version control software:

http://www.bazaar-vcs.org/

bzr branch http://code.mmediasys.com/installer3/dev my-local-branch

====

[1] This ships inside libarchive-2.4.0 named as 'bsdtar.exe'
http://gnuwin32.sourceforge.net/packages/libarchive.htm

[2] http://gnuwin32.sourceforge.net/packages/unzip.htm

--
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

From: "Luis Lavena" <luislavena@gmail.com>
>
> 1) VC6 vs. VC8 vs. MinGW have no room in this discussion. I took the
> MinGW path since was the only one that could be bootstrapped from
> scratch.

Just to clarify... What does "bootstrapped from scratch" mean
in this context?


Regards,

Bill

On 3 Jan 2008, at 16:45, Bill Kelly wrote:

>
> From: "Luis Lavena" <luislavena@gmail.com>
>>
>> 1) VC6 vs. VC8 vs. MinGW have no room in this discussion. I took the
>> MinGW path since was the only one that could be bootstrapped from
>> scratch.
>
> Just to clarify... What does "bootstrapped from scratch" mean
> in this context?

As low as he (we) can go. At the moment, it's in ruby (as per the code
presented and requirements), but some of my ideas had most of the
stack from a .cmd file - although that may be excessive.

.js or .vb via WSH anyone?

On Jan 3, 6:45 pm, Bill Kelly <bi...@cts.com> wrote:
>
Don't know if that is the right terminology (since I'm not a native
english speaker) but this means:

- Grab MinGW+MSYS
- Compile Ruby
- Package it
- Repeat!

Be usable so you can repeat the whole process with the new version of
the interpreter you just compiled :-)

So future version use the new one, is like what Rubinius is doing:
bootstrapping itself using a previous ruby installation.

Hope this clarify things a bit.

On Jan 3, 6:57 pm, James Tucker <jftuc...@gmail.com> wrote:
>
> As low as he (we) can go. At the moment, it's in ruby (as per the code
> presented and requirements), but some of my ideas had most of the
> stack from a .cmd file - although that may be excessive.
>
> js or .vb via WSH anyone?
>

The idea is Bootstrap Ruby with Ruby, like Rubinius guys are doing.

I must admit that have extensive batch files that I use to bootstrap
ruby with VC6 at office, but maintain .CMD and workaround its
limitations don't worth the effort.

Also, we are using Ruby for the fun of it, right? ;-)

On 3 Jan 2008, at 17:09, Luis Lavena wrote:

>>> Just to clarify... What does "bootstrapped from scratch" mean
> The idea is Bootstrap Ruby with Ruby, like Rubinius guys are doing.
>
> I must admit that have extensive batch files that I use to bootstrap
> ruby with VC6 at office, but maintain .CMD and workaround its
> limitations don't worth the effort.
>
> Also, we are using Ruby for the fun of it, right? ;-)

Right, shame one can't do the whole implement Jump, branch and then
move on up thing though... ;-)

Hi,

At Fri, 4 Jan 2008 05:29:58 +0900,
Luis Lavena wrote in [ruby-talk:285910]:
> So, now the code:
> 
> http://code.mmediasys.com/installer3/latest.zip
> 
> This is a pure-ruby solution that download and setup a mingw+msys
> environment to get Ruby build.

Seems manifest task in ruby18.rake doesn't close the target
file, so the file may be incomplete at the end of the task.

On Jan 4, 12:21 am, Nobuyoshi Nakada <n...@ruby-lang.org> wrote:
> > environment to get Ruby build.
>
> Seems manifest task in ruby18.rake doesn't close the target
> file, so the file may be incomplete at the end of the task.

Thank you Nobu, I didn't give attention to manifest since I was using
WiX Heat to collect all the files.

Change committed and pushed updated branch and zip file.

Regards,

Luis Lavena wrote:
[snip]

> 
> To bootstrap and use this project, you need:
> - Ruby 1.8.5 or greater
> - Rake 0.7.3 or greater
> - Tar [1] and Unzip [2] binary packages from GnuWin32

Aren't they in MSys?

[snip]


> I suggest to those who are interested, help me get this working.


This does look like fun. I have a spare Windows XP license I'm not using
and a Linux hosted VMware Workstation 6. :) Are the compiler, linker,
"tar" and zip/gzip/bzip2 the only external dependencies you have at the
moment?

On 4 ene, 03:03, "M. Edward (Ed) Borasky" <zn...@cesmail.net> wrote:
>
Chicken-egg situation:

They are inside the compressed tar.gz and tar.bz2 files... how I'm
supposed to get them if I don't decompress them first? :-D

So: if someone have a pure-ruby (with aditional gem or extension) that
allow us extract content from:

- .zip (deflate) files
- .gz (of .tar.gz) files
- .bz2 (of .tar.bz2) files

All without these dependencies, feel free to modify rake/
extracttask :-)

> This does look like fun. I have a spare Windows XP license I'm not using
> and a Linux hosted VMware Workstation 6. :) Are the compiler, linker,
> "tar" and zip/gzip/bzip2 the only external dependencies you have at the
> moment?

You don't need to manually download MinGW or MSYS, only you need get
your hands at tar and unzip.
(also, have a working ruby, which you could use OCI for that) ;-)

The compiler, linker and dependencies are downloaded automatically.

Right now only two "dependencies" are in place:

- Zlib 1.2.3
- Readline 5.0

I stopped at readline since I got some serious issues with
Readline.readline, as posted before to this list and ruby-core.

The thing is that those worked in the past, but right know they don't.

Each time I add a dependency (to get the bundled extension built), I
do a 'test-all' procedure and see if it's passing their self-tests.

Regards,

On 04/01/2008, Luis Lavena <luislavena@gmail.com> wrote:
> > Aren't they in MSys?
> - .zip (deflate) files
> - .gz (of .tar.gz) files
> - .bz2 (of .tar.bz2) files
>
> All without these dependencies, feel free to modify rake/
> extracttask :-)
>
If youcan afford relying on ruby stdlib, there is zlib which can do
deflate and gunzip.

There is minitar hosted on rubyforge under ruwiki (does not work on
unseekable files like stdin but works on gzipped files by using zlib).
There is rubyzip (untested, uses zlib for compression). bzip2 remains
to be solved but hopefully the basic stuff can be obtained as gzipped
archives.

http://rubyforge.org/frs/?group_id=84
http://rubyforge.org/frs/?group_id=909

Thanks

Michal

On 04/01/2008, Michal Suchanek <hramrach@centrum.cz> wrote:
> > >
> >
> There is minitar hosted on rubyforge under ruwiki (does not work on
> unseekable files like stdin but works on gzipped files by using zlib).
> There is rubyzip (untested, uses zlib for compression). bzip2 remains
> to be solved but hopefully the basic stuff can be obtained as gzipped
> archives.
>
> http://rubyforge.org/frs/?group_id=84
> http://rubyforge.org/frs/?group_id=909

And you can get bzip2 at
http://www.bzip.org/1.0.4/bzip2-104-x86-win32.exe - plain win32
executable.

Thanks

Michal

On 4 ene, 09:35, Michal Suchanek <hramr...@centrum.cz> wrote:
> http://rubyforge.org/frs/?group_id=84
> http://rubyforge.org/frs/?group_id=909
>

Thank you Michal,

Since I was moving from a batch-build process to a ruby/rake build
process, a few external tools remains as requirement.

MinGW packages are distributed in tar.gz format [1] (which I guess map
perfect for Minitar project), but MSYS provides tar.bz2 files [2],
which will still need the external tool to uncompress.

In this case, tar.bz2 files can be uncompressed with bsdtar, which is
the best implementation I've tested that work with all the file
formats. (tgz, tar.gz and tar.bz2)

There is also the memory usage and speed situation, extract 8MB files
can take a bit slow using a pure-ruby implementation.

[1] 
http://sourceforge.net/project/showfiles.php?group_id=2435&package_id=82723&release_id=237094
[2] 
http://sourceforge.net/project/showfiles.php?group_id=2435&package_id=24963&release_id=46827

Regards,

On 4 ene, 09:43, Michal Suchanek <hramr...@centrum.cz> wrote:
>
> And you can get bzip2 athttp://www.bzip.org/1.0.4/bzip2-104-x86-win32.exe- plain win32
> executable.
>

Thank you again.

I was using bsdtar from libarchive 2.4.0 mostly because it offered a
one-step unpack procedure:

tar xzf
(.tar, tar.gz and .tgz files)

tar jxf
(.tar.bz2)

Which is similar to the commands executed under *nix.

Anyhow this could be changed, patchs are welcome :-)

Regards,

On Jan 4, 2008 5:34 AM, Luis Lavena <luislavena@gmail.com> wrote:
> All without these dependencies, feel free to modify rake/
> extracttask :-)

.gz and .tar.gz files aren't hard to deal with since a working Ruby
1.8.5 is a prerequisite. Look at minitar, and the zlib library is part
of a working Ruby 1.8.5 install.

-austin

On 04/01/2008, Luis Lavena <luislavena@gmail.com> wrote:
> >
> perfect for Minitar project), but MSYS provides tar.bz2 files [2],
> which will still need the external tool to uncompress.
>
> In this case, tar.bz2 files can be uncompressed with bsdtar, which is
> the best implementation I've tested that work with all the file
> formats. (tgz, tar.gz and tar.bz2)
>
> There is also the memory usage and speed situation, extract 8MB files
> can take a bit slow using a pure-ruby implementation.

The decompression is handled by zlib so this should not be that much
of an issue. Still people extracting on slow machines could see the
difference, and there should be no problem with including these unpack
tools in the archive containing the build scripts.
On the other hand, the minitar might be useful if you wanted to do
something special with symlinks (currently it does not extract them at
all). Maintaining a patched minitar might be easier than maintaining a
patched bsdtar binary.

The rubyzip implementation does not include a working command line
tool as far as I can tell. A poor man's unzip using the library is
simple but I am not sure it is of much use. Perhaps an external unzip
tool would be easier.

Thanks

Michal

#unzip.rb
# The example:
  #Zip::ZipInputStream::open("my.zip") {
  #     |io|
  #
  #     while (entry = io.get_next_entry)
  #       puts "Contents of #{entry.name}: '#{io.read}'"
  #     end
  #   }

require 'zip/zip'

def open_wr fn
        elts = fn.split("/")
        dir = ""
        elts[0...-1].each{|d|
                raise RuntimeError, "Invalid filename '#{fn}'." if
                        d == "" or d == ".."
                dir << d << "/"
                Dir.mkdir dir unless File.directory? dir
        }
        raise RuntimeError, "File exists '#{fn}'." if
                File.exist? fn
        File.open(fn,"wb"){|f| yield f}
end

Zip::ZipInputStream::open(ARGV[0]) { |io|
        while (entry = io.get_next_entry)
                STDERR.puts entry.name
# directories are of type file, they just have a trailing slash
                next if entry.name[-1] == '/'[-1]
                open_wr(entry.name){|out| out << io.read}
        end
}

On 04/01/2008, Austin Ziegler <halostatue@gmail.com> wrote:
> >
> > All without these dependencies, feel free to modify rake/
> > extracttask :-)
>
> .gz and .tar.gz files aren't hard to deal with since a working Ruby
> 1.8.5 is a prerequisite. Look at minitar, and the zlib library is part
> of a working Ruby 1.8.5 install.
>
Yes, it just does not handle bzip2 which is used to compress the mingw
base that contains tar.

So you still need bzip2, and you can as well include a full
tar/gzip/bzibp2 suite then as it will probably make the decomression
somewhat faster and less memory intensive. It might waste a bit of
space as the minitar source would probably compress a few kilobytes
smaller than the tar binary. Didn't do any benchmarks, though.

Thanks

Michal

7za.exe from the 7z distribution is both open source, and really small.

Whilst it may not be a 'normal' tool, it very happily deals with all
of these issues in a single app, and common command line interface.

It's also pretty swift.

On 4 ene, 12:44, James Tucker <jftuc...@gmail.com> wrote:
> 7za.exe from the 7z distribution is both open source, and really small.
>
> Whilst it may not be a 'normal' tool, it very happily deals with all
> of these issues in a single app, and common command line interface.
>
> It's also pretty swift.
>

Until 'we' switch to other alternatives, either pure-ruby or externals
tools, I've uploaded a zip file which contains the needed files to get
you running:

http://code.mmediasys.com/installer3/deps.zip (320KB)

unzip into a new folder and put it in the PATH:

Example:

SET PATH=%PATH%;C:\installer3\dependencies

Then you can use rake without worring about how or here to get these
files.

Regards,

On Jan 4, 1:17 pm, Luis Lavena <luislav...@gmail.com> wrote:
> [...]
>
> Until 'we' switch to other alternatives, either pure-ruby or externals
> tools, I've uploaded a zip file which contains the needed files to get
> you running:
>
> [...]

No more!

(bumping this).

Thanks to Dennis Ranke Installer3 nows uses a pure-ruby solution to
download the required extracting tools using RubyZip code.

There is no need to install rubyzip gem!

You can grab exported code (revision 36) at the same URL:

http://code.mmediasys.com/installer3/latest.zip

Or checkout your own brach using bazaar:

bzr branch http://code.mmediasys.com/installer3/dev my-branch-feature

For those wanting to contribute, I'll be pushing further progress of
this project on rubyinstaller-devel mailing list:

http://rubyforge.org/mailman/listinfo/rubyinstaller-devel

Remember that the idea behind this is bootstrap the new One-Click
Installer for Windows.

Looking forward for contributors.

Regards,