Forum: Ruby-core Moving ruby 1.9.1 forward on windows

Ff880f3622a3ecd57d5e8d8721e4ef35?d=identicon&s=25 Charlie Savage (Guest)
on 2009-03-08 01:03
(Received via mailing list)
Attachment: smime.p7s (3 KB)
Hi everyone,

In consultation with Luis and Roger, I'd like to discuss moving Ruby
1.9.1 forward on Windows.  We have some concrete proposals that will
require changes to the way Ruby is built on Windows as well as changes
to rubygems.  First though, let me lay out our assumptions:

1.  The average windows user does not have a C compiler installed and
isn't necessarily all that interested in getting one.

2.  Because of #1, extensions for windows should be packaged as binary
gems (same as now)

3.  Ruby itself will likely be built by mingw32 (since the one click
installer is headed that way), but msvc builds should also be expected
and supported.

4.  Ruby extensions will be built with msvc (multiple versions),
mingw+msys or cross-compiled.


Proposals
==========

Based on these assumptions, we propose the following changes (and will
be happy to provide patches if the community can come to a consensus):


1.  Ruby library name

Proposal: The ruby runtime dll on Windows should *always* be named
ruby19.dll.  Currently it includes the runtime library name
(msvcrt-ruby19.dll for example).

Benefit: Having 1 runtime library name means gem developers only have to
create 1 binary for windows, not 1 for msvc builds, and 1 for mingw
builds.  This allows extension developers to use their toolchain of
choice.


2.  rubygems platform

Proposal:  The ruby platform should not be differ for msvc and mingw (so
no more mswin32-60 and mingw). We propose that the ruby gems platform
for windows is always:

<architecture>-<os>-<ruby version>

Examples:

x86-mswin-19
x86_64-mswin-19

Benefit: This change will allow developers to create binary gems using
their toolchain of choice (msvc, mingw/msys, cross-compile) and have it
work against ruby built with either msvc or mings/msys.

Adding the ruby-version will allow rubygems to distinguish between
binary gems built for ruby 1.8 versus 1.9 (which are clearly
incompatible).  Perhaps rubygems already has a mechanism for this - I
didn't see one in my research but could have missed it?


Mixing runtime libraries
========================

The most obvious criticism of this plan is that it will lead to mixing
of microsoft runtime c libraries.  From my experience this works as long
as extension developers follow the rules described here:

http://msdn.microsoft.com/en-us/library/ms235460(VS.80).aspx

To be more concrete, this boils down to two simple rules:

* If you call ALLOC or ALLOC_N, use xfree and not free
* Don't call sprintf or printf in an extension, instead use
rb_f_sprintf/rb_vsprintf/rb_io_printf

If an extension violate these two rules then its obvious, a segmentation
fault happens.  Thus these bugs are easy to find and easy to fix.

Since VC6 is thankfully no longer available, supporting msvc absolutely,
positively requires mixing c runtime libraries and therefore extension
writers must follow these two simple rules.  We don't view this as a
particularly difficult burden, especially in light of the changes
extension developers already frequently have to make for 1.9.1
compatibility.


Conclusion
==========

* Fixing the ruby runtime library name is a 1 line patch which we'll be
happy to provide
* Updating the rubygems platform is easy enough, but we assume deeper
changes would have to be made for rubygems to download the correct gem
binary.  We're happy to put together a patch if this seems like the
right way forward.

Anyone have questions, objections, or better ideas?

Thanks,

Charlie
47b1910084592eb77a032bc7d8d1a84e?d=identicon&s=25 Joel VanderWerf (Guest)
on 2009-03-08 03:16
(Received via mailing list)
Charlie Savage wrote:
> 1.  The average windows user does not have a C compiler installed and
> isn't necessarily all that interested in getting one.

What about packaging mingw32 with ruby OCI? That would avoid the need
for native gems, and it would permit run-time code generation (e.g.
rubyinline and cgen).
E7cff3cfd41c495e1012227d7dc24202?d=identicon&s=25 Luis Lavena (luislavena)
on 2009-03-08 04:13
(Received via mailing list)
On Sat, Mar 7, 2009 at 9:14 PM, Joel VanderWerf
<vjoel@path.berkeley.edu> wrote:
> Charlie Savage wrote:
>>
>> 1.  The average windows user does not have a C compiler installed and
>> isn't necessarily all that interested in getting one.
>
> What about packaging mingw32 with ruby OCI? That would avoid the need for
> native gems, and it would permit run-time code generation (e.g. rubyinline
> and cgen).
>

Packaging MinGW with OCI, while sounds tempting, put more in the plate
of One-Click Installer maintainers that goes a bit beyond the scope of
the project.

OCI will be built with MinGW, because several help mechanism has been
put in place that ease the build, test and release process for future
versions, but every developer should be able to pick a compiler to
create their project without being forced to one specific version of
it.

As a convenience tool, a devkit gem will be provided for those who
liek to install gems that are not binary-ready for Windows (gems from
source), but this will not be imposed or supported more than
occasional gem updates.
31ab75f7ddda241830659630746cdd3a?d=identicon&s=25 Austin Ziegler (austin)
on 2009-03-08 04:42
(Received via mailing list)
On Sat, Mar 7, 2009 at 7:01 PM, Charlie Savage <cfis@savagexi.com>
wrote:
> * If you call ALLOC or ALLOC_N, use xfree and not free
> developers already frequently have to make for 1.9.1 compatibility.
This works until you start linking third-party upstream source that
breaks these rules. If I were to make RbFoo that wraps libfoo, and
libfoo allocates memory through a method call, but doesn't provide an
explicit "free" mechanism, you're asking for trouble, and unless you
patch the upstream library, you won't have a fix.

-austin
F1d6cc2b735bfd82c8773172da2aeab9?d=identicon&s=25 Nobuyoshi Nakada (nobu)
on 2009-03-08 13:38
(Received via mailing list)
Hi,

At Sun, 8 Mar 2009 09:01:09 +0900,
Charlie Savage wrote in [ruby-core:22727]:
> 1.  Ruby library name
>
> Proposal: The ruby runtime dll on Windows should *always* be named
> ruby19.dll.  Currently it includes the runtime library name
> (msvcrt-ruby19.dll for example).

No.

> Mixing runtime libraries
> ========================
>
> The most obvious criticism of this plan is that it will lead to mixing
> of microsoft runtime c libraries.  From my experience this works as long
> as extension developers follow the rules described here:
>
> http://msdn.microsoft.com/en-us/library/ms235460(VS.80).aspx
>
> To be more concrete, this boils down to two simple rules:

errno is one of other problems.

> If an extension violate these two rules then its obvious, a segmentation
> fault happens.  Thus these bugs are easy to find and easy to fix.

Not all of the world follow the Microsoft's guideline.
Ff880f3622a3ecd57d5e8d8721e4ef35?d=identicon&s=25 Charlie Savage (Guest)
on 2009-03-08 20:47
(Received via mailing list)
Attachment: smime.p7s (3 KB)
> This works until you start linking third-party upstream source that
> breaks these rules. If I were to make RbFoo that wraps libfoo, and
> libfoo allocates memory through a method call, but doesn't provide an
> explicit "free" mechanism, you're asking for trouble, and unless you
> patch the upstream library, you won't have a fix.

Yes, that could happen.  But that would affect not only Ruby, but
potentially any program on Windows that uses the library.

Anyway, do you have any concrete examples?

Charlie
Ff880f3622a3ecd57d5e8d8721e4ef35?d=identicon&s=25 Charlie Savage (Guest)
on 2009-03-08 20:52
(Received via mailing list)
Attachment: smime.p7s (3 KB)
>> Proposal: The ruby runtime dll on Windows should *always* be named
>> ruby19.dll.  Currently it includes the runtime library name
>> (msvcrt-ruby19.dll for example).
>
> No.

Because?  You have an alternative solution?

> errno is one of other problems.

Ok, haven't run into that one yet, we'll add it to the list.  Any
specific gems/extensions that you know this affects?

>> If an extension violate these two rules then its obvious, a segmentation
>> fault happens.  Thus these bugs are easy to find and easy to fix.
>
> Not all of the world follow the Microsoft's guideline.

Sure.  But are they any specific cases that will cause us problems with
Ruby on windows with its normal set of extensions?

Charlie
31ab75f7ddda241830659630746cdd3a?d=identicon&s=25 Austin Ziegler (austin)
on 2009-03-08 21:09
(Received via mailing list)
On Sun, Mar 8, 2009 at 3:45 PM, Charlie Savage <cfis@savagexi.com>
wrote:
>> This works until you start linking third-party upstream source that
>> breaks these rules. If I were to make RbFoo that wraps libfoo, and
>> libfoo allocates memory through a method call, but doesn't provide an
>> explicit "free" mechanism, you're asking for trouble, and unless you
>> patch the upstream library, you won't have a fix.
> Yes, that could happen.  But that would affect not only Ruby, but
> potentially any program on Windows that uses the library.
>
> Anyway, do you have any concrete examples?

Current ones? Not offhand. Simply experience from trying to push Ruby
on Windows to be built with VS2005/VC8 instead of VC6 or mingw.

I recall seeing some libraries that I wanted to port to VC8 that
allocated memory on calls but didn't have a library-specific free
implementation. These things don't actually matter on most Unix
systems as there's only ever one C runtime, but I ended up giving up
on it because the task of maintaining third-party ports was too large
(pdcurses was one, I think, but I could be mistaken) and uninteresting
to me.

-austin
Ff880f3622a3ecd57d5e8d8721e4ef35?d=identicon&s=25 Charlie Savage (Guest)
on 2009-03-08 21:29
(Received via mailing list)
Attachment: smime.p7s (3 KB)
Hi Austin,

>> Anyway, do you have any concrete examples?
>
> Current ones? Not offhand. Simply experience from trying to push Ruby
> on Windows to be built with VS2005/VC8 instead of VC6 or mingw.

Do you by chance have any notes, blog posts, or can remember the various
issues you ran into when doing that?  Would be great to build off your
experiences.

> I recall seeing some libraries that I wanted to port to VC8 that
> allocated memory on calls but didn't have a library-specific free
> implementation. These things don't actually matter on most Unix
> systems as there's only ever one C runtime, but I ended up giving up
> on it because the task of maintaining third-party ports was too large
> (pdcurses was one, I think, but I could be mistaken) and uninteresting
> to me.

Yes, understood.  It would be helpful if we could put together a list of
problem libraries.

For pdcurses, or any of the default extensions that ship with ruby
(zlib, gdbm, iconv, etc.) there shouldn't be any problems since they
will all be compiled and distributed together as part of the one-click
installer.  Thus, they will all use the same C runtime library.

Charlie
31ab75f7ddda241830659630746cdd3a?d=identicon&s=25 Austin Ziegler (austin)
on 2009-03-08 21:39
(Received via mailing list)
On Sun, Mar 8, 2009 at 4:26 PM, Charlie Savage <cfis@savagexi.com>
wrote:
> Hi Austin,
>>> Anyway, do you have any concrete examples?
>> Current ones? Not offhand. Simply experience from trying to push Ruby
>> on Windows to be built with VS2005/VC8 instead of VC6 or mingw.
> Do you by chance have any notes, blog posts, or can remember the
> various issues you ran into when doing that?  Would be great to build
> off your experiences.

No more than what's on my site at halostatue.ca; I posted a long
series of notes that I put together when talking to Microsoft directly
about this problem. I stopped my participation in that primarily when
I switched to the Mac at home instead of continuing to frustrate
myself with Windows (I get paid to be frustrated with Windows at work;
I'd rather use something that frustrates me differently at home).

>
> For pdcurses, or any of the default extensions that ship with ruby (zlib,
> gdbm, iconv, etc.) there shouldn't be any problems since they will all be
> compiled and distributed together as part of the one-click installer.  Thus,
> they will all use the same C runtime library.

Well, my hope was to *not* have to rebuild things like pdcurses—but
remember that I was trying to build against VS2005, not mingw (I'm
*still* not convinced that mingw is a good idea because it depends on
such an old C runtime), so the incompatibility issue mattered much
much much more.

-austin
Ff880f3622a3ecd57d5e8d8721e4ef35?d=identicon&s=25 Charlie Savage (Guest)
on 2009-03-08 21:45
(Received via mailing list)
Attachment: smime.p7s (3 KB)
>
> Well, my hope was to *not* have to rebuild things like pdcurses—but
> remember that I was trying to build against VS2005,

Ah, I see.  The 1.9.1 one click installer will ship with new versions of
these libraries, so that problem goes away.

Charlie
9887130ab70f6e66b50f2c5e19272c67?d=identicon&s=25 Roger Pack (Guest)
on 2009-03-10 14:49
(Received via mailing list)
> Well, my hope was to *not* have to rebuild things like pdcurses—but
> remember that I was trying to build against VS2005, not mingw (I'm
> *still* not convinced that mingw is a good idea because it depends on
> such an old C runtime), so the incompatibility issue mattered much
> much much more.

I suppose you can link against a newer version if desired [1], though
it's true that the project goes stagnant quite a bit.

I suppose my only concerns with the change are
1) are there subtle errors, like you do an ALLOC_N and then the
library frees it, so you can't control it, resulting in a binary
incompatibility?

2) Apparently you cannot use structures across different runtimes is
that right [2]?  I suppose that's not a problem?

There is definitely some utility to this proposal, though:
1) It will official "bless" a specific msvcrt.dll version
[msvcr90.dll? [4]]  This apparently drops support for windows 98, but
who cares [3] :)

Binaries would need to be built [re-built] using VC2008 or mingw cross
compile linked to that dll, is that right?  I think that would bring
some sanity to the windows environment as currently it's pretty
fragmented.

2) It allows mingw built binaries to interop with VC ones--binaries
are much faster if built by gcc [1 #5].

Do the core folks favor VC2008 or any compiler at all?  It appears
that as long as you link to the same DLL the two are compatible.

Another option would be to release a mingw only OCI linked against
msvcr90.dll, then instruct extension developers on how to release
binaries that "pretend" to be mingw though they're only "mingw
compatible."

Or perhaps rubygems could be patched to recognize compatible binaries.

Thoughts?
-=r

[1] http://www.develer.com/oss/GccWinBinaries
[2] http://luabinaries.luaforge.net/manual.html#LuaBin...
[3] http://rcecafe.net/?p=33
[4] msvcr90.dll apparently? from
http://www.codeguru.com/forum/archive/index.php/t-... though
apparently it's possible to force link against msvcrt.dll with all its
inconsistencies :)  Would mingw be forced to link against msvcr90.dll
then?
See also http://jove.prohosting.com/iwave/ipython/issues.html as an
example that it is possible to at least link against msvc7.1.dll with
success.  See also
http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx for a
reiteration 'it is named msvcr90.dll now"
Ff880f3622a3ecd57d5e8d8721e4ef35?d=identicon&s=25 Charlie Savage (Guest)
on 2009-03-10 18:27
(Received via mailing list)
Attachment: smime.p7s (3 KB)
Hi Roger,

> I suppose you can link against a newer version if desired [1], though
> it's true that the project goes stagnant quite a bit.

If you want to use an unofficial version of gcc 4.x for mingw, then I'd
go here:

http://tdragon.net/recentgcc/

> I suppose my only concerns with the change are
> 1) are there subtle errors, like you do an ALLOC_N and then the
> library frees it, so you can't control it, resulting in a binary
> incompatibility?

If there are a segmentation fault will occur - so its not subtle.

> 2) Apparently you cannot use structures across different runtimes is
> that right [2]?  I suppose that's not a problem?

No, that is generally not true. VC and Mingw create compatible C binary
interfaces with one exception that I know of explained here (and there
is a patch to fix gcc):

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=36834
http://www.angelcode.com/dev/callconv/callconv.html

The only time I have run into this is when creating a Ruby extension
using VC2008 that called into the proj4 library built with MingW (so it
really didn't have anything to do with Ruby).  I don't know of any
examples where this is an issue with the Ruby's C api.

  > Binaries would need to be built [re-built] using VC2008 or mingw
cross
> compile linked to that dll, is that right?  I think that would bring
> some sanity to the windows environment as currently it's pretty
> fragmented.

Yes, it is possible to go try and go down the path that there is one
"blessed" compiler on Windows.  But the point of my proposal was to
avoid that, giving developers the choice to use whatever compiler they
want.

> 2) It allows mingw built binaries to interop with VC ones--binaries
> are much faster if built by gcc [1 #5].

That's an invalid comparison for a couple of reasons.  First you would
compare VC2008 versus gcc 4.3.  Except you can't, because gcc 4.x is not
officially released for windows so you're stuck with gcc 3.4.5.

> Do the core folks favor VC2008 or any compiler at all?  It appears
> that as long as you link to the same DLL the two are compatible.

I favor not favoring a compiler.

Now, you do bring up an interesting point, which is instead of favoring
a compiler favor a runtime library (msvcr90 being the obvious choice).
  In other words, VC2008 will always link against that one and you can
mingw to link against it also.  The downside is that you exclude VC2003
or the upcoming VC2010.

My vote is still to try not choosing a compiler.  I certainly think its
worth a try.  If it doesn't work, we change tactics.

> Another option would be to release a mingw only OCI linked against
> msvcr90.dll, then instruct extension developers on how to release
> binaries that "pretend" to be mingw though they're only "mingw
> compatible."

There is no pretending here.  Those would be mingw binaries linked
against msvcr90.dll.  That is supported.  I have no idea if it works
cross-compiling though.

> Or perhaps rubygems could be patched to recognize compatible binaries.

Not sure what you mean.  My goal is that an extension developer only has
  to release 1 binary for windows per ruby version.

Charlie
9887130ab70f6e66b50f2c5e19272c67?d=identicon&s=25 Roger Pack (Guest)
on 2009-03-10 20:18
(Received via mailing list)
>> 2) It allows mingw built binaries to interop with VC ones--binaries
>> are much faster if built by gcc [1 #5].
>
> That's an invalid comparison for a couple of reasons.  First you would
> compare VC2008 versus gcc 4.3.  Except you can't, because gcc 4.x is not
> officially released for windows so you're stuck with gcc 3.4.5.

Yeah I think the real comparison currently is VC2008 versus gcc 3.4.5
Here's a rough hack I did of it once [not a great comparison, but it
shows gcc 3.4.5 being still faster than VC2008
http://groups.google.com/group/ruby-benchmark-suit...
].


> Now, you do bring up an interesting point, which is instead of favoring a
> compiler favor a runtime library (msvcr90 being the obvious choice).  In
> other words, VC2008 will always link against that one and you can mingw to
> link against it also.  The downside is that you exclude VC2003 or the
> upcoming VC2010.

True it might be worthwhile to "bless" a runtime.  Theoretically this
could work with other compilers, as well, than its default.  I've seen
blog posts of people making VC2005 link against msvcrt.dll [i.e. VC6]
and also, apparently you can link mingw against other versions of
msvcrt.dll [1].


>> Another option would be to release a mingw only OCI linked against
>> msvcr90.dll, then instruct extension developers on how to release
>> binaries that "pretend" to be mingw though they're only "mingw
>> compatible."
>
> There is no pretending here.  Those would be mingw binaries linked against
> msvcr90.dll.  That is supported.  I have no idea if it works cross-compiling
> though.

I was just thinking that if desired, the current naming scheme could
be used and still fit.  Kind of :)

>> Or perhaps rubygems could be patched to recognize compatible binaries.
>
> Not sure what you mean.  My goal is that an extension developer only has  to
> release 1 binary for windows per ruby version.

For that one I was meaning rubygems could perhaps note "oh you're on
mingw, well I have a mswin32 binary that you can use even though it's
not your compiler" and install it appropriately.   Some change needs
to happen, not sure though.

Thanks!
-=r
[1] http://www.develer.com/oss/GccWinBinaries at the bottom.
This topic is locked and can not be replied to.