Win32 Extension Issues Wanted!

Ruby
1

Everyone. I had a conversation with folks from Microsoft today about
trying to move Ruby’s installer to Visual Studio 2005 and WiX (the MSI
installer). They’re quite serious about trying to get us shifted to
modern compiler technology rather than VC++6 or even MSYS/MINGW. I
can’t say that I disagree with them, either.

However, I indicated that one of the big problems that we had with
Ruby 1.8.2 was the use of VS.NET 2003 (VC++ 7.1) and its general
incompatibility – focussing on errno, because that’s the one item
which stuck in my mind – with earlier and later runtimes, as well as
a big problem of compatibility with MSYS/gcc and any MSVCRT (isascii,
as reported by Ara Howard).

They asked for as much information as I can gather on this matter. If
you have had problems trying to get Ruby or an extension compiled or
running on Windows for any reason, but especially because of Microsoft
runtime DLL differences, please provide me as much information as
possible so that I can pass it on to the VS team at Microsoft.

For what it’s worth, I have Ruby itself compiled with VS 2005,
although some of the extensions are not being built automatically,
from what I can tell. That may be a build script problem. However,
based on earlier reports of runtime version incompatibilities, it was
looking like I was going to have to recompile a lot of code. The
discussion today suggested that as long as function accesses are used
and not variable accesses, things will be okay (e.g., GetError() in
Windows, not errno). The problem, as I pointed out to them, is that
most Unix developers only ever have to worry about a single C
runtime being on their system and therefore don’t need to worry about
errno being in a different runtime DLL. Ideally, they would be able to
give us an external way of getting at the errno from a specific
runtime that may not be “our” runtime (e.g., the runtime with which we
were linked).

What other problems have people had and what can you provide me as
evidence? Also, can I give them your name and email for direct
contact? I will be headed to Europe soon and won’t be able to respond
quickly.

-austin

2

From: “Austin Z.” [email protected]

They asked for as much information as I can gather on this matter. If
you have had problems trying to get Ruby or an extension compiled or
running on Windows for any reason, but especially because of Microsoft
runtime DLL differences, please provide me as much information as
possible so that I can pass it on to the VS team at Microsoft.

I’m not sure if this is the sort of information you’re looking
for, but I’ve run into the occasional extension, and/or library
upon which an extension relies, that essentially requires
mingw/msys to build.

I got RMagick (ruby ImageMagick/GraphicsMagick) to build using
nmake/cl.exe, but it was a total hack. Building the extension
requires ./configure. The primary function of ./configure for
the RMagick build, is to analyze the GraphicsMagick/ImageMagick
header files and produce a config header file wherein about
200 configuration #defines are set or cleared, adapting RMagick
to whatever particular build of ImageMagick/GraphicsMagick it’s
working with.

(What I did was install the particular version of
GraphicsMagick I wanted, on my linux system, run the RMagick
./configure over there, then bring the generated
rmagick_config.h file back to my windows system. I also hacked
up RMagick’s extconf.rb in order to get it to work with
nmake/cl.exe.)

Some other extensions I suspect may require mingw/msys to build
include rcairo/cairo, DataFlow/PureData, ruby-gstreamer…

The more I run into this, the happier I would be to just have
ALL my ruby extensions (and ruby itself) built with
mingw/msys.

Regards,

Bill

3

On 6/26/06, Bill K. [email protected] wrote:

I’m not sure if this is the sort of information you’re looking
for, but I’ve run into the occasional extension, and/or library
upon which an extension relies, that essentially requires
mingw/msys to build.

Not specifically. However, part of what I really want to try to get
the MS team to help us out with is a situation wereh I have
ImageMagick built with MSYS, RMagick built with Visual Studio 2005 (cl
14) and Ruby built with VC++6 – and it all Just Works.

Right now, apparently, this is a nightmare and there’s “no way in
hell” that it would work. They’re interested in helping us; I’m not
about to turn down the help of the compiler writers. Which is why I
need more information.

-austin

4

Hello,

In message “Win32 Extension Issues Wanted!”
on Jun.27,2006 11:21:20, [email protected] wrote:
| What other problems have people had and what can you provide me as
| evidence? Also, can I give them your name and email for direct
| contact? I will be headed to Europe soon and won’t be able to respond
| quickly.

The problem of errno is not serious, I think.
We will be able to avoid the problem with some simple code
(for example, replace it with a function call by macro.)

The real problems are about various management areas that DLL
maintains internally and does not open to outside.
Such as, file descriptor table, malloc management table, etc.

The decision that the binary compatibility between versions of
runtime is not given is foolish.
Had they thought that passing file descriptors between DLL
became impossible?
As time goes by, there will be some cases to introduce such
incompatibility, I know.
However, VC7, VC7.1 and VC8 were shipped in only several years,
and they are mutually incompatible all.
It’s crazy, to say the least.

I decided to shut myself up with VC6 for the above-mentioned
reasons.
If MS keeps shipping incompatible versions at each upgrade,
I will throw away VC and shift to MinGW.
If MinGW also follows to MS, I’ll shift to Cygwin or throw away
Windows as development environment.

To hope to Microsoft is to prepare wrapper DLL to use MSVCR80.dll
named MSVCRT.dll.
If MS prepares such a mechanism, the binary compatibility will
be had.
If so, a certain DLL can use MSVCR80.dll and another DLL linked
with it can use MSVCRT.dll without any problem.

Sorry for my poor English.
Regards,

5

Hi all. What Nakamura says is in short are:

  • The main concern is that each DLL holds their own file descriptors
    table and malloc management areas (and so on), and those of MSVCRT[78]*
    are not compatible with current one.
  • Moving to MSVCRT[78]* breaks binary compatibility of extension
    libraries. Not only will they become incompatible with current mswin32
    version, also they will with mingw32. That’s not acceptable.
  • He is very angry for Microsoft continues shipping incompatible runtime
    every time.
6

Hi all. What Nakamura says is in short are:

I agree with all the statements here.

However there is one additional possibility, which is to compile with VC
.Net 200X , but ignore MSVCRT at linking and manually link to the 6.0
msvcrt.lib . A few functions are then missing such as _ftol2 but can be
recoded as a separate file. Please note that this does not work for all
C code. When compiling ZLib this way I hit into a function that I didn’t
know how to implement. However I compile compile the whole Neko
distribution this way.

That’s really a pitty that there is no VC backward compatibility
support, since I don’t want users of my software to have to install .Net
framework (containing MSCVCRT7X.dll).

Nicolas

7

On Tue, 27 Jun 2006, Austin Z. wrote:

Right now, apparently, this is a nightmare and there’s “no way in
hell” that it would work. They’re interested in helping us; I’m not
about to turn down the help of the compiler writers. Which is why I
need more information.

imho microsoft would be extremely hard pressed to deliver here: the
issue
isn’t so much with the compiler itself, it’s that any ruby built with ms
compilers is going to produce a badly crippled ruby. specifically i
mean that
a ruby that cannot

ruby extconf && make && make install

is broken. the point here is that it’s not so much the compilers
themselves
which are the issue, but the compiler toolchain, including a minimal set
of
tools to bootstrap a configuration from. it’s the lack of tools like
make,
yacc, sh, etc that make it impossible to do

cd gsl-1.8/
./configure --prefix=/usr/local && make && sudo make install

export LD_RUN_PATH=/usr/local/lib/
export LD_LIBRARY_PATH=/usr/local/lib/

cd ruby-gsl-1.8/
ruby extconf.rb && make && sudo make install

eg. a ‘usable’ ruby compiler should also compile third party libs that
the
extensions themselves are for. image magick is a perfect example of
this.

now, i’m well aware that mingw/msys is a bit of a mutt - but i
personally
think that simply compiling ruby is only half the issue. that ruby must
be
able to compile and link further extensions and the libs then depend on
to be
a ‘real’ ruby. so, unless the ms guys are also willing to help us put
together a free compiler toolchain that will work with an extconf.rb
(or
simlar) method of bootstrapping extensions and a means to compile
the libs
they bootstrap against it seems little will be gained.

hmmm. re-reading your post i see that you’re advocating a situation in
which
the compiler output itself is compatible and one might use a combination
of
mingw/msys and vc++ tools. that would indeed be a step forward - so are
you
advocating producing a ruby with vc++ compilers, or are you advocating
simply
having the various compilers out there produce binary abi compatible
output?

in any case, thanks for thinking of this - it’s definitely needed.

cheers.

-a

8

Austin Z. wrote:

They asked for as much information as I can gather on this matter. If
you have had problems trying to get Ruby or an extension compiled or
running on Windows for any reason, but especially because of Microsoft
runtime DLL differences, please provide me as much information as
possible so that I can pass it on to the VS team at Microsoft.
The only build issues I’ve had relate to nmake. It doesn’t seem to be
entirely compatible with make. For example, the last time I tried to
compile bdb or pl-ruby, it choked on the Makefile.

As far as running some of my extensions, sometimes I get random
segfaults if I don’t build with VC++ 6. The most notorious one is
win32-open3.

Note that Heesob had an interesting post that dealt with this issue, but
now I can’t find the dang thing.

Regards,

Dan

9

Austin Z. wrote:

On 6/26/06, Bill K. [email protected] wrote:

I’m not sure if this is the sort of information you’re looking
for, but I’ve run into the occasional extension, and/or library
upon which an extension relies, that essentially requires
mingw/msys to build.

Not specifically. However, part of what I really want to try to get
the MS team to help us out with is a situation wereh I have
ImageMagick built with MSYS, RMagick built with Visual Studio 2005 (cl
14) and Ruby built with VC++6 – and it all Just Works.

I’m not sure this is possible. This has come up quite often in the Perl
community. The problem is that the C library is responsible for
allocating and deallocating certain resources that cannot be shared
across different run-times (memory, I/O buffers, thread synchronization
objects for multi-threaded run-times, etc.) It can be worked around with
proper interfaces. For instance it wouldn’t matter what ImageMagick is
compiled with because it provides a completely encapsulated interface;
it handles its own resources. I’m not familiar with Ruby extensions, but
unless Ruby provides the same encapsulation of resource management it is
possible that corruptions will occur. E.g. Say you allocate memory from
one run-time which gives you a handle that is valid in that run-time;
then you try to free it in the other run-time, but in the other run-time
it points to some random area of memory, discontinuous with the first.
Boom! or worse.

I also seem to recall that there is some problem with redistribution:
msvcrt is available everywhere, but the newer run-times can’t be
distributed or something like that. I’m not sure of the particulars of
that issue.

I suggest contacting the folks at ActiveState.com. They have long dealt
with this issue with their distributions of Perl, Python, and Tcl/Tk.
They still use VC6 because of these issues.

Randy.

10

On 6/27/06, Randy W. Sims [email protected] wrote:

I’m not sure this is possible. This has come up quite often in the Perl
community. The problem is that the C library is responsible for
allocating and deallocating certain resources that cannot be shared
across different run-times (memory, I/O buffers, thread synchronization
objects for multi-threaded run-times, etc.)

I understand this. However, I spent 45 minutes on the phone with the
Visual Studio team yesterday, and they really seem interested in
trying to make things much easier for people to work with. Aside from
the ./configure && make && make install issues that Ara Howard pointed
out, the most likely setup with this would probably be Ruby and all
extensions compiled with VS 2005 using base libraries that may have
been compiled with any other compiler and runtime.

Remember: I’m collecting feedback for the VS team, not me directly.
I’m aware of a number of the problems, but am looking for
documentation so that we can try to have solutions made.

-austin

11

On 6/27/06, Daniel B. [email protected] wrote:

Austin Z. wrote:

They asked for as much information as I can gather on this matter. If
you have had problems trying to get Ruby or an extension compiled or
running on Windows for any reason, but especially because of Microsoft
runtime DLL differences, please provide me as much information as
possible so that I can pass it on to the VS team at Microsoft.
The only build issues I’ve had relate to nmake. It doesn’t seem to be
entirely compatible with make. For example, the last time I tried to
compile bdb or pl-ruby, it choked on the Makefile.

This may actually be a problem with the Makefile for bdb or pl-ruby,
because while nmake is a variant of make, most “Unix” makefiles are
actually GNUMake files.

As far as running some of my extensions, sometimes I get random
segfaults if I don’t build with VC++ 6. The most notorious one is
win32-open3.

If you’re able to collect any information about that, I’d appreciate it.

Note that Heesob had an interesting post that dealt with this issue, but
now I can’t find the dang thing.

That would be useful if it can be found.

-austin

12

Hi,

At Tue, 27 Jun 2006 11:21:20 +0900,
Austin Z. wrote in [ruby-core:08061]:

They asked for as much information as I can gather on this matter. If
you have had problems trying to get Ruby or an extension compiled or
running on Windows for any reason, but especially because of Microsoft
runtime DLL differences, please provide me as much information as
possible so that I can pass it on to the VS team at Microsoft.

Is this still open?

We need:

  • more unified I/O

    • nonblocking flag can be set on only sockets.
    • WaitForMultipleObjects doesn’t work with sockets.
    • APIs to tell how many bytes can be read immediately differ
      for ordinary file, console, socket and pipe.

  • pseudo tty
    pty feature is not supported at all now.

13

runtime DLL differences, please provide me as much information as

Do you want to find
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/190272 ?

Regards,

Park H.