OSX -flat_namespace issues show up in hitimes, nokogiri and amalgalite

Ruby
1

I’m not sure if it is the problem, but it appears that because ruby is
compiled
with -flat_namespace on Mac OS X that unexpected behavior can occur if
an
extension links against CoreServices. Here’s a fairly indepth example,
using
ruby 1.8.6. The exact same thing happens on 1.9.1.

% ./bin/ruby -v
ruby 1.8.6 (2009-08-04 patchlevel 383) [i686-darwin9.8.0]

% ./bin/ruby -rrbconfig -e ‘puts Config::CONFIG[“LDSHARED”]’
cc -dynamic -bundle -undefined suppress -flat_namespace

% ./bin/ruby -rubygems -e “require ‘hitimes’” -e “require ‘nokogiri’”
-e ‘puts “libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}”’
WARNING: Nokogiri was built against LibXML version 2.7.2, but has
dynamically loaded 2.6.16
HI. You’re using libxml2 version 2.6.16 which is over 4 years old and
has
plenty of bugs. We suggest that for maximum HTML/XML parsing
pleasure, you
upgrade your version of libxml2 and re-install nokogiri. If you like
using
libxml2 version 2.6.16, but don’t like this warning, please define the
constant
I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring
nokogiri.

libxml version : 20616

Change the order of the requires and all is good:

% ./bin/ruby -rubygems -e “require ‘nokogiri’” -e “require ‘hitimes’”
-e ‘puts “libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}”’
libxml version : 20702

A similar effect happens when using the Amalgalite gem, that case
complains
about the SQLite version. Both hitimes and amalgalite are my gems,
that’s how
I found this situation, but it could be exposed by someone elses gems
too.

Why does this happen?

The answer is, hitimes, on OSX links against ‘-framework CoreServices’
in order
to get access to the ‘AbsoluteToNanoseconds()’ and ‘UpTime()’ functions.

When hitimes loads and therefore the CoreServices framework loads,
CoreServices
brings along with it the system libxml and the sytem libsqlite3. We can
see
this by setting DYLD_PRINT_LIBRARIES=1:

% DYLD_PRINT_LIBRARIES=1 ./bin/ruby -rubygems -e "require 'hitimes'"

-e “require ‘nokogiri’” -e ‘puts “libxml version :
#{Nokogiri::LIBXML_PARSER_VERSION}”’
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/./bin/ruby
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /usr/lib/libobjc.A.dylib
dyld: loaded: /usr/lib/libgcc_s.1.dylib
dyld: loaded: /usr/lib/system/libmathCommon.A.dylib
dyld: loaded: /usr/lib/libauto.dylib
dyld: loaded: /usr/lib/libstdc++.6.dylib
dyld: loaded:
/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/thread.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/etc.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/stringio.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/syck.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/gems/1.8/gems/hitimes-1.0.4/lib/hitimes/1.8/hitimes_ext.bundle
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
dyld: loaded:
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
dyld: loaded:
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
dyld: loaded: /usr/lib/libicucore.A.dylib
dyld: loaded:
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
dyld: loaded: /usr/lib/libbsm.dylib
dyld: loaded: /usr/lib/libz.1.dylib
dyld: loaded:
/System/Library/Frameworks/Security.framework/Versions/A/Security
dyld: loaded:
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
–> dyld: loaded: /usr/lib/libsqlite3.0.dylib
dyld: loaded: /usr/lib/libresolv.9.dylib
–> dyld: loaded: /usr/lib/libxml2.2.dylib
dyld: loaded: /usr/lib/libxslt.1.dylib
dyld: loaded:
/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/gems/1.8/gems/nokogiri-1.3.3/lib/nokogiri/nokogiri.bundle
dyld: loaded: /opt/local/lib/libexslt.0.dylib
dyld: loaded: /opt/local/lib/libxslt.1.dylib
–> dyld: loaded: /opt/local/lib/libxml2.2.dylib
dyld: loaded: /opt/local/lib/libz.1.dylib
dyld: loaded: /opt/local/lib/libiconv.2.dylib
WARNING: Nokogiri was built against LibXML version 2.7.2, but has
dynamically loaded 2.6.16
HI. You’re using libxml2 version 2.6.16 which is over 4 years old
and has
plenty of bugs. We suggest that for maximum HTML/XML parsing
pleasure, you
upgrade your version of libxml2 and re-install nokogiri. If you
like using
libxml2 version 2.6.16, but don’t like this warning, please define
the constant
I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before
requring nokogiri.

dyld: loaded:

/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/racc/cparse.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/strscan.bundle
libxml version : 20616

You can see that the system libxml2, which is version 20616 is loaded
first and
then the version 20702 that we want to use, is loaded second.

So it appears that the -flat_namespace compiler paramter when compiling
ruby,
and the binary extensions,is shooting us in the foot. Lets make sure.
So
editing the distributed ‘configure’ script line 16991 from

darwin*)    : ${LDSHARED='cc -dynamic -bundle -undefined suppress

-flat_namespace’}
to
darwin*) : ${LDSHARED=‘cc -dynamic -bundle -undefined
dynamic_lookup’}

we have to change -undefined too or else we’ll get :

ld: can't use -undefined warning or suppress with

-twolevel_namespace

Now we have a new ruby that uses the two_level namespace on OS X

% ./bin/ruby -v -rrbconfig -e 'puts Config::CONFIG["LDSHARED"]'
ruby 1.8.6 (2009-08-04 patchlevel 383) [i686-darwin9.8.0]
cc -dynamic -bundle -undefined dynamic_lookup

And when we run the same thing that before blew up before all is well.

% DYLD_PRINT_LIBRARIES=1 ./bin/ruby -rubygems -e "require 'hitimes'"

-e “require ‘nokogiri’” -e ‘puts “libxml version :
#{Nokogiri::LIBXML_PARSER_VERSION}”’
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/./bin/ruby
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /usr/lib/libobjc.A.dylib
dyld: loaded: /usr/lib/libgcc_s.1.dylib
dyld: loaded: /usr/lib/system/libmathCommon.A.dylib
dyld: loaded: /usr/lib/libauto.dylib
dyld: loaded: /usr/lib/libstdc++.6.dylib
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/thread.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/etc.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/stringio.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/syck.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/gems/1.8/gems/hitimes-1.0.4/lib/hitimes/1.8/hitimes_ext.bundle
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
dyld: loaded:
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
dyld: loaded:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
dyld: loaded:
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
dyld: loaded: /usr/lib/libicucore.A.dylib
dyld: loaded:
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
dyld: loaded: /usr/lib/libbsm.dylib
dyld: loaded: /usr/lib/libz.1.dylib
dyld: loaded:
/System/Library/Frameworks/Security.framework/Versions/A/Security
dyld: loaded:
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
–> dyld: loaded: /usr/lib/libsqlite3.0.dylib
dyld: loaded: /usr/lib/libresolv.9.dylib
–> dyld: loaded: /usr/lib/libxml2.2.dylib
dyld: loaded: /usr/lib/libxslt.1.dylib
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/gems/1.8/gems/nokogiri-1.3.3/lib/nokogiri/nokogiri.bundle
dyld: loaded: /opt/local/lib/libexslt.0.dylib
dyld: loaded: /opt/local/lib/libxslt.1.dylib
–> dyld: loaded: /opt/local/lib/libxml2.2.dylib
dyld: loaded: /opt/local/lib/libz.1.dylib
dyld: loaded: /opt/local/lib/libiconv.2.dylib
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/racc/cparse.bundle
dyld: loaded:
/Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/strscan.bundle
libxml version : 20702

This time around even though the 20616 version is loaded first, and the
correct one,
20702, is what is used by the extension.

I guess my question is, what is the proper way to resolve this:

  1. should the ./configure script that is distributed with ruby be
    altered to use
    the two level namespaces?

  2. or should nokogiri and amalgalite and possibly others, warn the user
    when the
    library they are uses is not the one the extension expects?

Thoughts?

enjoy,

-jeremy

2

Hello

Twolevel namespace is used for quite some time in fink. It 's one of
the remaining build system adjustments in recent ruby.

Until now there was nothing I could point at that breaks with flat
namespace but twolevel namespace is the recommended linking on OS X
since some version like 10.2 or 10.3 and it does not seem to break
anything.

Note the -Wl prefix which allows building extensions that use libtool.

Thanks

Michal

— configure.in 2007-04-18 12:49:43.000000000 +0200
+++ configure.in 2007-04-19 14:23:04.000000000 +0200
@@ -1052,7 +1052,7 @@
rhapsody*) : ${LDSHARED=‘cc -dynamic -bundle -undefined
suppress’}
: ${LDFLAGS=""}
rb_cv_dlopen=yes ;;

  •   darwin*)        : ${LDSHARED='cc -dynamic -bundle -undefined

suppress -flat_namespace’}

  •   darwin*)        : ${LDSHARED='cc -dynamic -bundle

-Wl,-undefined,dynamic_lookup -Wl,-multiply_defined,sup
press’}
: ${LDFLAGS=""}
: ${LIBPATHENV=DYLD_LIBRARY_PATH}
rb_cv_dlopen=yes ;;

3

2009/8/24 Michal S. [email protected]:

Note the -Wl prefix which allows building extensions that use libtool.
            rb_cv_dlopen=yes ;;

  • Â Â Â darwin*) Â Â Â Â : ${LDSHARED=‘cc -dynamic -bundle -undefined
    suppress -flat_namespace’}
  • Â Â Â darwin*) Â Â Â Â : ${LDSHARED=‘cc -dynamic -bundle
    -Wl,-undefined,dynamic_lookup -Wl,-multiply_defined,sup
    press’}
    Â Â Â Â Â Â Â Â Â Â Â Â : ${LDFLAGS=“”}
    Â Â Â Â Â Â Â Â Â Â Â Â : ${LIBPATHENV=DYLD_LIBRARY_PATH}
    Â Â Â Â Â Â Â Â Â Â Â Â rb_cv_dlopen=yes ;;

It should get fixed in 1.8.8 hopefully.

It was rejected for earlier versions for compatibility reasons.

See http://redmine.ruby-lang.org/issues/show/1991

Thanks

Michal

4

On Wed, Aug 26, 2009 at 10:30:49PM +0900, Michal S. wrote:

?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??: ${LDFLAGS=“”}

It should get fixed in 1.8.8 hopefully.

It was rejected for earlier versions for compatibility reasons.

See http://redmine.ruby-lang.org/issues/show/1991

Yup, I saw the thread on ruby-core. thanks.

-jeremy