Gemspec option to prevent auto-require?


#1

Hello folks,

I’m wondering if anyone knows an option I can put in my gemspec to
prevent rubygems from requiring all my dependencies as soon as I
require my library.

It’s okay that gems will inforce their dependencies, (throw an error
if the gems arent there), but I don’t want them all to load at once.

In my library, I’ve got a bunch of requires in my methods, and I want
them to require those libs only when they are called.

so for instance :

require “rubygems”
require “ruport” #depends on fastercsv, but should not load it at this
point

Ruport::DataSet.load(“foo.csv”) #now fastercsv should be loaded

This works fine when running via ruby -Ilib over my sources, but when
I use it as a gem, it autoloads all it’s deps, which takes between 3
and 5 seconds and is really annoying.

Suggestions?
My Rakefile with gemspec is here:
http://rubyurl.com/nb2


#2

On Jun 13, 2006, at 3:21 PM, Gregory B. wrote:

so for instance :

require “rubygems”
require “ruport” #depends on fastercsv, but should not load it at
this point

ruport will be loaded here regardless of rubygems.

Ruport::DataSet.load(“foo.csv”) #now fastercsv should be loaded

This works fine when running via ruby -Ilib over my sources, but when
I use it as a gem, it autoloads all it’s deps, which takes between 3
and 5 seconds and is really annoying.

With older versions of rubygems require time was proportional to the
number of gems installed (I don’t know if this is still true).

You probably want to run gem cleanup.


Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


#3

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

them to require those libs only when they are called.

so for instance :

require “rubygems”
require “ruport” #depends on fastercsv, but should not load it at
this point

ruport will be loaded here regardless of rubygems.

That is not what I am referring to. I don’t want fastercsv to be
loaded at this point. Obviously ruport will be loaded :slight_smile:

Since the require “fastercsv” is within the DataSet.load method there
is no reason why fastercsv should load at this point. It does not
when I am not using gems.

With older versions of rubygems require time was proportional to the
number of gems installed (I don’t know if this is still true).

You probably want to run gem cleanup.

Will try this but these are generally on machines that only have
ruport and it’s dependencies installed :-/


#4

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

Since the require “fastercsv” is within the DataSet.load method there
is no reason why fastercsv should load at this point.

Yes, there is. ruport required it.

Eric, I think you are missing something here.
In ruby, if you put a require inside of a method, it is not required
until that method is called.

[sandal@harmonix ruport]$ irb --simple-prompt

def foo; require “rubygems”; end
=> nil

$LOADED_FEATURES.any? { |e| e =~ /rubygems/ }
=> false

foo
=> true

$LOADED_FEATURES.any? { |e| e =~ /rubygems/ }
=> true

require “ruport” does NOT call require “fastercsv”

Ruport::DataSet.load does

It does not when I am not using gems.

If you’re using ruport you’re using gems. ruport contains “require
‘rubygems’” somewhere inside.

yes, it contains
begin; require “rubygems”; rescue LoadError; nil; end

because Ruport is installable via setup.rb, and when I develop it, I
do not have the ruport gem installed, and sometimes have the other
dependant libs installed manually or not at all.

Do you understand what I’m trying to do now


#5

On Jun 13, 2006, at 3:45 PM, Gregory B. wrote:

once.

ruport will be loaded here regardless of rubygems.

That is not what I am referring to. I don’t want fastercsv to be
loaded at this point. Obviously ruport will be loaded :slight_smile:

ruport loads fastercsv.

Since the require “fastercsv” is within the DataSet.load method there
is no reason why fastercsv should load at this point.

Yes, there is. ruport required it.

It does not when I am not using gems.

If you’re using ruport you’re using gems. ruport contains “require
‘rubygems’” somewhere inside.


Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


#6

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

Ruport::DataSet.load does

It may also, but it is not the only place.

You are not working against trunk.

[sandal@harmonix ruport]$ irb -Ilib --simple-prompt -rruport

$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> false

[[1,2],[3,4]].to_ds(%w[a b]).as(:csv)
=> “a,b\n1,2\n3,4\n”

$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> true

I appreciate that you are trying to help, but you really aren’t
answering my question.


#7

On Jun 13, 2006, at 4:18 PM, Gregory B. wrote:

until that method is called.

require “ruport” does NOT call require “fastercsv”

$ cat x.rb
deps = %w[
/usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib
/usr/local/lib/ruby/gems/1.8/gems/pdf-writer-1.1.3/lib
/usr/local/lib/ruby/gems/1.8/gems/transaction-simple-1.3.0/lib
/usr/local/lib/ruby/gems/1.8/gems/color-tools-1.3.0/lib
]

deps.each { |d| $:.unshift d } # add paths manually outside rubygems

$" << ‘rubygems.rb’ # pretend we already required rubygems

require ‘ruport’ # raises an error, fastercsv not found

$ ruby x.rb
/usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/ruport/format/
plugin.rb:48:in `require’: no such file to load – fastercsv (LoadError)
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport/format/plugin.rb:48
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport/format.rb:73
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport/format.rb:72
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport.rb:60
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport.rb:59
from x.rb:12
$ cat -n /usr/local/lib/ruby/gems/1.8/gems/ruport0.4.5/lib/ruport/
format/plugin.rb | grep -A3 -B3 48
45
46
47 class CSVPlugin < Format::Plugin
48 require “fastercsv”
49
50 format_field_names do
51 FasterCSV.generate { |csv| csv << data.fields }
$ cat -n /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/ruport.rb
| grep -A3 -B3 60
57
58
59 %w[config report format query data_row data_set].each { |lib|
60 require “ruport/#{lib}”
61 }

Ruport::DataSet.load does

It may also, but it is not the only place.


Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


#8

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

You are not working against trunk.

Then you’ll need to provide code I can work with to discover your
real problem. Downloading an entire project without gems is a heavy
burden.

svn checkout svn://rubyforge.org//var/svn/ruport

Not exactly a heavy burden.

Note that Rubygems doesn’t do that much to require, it is roughly this:

  1. Call Kernel#require
  2. If we got a LoadError, go looking for a gem with that file
  3. If we found a gem, add its libraries to $LOAD_PATH
  4. Call Kernel#require

So I don’t believe rubygems is at fault here.

[sandal@harmonix pkg]$ irb --simple-prompt -rubygems -rruport

$LOADED_FEATURES.length
=> 166

[sandal@harmonix pkg]$ irb --simple-prompt -I…/lib -rruport

$LOADED_FEATURES.length
=> 80

(both against trunk)

Rubygems doesn’t do any auto-require unless you have set an
autorequire in your gemspec and use require_gem (IIRC, old versions
of rubygems would require everything in autorequire even when using
require).

no auto_require in my spec. No require_gem

[sandal@harmonix ruport]$ gem -v
0.8.11


#9

On Jun 13, 2006, at 4:52 PM, Gregory B. wrote:

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

Ruport::DataSet.load does

It may also, but it is not the only place.

You are not working against trunk.

Then you’ll need to provide code I can work with to discover your
real problem. Downloading an entire project without gems is a heavy
burden.

Note that Rubygems doesn’t do that much to require, it is roughly this:

  1. Call Kernel#require
  2. If we got a LoadError, go looking for a gem with that file
  3. If we found a gem, add its libraries to $LOAD_PATH
  4. Call Kernel#require

So I don’t believe rubygems is at fault here.

[sandal@harmonix ruport]$ irb -Ilib --simple-prompt -rruport

$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> false

[[1,2],[3,4]].to_ds(%w[a b]).as(:csv)
=> “a,b\n1,2\n3,4\n”

$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> true

I appreciate that you are trying to help, but you really aren’t
answering my question.

Rubygems doesn’t do any auto-require unless you have set an
autorequire in your gemspec and use require_gem (IIRC, old versions
of rubygems would require everything in autorequire even when using
require).


Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


#10

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

I don’t see fastercsv being required when rubygems is involved.

Okay, maybe i’m not being super clear. The problem is it’s loading
all dependencies.

I’m curious though, how is this command running at all:
ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems

It does not look like you’ve included the lib dir?

e.g -Ilib

or are you running this within lib?

I imagine that what is happening if you are running it within lib, is
that both commands are pulling ruport from the svn code directly, not
from a gem build and installed.


#11

On Jun 13, 2006, at 5:39 PM, Gregory B. wrote:

Not exactly a heavy burden.
I don’t see fastercsv being required when rubygems is involved.

$ svn info .
Path: .
URL: svn://rubyforge.org/var/svn/ruport/trunk/ruport/lib
Repository Root: svn://rubyforge.org/var/svn/ruport
Repository UUID: 0132c854-6f07-0410-9f12-e6e9c8fde30c
Revision: 300
Node Kind: directory
Schedule: normal
Last Changed Author: binarywaterfall
Last Changed Rev: 300
Last Changed Date: 2006-06-13 17:04:33 -0700 (Tue, 13 Jun 2006)

$ ruby -rrubygems -e “require ‘ruport’; $LOADED_FEATURES.each { |f|
puts f }” > rubygems
$ ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems
$ diff -u norubygems rubygems
— norubygems 2006-06-13 18:23:55.000000000 -0700
+++ rubygems 2006-06-13 18:23:48.000000000 -0700
@@ -1,4 +1,3 @@
-ruport.rb
rubygems.rb
rbconfig.rb
rubygems/rubygems_version.rb
@@ -25,6 +24,7 @@
fcntl.bundle
openssl/x509.rb
rubygems/custom_require.rb
+ruport.rb
ruport/config.rb
singleton.rb
ostruct.rb


Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


#12

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

I don’t see fastercsv being required when rubygems is involved.

$ ruby -rrubygems -e “require ‘ruport’; $LOADED_FEATURES.each { |f|
puts f }” > rubygems
$ ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems

I get this when I use either -Ilib or call from within lib, because
RubyGems will check the loadpath first, afaik.

but try this:
[sandal@harmonix lib]$ cd …
[sandal@harmonix ruport]$ ruby -rrubygems -e “require ‘ruport’;
$LOADED_FEATURES.each { |f|
puts f }” > rubygems
[sandal@harmonix ruport]$ diff -u rubygems lib/norubygems
— rubygems 2006-06-13 21:56:44.000000000 -0400
+++ lib/norubygems 2006-06-13 21:55:13.000000000 -0400
@@ -1,3 +1,4 @@
+ruport.rb
rubygems.rb
rbconfig.rb
rubygems/rubygems_version.rb
@@ -22,96 +23,6 @@
fcntl.so
openssl/x509.rb
rubygems/custom_require.rb
-faster_csv.rb
-English.rb
-enumerator.so
-date.rb
-stringio.so
-redcloth.rb
-color.rb
-color/rgb.rb
-color/rgb-colors.rb
-color/cmyk.rb
-color/grayscale.rb
-color/hsl.rb
-color/yiq.rb
-color/rgb/metallic.rb
-transaction/simple.rb
-pdf/writer.rb
-thread.rb
-open-uri.rb
-uri.rb
-uri/common.rb
-uri/generic.rb
-uri/ftp.rb
-uri/http.rb
-uri/https.rb
-uri/ldap.rb
-uri/mailto.rb
-pdf/math.rb
-pdf/writer/lang.rb
-pdf/writer/lang/en.rb
-zlib.so
-pdf/writer/arc4.rb
-digest/md5.so
-digest.so
-pdf/writer/fontmetrics.rb
-pdf/writer/object.rb
-pdf/writer/object/action.rb
-pdf/writer/object/annotation.rb
-pdf/writer/object/catalog.rb
-pdf/writer/object/contents.rb
-pdf/writer/object/destination.rb
-pdf/writer/object/encryption.rb
-pdf/writer/object/font.rb
-pdf/writer/object/fontdescriptor.rb
-pdf/writer/object/fontencoding.rb
-pdf/writer/object/image.rb
-pdf/writer/oreader.rb
-pdf/writer/object/info.rb
-pdf/writer/object/outlines.rb
-pdf/writer/object/outline.rb
-pdf/writer/object/page.rb
-pdf/writer/object/pages.rb
-pdf/writer/object/procset.rb
-pdf/writer/object/viewerpreferences.rb
-pdf/writer/ohash.rb
-pdf/writer/strokestyle.rb
-pdf/writer/graphics.rb
-pdf/writer/graphics/imageinfo.rb
-pdf/writer/state.rb
-rexml/document.rb
-rexml/element.rb
-rexml/parent.rb
-rexml/child.rb
-rexml/node.rb
-rexml/parseexception.rb
-rexml/namespace.rb
-rexml/xmltokens.rb
-rexml/attribute.rb
-rexml/text.rb
-rexml/entity.rb
-rexml/source.rb
-rexml/encoding.rb
-rexml/doctype.rb
-rexml/attlistdecl.rb
-rexml/cdata.rb
-rexml/xpath.rb
-rexml/functions.rb
-rexml/xpath_parser.rb
-rexml/syncenumerator.rb
-rexml/parsers/xpathparser.rb
-rexml/xmldecl.rb
-rexml/comment.rb
-rexml/instruction.rb
-rexml/rexml.rb
-rexml/output.rb
-rexml/parsers/baseparser.rb
-rexml/parsers/streamparser.rb
-rexml/parsers/treeparser.rb
-rexml/validation/validationexception.rb
-rexml/encodings/UTF-8.rb
-ruport.rb
ruport/config.rb
singleton.rb
ostruct.rb
@@ -119,6 +30,7 @@
erb.rb
strscan.so
yaml.rb
+stringio.so
yaml/error.rb
yaml/syck.rb
syck.so
@@ -128,6 +40,7 @@
yaml/stream.rb
yaml/constants.rb
yaml/rubytypes.rb
+date.rb
yaml/types.rb
logger.rb
monitor.rb
@@ -138,6 +51,8 @@
net/protocol.rb
socket.so
timeout.rb
+digest/md5.so
+digest.so
ruport/format.rb
ruport/format/builder.rb
ruport/format/open_node.rb


#13

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

So I don’t believe rubygems is at fault here.

I decided to make a simple example gem to illustrate this. Here is
the Rakefile with simple spec

require “rake/rdoctask”
require “rake/testtask”
require “rake/gempackagetask”
require “rubygems”

spec = Gem::Specification.new do |spec|
spec.name = “dummie”
spec.version = “0.0.1”
spec.platform = Gem::Platform::RUBY
spec.summary = “A dummy lib”
spec.require_path = “lib”
spec.files = “lib/dumb.rb”
spec.add_dependency(‘ruport’, ‘>= 0.1.0’)
end

Rake::GemPackageTask.new(spec) do |pkg|
pkg.need_zip = true
pkg.need_tar = true
end

lib/dumb.rb actually is just an empty file. There are no requires.
If you unpack the gem once installing it, you can see this.

Now take a look at this.

[sandal@harmonix ~]$ irb -rubygems --simple-prompt

require “dumb”
=> true

$LOADED_FEATURES.include? “faster_csv.rb”
=> true

Any reason why this should happen?

Because this sounds like a bug, here is some system info.

[sandal@harmonix ~]$ ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[sandal@harmonix ~]$ gem -v
0.8.11
[sandal@harmonix ~]$ rake --version
rake, version 0.7.1
[sandal@harmonix ~]$ uname -a
Linux harmonix 2.6.16-ARCH #1 SMP PREEMPT Wed May 31 06:41:53 UTC 2006
i686 Mobile AMD Sempron™ Processor 3000+ AuthenticAMD GNU/Linux


#14

On Jun 13, 2006, at 6:54 PM, Gregory B. wrote:

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

I don’t see fastercsv being required when rubygems is involved.

Okay, maybe i’m not being super clear. The problem is it’s loading
all dependencies.

Well, yes, I was saying I don’t see that behavior.

I’m curious though, how is this command running at all:
ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems

It does not look like you’ve included the lib dir?

e.g -Ilib

or are you running this within lib?

running within lib.

I imagine that what is happening if you are running it within lib, is
that both commands are pulling ruport from the svn code directly, not
from a gem build and installed.

I tried building a gem from SVN and got the same behavior, no
difference between rubygems and rubygems2. (This wouldn’t make a
difference, Rubygems doesn’t load anything extra.)


Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


#15

Gregory B. wrote:

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

I don’t see fastercsv being required when rubygems is involved.

I think the difference here is that Eric is probably running with a very
recent development version of RubyGems and Gregory is running the last
stable release.

I setup two gems, a and b, and had a depend upon b. Both libraries
announce themselves while loading:

– Using 0.8.11 -----------

$ gem -v
0.8.11
$ irb
irb(main):001:0> require ‘a’
Loading B
Loading A
=> true
irb(main):002:0>

– Using 0.8.99 -----------

$ gem -v
0.8.99
$ irb
irb(main):001:0> require ‘a’
Loading A
=> true
irb(main):002:0>


Version 0.9.0 should be out within the week, barring unexpected issues.
If you want the try the release candidate for 0.9.0 (which is version
0.8.99), do:

gem update --system --source http://onestepback.org/betagems

If you do try it, please report any issues to me or to the Rubygems
developer mailing list (on RubyForge).

Thanks.

– Jim W.


#16

A quick followup…

Jim W. wrote:

– Using 0.8.11 -----------

$ gem -v
0.8.11
$ irb
irb(main):001:0> require ‘a’
Loading B
Loading A
=> true
irb(main):002:0>

Note that you only get this behavior when the b package specifies an
autorequire file (which b does in this case, and which the fastercsv
package does as well). When RubyGems activates b, it does any
autorequires specified by b.

This is why:

(1) I really believe autorequire is evil. It isn’t the only subtle bug
it has caused.
(2) The soon to be released RubyGems removes a lot of the autorequire
behavior.

– Jim W.


#17

On Jun 13, 2006, at 7:23 PM, Gregory B. wrote:

require “rubygems”

spec = Gem::Specification.new do |spec|
spec.name = “dummie”
spec.version = “0.0.1”
spec.platform = Gem::Platform::RUBY
spec.summary = “A dummy lib”
spec.require_path = “lib”
spec.files = “lib/dumb.rb”
spec.add_dependency(‘ruport’, ‘>= 0.1.0’)

What happens if you remove this line?

Now take a look at this.

[sandal@harmonix ~]$ irb -rubygems --simple-prompt

require “dumb”
=> true

$LOADED_FEATURES.include? “faster_csv.rb”
=> true

Any reason why this should happen?

Because this sounds like a bug, here is some system info.

Do you have the RUBYOPT shell variable set? Do you have
a .gemrc? .irbrc?

[sandal@harmonix ~]$ ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[sandal@harmonix ~]$ gem -v
0.8.11
[sandal@harmonix ~]$ rake --version
rake, version 0.7.1
[sandal@harmonix ~]$ uname -a
Linux harmonix 2.6.16-ARCH #1 SMP PREEMPT Wed May 31 06:41:53 UTC 2006
i686 Mobile AMD Sempron™ Processor 3000+ AuthenticAMD GNU/Linux


Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


#18

On 6/13/06, Jim W. removed_email_address@domain.invalid wrote:

Gregory B. wrote:

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

I don’t see fastercsv being required when rubygems is involved.

I think the difference here is that Eric is probably running with a very
recent development version of RubyGems and Gregory is running the last
stable release.

Thank you jim.

This patch seems to resolve the problem, also:

— rubygems.rb 2006-06-13 22:48:48.000000000 -0400
+++ rubygems-fix.rb 2006-06-13 22:49:22.000000000 -0400
@@ -164,7 +164,7 @@

   # Load dependent gems first
   spec.dependencies.each do |dep_gem|
  •   activate(dep_gem, autorequire)
    
  •   activate(dep_gem, spec.autorequire)
     end
    
     # add bin dir to require_path

#19

On 6/13/06, Jim W. removed_email_address@domain.invalid wrote:

Version 0.9.0 should be out within the week, barring unexpected issues.
If you want the try the release candidate for 0.9.0 (which is version
0.8.99), do:

gem update --system --source http://onestepback.org/betagems

Yup, that did it! :slight_smile:

[sandal@harmonix pkg]$ irb -rubygems --simple-prompt

require “ruport”
=> true

$LOADED_FEATURES.length
=> 83

$LOADED_FEATURES.include? “faster_csv.rb”
=> false

[[1,2]].to_ds(%w[a b]).as(:csv)
=> “a,b\n1,2\n”

$LOADED_FEATURES.include? “faster_csv.rb”
=> true

quit
[sandal@harmonix pkg]$ gem -v
0.8.99

If you do try it, please report any issues to me or to the Rubygems
developer mailing list (on RubyForge).

I’m subbed to the list, I would have asked there first, but I didn’t
suspect a bug at first, just a problem with my Rakefile maybe.

Thanks Eric and Jim!


#20

On 6/13/06, Eric H. removed_email_address@domain.invalid wrote:

Already solved this, but for clarity, I’ll answer these questions too.

spec.add_dependency(‘ruport’, ‘>= 0.1.0’)

What happens if you remove this line?

Well, then ruport and it’s deps would disappear :slight_smile:

Do you have the RUBYOPT shell variable set? Do you have
a .gemrc? .irbrc?

[sandal@harmonix ~]$ echo $RUBY_OPT

[sandal@harmonix ~]$ cat ~/.gemrc
cat: /home/sandal/.gemrc: No such file or directory
[sandal@harmonix ~]$ cat ~/.irbrc
cat: /home/sandal/.irbrc: No such file or directory

This machine is pretty clean. I haven’t tweaked my install much
because it’s my primary dev box and I like to assume as little as
possible about other peoples configs.