Forum: Ruby-core Proposal: Autoload with block

= A proposal for autoload w/block:

== autoload(:Constant, &blk)
== autoload(&blk)

When Constant is accessed, the following happens:

1. The block is called with the constant name as a parameter and the 
scope
   where it's looking for the constant as self:

   scope.instance_exec(:Constant, &blk)

2. If the block raises an error, the error isn't rescued

3. If the block raises a specific error or throws something specific,
   it signals that the constant is missing unless it has been set 
earlier
   in the block:

3a. It checks if the constant is defined now
3b. If not, the constant doesn't exists and it'll continue searching for
    the constant in the constant chain.

4. If the block succeeds:
4a. It sets the constant to result of the block

If you don't specify a constant name to autoload, it will catch all
missing constants, but otherwise it works exactly the same as above.

There should also be a way to specify an autoload which triggers on all
scopes. So with the given code:

  # Not sure about how we should specify it though:
  autoload(true) do
    throw :stop
  end

  module Foo::Bar
    p Hello
  end

It will run the autoload block with respectivly Foo::Bar and Object as
scope (as opposed to Foo::Bar, Foo and Object which it'll search through
if you hack it up with const_missing).

There are basically two scenarios which this API covers:

- Easily use autoload without using #const_set:

  autoload(:C) { generate_C }

- Run code which might set the constant:

  autoload(:C) { set_C; throw :stop }

- The two scenarios above with an unspecified constant name.

None of these changes previous behaviour and you can also create a Ruby
monkey-patch which sort-of works similar through #const_missing (which 
you
can use where this proposal isn't implemented).

= Example of uses
== Camping

In Camping (http://whywentcamping.com) we use autoload to load optional
features which requires dependencies. Example:

  # in camping.rb
  module Camping
    autoload :Template, "camping/template"
  end

  # in camping/template.rb
  require 'tilt'
  module Camping
    Template = Tilt
  end

With the proposed changes to autoload, this could be written as:

  # in camping.rb
  module Camping
    autoload(:Template) { require 'tilt'; Tilt }
  end

Which is way easier to maintain and understand for everyone.

=== Rails

Rails overwrites const_missing in order to automagically load files
(from different locations) when a constant is accessed. Example of how
this could be implemented with the proposed autoload:

  autoload do |constant|
    # Convert FooBar to foo_bar.rb
    file = constant.to_s.underscore + '.rb'
    # Search through the list of directores:
    Rails.autoload_paths.each do |dir|
      full_path = File.join(dir, file)
      # Load the file and stop searching:
      File.exists?(full_path) && load(full_path) && break
    end

    throw :stop
  end

This is also more correct than Rails' version, which can't use the 
proper
cref because it uses const_missing:

  module Foo::Bar
    p Hello
  end

Rails will search through Foo::Bar, Foo and Object instead of Foo::Bar 
and
Object. This is really impossible to properly implement with
const_missing, but is doable with autoload with blocks.

=== Implementing regular autoload in Ruby

Just to show that this implementation is the general case of 
autoloading,
implementing regular autoloading is really simple with this API:

  autoload(:Constant) { load(file); throw :stop }

= Thoughts?

What are your thoughts on:

- A more general autoload solution?
- This proposal?
- Other ways to solve this "problem"?

// Magnus Holm

Sorry to plug my own stuff, but you might find subload of some interest 
here. It's unfinished, but provides some flexibility in these matters 
that might be of interest. I also have a fair amount of notes about 
possible other use cases that aren't covered yet in the subload code. 
Whilst on the topic, some consideration for thread safety might be worth 
the time - not that I'm proposing it can be 'fixed', merely considered 
to avoid worst cases.

Magnus, have you seen http://redmine.ruby-lang.org/issues/show/462 ?

That's interesting, but I don't buy matz' argument:

"the code in the block may not be executed (if the constant is defined
anywhere else), so it is not stable."

The same can be said about this code:

  autoload :Foo, "foo"
  Foo = 1
  p Foo   # => "foo.rb" is not evaluated

And given:

  autoload(:Foo) { Module.new }
  module Foo; end # => The block will be executed, just like autoload
works today.

// Magnus Holm

(2010/08/30 5:23), Magnus Holm wrote:
> "the code in the block may not be executed (if the constant is defined
> anywhere else), so it is not stable."
> 
> The same can be said

Yes, and quite simply we (at least me) now consider autoloading a bad 
idea.
You need to IMPROVE it, rather than just to EXTEND it, if you want it to
survive longer.

On Sun, Aug 29, 2010 at 23:52, Urabe Shyouhei <shyouhei@ruby-lang.org> 
wrote:
> (2010/08/30 5:23), Magnus Holm wrote:

> Yes, and quite simply we (at least me) now consider autoloading a bad idea.

Asking as someone who has switched to using autoload for basically
everything I write for Windows (work) because require is insanely slow
on Windows (especially on network shares) in both 1.8 and 1.9, can you
please expand on why you now consider autoloading to be a bad idea?

Thanks!

On 30 Aug 2010, at 06:09, Nikolai Weibull wrote:

> On Sun, Aug 29, 2010 at 23:52, Urabe Shyouhei <shyouhei@ruby-lang.org> wrote:
>> (2010/08/30 5:23), Magnus Holm wrote:
> 
>> Yes, and quite simply we (at least me) now consider autoloading a bad idea.
> 
> Asking as someone who has switched to using autoload for basically
> everything I write for Windows (work) because require is insanely slow
> on Windows (especially on network shares) in both 1.8 and 1.9, can you
> please expand on why you now consider autoloading to be a bad idea?

Autoload is a bad idea if you're doing loads inside of a threaded 
environment, as it can cause double loads to occur (but not on MRI I 
think), and this can result in some code that is evaluated twice to have 
some unintended effect. It can also result in incomplete constants being 
used such as to cause unusual errors, for example, methods not yet being 
defined.

The reality is, this is actually the same for any lazy load in a 
threaded environment. People seem to have taken the issue of autoload as 
if it is the cause of the problem, however, there is another opinion 
that the cause of the problem is allowing code to be loaded in a 
threaded environment. In my opinion (and a number of others) to load 
code in a threaded manner is fraught with potential issues that are hard 
to overcome, furthermore, the common runtime of most applications should 
be:

Load (code)
Setup (class heirachy, configurations, etc)
Spawn (threads)
Run (application logic)

If users keep to good order, problems can be avoided regardless of 
loading mechanisms.

I sympathise with your use of autoload, indeed I use autoload to save 
significant overhead in tests and such, and simply ensure that code is 
fully loaded and configured in any threaded production environment. 
Indeed a large part of the reason for subload[1] was to make it easier 
to override the loader semantics for production runtimes that would be 
threaded that could not ensure being fully loaded ahead of time. The 
solution worked very well there.

[1] http://github.com/raggi/subload

(2010/08/30 18:09), Nikolai Weibull wrote:
> Asking as someone who has switched to using autoload for basically
> everything I write for Windows (work) because require is insanely slow
> on Windows (especially on network shares) in both 1.8 and 1.9, can you
> please expand on why you now consider autoloading to be a bad idea?

Because by design one cannot eliminate its timing issue.  Imagine you 
have two
threads touching a same constant, and what if that constant is 
autoloaded...
some odd situations might be observed.

See this previous discussion:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/20238