Forum: Ruby-core autoload will be dead

Hi,

Today, I talked with NaHi about enhancing const_missing to enable
autoload-like feature with nested modules.  But autoload itself has
fundamental flaw under multi-thread environment.  I should have remove
autoload when I added threads to the language (threads came a few
months after autoload).

So I hereby declare the future deprecation of autoload.  Ruby will
keep autoload for a while, since 2.0 should keep compatibility to 1.9.
But you don't expect it will survive further future, e.g. 3.0.

I strongly discourage the use of autoload in any standard libraries.

              matz.

Does this also mean that we will not enhance const_missing to support
nested modules?

I think it is still possible to make require itself threadsafe, if we 
use a
single lock for requires, as Charlie Nutter has previously proposed.
Honestly, if we improve const_missing and make `require` capable of 
being
threadsafe, it will be easy to implement autoload in terms of those
primitives.

Yehuda Katz
(ph) 718.877.1325

Hi,

On Sat, Nov 19, 2011 at 16:21, Yehuda Katz <wycats@gmail.com> wrote:
> Does this also mean that we will not enhance const_missing to support nested
> modules?

No, it doesn't.  I explained your request at #2740 first, and we
discussed the way to solve it (no conclusion yet.)  I'll update #2740
later.

> I think it is still possible to make require itself threadsafe, if we use a
> single lock for requires, as Charlie Nutter has previously proposed.

I don't know that single lock solution is acceptable, because the
cross-require issue is caused by developer's error we think, and
should be easily avoidable (Just don't do!)  It would be too big
restriction to save this error.

In contrast to it, autoload can cause unintentional cross-require.  So
I talked to Matz about #2740.

> Honestly, if we improve const_missing and make `require` capable of being
> threadsafe, it will be easy to implement autoload in terms of those
> primitives.

I explained exactly along the same line, not sure Matz agreed or not
though.  Let's keep discussing in #2740.

Yehuda Katz
(ph) 718.877.1325


On Sat, Nov 19, 2011 at 12:06 AM, Hiroshi Nakamura 
<nakahiro@gmail.com>wrote:

>
> > I think it is still possible to make require itself threadsafe, if we
> use a
> > single lock for requires, as Charlie Nutter has previously proposed.
>
> I don't know that single lock solution is acceptable, because the
> cross-require issue is caused by developer's error we think, and
> should be easily avoidable (Just don't do!)  It would be too big
> restriction to save this error.
>

The few cases I have personally discovered in Rails were complex cases 
that
weren't obvious at first glance. When a lot of code is involved in the
autoload process, it's very easy for some cross-require cases to exist
(usually through intermediate code).

In my mind, making threadsafety guarantees about require, even if they 
come
with some caveats, is far better than simply warning developers not to 
do
"the wrong thing" in their apps or libraries in the case where code is
being loaded lazily in the presence of threads.

0［｀

Hi,

(2011/11/19 17:17), Yehuda Katz wrote:
>
> The few cases I have personally discovered in Rails were complex
> cases that weren't obvious at first glance. When a lot of code is
> involved in the autoload process, it's very easy for some
> cross-require cases to exist (usually through intermediate code).

Would you please collect and show us a concrete example of that
problem?  I can imagine artificial cases but we should make sure we
are solving the same problem.  (This discussion started from
'autoload' which is not related!)

Besides this, I discussed it with Matz again, and he seems to like the
single lock idea, unless it doesn't hurt anything I guess.  I still am
thinking that serializing all 'require' and 'load' could cause a few
(rare?) problem for existing lib/app, but I don't have any concrete
example, so let's implement and evaluate it first.

> In my mind, making threadsafety guarantees about require, even if
> they come with some caveats, is far better than simply warning
> developers not to do "the wrong thing" in their apps or libraries
> in the case where code is being loaded lazily in the presence of
> threads.

Somewhat agreed, but I want to see examples.  For me, loading code
lazily itself looks 'the wrong thing', too. :)

// NaHi

Yukihiro Matsumoto <matz@ruby-lang.org> wrote:
> Today, I talked with NaHi about enhancing const_missing to enable
> autoload-like feature with nested modules.  But autoload itself has
> fundamental flaw under multi-thread environment.  I should have remove
> autoload when I added threads to the language (threads came a few
> months after autoload).

Hi, many of my Ruby scripts/apps are single-threaded and I would
like to keep memory usage down.

How about keeping autoload unchanged for single-threaded use and have
modules registered via autoload instantly require everything registered
when Thread.new is called?

It is common to use autoload to register a number of incompatible 
options.
For instance, Rack registers all possible server adapters, and loading 
the
adapters has side-effects.

I looked into this when I worked on Rails threadsafetiness and it is 
simply
incorrect to preload anything registered as an autoload.

Yehuda Katz
(ph) 718.877.1325

On Nov 19, 2011 4:11 AM, "Yukihiro Matsumoto" <matz@ruby-lang.org> 
wrote:
> keep autoload for a while, since 2.0 should keep compatibility to 1.9.
> But you don't expect it will survive further future, e.g. 3.0.
>
> I strongly discourage the use of autoload in any standard libraries.
>
>

Thank you for the details matz, hope this means require and
$LOADED_FEATURES along $LOAD_PATH will get a performance refactoring.

Saying because lazy loading no longer be an option, startup times will
become a problem in some scenarios.

On Wed, Nov 23, 2011 at 04:34, Luis Lavena <luislavena@gmail.com> wrote:

> On Nov 19, 2011 4:11 AM, "Yukihiro Matsumoto" <matz@ruby-lang.org> wrote:

>> I strongly discourage the use of autoload in any standard libraries.
> Thank you for the details matz, hope this means require and $LOADED_FEATURES
> along $LOAD_PATH will get a performance refactoring.
>
> Saying because lazy loading no longer be an option, startup times will
> become a problem in some scenarios.

I second that.  I have mostly switched to using load and
require_relative, as they are the fastest alternative on Windows.
They still do an insane amount of (seemingly) unnecessary work, but at
least they’re better than require.  I currently still use autoload for
loading bigger features that aren’t immediately needed at start-up.  I
think that this use-case for autoload is still valid and perhaps
should be in the future, in the same way Yehuda mentioned regarding
Rack’s server adapters.

"autoloading" can still be done by putting the require within a method 
that
is called only as needed.

The downside of this is that requires get pushed down into deeper levels 
of
code, making requirements less obvious to developers. Documentation not
withstanding, it's also not hard to work around. Just link us something
like:

    $AUTOREQ = Hash.new{|h,k|h[k]=[]}

    def req(key, path)
      $AUTOREQ[key.to_sym] << path
    end

    def use(key)
      $AUTOREQ[key.to_sym].each{ |path| require path }
    end

Then

    req :rdoc, 'rdoc'
    req :markdown, 'redcarpet'

    class Tmpl
      initialize(type)
        use type

I'm sure this can be greatly improved upon, maybe even generalized (and
thread safe?) to make a useful library gem.