Strategies for autoloading ruby files

Ruby
#1

Hi all,

Following up on my post about complicated problems, here’s a much
simpler problem that I need to tackle.

There are many parts of my framework that are extensible, and I want
people to be able to create extensions without modifying the core
framework, akin to how you don’t have to specifically load each model
and controller in Rails.

Is there a relatively standard way of doing that with Ruby? This is
what I’m doing in one of my projects, Facter[1]:

     # Now see if we can find any other facts
     $:.each do |dir|
         fdir = File.join(dir, "facter")
         if FileTest.exists?(fdir) and FileTest.directory?(fdir)
             Dir.glob("#{fdir}/*.rb").each do |file|
                 # Load here, rather than require, because otherwise
                 # the facts won't get reloaded if someone calls
                 # "loadfacts".  Really only important in testing,
                 # but, well, it's important in testing.
                 begin
                     load file
                 rescue => detail
                     warn "Could not load %s: %s" %
                         [file, detail]
                 end
             end
         end
     end

I’ve tried further copying Rails by checking that loaded files create
the objects it seems they should create (e.g., here they should create a
fact named after the file), but users constantly complained about those
warnings.

Is this the best way? Any other recommended mechanisms?

I’d actually like the autoloading to work either on-demand (e.g., look
for ‘facter/myfact’ if I ask for the ‘myfact’ fact), or load all of them
right now.

Hmm. In fact, it’d be ideal if you could write a module that allows you
to register autoload paths (e.g., I’d use ‘facter’, ‘puppet/type’,
‘puppet/type/package’, ‘puppet/type/service’, and lots more), and then
just have hooks into that module so that you could autoload all
instances, or just load a specific name.

1 - http://reductivelabs.com/projects/facter

#2

On Wed, 2 Aug 2006, Luke K. wrote:

Is there a relatively standard way of doing that with Ruby? This is what
# but, well, it’s important in testing.
I’ve tried further copying Rails by checking that loaded files create the
register autoload paths (e.g., I’d use ‘facter’, ‘puppet/type’,
‘puppet/type/package’, ‘puppet/type/service’, and lots more), and then just
have hooks into that module so that you could autoload all instances, or just
load a specific name.

1 - http://reductivelabs.com/projects/facter

this may be helpful (or not! ;-))

http://codeforpeople.com/lib/ruby/dynaload/

i use it for plugins in my satellite processing system like so

class PlugIn
def self.inherited other
Dynaload::export self, ‘plugin’ => true
end
end

then plugins are written something like

require ‘plugin’

class MyPlugIn < Plugin
#
# stuff
#
end

then, assume that is saved in myplugin.rb, then i can do

require ‘dynaload’

loaded = Dynaload::dynaload ‘myplugin.rb’

plugins = loaded.select{|klass, attributes| attributes[‘plugin’] ==
true}

make sense?

the reason it’s at 0.0.0 is that it just worked for me - been using in
production for a long time.

cheers.

-a

#3

You can use Module.const_missing to autoload a file the first time
the class in that file is mentioned in the code. So you could write
an Object.const_missing(name) method that translates the constant
into a file name of an expected format, then load that file on-the-fly.

Amusingly, this is exactly the code example that’s given in the
Pickaxe version 2 under Module.const_missing, along with a note that
the code is in very poor style and is a “perverse kind of autoload
functionality”.

:slight_smile:

– Brian

#4

On Aug 1, 2006, at 5:04 PM, Brian P. wrote:

You can use Module.const_missing to autoload a file the first
time the class in that file is mentioned in the code. So you could
write an Object.const_missing(name) method that translates the
constant into a file name of an expected format, then load that
file on-the-fly.

Ruby has a function for this though:

$ ri -T Kernel#autoload
-------------------------------------------------------- Kernel#autoload
autoload(module, filename) => nil

  Registers filename to be loaded (using Kernel::require) the first
  time that module (which may be a String or a symbol) is accessed.

     autoload(:MyModule, "/usr/local/lib/modules/my_module.rb")

James Edward G. II

#5

On Aug 1, 2006, at 4:11 PM, James Edward G. II wrote:

    autoload(:MyModule, "/usr/local/lib/modules/my_module.rb")

James Edward G. II

Cool, I didn’t realize that was already baked in to Kernel.

On Aug 1, 2006, at 4:22 PM, Luke K. wrote:

Puppet supports lots of package types (e.g., dpkg, apt, rpm,
darwinports, gems) and service types (init, smf, etc.); I specify
the name when I create the class, and users can select those types
by name, either in the language or directly in Ruby if they’re
directly accessing the library.

In other words, my problem is not how to hook into the autoload,
it’s the guts behind it.

I misunderstood your question, sorry about that.

– Brian

#6

Brian P. wrote:

You can use Module.const_missing to autoload a file the first time the
class in that file is mentioned in the code. So you could write an
Object.const_missing(name) method that translates the constant into a
file name of an expected format, then load that file on-the-fly.

Ironically, I use almost no constants in my code. Parent classes get
constants, but subclasses are almost always created dynamically[1], like
so:

Puppet::Type.newtype(:user) do
…
end

Then I always refer to that class by the name ‘user’ and never by the
constant. This is partially because it’s easier for me, but also
because I’ve written a custom language (no, not a DSL[2]) for Puppet,
and most of those names are exposed in that language. E.g., you create
a ‘user’ type like above, and users can create user elements in Puppet:

user { luke:
comment => “Luke K.”,
uid => 100,
shell => bash,
ensure => exists
}

Puppet supports lots of package types (e.g., dpkg, apt, rpm,
darwinports, gems) and service types (init, smf, etc.); I specify the
name when I create the class, and users can select those types by name,
either in the language or directly in Ruby if they’re directly accessing
the library.

In other words, my problem is not how to hook into the autoload, it’s
the guts behind it.

Amusingly, this is exactly the code example that’s given in the Pickaxe
version 2 under Module.const_missing, along with a note that the code is
in very poor style and is a “perverse kind of autoload functionality”.

1 - I do assign constants to them, but only because class inspection
during error printing did not work consistently unless there’s a
constant assigned to the class. I just tried reproducing this but could
not; I don’t remember the details well enough, I guess.

2 - See the FAQ, http://reductivelabs.com/projects/puppet/faq.html, the
fourth question.

#7

On Wed, 2 Aug 2006, Luke K. wrote:

Yeah. Half of my question, though, was figuring out which files to load,
although that’s easy enough to resolve, I guess. I’m assuming it does get
moderately expensive as I do more of these, but I normally load by name
(e.g., look for ‘puppet/type/package/dpkg.rb’ anywhere in the search path),
and only sometimes load all files in all search directories.

indeed. that one’s pretty project specific. in my case i have a well
known
location(s) for plugins…

an array and then do shenanigans later to make it act like a hash. This is
clearly a trivial example, because ‘name’ can (usually) be autodetermined
from the class name, but you also can’t test whether methods are defined,
for instance. With the ‘new’ methods, I can do any initialization on
the class that I want (e.g., setting @name using accessor methods),
class_eval the passed block, and then do any analysis I need. See [1].

hmmm. that’s interesting. another approach would be

module Plugable
def self.included other
…
end
end

class MyPlugin
include Plugable
end

i’ve not run into that issue though.

When I switched from using inherited to creating ‘new’ methods, it
unleashed a flood of productivity. I was astounded – I barely slept for
two weeks because so much was suddenly so easy when it had been so difficult
for so long.

1 -
http://reductivelabs.com/downloads/puppet/apidocs/classes/Puppet/Type.html#M000597

Notice all of the other ‘new*’ methods around this one. I use it everywhere
and I love it.

yes - i’ve got a fair number of class factories lying around myself.
Class.new is powerfull since you can easily parameterize your classes.

on a related note i used ‘traits’ alot with this sort of thing since it
avoids
needing to initialize all those class vars directly down the inheritence
chain.

i’ll read over some over of you code but gotta run to get the kid…

ciao.

-a

#8

[email protected] wrote:

this may be helpful (or not! ;-))

http://codeforpeople.com/lib/ruby/dynaload/

Clearly, I should always check your codebase before asking any
questions. :slight_smile:

then plugins are written something like

make sense?
Yeah. Half of my question, though, was figuring out which files to
load, although that’s easy enough to resolve, I guess. I’m assuming it
does get moderately expensive as I do more of these, but I normally load
by name (e.g., look for ‘puppet/type/package/dpkg.rb’ anywhere in the
search path), and only sometimes load all files in all search
directories.

I started out using the self.inherited method for all of this kind of
work, but it just got too cumbersome eventually, at least partially
because the object passed to ‘inherited’ has not yet been initialized.
That is, with the following code:

class MyClass < ParentClass
@name = :myclass
end

When ‘inherited’ is called, ‘@name’ is not set, so you have to stick it
in an array and then do shenanigans later to make it act like a hash.
This is clearly a trivial example, because ‘name’ can (usually) be
autodetermined from the class name, but you also can’t test whether
methods are defined, for instance. With the ‘new’ methods, I can
do any initialization on the class that I want (e.g., setting @name
using accessor methods), class_eval the passed block, and then do any
analysis I need. See [1].

When I switched from using inherited to creating ‘new’ methods,
it unleashed a flood of productivity. I was astounded – I barely slept
for two weeks because so much was suddenly so easy when it had been so
difficult for so long.

1 -
http://reductivelabs.com/downloads/puppet/apidocs/classes/Puppet/Type.html#M000597

Notice all of the other ‘new*’ methods around this one. I use it
everywhere and I love it.

#9

Le 01 août à 23:53, Luke K. a écrit :

Is this the best way? Any other recommended mechanisms?

I don’t know about recommended, but I’ve made two or three programs
using plugins, and here’s my method, in the main script :

Dir.glob($conf[:path]) do |f|
$mtime = File.mtime(f)
begin
require “#{f}”
rescue LoadError => err
puts err
end
end
Plugin.plugins.each |p| do
…
end

class Plugin
@@plugins = []
@@mtimes = {}
def Grapher.inherited©
@@plugins.push©
@@mtimes[c.to_s] = $mtime
end
def Plugin.plugins
@@plugins
end
def Plugin.mtimes
@@mtimes
end
end

And the plugins just have to :

class LittlePlugin < Plugin
…
end

Note : I needed here to know the modification time of the plugin to
eventually reinitialize some data. I think that using a global variable
is more than clunky. Any thoughts on a better way ?

Fred

#10

[email protected] wrote:

module Plugable
def self.included other
…
end
end

class MyPlugin
include Plugable
end

At this point, I’m looking at creating an Autoload class, and classes
wanting to autoload will create an instance:

Puppet::Autoload.new(self, “puppet/type”)

They’ll then be able to store that instance, or retrieve it from the
class:

Puppet::Autoload[self].load(:package)

That’ll handle the mechanics of the loading. I’ll have an additional
:loadall method that will search through $:, loading everything that
matches.

I want to keep the work around plugin loading – that is, what to do
with the class and how to initialize it – in the class supporting
plugins (i.e., the class with the ‘new’ method), because most
initialization is pretty custom. I suppose I could start with a default
method in a module, though; I hadn’t thought much about that.

A significant part of this for me is that I have lots of classes that
are related to each other – the Puppet::Type class is related to its
subclasses, the Puppet::Type::Package class is related to the
Puppet::Type::package::PkgType classes (ugh, constants are hard), the
‘service’ class is related to each ‘svctype’ class, etc. Calling
‘new’ directly on the related class makes that relationship very
explicit and pleasantly clear.

yes - i’ve got a fair number of class factories lying around myself.
Class.new is powerfull since you can easily parameterize your classes.

I love Class.new and Module.new. Ruby was great before I discovered
them, but with them, I’m definitely hooked. Well, Class.new and
class_eval, anyway.

on a related note i used ‘traits’ alot with this sort of thing since it
avoids
needing to initialize all those class vars directly down the inheritence
chain.

Most of the variables involved are actually class instance variables
(which, I understand, others don’t use that much, but which I use
pervasively and could not live without).

I need to just add traits into my codebase; I think it would simplify a
lot of what I’m doing.

/me adds one more item to the todo

i’ll read over some over of you code but gotta run to get the kid…

Comments, positive or negative, are always appreciated.