Engines made up of..more engines

So the Rails Engines project looks pretty sweet. Duane J. seems
to be working on turning Typo into an engine, and it made me wonder if
it would be possible to distribute engines that include engines
themselves. In the Typo engine example, there would be no reason to
include the authentication in the blog itself, when he can just use
the available salted login generator engine. Also, it doesn’t provide
any flexibility for authentication mechanisms.

It’d be nice if there were a way to distribute engines, and include
other engines along with it. Perhaps a dependency kind of
thing…when you install the Typo engine, it automatically installs
the SHLG engine alongside. That would also make it really easy to
swap different authentication systems in and out, assuming they have
the same interface. Somehow though the engines would have to be smart
enough to look in the Typo engine dir for overrides on SHLG views and
controllers, rather than the main app dir.

I guess what I’m really saying here is that it’d be awesome if there
were very basic building blocks available, and you can put those
together to create slightly more complex blocks with integrated
functionality, that can be configured easily by just substituting
different engines in for the very simple functionality.

I don’t think I could implement any of this really, and not sure if
I’ve explained it well enough…hopefully James A.s, and whoever
else may decide to contribute, can use this as a launching pad of
sorts.

Pat

On Nov 7, 2005, at 2:29 PM, Pat M. wrote:

thing…when you install the Typo engine, it automatically installs
different engines in for the very simple functionality.

Yes!

I think this is a great idea. I was thinking of the very same thing
when I started with the Typo Engine project. There are still a lot
of pieces yet to be put in to place, but we can move in this
direction. I’ve started with a patch that gives the template
rendering methods a “chain of command” so that templates will be
rendered from alternate directories if not found in the view’s
primary location [1].

Thinking about this further, here’s what still needs to be done to
make recursive engines a reality:

  1. Routes need some kind of recursive configuration ability.

As they are, routes limit us in two significant ways:
a. They are configured once (at boot-time) and in one location
only (routes.rb’s draw method call).
b. They assume an absolute base path from which each route is
interpreted.

The first limitation has (possibly?) been overcome by the productized
Rails source code–there are some methods I wrote in that gem that
permit second-tier applications to re-open the routes table and add
their own rules from within a separate routes.rb file.

Overcoming the second limitation has not (to my knowledge) been
attempted. The problem is that a route such as this:

map.connect ‘articles/page/:page’,
:controller => ‘articles’, :action => ‘index’,
:page => /\d+/

has an assumed base path: ‘/’. What would be useful is if we could
set up a route in the main routes.rb file that then passes off
control to a sub-router. For example:

config/routes.rb:
ActionController::Routing::Routes.draw do |map|
map.connect ‘blog/*anything’, :rerouter => ‘TypoEngine::Routing’
end

vendor/plugins/typo_engine/config/routes.rb:
class TypoEngine::Routing < ActionController::Routing
# Not sure what goes here… maybe some way to prepend a
# ‘/typo/’ module to all controllers?
end

TypoEngine::Routing::Routes.draw do |map|
map.connect ‘:controller/:action/:id’
end

This might work, but then it assumes that sub-engines need to know
nothing of their parent. Ideally, routing should be just one
attribute of the recursive engine configuration tree.

  1. We need a globally accessible “this_engine” variable.

This has come up all over the place as I’ve tried to make Typo in to
a well-behaved engine. What I’m talking about is a way of accessing
everything to do with the engine (including its relationship to other
engines) in a object-oriented “self” or “this” kind of way.

For example, in Typo there is a “#{RAILS_ROOT}/themes” folder unique
to that application. Models, Views and Controllers, as well as any
config modules or libraries all need to access the configuration
information that pertains to “this engine”.

Using this_engine.parent or this_engine.children I imagine we could
also gain access to the configuration information pertaining to
related engines.

  1. Something has to be done about namespace pollution.

Typo has more than 20 models, including such generic ones as
“Category”, “User” and “Tag”. Undoubtedly this is going to cause
some problems when integrating in to other applications. So far, I
haven’t found a really elegant way of fixing this problem. Should we
just require that engine developers prepend the engine name in front
of all model and controller classes?

For example:
/vendor/plugins/typo_engine/app/models/category.rb:
class Typo::Category < ActiveRecord::Base; end
/vendor/plugins/typo_engine/app/models/tag.rb:
class Typo::Tag < ActiveRecord::Base; end

etc.
(Does anyone know if this can be done currently?)

And for controllers:

/vendor/plugins/typo_engine/app/controllers/typo/articles_controller.rb:
class Typo::ArticlesController < ApplicationController; end
/vendor/plugins/typo_engine/app/controllers/typo/admin/
blacklist_controller.rb:
class Typo::Admin::BlacklistController < ApplicationController; end

  1. The engines plugin needs to load recursively.

The engines plugin would have to load engines recursively. I’m sure
there are some other issues involved with this step that I haven’t
thought of yet.

  1. Other issues: What about library version conflicts?

Typo has several dependencies, all in the vendor directory. What if
an application needs one version of RubyPants while the Typo engine
needs another?

I don’t think I could implement any of this really, and not sure if
I’ve explained it well enough…hopefully James A.s, and whoever
else may decide to contribute, can use this as a launching pad of
sorts.

Ideas are good… source code is better! :wink:

Duane J.
(canadaduane)

[1] Peak Obsession

The plugin dependencies ‘problem’ (which is a superset of any engines
dependencies problem) is one that is not likely to see a solution as
part of core Rails in the near future - understandably, nobody wants
to end up re-implementing the same kind of work that’s been done with
rubygems. So for the moment, if one plugin needs another plugin (and
therefore ditto for engines), it’s up to the engine developer to make
that very clear in their documentation, and up to the application
developer to actually read that documentation and follow up on those
requirements. Documentation is crucially important for plugins and
anything else like them. Did I mention that documentation is
important? :slight_smile:

A patch was applied recently which enabled a plugin to ensure another
plugin was loaded before it by called 'load_plugin ’ in the
calling plugin’s init.rb file. This is something of a temporary fix,
however.

It’s very simple to build an engine on top of another one; one
engine’s controllers/views/helpers/models can override code in an
underlying engine in the same way that you can override the code in
your application’s /app directory. I’ll hopefully have an example of
this available by the end of the week. What you as a developer must
bear in the mind is that the order in which engines are ‘started’ is
what defines the order of precedence in which code will be ‘dominant’
in the final running system. E.g., in environment.rb, if you call:

Engines.start :engine_one, :engine_two

or

Engines.start :engine_one
Engines.start :engine_two

… code in engine two will take precedence over that in engine one;
if two identical methods in two identically-named controllers exist,
the one from the latest-loaded engine will be used when your system
runs.

[Someone put this up on the wiki?]

In a nutshell… I doubt you’ll be able to automatically pull all the
engines you need for a certain system any time soon, but if developers
make it very clear which building blocks you need (engines AND
plugins), then you are free to plug them together in whatever order
your heart desires…

  • james

/vendor/plugins/typo_engine/app/models/category.rb:
class Typo::Category < ActiveRecord::Base; end
/vendor/plugins/typo_engine/app/models/tag.rb:
class Typo::Tag < ActiveRecord::Base; end

etc.
(Does anyone know if this can be done currently?)
No, unfortunately not.

http://dev.rubyonrails.com/ticket/2147

Some personal opinions…

On 11/7/05, Duane J. [email protected] wrote:

I’ve started with a patch that gives the template
rendering methods a “chain of command” so that templates will be
rendered from alternate directories if not found in the view’s
primary location [1].

This is basically what the Engines plugin does right now anyway, as
far as I can tell.

  1. Routes need some kind of recursive configuration ability.

I think there’s something to be said for making the developer do some
work themselves here. If your engine provides too much magic, when
bugs crop up (and the invariably will), it will be harder and harder
for the developer to trace where the issue actually lies. Is it so bad
to say to the engine user “When you install this engine, you’ll want
to add something like this to your application routes.rb file”?

The login engine currently requires developers to include modules in
the ApplicationController and ApplicationHelper. It’s technically
possible to have this happen automatically, but I fear that if you
hide too much from the guy or gal who is actually using your engine in
their application, it will become harder for them to understand what
your engine does, how it works and how they might be able to resolve
any issues themselves. That’s my gut feeling anyway.

  1. We need a globally accessible “this_engine” variable.

This might happen, it’s something I’ve toyed with. Stay tuned, as they
say.

  1. Something has to be done about namespace pollution.

I wish I had something profound to say about this right now, but alas

  • I don’t. Try stuff out and see what works?
  1. The engines plugin needs to load recursively.

Not recursively, but sequentially and in a defined order is possible
right now. Do you really need it to be recursive?

  • james

On Nov 10, 2005, at 4:07 AM, James A. wrote:

=> #<Engine:0x236bc00 @name=“login”,
@root=“script/…/config/…/config/…/vendor/plugins/login_engine”>

  • james

Sweet deal! Thanks, James.

Duane J.
(canadaduane)

On 11/7/05, Duane J. [email protected] wrote:

  1. We need a globally accessible “this_engine” variable.

In SVN now:

Engines.current
=> #<Engine:0x236bc00 @name=“user”,
@root=“script/…/config/…/config/…/vendor/plugins/user_engine”>

Engines.get(:login)
=> #<Engine:0x236bc00 @name=“login”,
@root=“script/…/config/…/config/…/vendor/plugins/login_engine”>

  • james

If when you brain sees ‘engines’ it reads ‘big clunking high-level
monolithic components’, please read this:

http://rails-engines.rubyforge.org/wiki/wiki.pl?OhGodWhatHaveWeDone

  • james

On the subject of engines, I’ve written
http://weblog.rubyonrails.org/articles/2005/11/11/why-engines-and-components-are-not-evil-but-distracting

It’s a tale of caution, not condemnation.

David Heinemeier H.
http://www.loudthinking.com – Broadcasting Brain
http://www.basecamphq.com – Online project management
http://www.backpackit.com – Personal information manager
http://www.rubyonrails.com – Web-application framework

Duane said he wants to turn Typo into an engine…that seems like a
pretty big system to me. Not huge, but definitely more comprehensive
than the simple LoginEngine. You mention in your wiki entry that you
hope developers will use common sense as far as what to convert into
an engine, and what not to. Considering that you haven’t raised any
objections to the TypoEngine project, I’m really just not sure how big
is too big.

The main reason that the engines appeal to me is that I can write one
app and then have as many custom views for it as I want, and add on a
bit of custom functionality if I need to. Previously I was using the
Productized Rails, because that fit my need, but engines seem less
monolithic, and more flexible. I like the idea of having an entire
application that I can just load as an engine, and I’m up and running.
Then I can override the views as I see fit.

I suppose it’d be possible to write a plugin to handle the model
(well, offload functionality to a module) and the controller, and use
a generator to create initial view files? Just seems like it’s a lot
easier/smarter to just include the views in an engine and then
override them in the app.

Am I missing the point of engines?

Pat

On 11/12/05, Pat M. [email protected] wrote:

I suppose it’d be possible to write a plugin to handle the model
(well, offload functionality to a module) and the controller, and use
a generator to create initial view files? Just seems like it’s a lot
easier/smarter to just include the views in an engine and then
override them in the app.

Pat, I think you’ve grasped the point of Engines here exactly. In
fact, you’ve made my day by hitting the nail on the head so well.

Duane said he wants to turn Typo into an engine…that seems like a
pretty big system to me. Not huge, but definitely more comprehensive
than the simple LoginEngine. You mention in your wiki entry that you
hope developers will use common sense as far as what to convert into
an engine, and what not to. Considering that you haven’t raised any
objections to the TypoEngine project, I’m really just not sure how big
is too big.

It’s not up to me to say who can make any particular Engine… Duane
is a smart guy, and he’s given me invaluable help improving the way
that the Engines plugin itself works. His “TypoEngine” might not be
suitable for anyone else in the world to use, but if it works for the
specific project he wants it for, who is anyone to tell him he can’t
do it? It’s worth doing it for the ride alone - I’m sure as a result
of his efforts we’re learning learning more about the guts of Rails
than ever before. And that can be no bad thing at all.

  • james

pergesu wrote:

Duane said he wants to turn Typo into an engine…that seems like a
pretty big system to me. Not huge, but definitely more comprehensive
than the simple LoginEngine. You mention in your wiki entry that you
hope developers will use common sense as far as what to convert into
an engine, and what not to. Considering that you haven’t raised any
objections to the TypoEngine project, I’m really just not sure how big
is too big.

If engines aren’t the right way to use multiple “big” Rails applications
on one website (e.g. Blog, Forum and Wiki), then what IS the right way?

James A. wrote:

His “TypoEngine” might not be
suitable for anyone else in the world to use, but if it works for the
specific project he wants it for, who is anyone to tell him he can’t
do it?

Actually, I think you’ve hit the nail on the head. I also am a former
user of
Duane’s clever Product generator (before it was a generator, even). Now
I’m
refactoring that common code into multiple engines that I can bolt on
the
various sites.

The interesting part is that I’m coming up with the same sorts of
engines that
(presumably) everyone else is: user/auth/ACL, blog, wiki, image gallery,
music
server, etc. None of them are terribly featureful at the moment, so I
would
happily swap-out my home brewed blog engine for Typo, if it was a nice
simple
drop-in module. I’d add Trac just for my own use, if it was an engine.
Since
it’s so easy to override stuff, I’ll probably switch to LoginEngine, and
just
override the parts where I like to do things differently. Over time,
others
will probably add features, which I’ll get for free. What’s not to
like?
It’s code I don’t have to write.

I guess that’s my whole point is: no such thing as an engine that’s “too
big”.
If it has stuff you don’t want, disable it. If it’s missing
functionality,
add it. If that’s too much trouble, don’t use it at all.

Anything that facilitates sharing and reusing code is a Good Thing[tm].

==============================================================================
Steve S. Hacker / Drummer / Super
Genius

On 12/1/05, Andreas S. [email protected] wrote:

on one website (e.g. Blog, Forum and Wiki), then what IS the right way?
I don’t have any problem with doing so, and in fact I’m using the
engine approach for writing apps like this. I just remember seeing
some comment in the past. Anyway, I like Engines…and one thing I’ve
found is that it removes a lot of conflict between applications. Just
include the LoginEngine, and you can use however many other engines
you want easily, all using the same backend, instead of having to
strip user code from your blog, or move user code into a separate app
entirely. Pretty sweet.

Pat

For those who’ve been following this thread and are curious about the
state of the TypoEngine project:

On Nov 11, 2005, at 8:55 PM, James A. wrote:

Duane said he wants to turn Typo into an engine…that seems like a
that the Engines plugin itself works. His “TypoEngine” might not be
suitable for anyone else in the world to use, but if it works for the
specific project he wants it for, who is anyone to tell him he can’t
do it? It’s worth doing it for the ride alone - I’m sure as a result
of his efforts we’re learning learning more about the guts of Rails
than ever before. And that can be no bad thing at all.

I’ve spent quite a bit of time and effort trying to turn Typo in to a
pluggable engine. I do think it is possible… eventually; however,
for the amount of effort it’s taken to convert it to an engine, I
could have rewritten it. In fact, I did basically that in the end,
by pulling pieces of it out and putting it in to my app.

The problem is not so much in the area of enginifying it–though
there are still some hiccups there–but in making it useful to our
application as an engine. James got it exactly right. For example:
in our application, we need multiple blogs. That seems like a nice,
easy thing to ask for, but unfortunately it means changing
everything to include a blog_id, both on the user side and on the 9
admin pages, as well as the api, xml and feed code. In addition, we
also require certain security measures that are not present as is.

So while I think it’s still possible to make Typo in to an engine,
I’ve had to stop myself and ask why I’m doing it. What I really
wanted was a MultipleBlogWithUniqueSecurityForMyAppEngine. Sigh. I
don’t think anyone’s built it yet :wink:

Duane J.
(canadaduane)

P.S. If anyone would like to take a look at the partly functioning
TypoEngine code, email me. I won’t be developing it any longer,
however.

On 12/1/05, Duane J. [email protected] wrote:

is too big.

easy thing to ask for, but unfortunately it means changing
(canadaduane)

P.S. If anyone would like to take a look at the partly functioning
TypoEngine code, email me. I won’t be developing it any longer,
however.

Were you trying to use multiple blogs in the same database? That’s
the only reason I can see why you’d need a blog_id all over the place.
Otherwise you “just” include the engine in the app, and specify the
db connection details.

Here’s how I’m doing things. Let’s say I have a blog engine, a wiki
engine, and a bug tracker engine. One of my users wants just a blog
and a wiki, and one wants all three. I’d create a project for each
one, and include the engines for the functionality he wants. Each
project is separate, so I can use a different database for each site,
and there’s no need to manage things like a blog_id.

I suppose that I use the Engines in a very similar light to how I was
using your Productized Rails…allowing me to have core functionality
which I extend where necessary, and override the views easily. The
difference is that the Engines make a lot more sense to me for that
purpose. I want to keep projects separate in case I deploy them to
different servers. I want to be able to include only the components I
want in each project.

I haven’t taken a look at your TypoEngine code, and I’m sure it’s no
small task converting Typo into an engine. I’m just curious what your
exact requirements are that excludes TypoEngine as a possible solution

  • namely your need for a blog_id, which I imagine is as a result of
    storing multiple blogs in a single database.

Pat

On Dec 1, 2005, at 11:40 AM, Pat M. wrote:

in our application, we need multiple blogs. That seems like a nice,
Duane J.

difference is that the Engines make a lot more sense to me for that

Yes, you’ve got it exactly. It’s just a different requirement than
what Typo was originally intended to solve.

Duane