Forum: Rails Engines development Introduction: modular methods (feedback required!)

A few thoughts on engines first.

When you install Rails, it does not do anything. This is what I like. To 
make it do anything, you need to write code.

It is extermely simple to write simple code in RoR. I like this too. You 
can start with scaffold generator, then adjust some methods, add some 
relations etc. Rails supports you on your way. If you need a list, you 
have acts_as_list. If you need date/time fields on your forms, there's a 
date helper.

However doing complex things is, of course, not that easy. There's no 
acts_as_customer_info_form_with_tax_reports_and_employment_histories. 
This should NOT be done with one line of code. To summarize, Rails 
provides building blocks, and it is you who assembles your app.

Now, engines are completely opposite to this. You add a LoginEngine and 
your application gets very complex login logic at once, without you 
having a slightest idea on how it works or how it can be customized. 
What's more, there are no building blocks for this â?? if you don't like 
anything lying deeply inside LoginEngine, your only option is to rewrite 
all or most of it. This is too bad.

LoginEngine makes 2 mistakes. First, it makes very complex things 
(salted login support with email confirmation, password recoveries, 
security tokens etc) too simple. Second, it does not make "non-default" 
simple things enough simple.

Of course, engines are perfectly suitable when you really want to reuse 
some business logic, e.g., in a series of similar applications. 
Probably, Wiki or forum engine is enough self-contained to be an engine. 
(Or, maybe, not. The key is reusability. It's very hard to make reusable 
business logic. It's much more useful to provide reusable building 
blocks. If one wants a Wiki, he should spend a day rolling out his own 
Wiki using Wiki building blocks, rather than getting an uncustomizable 
monster at once.)

Testing is not addressed either. There's no point in testing existing 
functionality of an existing engine: it's authors should have tested it 
well (after all, they are the authors of the testsuite, so the engine 
already passes it probably). What should really be tested are your 
customizations and their relations to standard parts of the engine. 
However nothing helps you to do this.

Now I introduce a concept of "modular methods". These are class methods 
for your models, controllers and testcases that are delivered via 
plugins, support reflection (if you know what this means) and serve as 
building blocks rather than complete solutions.

An example is:

class Person < ActiveRecord::Base
  stores_encoded_representation_of :password, :salt => "Foo123"
end

Observe that stores_encoded_representation_of modular method is called. 
There is nothing unusual here. In fact, I do not offer any new ideas, 
the key point is just to think in terms of building blocks.

Modular methods can interact with each other:

class Person < ActiveRecord::Base
  generates_per_record_salt
  stores_encoded_representation_of :password, :salt => "Foo123"
end

Here, stores_encoded_representation_of method detects that 
has_per_record_salt has been called and makes use of per-record salts to 
make passwords non-interchangable between records (just like LoginEngine 
does).

Yet another example:

class Person < ActiveRecord::Base
  has_per_record_salt :salt
  stores_encoded_representation_of :password, :salt => "Foo123"
  validates_length_of :password, :within => 6..20, :if => 
:password_changed?
  validates_confirmation_of :password, :if => :password_changed?
end

Here standard methods validates_length_of and validates_confirmation_of 
perform validation only if a new password is being set. 
"password_changed?" instance method is generated by "tracks_changes_of" 
modular method, which is being invoked by 
"stores_encoded_representation_of". Tracks_changes_of is clever enough 
to ignore multiple invocations with the same field. (Yes, all those 
communications are possible because of reflections. Rails already 
supports reflections on aggregations and assosications, and modular 
methods add support for reflections on methods.)

There's a modular_methods plugin which defines some support classes. 
Probably the most complex support is provided for testcase modular 
methods. Given a model like this:

class PersonForEncodedRepresentation < ActiveRecord::Base
  validates_length_of :password, :if => :password_changed?, :within => 
3..10
  validates_length_of :salted_password_1, :if => 
:salted_password_1_changed?, :within => 3..10
  validates_length_of :salted_password_2, :if => 
:salted_password_2_changed?, :within => 3..10

  generates_per_record_salt :salt => 'WOO-HOO!'

  stores_encoded_representation_of :password, :use_per_record_salt => 
false
  stores_encoded_representation_of :salted_password_1, :salt => 'xxx', 
:use_per_record_salt => true
  stores_encoded_representation_of :salted_password_2, :salt => 'yyy'
end

it allows to write testing code like this:

class PersonForEncodedRepresentationTest < Test::Unit::TestCase
  fixtures :people_for_encoded_representation

  OPTIONS = {
      :encoded_length => 40,
      :valid_values => ['secret', 'topsecret'],
      :invalid_values => ['x', 'verylong' * 10],
      :base_fixture => :bob,
      :samples => [:david, :andrey]
  }

  tests_encoded_representation_of :password, OPTIONS
  tests_encoded_representation_of :salted_password_1, OPTIONS
  tests_encoded_representation_of :salted_password_2, OPTIONS

  tests_encoding_incompatibility_of :password, :salted_password_1,
      :salted_password_2,
      :base_fixture => :bob, :value => 'secret'

  ATTRS = [:name, :password, :salted_password_1, :salted_password_2]

  tests_encoding_compatibility_across_records_of :password,
      :sample => :david, :sample_attrs => ATTRS

  tests_encoding_incompatibility_across_records_of :salted_password_1,
      :sample => :david, :sample_attrs => ATTRS
end

Note all this fixtures and "samples" stuff. It's needed. Suppose you've 
got a LoginEngine, and you add a new column to your users table and a 
corresponding validation to your model. How can LoginEngine save any 
rows now? To pass validation, it must fill in your new field, but it 
does not have a slightest idea on how to do that.

With modular testing methods, it's you who writes the fixtures, so you 
can provide all necessary fields there. You then give the names of your 
fixtures, and they get loaded automatically. Also you sometimes provide 
a set of valid and invalid values, for the methods to use them when they 
need a valid or invalid record. (For example, 
tests_encoded_representation_of uses invalid values to check that, after 
an unsuccessful validation, the value of the column does not get encoded 
even if it has been changed.)

(Internally, modular methods are implemented using corresponding 
classes, like StoresEncodedRepresentationOf. There is some syntatic 
sugar for modular method writers, for example, options are automatically 
parsed and assigned to instance attributes. There are some helper 
methods, and some activities are automated.)

I'm currently working to implement this idea. I don't have much time, 
though. Anyway, feedback and discussions are welcome! (Anyone willing to 
help me is especially welcome.)

You can look at the code via SVN:

svn://82.146.42.23/webdevel/rails/plugins/modular_methods/trunk
(the plugin itself)

svn://82.146.42.23/webdevel/rails/plugins/modular_methods_test/trunk
(a test application for the plugin, has plugin in svn:externals)

(Sorry for IP's, have not bought a domain yet. Also available as 
andreyvit.firstvds.ru instead of IP if somebody cares. Sorry, cannot run 
Apache 2 now so cannot provide http access to the repository.)

The code is far from being ready for real-world usage, but the idea 
should be clear.

Andrey.

Hi Andrey,

I don't have very much that I can usefully add to your ideas, save a
few comments on what you've said about engines:

On 3/7/06, Andrey Tarantsov <andreyvit@gmail.com> wrote:
> Now, engines are completely opposite to this. You add a LoginEngine and
> your application gets very complex login logic at once, without you
> having a slightest idea on how it works or how it can be customized.

....this is always the case when you take code from an outside source.
The onus is on you to examine it, and understand how it will interact
with your application. But distributing black-box components is NOT
the purpose of the engines plugin, but rather a secondary effect with
unique benefits and of course, some unique drawbacks.

> What's more, there are no building blocks for this ? if you don't like
> anything lying deeply inside LoginEngine, your only option is to rewrite
> all or most of it. This is too bad.

This is both true and not true. You can override elements within an
engine at the method and view/partial level - very easily I might add
- so what you say above is false if you are applying it to engines in
general.

The LoginEngine itself, however, does have aspects where it's
implementation doesn't lend itself well to being overridden. That's
because it's only an *example*, and represents the minimum amount of
work I had to do to transform the authentication-system-du-jour (at
the time) into an engine for the purposes of illustrating what it is
that Engines are, and why they might be useful to some folks.

To make this absolutely clear, the point of engines is NOT so I can
write a bunch of components for you guys to plug together in weird and
wonderful ways. It's so YOU can develop your applications in a modular
way and reuse those modules across your applications as YOU determine
might be appropriate. The community sharing is essentially an
afterthought, but if you're written something that other people find
useful then that presents a not-inconsiderable secondary bonus.

Producing engines for the world at large is not, for me at least, a
quest to give birth to something that will work for everyone. If you
find yourself having to rewrite a lot of some component, chances are
you would be better off not using it. However, at the same time you
might realise that you're going to need this new module, including its
views and other assets, in your next project. The point of the engines
plugin, and the 'engines' development technique - to help you do
exactly that.

Regarding the LoginEngine in particular, I have no doubt that a
better, more modular login system can be developed and used in place
of the LoginEngine, but lets please be clear  when we're talking about
engine development, and be sure to distinguish it from the particular
implementation within a given engine. Engines != [LoginEngine,
UserEngine].

As a final note which relates somewhat more directly to your ideas, I
think you're right to bring focus on testing and engines, but it
should be borne in mind that there's absolutely nothing stopping a
developer from writing their own set of fixtures in the application
root, and a corresponding set of methods to test any methods they have
changed or added. They should hopefully not have to test everything
(no need for a test_user_present_in_session_after_login or anything
like that) - just what's appropriate for their application
(test_non_admin_user_can_edit_wiki_page or whatever). Perhaps the
upcoming integration testing will light a clearer path to modular
testing nirvana...

Sorry if this doesn't seem quite on-topic, I just feel it's essential
not to bring criticisms of the LoginEngine's "lack of modularity"
against engines in general.

If, as I hope, you are simply chastising the poor implementation of
that engine, believe me that you are not alone in feeling this way.
Onwards and upwards. Good luck exploring your modular methods idea!

- james

Hi!

I think the engines are nice to:
1) share some common logic between several applications on your own;
2) make an example implementation of something available to the world.

Engines, at my opinion, are not good for (and I'm going to give
arguments for this):
3) making an implementation of something that is usable in a wide range
of applications.

Overriding elements within an engine works good for the first case. If
something gets unoverridable, you refactor your engine a bit and stay
happy.

Overriding *CAN* work for the third case *IF* the engine is designed
with that in mind. Doing this is difficult, but even the more important
problem is that this way is not in the spirit of Rails, and is not easy
for users. I've already told why in the first post. (Rails does not
provide ready-made implementations, it generally focuses on providing
building blocks, and the assembly process stays in the hands of
developer. This way the developer knows how his application works.)

The very important thing you must realize is that people tend to take
the 2nd case as if it was the 3rd case. I've seen this happening in
other projects, and now this starts to happen with Rails. Instead of
taking the LoginEngine as an example and rolling their own engine for
their specific tasks (which will probably be common for a set of
applications), they treat it as an unmodifiable black box and search for
a way to customize it. If you look through the engine users mailing list
(and I guess you do :), you already know this is happening.

What's more, people take LoginEngine as an example of a proper engine,
and start distributing their own engines similar to it. But this is not
a correct way! Their code would be much, much more reusable if they
would split it into several building blocks, and make such blocks
available.

Not only splitting up complex logic into building blocks helps
reusability, it also helps to produce somewhat more managable code,
which better adheres to the "one module (class) - one task" rule.

> But distributing black-box components is NOT
> the purpose of the engines plugin, but rather a secondary effect with
> unique benefits and of course, some unique drawbacks.

But this side-effect has very important consequences, which can even
ruine the whole Rails plugins idea. One must be very sure before he
starts to wrap complex logic into an opaque component for
redistribution.

> The LoginEngine itself, however, does have aspects where it's
> implementation doesn't lend itself well to being overridden.

The problem is that you believe one can make a Good Engine that will
solve this problem. What I say is that making public reusable engines in
not practical at all, because such complex logic needs a higher level of
customization than is possible with engines-style overriding and/or
configuration.

> To make this absolutely clear, the point of engines is NOT so I can
> write a bunch of components for you guys to plug together in weird and
> wonderful ways. It's so YOU can develop your applications in a modular
> way and reuse those modules across your applications as YOU determine
> might be appropriate.

Yeah, this is all right, and I fully agree with this. However I'm
worried by all the newly appearing public engines which are nearly
uncustomizable.

> The community sharing is essentially an
> afterthought, but if you're written something that other people find
> useful then that presents a not-inconsiderable secondary bonus.

I think it must be stated somewhere that engines (unlike plugins) are
not meant to provide drop-in functionality from public sources. Doing
this can bring serious scaling (in the sense of requirements changes)
and customization problems to your application.

> If you find yourself having to rewrite a lot of some component,
> chances are you would be better off not using it.

Yeah, that is *exactly* what I've talking about: if the only thing you
like about, say, LoginEngine is the way it handles account deletion, you
cannot use just that one thing from it (and *still* have benefit of
updates of that feature, that is, still get everything one gets when he
moves from code generators to engines).

Engines are good as a way to share complete business logic betwen your
projects. (For example, I have a credit-card processing engine that
works with my specific bank and with my specific sschemas. I'm not going
to make it public because I'm not going to maintain it as a public code
with a stable interface. *And* it is inefficient to do so, even if I
wanted to.)

However, then, a way to share truly reusable building blocks is still
required for Rails. (Another reason engines are not good for this is
that you can have only one copy of an engine, while you could use the
same building blocks to build two different parts of your application.)

> Sorry if this doesn't seem quite on-topic, I just feel it's essential
> not to bring criticisms of the LoginEngine's "lack of modularity"
> against engines in general.

I hope that I've explained my thoughts better, and that I've made clear
that the problem is not specific to {Login,User}Engine, but is much more
general and relates to all public engines.

Yes, one can build a modular engine, but both it's usage and development
(and even usage documentation) would be harder than splitting the
modules and implementing them as separate non-engine plugins.

> Good luck exploring your modular methods idea!

Thank you, I hope I'll come up with a good solution to the "reusable
blocks" problem I've stated.

Still another question: being tired of edge rails and trunk engines
(in)compatibilities, is there a plan to integrate Engines plugin into
the core?

Andrey.

On 3/20/06, Andrey Tarantsov <andreyvit@gmail.com> wrote:
> Still another question: being tired of edge rails and trunk engines
> (in)compatibilities, is there a plan to integrate Engines plugin into
> the core?

There is no plan to integrate the functionality that the Engines
plugin provides into the core.

The engines plugin is currently compatible with edge rails, and will
absolutely be compatible with Rails 1.1 - when using the edge of
anything, you MUST expect issues - it's not called the bleeding edge
for nothing. However, if you are seeing problems, please file a report
at http://dev.rails-engines.org, and we'll see what can be done.

--
* J *
  ~