Ruby Forum Ruby-core > Re: new method dispatch rule (matz' proposal)

This is a late response to the very long thread that started back in
January with this message:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/9996

As I understand it, the problem Matz is seeking to address is that
private methods are inherited by and may be overridden by subclasses.
This means that a subclass writer can inadvertently alter the behavior
of its superclass simply by defining a method that has the same name as
an internal helper method of the superclass.  The only way to defend
against this is for subclass writers to be familiar with the internal
implementation of the superclass.

Matz's proposal was to resolve this problem by altering the method
resolution algorithm.  If I understand correctly, the problems were:

1) It was kind of complicated
2) The method lookup algorithm depended on whether a method was called
functionally without an object prefix or in OO form with an object
prefix.  (That is called as foo() or o.foo())
3) There were backward compatibility issues

Part of the discussion was about the question of whether to alter the
semantics of private methods, or add a new "local" or "personal"
visibility level for methods that were truely local to the defining
class and could not be inherited or overridden.

With all that as review, here are my thoughts on the matter.

1) It is reasonable to expect private methods (or local methods if we
call them that) to be looked up and dispatched differently than
non-private methods.

2) The problem, though, is that since Ruby is a dynamic language, the
interpreter can't tell whether a method is private or not until it has
looked it up.

3) Therefore, solutions like Matz's proposal depend on the method
invocation syntax (functional style versus object-oriented style) to
determine which method lookup algorithm is to be used.  This leads to
the confusing and hard-to-justify situation in which the invocation
"foo" might do something different than the invocation "self.foo".

I propose, therefore, that some new syntax be introduced (avoiding
backward incompatibility) either for making local method names obviously
different from other method names.  Or that a  new method invocation
syntax be introduced and that this new syntax be used when we want to
invoke a local method defined in (or inherited by) the caller without
considering any methods defined further down the class hierarchy.

I don't grok parse.y well enough to know whether any syntax I would
propose would actually result in an unambiguous grammar, but my
suggestion is that local methods have an @ prefix just like instance
variables do.  Then the interpreter knows from the name of the method
that it is to be dispatched differently.

In fact, I might actually go further than this and propose that local
methods are not actually methods, but are syntactic sugar for lambdas.

In Ruby 1.8, we can write code like this:

class Test
   def initialize(greeting)
     @greeting = greeting
     @greeter = lambda { |x| puts "#@greeting #{x}" }
   end

   def greet(x)
     @greeter[x]
   end
end

t = Test.new("hello")
t.greet("world")

In this code, the instance variable @greeter refers to a local function
that is completely hidden from subclasses and cannot be altered.

Perhaps Ruby 1.9 could take this idiom and add syntax sugar to make the
definition and invocation of local functions more like the definition
and invocation of regular methods.

   David Flanagan

On 6/13/07, David Flanagan <david@davidflanagan.com> wrote:
> against this is for subclass writers to be familiar with the internal
>
>
> I propose, therefore, that some new syntax be introduced (avoiding
> that it is to be dispatched differently.
>    end
> that is completely hidden from subclasses and cannot be altered.
I strongly disagree here. While I think it is good to protect private
methods against inadvertently overriding them, I consider it harmful
to completely hide them from subclasses. If I really want to change
something I should be able to do so (i.e. via reflection). In
languages with strict privacy concepts like Java it is a pain to
extend a class in ways other than intended by the author of that
class.

-- henon

Meinrad,

I favor leaving private methods exactly as they currently are, but
adding a new mechanism or visibility level that a class writer can use
to ensure that local helper methods are not overwritten.  Surely the
author of a class ought to have the right to decide whether subclassers
are allowed to meddle!

     David

Perhaps an underscore:

  class X
    def _foo
      puts "I'm local to X."
    end
  end

I suggest that b/c the idea of local instance vars in the form of @_a
has also been given consideration.

Though, I sort of like the idea of:

  def @foo
    ...
  end

But maybe that only makes sense if all instance vars are local and
accessors are required to communicate between class and subclass.

T.

David Flanagan wrote:
>   end
> end
> 
> t = Test.new("hello")
> t.greet("world")
> 
> In this code, the instance variable @greeter refers to a local function 
> that is completely hidden from subclasses and cannot be altered.

class Test2 < Test; end
t2 = Test2.new("hello")
t2.greet("world") #=> "hello world"

How is this hidden from subclasses? Or did I miss something?

But this touches on an idea I was thinking about recently... how about
giving a warning when a subclasses overrides a method and doesn't use
"super"? IMHO in 99% of cases if you override a method you should call
super. In the other 1% of cases you could use an idiom like "super if
false" which makes it very clear when reading the code that we are
(dangerously) skipping the normal inheritance chain. It solves the same
problem as the new method dispatch rule but for *both* public and
private methods, while still leaving the programmer freedom to override
private methods if he knows what he's doing. Comments?

Daniel

On 6/13/07, David Flanagan <david@davidflanagan.com> wrote:
> Meinrad,
>
> I favor leaving private methods exactly as they currently are, but
> adding a new mechanism or visibility level that a class writer can use
> to ensure that local helper methods are not overwritten.  Surely the
> author of a class ought to have the right to decide whether subclassers
> are allowed to meddle!
>
>      David
>
Hmm, but if there is something about a superclass which cannot be
changed (not even by using some backdoor) the re-usability of that
class is poor because it might not be possible to adapt the class to
specific needs. It is impossible for the author to predict all
possible use cases. That's why everything should be allowed. Nobody
wants that others decide what he needs or needs not.

I agree with you that prevention of *accidental* overwriting of
private methods is a good thing though.

-- henon

Daniel DeLorme wrote:
>>     @greeter[x]
> t2 = Test2.new("hello")
> problem as the new method dispatch rule but for *both* public and 
> private methods, while still leaving the programmer freedom to override 
> private methods if he knows what he's doing. Comments?
That ends up looking a lot like a magical side-effect to me.  Besides, I
don't think the assumption (that you should often be calling super)
necessarily valid.

What it sounds to me like you're pining for is the "override" keyword
from C#.  I guess an equivalent implementation in Ruby might be for
methods defined with "def" to pop a warning when they redefine a
superclass method, and to introduce a "redef" keyword to do specifically
that.

Not sure I like the look of it (introducing new keywords, bad), but I
think it would fit the bill here.

On 6/13/07, David Flanagan <david@davidflanagan.com> wrote:
> against this is for subclass writers to be familiar with the internal
> implementation of the superclass.

I would argue that there's really no technical defense against the
issues raised by inheriting from classes outside of your control, or
having classes outside of your control subclass your class.
Inheritance is implementation sharing and it inherently breaks
encapsulation.  That's why it's almost always better to delegate to
classes you 'buy' rather than subclassing them.

Back when I was evangelizing OOP for IBM, I used to talk about this
and say that inheritance was like sex, it's much more socially
acceptable when it's done between committed consensual adults.

This isn't to say that such subclassing is always bad, just that it
requires care.

The case isn't that subclass writers can inadvertently alter the
behavior of a superclass by overriding a private helper method.
Without actually opening up the superclass and changing it, the
superclass and it's instances will still work.  What gets broken is
the subclass and its instances, and I'd argue that that's no different
than any other bug in the subclass, and it's not limited to
inadvertantly changing a PRIVATE method. Of course private methods are
more likely to be undocumented and missed by a drive-by subclasser.

And if the concern about breaking classes is real, then the solution
would need to include doing away with the ability to open existing
classes. That would really mean that Ruby wouldn't be Ruby anymore
IMHO.

My sense is that if the changes were to be implemented they would just
provide a false sense of security, better to avoid introducing further
complexity, I'd argue for better "inheritance education" which
advocated delegation over promiscuous inheritance, and
"safe-inheritance" practices using the existing mechanisms.

> semantics of private methods, or add a new "local" or "personal"
> visibility level for methods that were truely local to the defining
> class and could not be inherited or overridden.
>
> With all that as review, here are my thoughts on the matter.
>
> 1) It is reasonable to expect private methods (or local methods if we
> call them that) to be looked up and dispatched differently than
> non-private methods.

Personally, my instincts say no.

> 2) The problem, though, is that since Ruby is a dynamic language, the
> interpreter can't tell whether a method is private or not until it has
> looked it up.

And here I think that we run into the tensions between the concepts of
static and dynamic languages. Introducing static features can have
some unpleasant effects.  There are already areas where Ruby falls
down slightly on properly handling dynamic changes, for example the
double module inclusion problem where a change to module A to include
module B isn't seen by modules/classes which have already included
module A:
http://eigenclass.org/hiki/The+double+inclusion+problem

It seems to me that this is just an implementation issue, fixing it
would involve a little more internal bookkeeping so that the method
lookup chain of mdules including A when A was changed.

Or the issues involved in handling "re-including" a module which is
already included by an ancestor:
http://talklikeaduck.denhaven2.com/articles/2006/10/09/a-subtle-change-to-mixin-semantics-in-ruby-1-9

Ruby 1.8 ignores the 're-inclusion' at one point 1.9 altered this
behavior, then it went back, I haven't re-built 1.9 in a while so I'm
not sure where the matter lies now.

The 1.8 semantics on module 're-inclusion' always seemed wrong to me,
I'm not sure I fully understand the rationale, I haven't thought it
through but I thnk that it might be wrapped up in a relationship with
the double inclusion problem.

These edge-cases are really small flaws which make Ruby just a little
brittle and less dynamic than it could be.  I think of them like the
stress hardening which happens when you bend a piece of metal back and
forth.  Do it enough and the metal breaks at those hardened points.
Fixing them would be like annealing the metal to get back its original
malleability.

Getting back  to my main point my fear is that implementing the kind
of tinkering with method lookup being proposed will  increase the
number of these edge cases.

> invoke a local method defined in (or inherited by) the caller without
> considering any methods defined further down the class hierarchy.

IMHO, Ruby already has enough syntax, of course that comes from my
Smalltalk background.


> I don't grok parse.y well enough to know whether any syntax I would
> propose would actually result in an unambiguous grammar, but my
> suggestion is that local methods have an @ prefix just like instance
> variables do.  Then the interpreter knows from the name of the method
> that it is to be dispatched differently.
>
> In fact, I might actually go further than this and propose that local
> methods are not actually methods, but are syntactic sugar for lambdas.

Which means that the same thing should be possible with DSL like
metaprogramming and without syntax changes to the language.

>    end
> end
>
> t = Test.new("hello")
> t.greet("world")
>
> In this code, the instance variable @greeter refers to a local function
> that is completely hidden from subclasses and cannot be altered.

Actually, it can since it's an instance variable.

class TestSub < Test
     def initialize
          super
          @greeter = { rand }
      end
end

or even

class TestSub2 < Test
    def set_greeter(&b)
        @greeter = lambda(&b)
    end
end


> Perhaps Ruby 1.9 could take this idiom and add syntax sugar to make the
> definition and invocation of local functions more like the definition
> and invocation of regular methods.

My opinion is that the cure for this 'problem' is worse than the 
disease.


--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

Rick DeNatale wrote:

<snip>

> My opinion is that the cure for this 'problem' is worse than the disease.

I'd settle for a warning when run with -w. This code emits a warning
when run with -w:

# redef.rb
class Foo
    def bar
       puts "hello"
    end

    def bar
       puts "world"
    end
end

redef.rb:6: warning: method redefined; discarding old bar

But this does not:

# redef2.rb
class Foo
    def bar
       puts "hello"
    end
end

class Baz < Foo
    def bar
       puts "world"
    end
end

Is there any way to make the 2nd case emit a warning without major
changes to core Ruby?

Dan

On 6/14/07, Daniel Berger <Daniel.Berger@qwest.com> wrote:
> class Foo
>
>     def bar
>        puts "world"
>     end
> end
>
> Is there any way to make the 2nd case emit a warning without major
> changes to core Ruby?

I'm not sure it should. Overriding a method is perfectly natural,
sometimes you call super as part of that,when you are augmenting the
inherited method, but sometimes you ARE replacing it and that's fine.

 Also what should happen in a case like this:

class Foo
end

class Baz < Foo
  def bar
     puts "world"
  end
end

class Foo
   def bar
       puts "hello"
   end
end

Note that this is a temporal ordering, the code need not appear in the
same file, for example the bar method might be added to the base class
foo in a gem.

To make it more explicit, should Ruby examine all existing
superclasses AND subclasses AND including classes when changing a
class/module?

What about singleton methods and classes?

There be dragons!

There's no magic bullet to warding off changes due to changing
ancestors and descendants, the best one can do is to be aware of what
CAN happen, and have a good set of regression tests to detect
problems.


--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

Daniel DeLorme wrote:
> How is this hidden from subclasses? Or did I miss something?
> 

Sorry, everyone!  As Daniel and others have pointed out my analogy of
non-inheritable local methods to instance variables makes no sense since
instance variables are inheritable.  I'm feeling pretty embarassed by 
that.

However, I do stand by my argument that it would be good to have some
kind of non-inheritable really-private kind of method in Ruby.  And I
think that distinguishing those methods with punctuation in their names
would be an easy way to get there without having to jigger with the
existing method dispatch rules.

  David

Rick DeNatale wrote:
> I'm not sure it should. Overriding a method is perfectly natural,
> sometimes you call super as part of that,when you are augmenting the
> inherited method, but sometimes you ARE replacing it and that's fine.

Sometimes but IMHO rarely. If the parent class has some implementation,
subclasses are more likely to augment/supplement that implementation
rather than discard it completely. IMHO discarding the parent method has
a strong risk of breaking stuff if the implementation of that parent
method ever has to change.

However I have to admit that it could get complicated when method
aliasing is used instead of super, like rails does a lot.

> There be dragons!
> 
> There's no magic bullet to warding off changes due to changing
> ancestors and descendants, the best one can do is to be aware of what
> CAN happen, and have a good set of regression tests to detect
> problems.

It might be tricky in some edge cases but a solution doesn't need to be
a magic bullet in order to be useful. Such a warning would have been
useful to me when I was a newbie and wondering why the module I was
including in a class wasn't overriding the methods I wanted it to.

Daniel

David Flanagan <david@davidflanagan.com> writes:
> kind of non-inheritable really-private kind of method in Ruby.  And I
> think that distinguishing those methods with punctuation in their
> names would be an easy way to get there without having to jigger with
> the existing method dispatch rules.
>
>   David

Maybe I'm confused, which of these are you talking about?

class A
  def m; @f(); end
  def @f; true; end
end

class B < A
end

B.new.m  #=> Error: The mighty class author requires that you
             reimplement @f.

or

class B < A
  def @f; false; end
  #=> Error: The mighty class author forbids you to reimplement @f
end

Steve

On 6/16/07, Steven Lumos <steven@lumos.us> wrote:
> > However, I do stand by my argument that it would be good to have some
>   def m; @f(); end
>
> class B < A
>   def @f; false; end
>   #=> Error: The mighty class author forbids you to reimplement @f
> end
>
> Steve

I think what is meant is:

   #=> Error: The mighty class author forbids you to reimplement @f.

I was thinking about this some more. While I understand the argument
that no instance method should truly be untouchable, it seems to me
that these local-private methods have an even stronger use case in
mixins. For example, right now I'm working on a Package class. It has
lots of attributes. I would like to add a #build method to it,
however, there is a lot of support structure that goes with #build,
such as the inclusion of FileUtils. While I could probably get away
with it, I feel skittish about adding that many more methods to
Package. It's just getting to be too much. So I made a Builder class
and have elected to delegate. That works, but its a bit more work. If
instead I could make those support methods local-private to the
module, than I could just include the module and achieve the same
effect much more easily.

T.

Steven Lumos wrote:
> B.new.m  #=> Error: The mighty class author requires that you
> 
> 

Steve,

I don't intend either of those, really.  My proposal is that the method
@f would be completely local to the defining class.  There is no
inheritance of these hypothetical methods, and so the class hierarchy is
irrelevant.  If you try to invoke @f in a class B that does not define
@f you get a NoMethodError or whatever. And you can define @f in a class
B regardless of what is defined in the superclass, so you'd never get an
error message like your second one.

I think that the intent of the Matz's proposal from 6 months ago was to
acheive something like this, and there seemed to be some consensus that
that would be a good thing.  What I'm adding with my proposal here is an
explicit punctuation prefix that would, I think, make it possible to
implement this kind of local method without altering the method name
resolution algorithm that is currently used for private methods.

  David

On 6/17/07, David Flanagan <david@davidflanagan.com> wrote:
> >
> > Steve
> B regardless of what is defined in the superclass, so you'd never get an
> error message like your second one.
>
> I think that the intent of the Matz's proposal from 6 months ago was to
> acheive something like this, and there seemed to be some consensus that
> that would be a good thing.  What I'm adding with my proposal here is an
> explicit punctuation prefix that would, I think, make it possible to
> implement this kind of local method without altering the method name
> resolution algorithm that is currently used for private methods.

One worry I have about the prefix is how would one include a module as
local-private? Ie.

  module M
    def x; ...; end
  end

  class R
    include_as_local M
  end

Would #include_as_local change the names of M's methods somehow?

T.

David Flanagan <david@davidflanagan.com> writes:
>>
>> Steve
> define @f in a class B regardless of what is defined in the
> superclass, so you'd never get an error message like your second one.

I think I get it now.

class B < A; end
B.new.m  #=> true

class B < A
  def m; @f(); end
end
B.new.m  #=> NoMethodError

class B < A
  def @f; false; end
end
B.new.m  #=> true

class B < A
  def m; @f(); end
  def @f; false; end
end
B.new.m  #=> false

Steve