Forum: Ruby-core Module#prepend and Array#prepend

Matz,
As we discussed at LoneStar, I have the following proposal:

given the following class:

class Person
  def speak(words)
    puts words
  end
end

It should be possible to do the following:

module Exclaimer
  def speak(words)
    super("#{words}!")
  end
end

class Person
  prepend Exclaimer
end

and then have:

Person.new.speak("matz") #=> matz!

Effectively, prepend would prepend a module to the class' ancestor 
chain.
Implementation-wise, it means that all methods on a class are put into 
an
implicit module, or an implicit subclass is created for each new class 
that
can have modules mixed into it (via prepend).

We also discussed aliasing unshift to prepend for Arrays, since
shift/unshift is a source of much confusion.

Yehuda Katz wrote:
> class Person
>   prepend Exclaimer
> end

Is there some way to make the relationship more visually obvious?

class Person > Exclaimer
end

class Person
   self >> Exclaimer
end

Whatever you call it, I like the idea--it beats extending each instance
with the module.

Hi Yehuda and all,

I started burbling away and haven't done any intros.  That's a bit
rude of me!

I've been using Ruby for almost seven years.  In large part it is
Ruby's lispiness that sucked me in---after having used lisp at uni.
For that time I've mostly been lurking, but feel like I half-know what
I'm talking about now!  And so, after a self-imposed exile from ruby-
talk after I was given the shits, I thought I should properly jump
back in.  (I needed to supply the list bot with a surname, so I chose
"because", for reasons which may be apparent to some.)

In that time, I've been bad by having been a bit of a hoarder, but I
will release at least a few thousand of lines of code in the context
of a code distribution system I've been working on for a while (It
plays in a rudimentary way nicely with Rubygems and I'm going to have
to do something with git too...)(HINT: $REMOTE_LOAD_PATH), including
several hundreds of additional methods on the standard classes, some
of which is sugar, many of which are small, a few outright rip-offs
(mostly acknowledged), and some simply inspired by other code (also
mostly acknowledged).  I just need to spend a month or two cleaning it
up...

(NB. Someone harass me please!  Perhaps start by harassing me about
Options (a gorgeous little wrapper for OptionParser) and Attributes,
which creates default attribute values, sets up getters and setters,
setter and getter aliases and integrates with Options optionally for
the defaults.  I need a bit of encouragement...  K?  :-)

Anyway, to the matter at hand...

I don't mind shift/unshift, though I do have aliases Array#first!/
last! for Array#shift/pop in my standard kit as it seems to make more
sense sometimes that way.  Programming Ruby has Array#unshift as
"Prepends object(s) to arr."  Don't fight it huh?

Might this feature also go some ways to addressing my desire for
module-specific behaviours on objects?  However this is, in a sense,
more explicit than I was desiring.

Is it preferable that this be as explicit as Module.append/prepend (I
might add those aliases to Array as well, since I made some methods up
for String by those names.) only, rather than include an implicit
append/prepend by virtue of definitions for methods in the context of
a module?

I don't mind both, since removing a feature by such a means might be
preferable over eval'ing symmetric code.

This also relates to my quizzications over Module.ancestors.  (I note
that Programming Ruby states that it "Returns a list of modules
included in mod (including mod itself)."  So the behaviour of
returning itself as an ancestor is apparently OK!

So, is it too much to have:

Module.mixins;
Module.ancestors/inherits/inherits_from (which only has the super
class chain, but flattened to a list);
Module.influences/behaves_like/friends_and_family (for mixins and
ancestors combined); and
Module.behaviours/method_sources (for the lot, as per ancestors now)?

Or something like that.  (Naming things is hard sometimes.)

I use introspection quite a bit, so this is very much an area of
interest for me.  I've written a number of methods on ObjectSpace and
elsewhere to do somewhat related stuff to this, so as I can automate
discovery of behaviour.  It is slow, but fun.

Anyone for "implements"?


t

On Sun, Aug 30, 2009 at 10:09 PM, Joel VanderWerf
<vjoel@path.berkeley.edu>wrote:

> end
>
> class Person
>  self >> Exclaimer
> end
>

Hmmm... seems potentially a bit visually confusing. I'm happy with 
prepend,
in particular if we also make prepend a method on Array.

-- Yehuda

Hi --

On Mon, 31 Aug 2009, Yehuda Katz wrote:

> Is there some way to make the relationship more visually obvious?
> in particular if we also make prepend a method on Array.
I like the idea of the reverse #include but I'm not sure about the
name. The problem, for me, is that it's *too* array-like. The other
related operations don't have that same spatial/collection feel to
them. Somehow prepend doesn't sound like it's in the same semantic
space as include.

Now I'm supposed to pony up a brilliant alternative :-) include! came
to mind. The "dangerous" aspect is that it occludes the current
version of the method.

   module M
     def x
     end
   end

   class C
     def x
     end

     include M       # doesn't affect instances
     include! M      # blocks this class's same-named methods
   end

Mind you, I've always been intrigued by the idea of ancestors being
a fully read-write array. But the current semantics of include are not
very array-like, so I'm not sure about "prepend".


David

--
David A. Black / Ruby Power and Light, LLC / http://www.rubypal.com
Ruby/Rails training, mentoring, consulting, code-review
Latest book: The Well-Grounded Rubyist (http://www.manning.com/black2)

September Ruby training in NJ has been POSTPONED. Details to follow.

Hi,

2009/8/31 Yehuda Katz <wycats@gmail.com>:
> Matz,
> As we discussed at LoneStar, I have the following proposal:
(snip)
> Effectively, prepend would prepend a module to the class' ancestor chain.


I cannot guess the problem you want to solve.  I know this mail
is just discussion log or reminder, but I'd like you to elaborate
your proposal completely when sending to ruby-core.

Why don't you define subclass explicitly?

  class Person
    def speak(words)
      puts words
    end
  end
  module Exclaim
    def speak(words)
      super("#{ words }!")
    end
  end
  class Exclaimer < Person
    include Exclaim
  end
  Exclaimer.new.speak("matz") #=> matz!

As far as I read your mail, I think this is the correct way.

Do you want to define Person by including Exclaimer?  If so,
how about this?

  class Person
    def speak(words)
      puts adjust_words(words)
    end
  end
  module Exclaimer
    def adjust_words(words)
      "#{ words }!"
    end
  end
  class Person
    include Exclaimer
  end
  Person.new.speak("matz") #=> matz!

On Mon, Aug 31, 2009 at 8:39 AM, Yusuke ENDOH<mame@tsg.ne.jp> wrote:
>  end
>  module Exclaim
>    def speak(words)
>      super("#{ words }!")
>    end
>  end
>  class Exclaimer < Person
>    include Exclaim
>  end
>  Exclaimer.new.speak("matz") #=> matz!

Perhaps I'm mistaken but I think that besides the 'syntax' difference,
Yehuda's intent was that something like this would work:


class Person
    def speak(str)
         str
    end
end

p = Person.new

p.speak("hello") #=> "hello"

module LoudSpeaker
   def speak(str)
         super(str.upcase)
   end
   end
end

class Person
    prepend Voice
end

p.speak("hello") #=> "HELLO"

If this is the intent, it's more than adjusting the ancestor chain of
the class when it prepends a module, but adjusting the ancestor chain
of any present and future instances of that class.

I think that this is the intent because of something Yehuda said in
the post which started this thread: "Effectively, prepend would
prepend a module to the class' ancestor chain. Implementation-wise, it
means that all methods on a class are put into an implicit module, or
an implicit subclass is created for each new class that can have
modules mixed into it (via prepend)."

Although hidden, or maybe transparent, might be a better word here than 
hidden.

I think there are several interesting discussion topics about this idea.

First, is it useful enough to carry it's weight.

Second, if it is what's the 'right' syntax.

Third, what's the impact on implementation and potential for edge cases.

I think that the Yehuda's implementation suggestions are that either:

   The existing hash in class objects which maps method selectors to
methods would actually be empty, or at least not consulted directly
during method lookup and the class would start out with klass pointing
to something akin to a module proxy, really a class proxy pointing to
the instance method hash of the class.  The prepend method would
insert a module proxy for the module being prepended at the head of
the chain started by the klass field of the class. The current include
method would search the klass chain for the special class proxy and
insert the module proxy just after that.

or

 A hidden subclass with no instance methods be created for each class
which could "have modules mixed into it (via prepend)" and  instances
of the class would actually be directly instances of this hidden class
instead of the class itself.  Prepend would insert a module proxy
between this hidden class and the actual class.  I think that this
would need to be the case for every class since I don't know how you
would predict if prepend would be sent to the class in future.

These two alternatives seem to me to be only subtly different, I
suspect that the first one is cleaner, but I don't know.

Some questions that occur to me:

1) Since prepend is really a variation on include, if you can prepend
a module to a class, why can't you prepend a module to a  module

2) Is there a similar variant for extend?

3) Which is related to the question of should you be able to prepend
to a singleton-class?

4) Are there edge-cases related to handing super?  Right now unless
it's changed (again) in Ruby 1.9, include takes pains not to have a
proxy to the same module show up twice in an ancestor chain because of
the way super method lookup works by searching first from the
beginning of self's klass chain for the current method, then searching
again for the next one, which can cause unexpected results if super is
used in a method invoked itself from super earlier in the chain.


--
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale
WWR: http://www.workingwithrails.com/person/9021-rick-denatale
LinkedIn: http://www.linkedin.com/in/rickdenatale

2009/8/31 Rick DeNatale <rick.denatale@gmail.com>:
>    def speak(str)
>         super(str.upcase)
>   end
>   end
> end
>
> class Person
>    prepend Voice
> end
>
> p.speak("hello") #=> "HELLO"


Even if it is Yehuda's intent, I cannot see yet when it is necessary.

Indeed, we can define and refine classes dynamically, but I don't think
that the feature's abuse is recommended.  I think we should consider
more actual circumstances than Person#speak to determine whether it is
surely needed.

2009/9/1 Yusuke ENDOH <mame@tsg.ne.jp>:
>> class Person
>>   def speak(str)
>
>
> Even if it is Yehuda's intent, I cannot see yet when it is necessary.
>
> Indeed, we can define and refine classes dynamically, but I don't think
> that the feature's abuse is recommended.  I think we should consider
> more actual circumstances than Person#speak to determine whether it is
> surely needed.


Sorry I forgot to say thank you for giving your reasoning.
Thank you.

On Mon, Aug 31, 2009 at 4:09 AM, David A. Black <dblack@rubypal.com> 
wrote:

>>            class Person
>>  self >> Exclaimer
> related operations don't have that same spatial/collection feel to
>  end
> a fully read-write array. But the current semantics of include are not
> very array-like, so I'm not sure about "prepend".
>

We asked Matz about that, and he felt it was a bit *too* much. One major
problem is that it would affect classes and not just modules, with more
dubious semantics.

Yusuke,

The reason that something like #prepend would be needed is so that
ruby can provide a good way to do AOP. Your example is not exactly a
1-1 mapping onto the prepend feature. Instead, you would need to do
this:

>  class Person # Empty class that only includes the module
>    include Speak
>  end
>   class Exclaimer < Person
>    include Exclaim
>  end


If your classes are NOT empty and only include modules, then it is
impossible to include a module "under" methods defined directly on the
class. This causes pain when you are using a ruby library that does
not assume that you will be extending classes. Rails "solved" this
problem using alias_method_chain, but this method can cause confusion.

I hope I explained my thoughts clearly.

Regards,
Carl Lerche

Indeed,
The existing in-ruby solutions only work if the original author takes 
pains
to supports extensions. In Rails 3, we do this. However, this is a large
burden on Ruby programmers, and we could provide an in-ruby solution
(prepend) that guarantees that any class can be extended.

Consider the following (extremely simplified) scenario:

class ActionController
  def render(template)
    find_template(name).render(self)
  end

  def find_template(name)
    return Template.new(name)
  end
end

Again, this is an extremely simplified example. Now what if I want to 
add
support for layouts to ActionController? Because render is defined on 
the
class itself, I cannot do so. As the creator of ActionController, I can 
do:

module Renderer
  def render(template)
    find_template(name).render(self)
  end

  def find_template(name)
    return Template.new(name)
  end
end

class ActionController
  include Renderer
end

And then to extend, I can do:

module Layouts
  def render(template, layout)
    find_template(layout) { super }
  end
end

class ActionController
  include Layouts
end

Because the original creator of ActionController put the implementation 
in a
module, another Ruby programmer can extend it. But frequently, Ruby code
does not do this. I would like to be able to do:

module Layouts
  def render(template, layout)
    find_template(layout) { super }
  end
end

class ActionController
  prepend Layouts
end

which would have the same effect as when the implementation was 
explicitly
inserted into a module, but it'll work on *any* class, not just ones
specially designed.

-- Yehuda

On Mon, Aug 31, 2009 at 4:09 AM, David A. Black<dblack@rubypal.com> 
wrote:
>>            class Person
>>  self >> Exclaimer
> them. Somehow prepend doesn't sound like it's in the same semantic
>
> very array-like, so I'm not sure about "prepend".
Include is gravy around append_features, so prepend_features feels
natural to me.

Also intriguing (and tangential) would be allowing
ancestors.unshift/push directly.

jeremy

Hi,

2009/9/1 Carl Lerche <clerche@engineyard.com>:
>>  end
>>  end
>
>
> If your classes are NOT empty and only include modules, then it is
> impossible to include a module "under" methods defined directly on the
> class.


Do you talk about the following behavior as a problem?

  class Person
    def speak(words)
      puts words
    end
  end
  module Exclaim
    def speak(words)
      super("#{ words }!")
    end
  end
  class Person
    include Exclaim
  end
  Person.new.speak("matz") #=> "matz", not "matz!"


> This causes pain when you are using a ruby library that does not
> assume that you will be extending classes. Rails "solved" this problem using
> alias_method_chain, but this method can cause confusion.


Thank you, now it's starting to make sense.

However, I think that module inclusion is not such a patching tool;
open class is.  In the above case, we can achieve our intent by
redefining `speak' directly:

  class Person
    def speak(words)
      puts "#{ words }!"
    end
  end
  Person.new.speak("matz") #=> "matz!"

When `speak' has a more complex definition, this way is a pain because
we cannot use super.  It is possible to solve the pain by preserving
the original definition of speak as a alias, for example, old_speak.
But I admit the solution is ugly.

Then, I guess that what we really need is a method wrapping feature:

  class Person
    refine_method(:speak) do |old_method, words|
      # old_method is a Method of original definition
      old_method.call("#{ words }!")
    end
  end
  Person.new.speak("matz") #=> "matz!"

Hi,

In message "Re: [ruby-core:25228] Re: Module#prepend and Array#prepend"
    on Tue, 1 Sep 2009 03:24:30 +0900, Yusuke ENDOH <mame@tsg.ne.jp> 
writes:

|However, I think that module inclusion is not such a patching tool;
|open class is.

|Then, I guess that what we really need is a method wrapping feature:
|
|  class Person
|    refine_method(:speak) do |old_method, words|
|      # old_method is a Method of original definition
|      old_method.call("#{ words }!")
|    end
|  end
|  Person.new.speak("matz") #=> "matz!"

Can you elaborate why you think module inclusion is not such a
patching tool, and the above "refine_method" is sought solution than
Module#prepend?

              matz.

2009/9/1 Yukihiro Matsumoto <matz@ruby-lang.org>:
> |  class Person
> |    refine_method(:speak) do |old_method, words|
> |      # old_method is a Method of original definition
> |      old_method.call("#{ words }!")
> |    end
> |  end
> |  Person.new.speak("matz") #=> "matz!"
>
> Can you elaborate why you think module inclusion is not such a
> patching tool,

I have understood that module mix-in is just adding a concern because,
as far as I know, the built-in modules (i.e., Comparable, Enumerable,
etc.) merely add methods and do not modify existing methods of original
class.


> and the above "refine_method" is sought solution than
> Module#prepend?

I thought:

- we should use open class or reflection to redefine an existing method,

- but it is burdensome to redefine a method with using its original
  behavior,

- then, the method to do it may be helpful.

Hi --

2009/9/1 Yehuda Katz <wycats@gmail.com>:
> The existing in-ruby solutions only work if the original author takes pains
> to supports extensions. In Rails 3, we do this. However, this is a large
> burden on Ruby programmers, and we could provide an in-ruby solution
> (prepend) that guarantees that any class can be extended.

I understood very well.  Thank you!

I think that module has been a mechanism to add a concern to a class,
not to modify a class itself.  I understand that your proposal is to
allow module to change a class.

It seems to me to be a big and philosophical change.  I wait matz's
idea.

Hi,

In message "Re: [ruby-core:25230] Re: Module#prepend and Array#prepend"
    on Tue, 1 Sep 2009 03:51:35 +0900, Yusuke ENDOH <mame@tsg.ne.jp> 
writes:

|It seems to me to be a big and philosophical change.  I wait matz's
|idea.

I am not happiest by this idea, since I am a true believer of method
combination a la CLOS.  But I admit it solves the important problem far
easier than method combination at the same time.

As a result, I am positive, but I still need a deep think.

              matz.

Hi,

2009/9/1 Yukihiro Matsumoto <matz@ruby-lang.org>:
> As a result, I am positive, but I still need a deep think.
I see your agonizing choice.

But Module#prepend seems to me to be ad-hoc.  I hope this is a good
chance to rethink Ruby's module mechanism, including classbox :-)

On Mon, Aug 31, 2009 at 2:24 PM, Yusuke ENDOH<mame@tsg.ne.jp> wrote:
> Hi,
>
> 2009/9/1 Carl Lerche <clerche@engineyard.com>:

>
>> This causes pain when you are using a ruby library that does not
>> assume that you will be extending classes. Rails "solved" this problem using
>> alias_method_chain, but this method can cause confusion.
>
...
>
> Then, I guess that what we really need is a method wrapping feature:
>
>  class Person
>    refine_method(:speak) do |old_method, words|
>      # old_method is a Method of original definition
>      old_method.call("#{ words }!")
>    end
>  end
>  Person.new.speak("matz") #=> "matz!"

Yusuke-san,

Are you familiar with alias_method_chain in Rails which Carl mentioned.

This is a common rails extension idiom which you'd use like this

module Exclaimer
   def speak_with_exclaimer(words)
        speak_with_exclaimer("#{words}!")
   end

   def.self.included(base)
       base.alias_method_chain :speak, :exclaimer
   end
end

and alias_method_chain aliases the existing speak method to
send_without_exclaimer, and then the speak_with_exclaimer method to
speak

This allows independently developed class extension modules to be
composed, but it can be a bit confusing at times, particularly when
debugging because of the aliasing.


--
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale
WWR: http://www.workingwithrails.com/person/9021-rick-denatale
LinkedIn: http://www.linkedin.com/in/rickdenatale

Hi --

On Tue, 1 Sep 2009, Yehuda Katz wrote:

>             <vjoel@path.berkeley.edu>
>             class Person > Exclaimer
>             Array.
> version of the method.
>    include M       # doesn't affect instances
> dubious semantics.
True.

One question about the prepend idea:

   module M
     def x; "M's x"; end
   end

   module N
     def x; "N's x"; end
     prepend M   # or whatever it's called :-)
   end

   class C
     include N
   end

Would N have sacrificed itself, so to speak, to M, so that:

   C.new.x

would print "M's x"?


David

--
David A. Black / Ruby Power and Light, LLC / http://www.rubypal.com
Ruby/Rails training, mentoring, consulting, code-review
Latest book: The Well-Grounded Rubyist (http://www.manning.com/black2)

September Ruby training in NJ has been POSTPONED. Details to follow.

Hi --

On Tue, 1 Sep 2009, Jeremy Kemper wrote:

>>>      Yehuda Katz wrote:
>>> class Person
>> related operations don't have that same spatial/collection feel to
>>  end
>> a fully read-write array. But the current semantics of include are not
>> very array-like, so I'm not sure about "prepend".
>
>
> Include is gravy around append_features, so prepend_features feels
> natural to me.

I could imagine modules having a prepend_features hook, but I'd still
prefer something that lined up better with include for the actual
operation.

> Also intriguing (and tangential) would be allowing
> ancestors.unshift/push directly.

That was along the lines I was thinking with the read-write array
thing... though as Yehuda points out, it would be a bit weird to move
classes around. But maybe if it was strictly linear with no
leap-frogging.


David

--
David A. Black / Ruby Power and Light, LLC / http://www.rubypal.com
Ruby/Rails training, mentoring, consulting, code-review
Latest book: The Well-Grounded Rubyist (http://www.manning.com/black2)

September Ruby training in NJ has been POSTPONED. Details to follow.

On Tue, Sep 1, 2009 at 5:37 PM, David A. Black <dblack@rubypal.com> 
wrote:

>>
>>                       end
>>            end
>> related operations don't have that same spatial/collection feel to
>>  end
>> a fully read-write array. But the current semantics of include are not
> One question about the prepend idea:
>  class C
>    include N
>  end
>
> Would N have sacrificed itself, so to speak, to M, so that:
>
>  C.new.x
>
> would print "M's x"?


Yep. But keep in mind that M could call super, which would go to N's x.

-- Yehuda

When module method redefines original instance method, it's impossible 
to invoke it after and it's accessible only through super?