Forum: Ruby-core Refinements and nested methods

Issue #4085 has been updated by headius (Charles Nutter).


Escaping the "should we or shouldn't we" question for a bit, I thought 
of an alternative implementation, building off ko1's idea.

ko1's suggestion, as I understand it, was to add a flag to the method 
table (or method entry) of a refined class/method as a trigger for the 
call site to search refinements. While writing my blog post, I started 
to type the sentence "the methods defined in the refinement do not 
actually go on the class in question", and then I realized: why not?

Currently, Ruby implementations structure the method table as a simple 
map from names to method bodies. If instead the method table was a map 
from names to collections of methods, we could use that to choose the 
appropriate method for a given context.

So, for code like this:

module X
  refine String do
    def upcase; downcase; end
  end
end

String's method table would contain an entry like this:

{:upcase => {
  :default => <builtin upcase>,
  X => <upcase patched>}
}

Method lookup would then proceed as normal in all situations. The result 
of lookup would be a table mapping refinements to methods with a default 
entry if the method is defined directly on String.

After lookup, call sites would know there's potentially refinements 
active for the given method. The calling scope (or parent scopes) would 
have references to individual refinements, and if there were an entry 
for one of them it would be used.

This still requires access to the caller scope, of course, to understand 
what refinements are active. However, because refinement changes would 
invalidate the String class directly (since they actually modify the 
method table), the method (refined or otherwise) could be cached as 
normal. The caller's scope never changes (statically determined at 
compile time), so it does not participate in invalidation.

This also works for refinements added to classes after the initial run 
through. If we cache the default downcase method from String, and then 
the refinement is updated to add downcase, we would see that as an 
invalidation event for String's method table. Future calls would then 
re-cache and pick up the change.

This also feels a bit more OO-friendly to me. Rather than storing 
patches on separate structures sprinkled around memory, we store the 
patches directly on the refined class, only using the module containing 
the refinements as a key. The methods *do* live on String, but depending 
on the *namespace* they're looked up from we *select* the appropriate 
implementation. It's basically just double-dispatch at that point, with 
the selector being the calling context.

It also makes available an interesting possibility for #method and 
friends: return all methods. So...

using X

String.instance_methods(:upcase) # => {:default => <builtin upcase>, X 
=> <new upcase>}

Note that this is "instance_methods", plural, to avoid breaking 
instance_method and to make it explicit that we're asking for all 
implementations of a given method. This allows accessing the original 
method even if refinements are active, and still also allows searching 
for the refined method active in the current scope.

I admit I am a bit reluctant to suggest this, because I still have 
concerns about the feature itself. But it would be possible for call 
sites to only need a reference to their calling scope (determined at 
parse time) to implement dynamic refinements without severe impact to 
normal code. Dynamic refinements, as in module_eval, would work by 
simply invalidating the call sites they contain. This could be done 
actively, walking all call sites and resetting them. This could also be 
done by invalidating the classes refined. An example in pseudo-code:

def X.module_eval_refined(&block)
  unless block.using? self
    refinements.each_key {|cls| cls.touch } # invalidate all refined 
classes
    block.using(self)
  end
  module_eval &block
end

This is obviously not a thread-safe mechanism. An alternative that 
invalidates the block's call sites (this would require more work and be 
more expensive at invalidation time, but less globally-damaging):

def X.module_eval_refined(&block)
  unless block.using? self
    block.invalidate
    block.using(self)
  end
  module_eval &block
end

Proc#using would either mutate the block's already-present scope 
(permanently adding the refinement) or duplicate the block and its scope 
and tweak it (more expensive, of course).

---

In any case, I would really like more time for this dialog to continue. 
If we push refinements into Ruby in their current form, we're not giving 
adequate time to flesh out the edge cases. If we push a partial 
implementation now, we may be making a future implementation harder and 
we would not be protecting ourselves from mistakes. I want to work with 
you to find a definition and implementation of refinements that meets 
requirements without punishing future Rubyists.

I also must apologize for not joining the dialog sooner. This bug was 
filed in 2010, and the current refinements implementation was pushed to 
master a few months ago. We should have started discussing a long time 
ago.
----------------------------------------
Feature #4085: Refinements and nested methods
https://bugs.ruby-lang.org/issues/4085#change-33640

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal
Assignee: matz (Yukihiro Matsumoto)
Category: core
Target version: 2.0.0


=begin
 As I said at RubyConf 2010, I'd like to propose a new features called
 "Refinements."

 Refinements are similar to Classboxes.  However, Refinements doesn't
 support local rebinding as mentioned later.  In this sense,
 Refinements might be more similar to selector namespaces, but I'm not
 sure because I have never seen any implementation of selector
 namespaces.

 In Refinements, a Ruby module is used as a namespace (or classbox) for
 class extensions.  Such class extensions are called refinements.  For
 example, the following module refines Fixnum.

   module MathN
     refine Fixnum do
       def /(other) quo(other) end
     end
   end

 Module#refine(klass) takes one argument, which is a class to be
 extended.  Module#refine also takes a block, where additional or
 overriding methods of klass can be defined.  In this example, MathN
 refines Fixnum so that 1 / 2 returns a rational number (1/2) instead
 of an integer 0.

 This refinement can be enabled by the method using.

   class Foo
     using MathN

     def foo
       p 1 / 2
     end
   end

   f = Foo.new
   f.foo #=> (1/2)
   p 1 / 2

 In this example, the refinement in MathN is enabled in the definition
 of Foo.  The effective scope of the refinement is the innermost class,
 module, or method where using is called; however the refinement is not
 enabled before the call of using.  If there is no such class, module,
 or method, then the effective scope is the file where using is called.
 Note that refinements are pseudo-lexically scoped.  For example,
 foo.baz prints not "FooExt#bar" but "Foo#bar" in the following code:

   class Foo
     def bar
       puts "Foo#bar"
     end

     def baz
       bar
     end
   end

   module FooExt
     refine Foo do
       def bar
         puts "FooExt#bar"
       end
     end
   end

   module Quux
     using FooExt

     foo = Foo.new
     foo.bar  # => FooExt#bar
     foo.baz  # => Foo#bar
   end

 Refinements are also enabled in reopened definitions of classes using
 refinements and definitions of their subclasses, so they are
 *pseudo*-lexically scoped.

   class Foo
     using MathN
   end

   class Foo
     # MathN is enabled in a reopened definition.
     p 1 / 2  #=> (1/2)
   end

   class Bar < Foo
     # MathN is enabled in a subclass definition.
     p 1 / 2  #=> (1/2)
   end

 If a module or class is using refinements, they are enabled in
 module_eval, class_eval, and instance_eval if the receiver is the
 class or module, or an instance of the class.

   module A
     using MathN
   end
   class B
     using MathN
   end
   MathN.module_eval do
     p 1 / 2  #=> (1/2)
   end
   A.module_eval do
     p 1 / 2  #=> (1/2)
   end
   B.class_eval do
     p 1 / 2  #=> (1/2)
   end
   B.new.instance_eval do
     p 1 / 2  #=> (1/2)
   end

 Besides refinements, I'd like to propose new behavior of nested 
methods.
 Currently, the scope of a nested method is not closed in the outer 
method.

   def foo
     def bar
       puts "bar"
     end
     bar
   end
   foo  #=> bar
   bar  #=> bar

 In Ruby, there are no functions, but only methods.  So there are no
 right places where nested methods are defined.  However, if
 refinements are introduced, a refinement enabled only in the outer
 method would be the right place.  For example, the above code is
 almost equivalent to the following code:

   def foo
     klass = self.class
     m = Module.new {
       refine klass do
         def bar
           puts "bar"
         end
       end
     }
     using m
     bar
   end
   foo  #=> bar
   bar  #=> NoMethodError

 The attached patch is based on SVN trunk r29837.
=end

On 22.11.2012 17:40, headius (Charles Nutter) wrote:

> After lookup, call sites would know there's potentially refinements active for 
the given method. The calling scope (or parent scopes) would have references to 
individual refinements, and if there were an entry for one of them it would be 
used.
>
> This still requires access to the caller scope, of course, to understand what 
refinements are active. However, because refinement changes would invalidate the 
String class directly (since they actually modify the method table), the method 
(refined or otherwise) could be cached as normal. The caller's scope never changes 
(statically determined at compile time), so it does not participate in 
invalidation.
>
> This also works for refinements added to classes after the initial run through. 
If we cache the default downcase method from String, and then the refinement is 
updated to add downcase, we would see that as an invalidation event for String's 
method table. Future calls would then re-cache and pick up the change.
>
> This also feels a bit more OO-friendly to me. Rather than storing patches on 
separate structures sprinkled around memory, we store the patches directly on the 
refined class, only using the module containing the refinements as a key. The 
methods *do* live on String, but depending on the *namespace* they're looked up 
from we *select* the appropriate implementation. It's basically just 
double-dispatch at that point, with the selector being the calling context.

This (together with Module.prepend) is reminding me a bit of AspectJ's
pointcuts. Which in turn leads me to think that we are missing something
here:
We don't know and cannot know in advance which kind of scopes the
developer will need to apply his patches.

We have many different ideas flying around how to determine the scope of
the refinement.

a) Local only? Maybe even constrained inside a block?

Good for builder DSLs or the like where you basically want to extend
core objects to make it look more like written sentences than code.

b) Class inheritance?

E.g. If you want to provide some nice class configuration syntax

c) Module namespace?

If you like to use some convenience methods throughout your project
without creating conflicts with extensions that libraries might use in
their own code

d) Stack-down X frames

Black Magic: Patch some behavior inside a single method by wrapping it

e) Thread local

More black magic: Fix some broken interaction between library code.
Stub any kind of method out temporarily without breaking other things in
multi-threaded environments.



So what I am saying is that we don't just need a way to define
refinement namespaces. We also need to let the programmer define where
and when those namespaces get applied. And we need the common cases to
be fast. The madness-driven ones (d and e) can be slow, but can only be
allowed to be slow at those callsites that are affected, not globally.

So I would suggest not providing *any* inheritance at all. Refinements
scopes must be activated in every single module (or possibly even
method) that they should be applied to. They shouldn't even apply to
methods overriding another method when the super-method is
refinement-scoped. If you want to apply them in many places at once you
can do so via metaprogramming.

   module FooExt
     refine String do
       def downcase
         upcase
       end
     end
   end



case a)

   class ClassA
     def bar; end

     # apply to all methods when no block is passed
     using(FooExt)

     def baz; end

     # both .bar and .baz are refined now.
     # we can apply them retroactively!
     # this is important for monkey-patching
   end

   class ClassB
     def foo
       "x".downcase => "x"
       using_refinements(FooExt) do
         "x".downcase => "X"
       end
     end
   end

   class ClassC
     def bar
       "x".downcase
     end
     def baz
       "x".downcase
     end
   end

   o = ClassC.new

   o.bar # => "x"
   o.send(:using, FooExt, :on => :bar)
   o.bar # => "X"
   o.baz # => "x"

   # acts as instance_eval with refinements
   Object.new.using_refinements(FooExt) do
     "x".downcase # => "X"
   end


case b)

   class MyModel < ActiveRecord::Base; end
   class SubModel < MyModel; end
   class OtherModel < ActiveRecord::Base; end

   ActiceRecord::Base.descendants.each do |c|
     c.send(:using, FooExt)
   end

case c)

   # assume this traverses the constants downwards
   MyApplicationNamespace.recursive_submodules.each do |mod|
     mod.send(:using, FooExt)
   end

   # only modify callsites for a single method
   Bar.instance_method(:test).using(FooExt)

case d)
case e)

   # applies refinement to callsites in method :bar in MyClass
   # but only if the guard condition is true
   # otherwise the unrefined method is used
   MyClass.send(:using, FooExt, :on => :bar, :if => lambda do
      Thread[:use_string_patches?]
   end)

   # applies FooExt a dynamic refinement scope
   # to *all* String.downcase callsites throughout the application
   FooExt.send(:use_everywhere) do
      Thread[:use_string_patches?] && caller[1]["<stack match here>"]
   end



I think this should demonstrate the power of letting the programmer
decide how refinement scopes are determined instead of having the
language dictate a fixed lookup strategy.

Cases d) and e) are just for demonstration and don't have to be taken
seriously!


But the metaprogramming issues with __send__, respond_to? and
Symbol.to_proc would still remain.