Forum: Ruby-core Refinements and nested methods

Issue #4085 has been updated by headius (Charles Nutter).


Anonymous wrote:
>  From usability perspective, all retrospective methods e.g. respond_to?
>  methods etc. should reflect refinements. But for 2.0, I don't make
>  them check refinements, because of performance and complexity.  It
>  should be issue of future version.

respond_to? brings up a really good point: there's lots of methods that 
might need special refinement care...but I think it makes things more 
confusing rather than less.

Let's take the respond_to? example. You want respond_to? to reflect 
refinements. That seems reasonable on the surface, even though 
respond_to? is not refined itself. The first peculiarity there is that 
you could no longer wrap respond_to?, respond_to_missing?, method, 
instance_method, and so on, because wrapping them would break their 
visibility to refinements (due to the intervening unrefined Ruby frame). 
We'd be reducing the metaprogrammability of many, many core methods.

And then there's methods that *call* respond_to? (or coercion methods 
like to_s). Example:

class X; end

module M1
  refine X
    def to_path; '/tmp'; end
  end
end

using M1

File.open(X.new) # ????

File.open checks respond_to?(:to_path) and if that succeeds it calls 
to_path. But the above code will never work unless File.open is also 
made refinement-aware.

I think this is a pretty significant problem, and it shows how much more 
limited refinements will actually be. The bottom line is that making any 
core methods reflect refinements may be *more* confusing, because only 
direct invocations will work...not wrapped invocations, called-method 
invocations, or double-dispatched invocations.

Again it may be good to look at C# extension methods, which are not 
reflected: 
http://stackoverflow.com/questions/299515/c-sharp-...

More and more I see the only value of refinements as providing extension 
methods for specific classes in specific scopes. If we want reflective 
access, we should provide something like C# that allows getting the 
available refined methods and a mechanism that returns the current 
active refinements. Users can figure out what they need from there 
without special-casing a whole bunch of core methods.

class Module
  def refined_methods ...
  def refined_instance_methods ...
end

module Kernel
  def active_refinements ...
end

More coming...
----------------------------------------
Feature #4085: Refinements and nested methods
https://bugs.ruby-lang.org/issues/4085#change-34236

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal
Assignee: matz (Yukihiro Matsumoto)
Category: core
Target version: 2.0.0


=begin
 As I said at RubyConf 2010, I'd like to propose a new features called
 "Refinements."

 Refinements are similar to Classboxes.  However, Refinements doesn't
 support local rebinding as mentioned later.  In this sense,
 Refinements might be more similar to selector namespaces, but I'm not
 sure because I have never seen any implementation of selector
 namespaces.

 In Refinements, a Ruby module is used as a namespace (or classbox) for
 class extensions.  Such class extensions are called refinements.  For
 example, the following module refines Fixnum.

   module MathN
     refine Fixnum do
       def /(other) quo(other) end
     end
   end

 Module#refine(klass) takes one argument, which is a class to be
 extended.  Module#refine also takes a block, where additional or
 overriding methods of klass can be defined.  In this example, MathN
 refines Fixnum so that 1 / 2 returns a rational number (1/2) instead
 of an integer 0.

 This refinement can be enabled by the method using.

   class Foo
     using MathN

     def foo
       p 1 / 2
     end
   end

   f = Foo.new
   f.foo #=> (1/2)
   p 1 / 2

 In this example, the refinement in MathN is enabled in the definition
 of Foo.  The effective scope of the refinement is the innermost class,
 module, or method where using is called; however the refinement is not
 enabled before the call of using.  If there is no such class, module,
 or method, then the effective scope is the file where using is called.
 Note that refinements are pseudo-lexically scoped.  For example,
 foo.baz prints not "FooExt#bar" but "Foo#bar" in the following code:

   class Foo
     def bar
       puts "Foo#bar"
     end

     def baz
       bar
     end
   end

   module FooExt
     refine Foo do
       def bar
         puts "FooExt#bar"
       end
     end
   end

   module Quux
     using FooExt

     foo = Foo.new
     foo.bar  # => FooExt#bar
     foo.baz  # => Foo#bar
   end

 Refinements are also enabled in reopened definitions of classes using
 refinements and definitions of their subclasses, so they are
 *pseudo*-lexically scoped.

   class Foo
     using MathN
   end

   class Foo
     # MathN is enabled in a reopened definition.
     p 1 / 2  #=> (1/2)
   end

   class Bar < Foo
     # MathN is enabled in a subclass definition.
     p 1 / 2  #=> (1/2)
   end

 If a module or class is using refinements, they are enabled in
 module_eval, class_eval, and instance_eval if the receiver is the
 class or module, or an instance of the class.

   module A
     using MathN
   end
   class B
     using MathN
   end
   MathN.module_eval do
     p 1 / 2  #=> (1/2)
   end
   A.module_eval do
     p 1 / 2  #=> (1/2)
   end
   B.class_eval do
     p 1 / 2  #=> (1/2)
   end
   B.new.instance_eval do
     p 1 / 2  #=> (1/2)
   end

 Besides refinements, I'd like to propose new behavior of nested 
methods.
 Currently, the scope of a nested method is not closed in the outer 
method.

   def foo
     def bar
       puts "bar"
     end
     bar
   end
   foo  #=> bar
   bar  #=> bar

 In Ruby, there are no functions, but only methods.  So there are no
 right places where nested methods are defined.  However, if
 refinements are introduced, a refinement enabled only in the outer
 method would be the right place.  For example, the above code is
 almost equivalent to the following code:

   def foo
     klass = self.class
     m = Module.new {
       refine klass do
         def bar
           puts "bar"
         end
       end
     }
     using m
     bar
   end
   foo  #=> bar
   bar  #=> NoMethodError

 The attached patch is based on SVN trunk r29837.
=end

On 30.11.2012 20:41, headius (Charles Nutter) wrote:
>      def to_path; '/tmp'; end
>    end
> end
>
> using M1
>
> File.open(X.new) # ????
>
> File.open checks respond_to?(:to_path) and if that succeeds it calls to_path. 
But the above code will never work unless File.open is also made refinement-aware.
>
> I think this is a pretty significant problem, and it shows how much more limited 
refinements will actually be. The bottom line is that making any core methods 
reflect refinements may be *more* confusing, because only direct invocations will 
work...not wrapped invocations, called-method invocations, or double-dispatched 
invocations.

The common problem of all these methods is that they happen down-stack.
The obvious solution that problem would be the option to make
refinements optionally apply on a thread-scope too, but the performance
implications obviously are horrible as they would could impact any
callsite anywhere in the whole application.

So I think the proper solution is to provide a much much more low-level
metaprogramming alternative: The modification of individual, selected
callsites and methods. I.e. Allow metaprogramming on the AST itself.

Once we can do that we can also inspect the stack - or as potentially
more performant alternative as it wouldn't need to reify the whole stack
- examine thread-local variables.

Modifying the AST obviates the problem of inheritance as the programmer
has precise control of which method or callsite gets modified where.

To express some commonly thrown-around refinement examples as
AST-transforms:

(Concepts shamelessly stolen from Java's MethodHandle/invokedynamic and
AspectJ)


case a) Traditional monkeypatching

    class A
      def foo
        puts "in A"
        raise "woops"
      end
    end

    class B < A
      def foo
        super
        puts "after exception!"
      end
    end

    Ruby::AST.methods(A, :foo).enter do |method_name, *args|
      puts "inserted in #{self.name}" # we're inside the method here
    end

    A.new.foo
    # inserted in A
    # in A
    # Exception: woops

    # match super call
    sites = Ruby::Ast.methods(B, :foo).callsites(A, :foo)
    # match existing AST callsites and those defined in the future
    transform = sites.transforms(:existing, :future)

    # add a new transform
    sites.wrap do |source_self,target_self,target_as_proc, *args, 
&block|
      nil # don't execute target method
    end

    B.new.foo
    # after exception!


case b) Simple, scoped Intercept

    class C
      def m1
        "Foo".downcase
      end
    end

    # match methods in C, then select callsites inside those methods
    sites = Ruby::AST.methods(C, [:m1,:m2]).callsites(String, :downcase)

    # don't modify existing code
    sites.transforms(:future).after do |target, result, *args|
      result + "x"
    end

    class C
      def m2
        "Foo".downcase
      end

      def m3
        "Foo".downcase
      end
    end


    C.new.m1 # => foo
    C.new.m2 # => foox
    C.new.m3 # => foo



case c) advanced example, down-stack "refinement"

    f = future_transforms = []

    # not scoped to specific methods here!
    sites = Ruby::AST.callsites(String, :dasherize)
    transform = sites.transforms(:future, :existing)
    transform = transform.guard{Thread.token?(:refine_dasherize)}


    # modifies callsites in the whole application
    # megamorphic ones will suffer a performance loss from a typecheck.
    # optimized, monomorphic callsites will only
    #  suffer from the thread-local variable check
    # since it's not volatile it can be hoisted in loops
    f << transform.wrap do |target,target_method_as_proc, *args, &block|
       if target_method_as_proc != nil
          # ok, target already exists, let's call that
          target_method_as_proc.call(*args, &block)
       else
          # roll our own implementation
          target.gsub(%r_/, '-')
       end
    end

    sites = Ruby::AST.callsites(String, :send)
    transform = sites.transforms(:future, :existing)
    transform = transform.thread_guard(:refine_dasherize)

    # same performance impact as above
    f << transform.wrap do |target, target_method_as_proc, *args, 
&block|
      if args[0] == :dasherize
        # invoke directly -> this is a normal callsite
        # -> gets wrapped by previous transform
        target.dasherize
      else
        target_method_as_proc.call(*args,&block)
      end
    end

    # above deals with .send()
    # similar could be achieved with Module.prepend
    # but callsite modification is invisible to the stack
    # and to the module hierarchy
    # it's a matter of picking the right tools for the right job




    transforms = Ruby::AST.methods(D).transforms(:future)

    # enable refinement for all methods in ClassB and downstack
    transforms.enter do |method_name, *args, &block|
      # thread tokens should be a MultiSet to allow reentrancy
      Thread.add_token(:refine_dasherize)
    end
    transforms.exit do |method_name, return_value, *args, &block|
      Thread.remove_token(:refine_dasherize)
      return_value
    end

    class D
      def m1
        "a_b".dasherize
      end

      def m2
        "a_b".send(:dasherize)
      end
    end

    # transforms(:future) -> code gets patched
    require "some_gem"


    future_transforms.each{|t| t.suspend}

    # code doesn't get patched anymore
    require "other_gem"


Is this complex? Yes
Is this verbose? Yes
Can you shoot yourself in the foot with it? Yes

This is intentional. It's a low-level API meant to provide as much
freedom to the programmer as possible. Higher-level abstractions (such
as refinements) can be built ontop of it. These abstractions could cover
many of the conflicting use-cases we have discussed in this thread.

Inheritance? A matter of module/method selectors
File scope? A matter of existing/future transform selectiveness
Super calls? You have full control over them, they're just callsites
OOP integration? Only when you want to go the extra mile

Actually, if we combine AST modification and Module.prepend we can
actually get down-stack refinements with complete OOP-integration if
needed. Assuming an .unprepend is also possible.

In fact, let me write this down:

case c) now with metaclass coercion:

   module StringRefinement
     def downcase
       super + "x"
     end
   end

   Class E
     def m1
       "Foo".send(:downcase)
     end
   end

   sites = Ruby::Ast.methods(E).callsites(String)
   transforms = sites.transform(:existing)

   # an alternative to .wrap
   # similar to method .enter/.exist. Except it's for callsites
   transforms.before do |target,method_name, *args, &block|
     target.metaclass.send(:prepend, StringRefinement)
   end
   transforms.after do |target,method_name, return_value, *args, &block|
     target.metaclass.send(:unprepend, StringRefinement)
     return_value
   end

   E.new.m1 # => foox



I think such a fine-grained API is what we will need. No single
high-level API can cover all the use-cases provided for Refinements and
retain performance at the same time. Simply because it is too coarse.