Matz says namespaces are too hard to implement - why?

On 22.12.2007 20:14, Charles Oliver N. wrote:

old definition would apply again; this also imposes the interesting
question if recursion with redefined methods is still possible… :-))
A lot of interesting problems to solve. :slight_smile:

Yes, lexically scoped namespaces would have the same performance
implications call-stack scoped namespaces iff they were applied
dynamically at runtime. If they were applied statically at parse time,
via a keyword or other syntax, the overhead of checking for a namespace
would be limited to specific chunks of code:

Yeah, you’re right. I looked more to the receiving side but did not
think about the call location.

namespace String => StringDecorate {
namespacing is actually useful.
Yeah, and then there is still the question what happens to recursion?
Since the checks are lexically, you could override a recursive method
with another one and the second invocation ends up calling the old
version. Spooky although I guess you could do some cute tricks with
this. :slight_smile:

Kind regards

robert

To find a solution to my want/need for namespaces,

Are there any module-specific variables? Kind of like a controlled
global?

I’ve been looking into using Fluid to get what I’m looking for, but
I was hoping that the variable usage could span multiple files.

Help?
-------------------------------------------------------|
~ Ari
if god gives you lemons
YOU FIND A NEW GOD

On Sun, Dec 23, 2007 at 09:39:46AM +0900, Charles Oliver N. wrote:

Robert K. wrote:

Yeah, and then there is still the question what happens to recursion?
Since the checks are lexically, you could override a recursive method with
another one and the second invocation ends up calling the old version.
Spooky although I guess you could do some cute tricks with this. :slight_smile:

One way or the other :slight_smile: I don’t claim to know where selector namespaces are
actually useful, so I can’t say which way would be better to go. But I’ll
gladly illustrate how to implement them either way.

Here’s a ruby-core thread which looks related. The features described in
there
could likely be implemented through selector namespaces.

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/9960

On Dec 27, 11:10 am, Eivind E. [email protected] wrote:

end
group of methods that you wouldn’t otherwise get access to. You can
do namespacing manually by calling all your methods
“mynamespace_method” instead of just “method”, and checking for this

Well, but that’s the trivial case that Charles talked about. The
practical case would be where all the code called from a given
selector namespace respects the definitions / constants / variables of
that namesapce; so if you define Array#index to return “foo”, then any
methods called within that namespace will use the modified
Array#index. And that’s just where it gets hairy – causing extra
lookups (at best, one; more likely, several) for every vcall, to
determine if we’re in a namespace. (Someone correct me if I’ve not
followed the thread right.)

expect. It seems like if selector namespacing is useful (which I’m
can’t do it across libraries or calls without installing a namespace
prefixing to create an explicit namespace might be as useful as adding
implicit ones, and all of this might only be useful in connection with
adding some sort of protocol (interface definition) support.

Eivind.

Regards,
Jordan

On Dec 22, 2007 4:22 PM, Charles Oliver N. [email protected]
wrote:

}

The default implementation of Enumerable#collect calls each. Would you
expect collect to see the original each method or the one you’ve
provided in the namespace?

Original each.

The way I think about it, importing a namespace is gaining access to a
group of methods that you wouldn’t otherwise get access to. You can
do namespacing manually by calling all your methods
“mynamespace_method” instead of just “method”, and checking for this
in method_missing, a la (but mind overriding $1 and fixing up
exception backtraces):

def method_missing(method, rest)
if method.to_s =~ /^mynamespace_(.
)/
send($1, *rest)
else
super
end
end

I would also believe that with type inference, namespacing should be
able to make code go faster, not slower. When you do namespace
lookups, you can do early resolution of calls more easily, as you can
find the method being available through the namespace and short
circuit. (I have no idea if you’re doing inference in JRuby - are
you?)

See the above example; if you only want namespaces so that within a
given block of code method calls to where you want them to, that’s
simpler to implement. But it breaks some amount of consistency you might
expect. It seems like if selector namespacing is useful (which I’m
unsure of) it would only be generally useful if it could also affect
calls further down the chain. Maybe I’m wrong?

You’re expecting consistency at a different level than me, at least.

At some point you have to be able to say “this code is being
namespaced”. If you want to do that at runtime, then either you need to
modify already-parsed code (which won’t work across libraries or calls)
or you need every invocation to check for namespaces. If you want to do
that at parse/compile time you need a pragma or keyword, and you still
can’t do it across libraries or calls without installing a namespace
check for every invocation.

I think the appropriate point in time is compile time; not being able
to modify name spaces at run time may be a tolerable cost. The point
of name spaces, as I see them, is to avoid name conflicts while still
retaining “nice” names; while it might be fun to be able to play with
them at runtime, ones that are restricted to compile time may be
better than not having name spaces at all.

I’m still on the fence about how useful they are at all - use of plain
prefixing to create an explicit namespace might be as useful as adding
implicit ones, and all of this might only be useful in connection with
adding some sort of protocol (interface definition) support.

Eivind.

On Sat, Dec 22, 2007 at 11:50:41PM +0900, Charles Oliver N. wrote:

The complication is that under normal circumstances a method has no
knowledge of whether it’s being called inside a namespace or not, since
the installation of the namespace itself is just another method call. So
every method in the system would have to check whether they are being
called under a namespace.

The technique I used in:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/161439

was to modify caller’s frame to signify that a particular namespace is
active.

If use and namespace were keywords rather than method calls, they could
do this without the use of caller_binding. :slight_smile:

This technique is slow, but iirc only for methods defined in a
namespace (but the cost is paid regardless whether the namespace is
active).

If namespaces were lexical elements, I think the performance penalty
could be minimized, as it would be known at compile time which method
lookup tables to use for a given call.

Paul

On Dec 27, 2007 7:09 PM, MonkeeSage [email protected] wrote:

On Dec 27, 11:10 am, Eivind E. [email protected] wrote:

On Dec 22, 2007 4:22 PM, Charles Oliver N. [email protected] wrote:
[… example calling collect from something importing a namespace
snipped …]

Well, but that’s the trivial case that Charles talked about. The
practical case would be where all the code called from a given
selector namespace respects the definitions / constants / variables of
that namesapce;

My point was that I do not see why this would be any more practical.

I don’t know what your usecase is; from my point of view, the idea of
namespaces is to separate the call space for implementations of
different extensions to a set of types (in a more static language,
each type would be a different class; this is less relevant for Ruby),
so that different extensions do not step on each other.

As far as I can tell, there are slightly different tradeoffs between
using lexical scoping (which I propose) and dynamic scoping (which you
propose), with the dynamic scoping enabling some hacks that the
lexical scoping doesn’t, and the lexical scoping having the benefits
of ease of understanding and speed of code.

This is the standard tradeoff between lexical and dynamic scoping; in
languages that provide both dynamic and lexical scoping for variables
(Perl, Lisp) my experience is that we end up using about 1000x more
lexical than dynamic scoping, and in languages that only provide
lexical scoping (e.g, Ruby), we easily work around the lack of dynamic
scoping and hardly notice that it isn’t there.

If you (or anybody) can show me usecases that show that dynamic
scoping is essential for implementation of namespaces in Ruby, of
course I’m all ears. So far, it’s just seems like the need for
dynamic scoping has been taken as a given, and I personally feel that
dynamic scoping make programs harder to understand than lexical
scoping, and that without compelling usecases it seems more reasonable
to use lexical scoping, which also avoids the speed penalty.

Eivind.

On Sat, Dec 22, 2007 at 12:18:24PM +0900, Stefan R. wrote:

So now I wonder, am I missing something? Is the devil in the details, or
is one of the incomplete things in my proof of concept the show stopper?
I see that it would be some work, but I don’t see how it would be hard.
Please enlighten me :slight_smile:

AFAICT, one problem with your implementation is it’s not possible to
have multiple namespaces active at the same time. This is something
that I think would be very desirable, otherwise it’s not possible for a
single method to make use of methods defined in two different namespaces
(libraries) simultaneously.

Paul

Paul B. wrote:

If namespaces were lexical elements, I think the performance penalty
could be minimized, as it would be known at compile time which method
lookup tables to use for a given call.

Yep, this would work, but it only namespaces for a single frame. Is that
what we want?

  • Charlie

On Fri, Dec 28, 2007 at 10:15:01AM +0900, Charles Oliver N. wrote:

Paul B. wrote:

If namespaces were lexical elements, I think the performance penalty
could be minimized, as it would be known at compile time which method
lookup tables to use for a given call.

Yep, this would work, but it only namespaces for a single frame. Is that
what we want?

I think what I would want is:

  • apply a namespace to a particular scope
  • apply a namespace to a single method
  • apply a namespace to an entire class definition

All of these can be determined at compile time.

What benefit would there be to a namespace being active for child
frames? I think it might cause side-effects that are hard to track.

I wonder what Smallscript does.

Paul

Eivind E. wrote:

p Array.new.collect
do namespacing manually by calling all your methods
“mynamespace_method” instead of just “method”, and checking for this
in method_missing, a la (but mind overriding $1 and fixing up
exception backtraces):

I was thinking about this a bit more today and I don’t see how it’s
particularly useful. If the namespacing is only local to some block or
other lexical structure…why don’t you just call the methods directly?
What does it gain you?

It seems like namespacing is only “really” useful if you can also have
namespace changes apply to calls further down the stack. In a lexical
context, I don’t see what it gains you over just calling different
methods.

I would also believe that with type inference, namespacing should be
able to make code go faster, not slower. When you do namespace
lookups, you can do early resolution of calls more easily, as you can
find the method being available through the namespace and short
circuit. (I have no idea if you’re doing inference in JRuby - are
you?)

Not…yet :slight_smile:

of name spaces, as I see them, is to avoid name conflicts while still
retaining “nice” names; while it might be fun to be able to play with
them at runtime, ones that are restricted to compile time may be
better than not having name spaces at all.

I’m still on the fence about how useful they are at all - use of plain
prefixing to create an explicit namespace might be as useful as adding
implicit ones, and all of this might only be useful in connection with
adding some sort of protocol (interface definition) support.

Ditto. If they’re only lexically scoped to make “nice” names, I don’t
see the benefit. I could see how they might be useful if you wanted to
run an entire call stack with temporary modifications, but of course
that’s the trickier version.

  • Charlie

Paul B. wrote:

I do see the point about #each, but I’m having a hard time imagining
what the implications are of having the namespace apply further down the
stack. What do we do if I call a method that imports a namespace which
conflicts with the one I’ve just imported?

Namespaces in Ecmascript appear to be lexically scoped:

I’m curious if implementors of other languages have thought about this
issue.

I really don’t know, and to be honest I haven’t seen a particularly
compelling use case for either lexically scoped namespaces
or…what…temporally scoped?

So we want to limit class modifications to a specific range of
execution. Then there’s two cases:

  1. we want to do it lexically-scoped. In that case, we can just call our
    own utility methods directly, and if the object isn’t typed as we
    expect, it passes through. Sure, syntactic sugar…but who cares, is it
    worth impacting the rest of the language impl to support it?
  2. we want to do it temporally-scoped, affecting downstream calls too.
    Well then, I’d love to see the use case for this, and you may have a
    hard time convincing me that it’s worth crippling every invocation with
    some indirection mechanism that can be overridden on a whim.

What is the use case? I’d love to see.

  • Charlie

On Fri, Dec 28, 2007 at 11:51:17AM +0900, Charles Oliver N. wrote:

I was thinking about this a bit more today and I don’t see how it’s
particularly useful. If the namespacing is only local to some block or
other lexical structure…why don’t you just call the methods directly?
What does it gain you?

It seems like namespacing is only “really” useful if you can also have
namespace changes apply to calls further down the stack. In a lexical
context, I don’t see what it gains you over just calling different methods.

It might not be desirable to call the methods directly if you don’t know
what the type of the object is. For example, if I have a method that
was passed an object as a parameter, I might import a namespace to get a
particular behavior for arrays in case the object passed in is an array.
But the object might not be an array, in which case I can’t just change
the name of the method I call.

I do see the point about #each, but I’m having a hard time imagining
what the implications are of having the namespace apply further down the
stack. What do we do if I call a method that imports a namespace which
conflicts with the one I’ve just imported?

Namespaces in Ecmascript appear to be lexically scoped:

http://docs.huihoo.com/web/js/es4/core/namespaces.html

I’m curious if implementors of other languages have thought about this
issue. I wonder if we could get David Simmons to chime in, since he’s
the one who pointed us here in the first place.

Paul

On Dec 28, 2007 10:03 AM, Paul B. [email protected] wrote:

Ideally this should work:

class Foo
def foo
# [ruby-talk:69859]
use namespace Mathn
m = Matrix[[1,2], [3,4]]
return m*m.inv #=> [[1,0], [0,1]]
end
end

but what about this:

def divide(a, b)
use namespace Mathn
a / b
end

Now, I’d think you would want:

1 / 2 #=> 1

divide(1,2) #=> 1 / 2

The problem here it seems that you would need more than attach a
namespace to an object. Here we want the Integer/Fixnum classes to
act differently in the context of executing the divide method, even
for objects instantiated outside and passed in.


Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

On Fri, Dec 28, 2007 at 04:20:11PM +0900, Charles Oliver N. wrote:

I really don’t know, and to be honest I haven’t seen a particularly
compelling use case for either lexically scoped namespaces
or…what…temporally scoped?

What is the use case? I’d love to see.

I would like other users to be able to import mathn without it breaking
my code.

Ideally this should work:

class Foo
def foo
# [ruby-talk:69859]
use namespace Mathn
m = Matrix[[1,2], [3,4]]
return m*m.inv #=> [[1,0], [0,1]]
end
end

class Bar
def bar
return 5/2 #=> 2
end
end

but I guess this is dynamic scoping rather than lexical scoping.

(“temporal” to me implies that the change applies outside the current
thread – see also [ruby-talk:196082])

While potentially more powerful, dynamic scoping makes it harder to know
what side-effects importing the namespace might have.

But what if the namespace could be turned on/off at the object or method
level? And what if that namespace change automatically be propogated to
objects or methods created within a scope where that namespace were
active?

In the above example, m would automatically get the Mathn namespace
since it was created in a scope where that namespace was active. Then
the code would work as expected.

This could be implemented similar to the way mix-ins are implemented
today, avoiding the performance penalty for every object.

Paul

On Sat, Dec 29, 2007 at 12:32:10AM +0900, Rick DeNatale wrote:

divide(1,2) #=> 1 / 2

The problem here it seems that you would need more than attach a
namespace to an object. Here we want the Integer/Fixnum classes to
act differently in the context of executing the divide method, even
for objects instantiated outside and passed in.

I’m having trouble understanding why this case wouldn’t work as
expected. Can you elaborate?

Thanks,

Paul

On Dec 28, 2007 10:37 AM, Paul B. [email protected] wrote:

1 / 2 #=> 1

divide(1,2) #=> 1 / 2

The problem here it seems that you would need more than attach a
namespace to an object. Here we want the Integer/Fixnum classes to
act differently in the context of executing the divide method, even
for objects instantiated outside and passed in.

I’m having trouble understanding why this case wouldn’t work as
expected. Can you elaborate?

What causes the integer 1 to use the ‘/’ method monkeypatched in Mathn
in one case and not the other?


Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

On Sat, Dec 29, 2007 at 12:54:28AM +0900, Rick DeNatale wrote:

def divide(a, b)
use namespace Mathn
a / b
end

Now, I’d think you would want:

1 / 2 #=> 1

divide(1,2) #=> 1 / 2

What causes the integer 1 to use the ‘/’ method monkeypatched in Mathn
in one case and not the other?

In the first case (1 / 2), the Mathn namespace is not active, so the
usual integer division method applies.

In the second case (divide), the Mathn namespace is active, so Integer#/
defined in the Mathn namespace applies.

Paul

On 23/12/2007, Charles Oliver N. [email protected] wrote:

So you’re saying the stdlib delegator is slow? Do you have some
benchmarks for this? I’d like to examine it and see if I can improve it
or speed it up in JRuby. I believe there are parts of Rails that use
delegator too (or perhaps only used to), so it would be nice to see if
it’s a real bottleneck.

I tried to benchmark some code using the delegator, and it turns out
to be quite fast. So the problem was probably with actually using the
methods for converting stuff or the methods stuck to the original
class by the delegator.

So we get a replacement of the stack-scoped namespace quite easily,
and I would really like to see a usecase for stack-scoped namespace
that cannot be implemented using a delegator.

Thanks

Michal

            user     system      total        real

UString 1.790000 0.100000 1.890000 ( 1.992710)
Delegate 1.840000 0.090000 1.930000 ( 2.039103)
Debug 1.810000 0.070000 1.880000 ( 1.984717)
String 2.760000 0.090000 2.850000 ( 3.191666)
user system total real
UString 1.680000 0.100000 1.780000 ( 1.901071)
Delegate 1.730000 0.100000 1.830000 ( 1.959464)
Debug 1.790000 0.100000 1.890000 ( 2.284965)
String 2.790000 0.130000 2.920000 ( 3.415921)
user system total real
UString 1.700000 0.100000 1.800000 ( 1.903906)
Delegate 1.740000 0.100000 1.840000 ( 1.957677)
Debug 1.770000 0.110000 1.880000 ( 1.975056)
String 2.840000 0.110000 2.950000 ( 3.165277)

some utf-8 scattered tea in a string

str=“これはムルチバイトのストリング。このストリングに採番する事が出来ません。”

require ‘delegate’
require ‘benchmark’

class UString < Array
def self.from_str str
return UString.new str.unpack(“U*”)
end
end

class UStringDelegate < Array
def self.from_str str
return self.new str.unpack(“U*”).map{|x| SimpleDelegator.new x}
end
end

class UChar < SimpleDelegator
def inspect
“'”+pack(“U”)+“'”
end
def to_s
pack(“U”)
end
end

class UStringDebug < Array
def self.from_str str
return self.new str.unpack(“U*”).map{|x| UChar.new x}
end
end

3.times {Benchmark.bm(10){|x|
s1 = UString.from_str str
x.report(“UString”){ s2 = str[1…-2]; 100_0000.times{ s2 == s1[1…-2]
}}
s1 = UStringDelegate.from_str str
x.report(“Delegate”){ s2 = str[1…-2]; 100_0000.times{ s2 == s1[1…-2]
}}
s1 = UStringDebug.from_str str
x.report(“Debug”){ s2 = str[1…-2]; 100_0000.times{ s2 == s1[1…-2] }}
s1 = str
x.report(“String”){ s2 = str[1…-2]; 100_0000.times{ s2 == s1[1…-2]
}}
}}

On Dec 28, 2007 1:28 PM, Paul B. [email protected] wrote:

divide(1,2) #=> 1 / 2

What causes the integer 1 to use the ‘/’ method monkeypatched in Mathn
in one case and not the other?

In the first case (1 / 2), the Mathn namespace is not active, so the
usual integer division method applies.

In the second case (divide), the Mathn namespace is active, so Integer#/
defined in the Mathn namespace applies.

Okay, I read your pastie again, and I understand the unnatural acts
being performed a bit more. All I can say is that it makes me want to
run away in horror.

There are also some other little issues, for example, the way mathn
changes Fixnum and Bignum

class Fixnum
alias / quo
end

class Bignum
alias / quo
end

and in Ruby 1.9 these become:

class Fixnum
remove_method :confused:
alias / quo
end

class Bignum
remove_method :confused:
alias / quo
end


Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/