NOTE: The SVN repository isn’t up to date with these changes. It will be
sometime over the weekend.
Thanks & happy reading,
-John
286238: (dinov)
Currently ngen will try and produce an image that’s probably around 1gig
in size (or larger) for Microsoft.Scripting.dll due to the reflected
caller additions. This cuts back the number of FastCreate’s we’ll do
and gets this back to a reasonable level. It also adds a cache for
non-dynamicmethod’s so that we only create one per method. This should
help w/ both the throughput regressions as well as the fact that I broke
ngen.
Also included is the fix for the out param reported on the mailing list
yesterday.
286963: (tomat) Ruby bug fixes
Fixes the following Ruby bugs (no major change in design needed).
ID Assigned To Title
304051 Tomas M. ruby: explicitly set $! in the ensure
clause should not be preserved
303812 Tomas M. ruby: $! is not preserved in the
ensure clause when the exception is thrown in the else clause
303774 Tomas M. ruby: redo should not re-evaluate the
loop condition
303703 Tomas M. ruby: “break” does not break out from
the lambda call
298222 Tomas M. ruby: explicitly setting $! in the
rescue exception parameter does not preserve $!
297959 Tomas M. ruby: $! in the ensure clause should
not be nil when an exception is raised, but no matched rescue clause is
found
293040: (dinov) Preparation to fix infinite recursion issue
These are the final set of small fixes which are necessary before we can
fix the infinite recursion issue.
1. Switch from using Type.Missing to our own MissingParameter type.
We can’t pass Type.Missing through Reflection.Invoke calls.
2. Fix Python’s DoOperationBinderHelper to avoid casts which
box/unbox values (this is just due to an identity of integer test in
test_copy)
3. Add a missing conversion in Ruby
4. Check IsInstanceOfType in Cast to catch COM objects interface
tests correctly
5. Makes evaluation of addresses work the same as emitting - we can
end up evaluating an expression before getting the address and that
expression can throw.
6. Fix an issue w/ return types on properties not going through the
binder’s converter
7. Add support for strongly typed delegates in the eval case
8. Fixed an issue w/ rethrowing exceptions in evaluate mode
9. Added support for returning from a VoidExpression
10. Fix an issue w/ arrays in evaluate mode where we try and cast to
object[] when we have an array of byte[]
Also fixing this bug:
304475 dir(System.Reflection.Emit.AssemblyBuilder) throw TypeError
Where we just need to check for null in DynamicMixin.
293085: (tomat) Fixes RubyForge bug #14425.
Ruby allows jump statements (return, redo, retry, break and next) to
appear in Boolean expressions as right operands.
Examples:
def foo x
x and return ‘true’
x or return ‘false’
‘Unreachable’
end
foo true # “trueâ€
foo false # “falseâ€
This shelveset changes the grammar to support this feature and adds AST
node ConditionalJumpExpression that represents and/or
expression. Also adds a unit test of the node.
293153: (jomes) generics: implements support for constructing generic
types
Pretty simple, implements support for creating generic .NET types like
List, Dictionary<K,V>, etc.
The syntax in Ruby is something like this:
require “mscorlib”
IntList = System::Collections::Generic::List.of(System::Int32)
a = IntList.new
a.add 1
a.add 2
a.add 3
assert_equal(a.count, 3)
I also added support for the [] syntax that IronPython uses:
System::Collections::Generic::Dictionary[String, String]
293329: (mmaly) Bye bye, WeakCondition
This change gets rid of the WeakCondition and updates all former calls
to that to generate proper casts. Ruby has factory it wanted which
generates the casts and up-casts to object. The cast to bool must be
dynamic so that the helper gets called. Alternative would be calling the
helper explicitly or doing something yet smarter.
There is one temporary change in the ReferenceArgBuilder.cs The builder
generates code:
ElementType tmp;
arg is StrongBox ? tmp =
RuntimeHelpers.GetBox((StrongBox)parameter),
tmp : RuntimeHelpers.IncorrectBoxType(StrongBox);
Notice the comma in the first expression. If we cast the whole thing to
object, the currently not quite complete implementation of calling
methods with by ref arguments will do the wrong thing. Temporarily we
cast the 2nd expression to ElementType as it will throw inside the
helper anyway and the cast will never execute. The ideal solution is for
the binder to generate a better tree all around, which will play better
with the upcoming “emit as†implementation overhaul.
293941: (tomat) Ruby libraries
Prepares built-ins for a split: the types and functionality that the
compiler directly depends on will be kept in Ruby.dll while all Ruby
visible methods will be factored out to Ruby.Libraries.dll. Also adds
Ruby.Libraries and ClassInitGenerator projects to the Rowan solution –
they depend on Ruby.dll and M.S.dll, so they should be rebuilt each time
the dependencies are rebuilt.
293960: (jomes) fixes inheriting from a .NET type
Very simple change, fixes inheriting from a .NET type in Ruby
- We weren’t handling TypeTrackers
- We weren’t exposing CLS members on the derived type
294389: (mmaly) Switch statement upgrade
I managed to rewrite the switch statement yesterday. While this change
doesn’t yet add the missing interpreted mode, it removes all
Jscript-isms. The switch statement is true switch. Takes an int and
routes to given label. The labels are marked with constant values.
Switch will choose dynamically whether to emit as jump table or “ifâ€
statements.
If the labels are integer-like constants, it will generate the DLR
switch. If not, it will also generate the DLR switch (cool, huh?) except
preceded with an if statement which ‘encodes’ the switch labels into a
constant integer and then proceeds to the direct switch with that
constant. This eliminates the need for goto to do the fall through.
In the second (via encoding) case, you’ll see the if … elseif … elseif
pattern first, in which the bodies are only (.bound #2) = … and then
switch over the #2 temp.
DLR switch now checks for unique label values, and does it in 3 ways:
• If there are small number of cases (< 10), it uses dumb O(N^2)
algorithm
• If there are cases values of which are in reasonable range (max – min
< 1024), it uses BitArray to track the unique values
• Else, it falls back to dictionary
Only the generation into the pure switch is affected. When generating
the if …elif …elif prefix, the generated label numbers are unique by
definition so that path remains the same. However, for the switch table
code path, the code merges bodies of the case clauses. Essentially each
case clause whose label is not unique is merged with the previous
clause’s body. The bodies must stay in place because of the
fall-through, but no one can ever jump to the label of the non-unique
statement so it is safe to merge them with the previous bodies.
294586: (jomes) ruby extension methods on interfaces
Implements extension methods on interfaces for Ruby. What this means it
that .NET types will get Ruby methods injected onto them if they
implement certain interfaces. For example, types that implement
IEnumerable will behave just like a Ruby classes that mix in Enumerable.
I implemented this by treating interfaces as if they were mixed in
modules.
These interfaces have extension methods now:
• IEnumerable
• IComparable
• IList
• IDictionary<object, object>
IList and IDictionary<object, object> were factored out of Array
and Hash, respectively. Some investigation remains to see if these can
be made to support all generic types or if they could use IList and
IDictionary instead, but I didn’t attempt that yet.
Detailed change description:
• Added a new attribute RubyExtensionInterfaceAttribute, updated
ClassInitGenerator & Initializer to understand it
• Added the MixinInterfaces property to RubyClassAttribute, which
tells ClassInitGenerator to inject all valid interface methods it finds
onto the type (useful if you’re extending a CLR type and you want
everything).
o StringOps uses this feature to get Enumerable & Comparable
support.
• RubyExecutionContext keeps track of interface mappings & mixes
in the interface’s RubyModule onto new .NET types
• IEnumerableOps implements each & includes Enumerable
• IComparableOps implements <=> & includes Comparable
• IDictionaryOps takes methods from HashOps (which were already
written to operate on IDictionary<object, object>)
• IListOps takes methods from ArrayOps
o Helpers are needed for AddRange, GetRange, InsertRange, and
RemoveRange, which IList doesn’t support. Also, several methods are
different between ArrayOps and IListOps
294789: (dinov) Infinite recursion fix
This changes rule creation so it only runs the test once and evaluates
the target to avoid the test changing and getting infinite recursion.
This is done by pushing the update & invoke out of
DynamicSite.UpdateBindingAndInvoke and all the way down into the
internal portions of ActionBinder and the rule caching mechanism. We
now pass in the site, the site’s target (by-ref), and the site’s rule
list (by-ref) and let the action binder update these for us. This
reduces a bunch of the duplicated code in DynamicSite.Generated.cs.
Part of the reason to push this in is that the site HAS to have its
target delegate set before executing the target of the rule. If not the
site could recurse on its self, not have the target set, and therefore
we limit the amount of stack that can be used - as well as increase the
of calls which don’t use the optimized target.
294962: (dinov) Updates ExtensionTypeAttribute
This updates ExtensionTypeAttribute to no longer have any DynamicType
dependencies. It also removes some Python specific pieces to it
(support for deriving from int, etc…) and removes “using M.S.T;†from
everyone using it just for ExtensionTypeAttribute (this is most of the
files changed).
294999: (dinov)
This change implements the helper methods on PropertyTracker and
switches GetMemberBinderHelper over to using these helper methods. This
change ultimately removes dependencies on ReflectedProperty and
ReflectedIndexer. Mostly this is moving the logic in
GetMemberBinderHelper over into PropertyTracker.
Add new ErrorInfo class which is used for reporting information about
how an error should be reported to the user. The error can either be
reported as an exception which gets thrown or a value that gets
returned. Added new methods to ActionBinder which enable languages to
produce errors. Eventually all of the error production will move into
this format.
Small tweaks:
Move MakeCallExpression from BinderHelper onto Binder.
The only reason it was on BinderHelper was because it needed to access
the Binder to provide conversions.
Fixed the iteration in MethodBinder over the dictionary
(which is bogus, we have a CodePlex bug on this reported by a user)
Implemented Call on MethodTracker – this is used when
PropertyTracker falls back for private binding.
Exposed IsStatic directly off of PropertyTracker.
295337: (mmaly) Redesigning the DLR AST Walkers
(Walker6 passed SNAP already, Walker7 contains few cosmetic updates such
as missing comments, removing few redundant argument checks, adding few
extra ones etc. I hope it also passes)
To fix the “non-empty stack upon entry to try†bug, I am going to need a
better walker than what is easy to implement in our system (one that I
can use to re-generate the AST along the way). The current walker design
doesn’t lend itself easily to that purpose (the Walk methods on the
nodes do just that – walk, but I’d have to write a type-based switch to
implement my walker, rather than an enum-based switch).
Second motivation is LINQ which doesn’t have walker on its nodes so to
get closer to that model, our walkers must live completely outside of
the AST.
This checkin achieves these two goals in a following ways:
-
Each node is given an enum (this enum started with LINQ’s
values, some are commented out as we don’t use them (yet – CheckedAdd
for example), some are added (statements). The UnaryOperator and
BinaryOperator enums got swallowed by the big enum which nicely plays
that role.
2) The walker is combination of generated and hand-written code and
the Walk methods from the AST nodes are gone. The individual walkers
haven’t changed except to call WalkNode(node) instead of node.Walk(this)
to perform the node walk. The WalkNode will switch based on the node
type (enum), direct cast and perform walk identical to that of the
original node’s walk.
What I like about having the walker completely outside of the tree is
that the AST nodes don’t dictate the order of walking or don’t specify
one way as ‘primary’. This way any walker (and we’ll eventually need a
reverse walker) plays by the same rules on the same playing field.
Next steps in this direction:
• Make all leaf AST nodes sealed
• Remove codegen from the AST nodes
• Merge AST node that can be merged (say, after we remove the
codegen out, there is no need to have break and continue as separate AST
nodes because they’ll hold identical information, same for
CodeContextExpression and EnvironmentExpression for example). So we’ll
end up with fewer nodes as a result, which is a goodness.
295459: (dinov)
This change is just re-organizing DynamicHelpers and RuntimeHelpers in
preparation for the removal of DynamicType. Previously these two
classes had no real strong distinction. Why does something go into
DynamicHelpers? Why does it go into RuntimeHelpers? Who knows. There
is now one distinction: Things in DynamicHelpers are going away, things
in RuntimeHelpers are staying. DynamicHelpers its self moves into
M.S.Types indicating that it will be removed real soon now (I don’t
actually move the file, it’ll be gone in a day or two).
What this is ultimately going to accomplish is that all of the
IronPython type system will be able to come up in one chunk w/ hopefully
no code changes. From there it’ll be a few renames away from being
integrated into the IronPython code base. Because there are a whole lot
more references to DynamicHelpers.GetDynamicType* then there are to the
other functionality in DynamicHelpers it seems natural to keep
DynamicHelpers as the home of GetDynamicType and friends.
So most of this change is just moving things from DH to RH and adding
“using M.S.Types;†to IronPython. There’s one additional tweak in that
I remove LanguageContext.DeleteMember. JS was using DynamicType for its
implementation and the only real caller was IronPython. This gets
replaced by a PythonOps.DeleteAttr which closely mimics the TryGetAttr
method we already have.
Also some code only used by Python (CheckTypeVersion) gets pulled up
into PythonOps and a couple of places where we stop getting type names
from DynamicType.
295467: (jomes) inject onto IList vs. IList
Pretty simple, just changes the IList member injectors to inject
onto IList instead, because a lot more things implement IList
(especially notable are ArrayList and List for any type T).
I needed a couple of minor bug fixes—the interface injection was
injecting things onto non-concrete generic types. I also changed the
generator back to using a sorted dictionary so Initializer.Generated.cs
doesn’t change so much each time (which you can see if you look at the
diff, some stuff changed that shouldn’t have).
295500: (dinov)
This starts exposing MethodGroup’s instead of BuiltinFunction’s.
First off one “big†change is FastCallable is getting moved into
M.S.Types (which will get pulled up). This also causes a bunch of files
to get using M.S.Types for references to FastCallable. CallType remains
in Microsoft.Scripting as it’s a component of the MethodBinder.
This change alters some MemberTracker setup. I’ve made MemberGroup’s no
longer a MemberTracker. Instead MemberGroup’s are just ways that we
talk about groups of members. I’ve added a MethodGroup which is a
collection of methods. MethodGroup’s are also unique per actual .NET
method (and generic instantiation for generic methods). That allows us
to use a simple comparison on them to check for identity (this is
enforced by the ReflectionCache).
From here it’s basically pretty straight-forward: The DLR gets
MemberGroup’s, for method binding it creates MethodGroup’s (once we’re
resolved to just a bunch of methods). All languages except for Python
currently expose MethodGroup’s directly to the user. Python transforms
these into BuiltinFunction’s.
There’s still more work to be done: This change doesn’t remove
CallBinderHelper’s recognition of
BuiltinFunction/BuiltinMethodDescriptor’s. We should also more strictly
base Python’s identity of BuiltinFunction’s on MethodGroup’s and then we
can also use simple object identity for tests against BuiltinFunction’s
– currently Python is sharing the hash-key w/ ReflectionCache which
isn’t necessary. Finally we don’t have perfect one-to-one mappings
between methods and MethodTracker’s. This is because we use the
MethodInfo as the key which can be duplicated based upon from what type
you got the MethodInfo. But this change is already big enough that it
seemed prudent to wait on those next steps.
296822: (jomes) Ruby Class Variables
Implements support for Ruby class variables (aka static fields)
Mostly, this is straightforward. We add a dictionary to RubyModule for
storing the values, and add methods for getting and setting the
variables. The class variable AST node turns into get/set calls. Ruby
resolves class variables in method resolution order, so it searches all
base classes & mixins. Setting variables is interesting—the MRO is
searched, and if the variable is found it’s modified, otherwise, it’s
added to the current Module/Class.
A few AST changes are needed—unlike instance variables (which are
resolved on the current “selfâ€), class variables are lexically bound to
the enclosing module/class. So we need scoping for modules, and methods
need the ability to be closures. We close over the module’s “selfâ€, but
because the DLR merges variables by name, we need to generate a new name
for each module. Also, DLR closures don’t work unless you flow in the
correct CodeContext, so RubyMethodInfo’s need to hold on to their
CodeContext, and flow in the right one.
297031: (jomes) ruby monkey patching of .NET types
A few minor fixes to allow Ruby to add methods to .NET types, aka
“monkey patchâ€. The reason this wasn’t working is that Ruby exposes .NET
namespaces as NamespaceTrackers & .NET types as TypeTrackers. We were
already converting TypeTrackers to RubyClass under certain
circumstances; now we also convert NamespaceTrackers to RubyModules. For
this to work, a RubyModule can have a NamespaceTracker backing it. When
we go to lookup a constant on the module, we check the module’s
constants, and then fall back to the namespace.
This design is not optimal… we need to come up with a better way for
language’s type objects to interoperate. But it’s intended to unblock
some of our .NET interop scenarios in the short term.
297060: (jomes) fix scoping of constant variables
Ruby constants are supposed to bind lexically to the enclosing
class/module, but this wasn’t working. Pretty simple change,
ConstantVariable now binds to the lexically enclosing module, or Object
if an enclosing module isn’t present.
There was already a test for this, but it wasn’t running against
IronRuby, so I enabled it & added a few more test cases.
297066: (jomes) Ruby instance_eval/class_eval/module_eval support (for
the non-string overloads)
Implements basic support for the instance_eval, class_eval, module_eval
overloads that take a block (not the ones that take strings, that
requires full “eval†support). This is pretty easy to do, because we
just create a new Proc with a different “self†and call the block.
instance_eval on a Module/Class seems to work a bit differently, so I’m
throwing NotImplemented until we can make the necessary infrastructure
changes to handle it correctly.
297179: (dinov)
Next change in the line of changes to remove DynamicType. This one
removes ReflectedEvent/BoundEvent from the parlance of the DLR:
ReflectionCache.GetReflectedEvent moves to DynamicTypeOps and callers
are updated.
Python gets a PythonSetMemberBinderHelper for dealing w/ the assignment
logic back to a ReflectedEvent.
We now pass the type we’re doing the lookup from for ReturnMemberTracker
(this is our version of the .NET ReflectedType property which shows
where each member came from)
The DLR currently generates calls to static helper methods for doing the
addition/subtraction to the event object.
Fixed an issue where getting a TypeGroup’s DeclaringType/Name could
throw
And finally manifest the in-place add/subtract methods onto
EventTracker’s and BoundEventTrackers (these aren’t directly defined to
deal w/ typing issues where we need the delegate to be strongly typed).
2975790: (jflam) Enables support for Rubinius spec suite
Adds Rubinius spec suite for Array, Hash and String. Dir is also added,
but tests are not enabled for this yet.
Adds a modified version of the Rubinius spec runner that uses only
features implemented by IronRuby. These are the mini_mock, mini_rspec,
mspec, mspec_helper, rspec_helper, simple_mock, and spec_helper files in
\Tests.
Adds/fixes features to IronRuby to support running the test suite:
- Fixes require behavior in loader.cs to allow files to be loaded more
than once - Fixes FILE
- Adds File.dirname() method
- Adds RubyExtensionModuleAttribute and marks IListOps,
IComparableOps, IDictionaryOps, IEnumerableOps using this attribute. - Adds implementation of const_defined?, alias_method and
remove_method
Adds a new utility (IronRuby.Libraries.Scanner) that generates a YAML
file that contains a list of all implemented singleton and instance
methods (note that this is written in C# 3.0).
297638: (mmaly) Removing callwiththis
I’ve had CallWithThisExpression in my sight for a while now and as
Nandan and Jitu made progress on the binder, it is no longer used so it
can go now.
Additionally, MSAst.Arg was used only as a temporary information storage
and was never actually part of the AST (CallSignature would throw it
away) so I got rid of it too. In Python it meant even cleaner code.
297654: (jomes) splat and send
This change enables splatting of argument lists when calling Ruby
library methods that are implemented in C#. Before this change,
splatting only worked if you’re calling methods defined in Ruby. Fixing
this is pretty straightforward: I just flatten out any splatted lists
before passing them to the MethodBinder. (The method binder expects
things flattened down to Types—it doesn’t use ArgumentKinds).
I also implement Ruby’s send and send methods, which are used by a
lot of tests that John L. wants to enable. The implementations are very
simple & slow: they just create a new site with the correct
InvokeMemberAction and Invoke it. Not optimal, but it should be okay
until we integrate send directly into the method binder (Tomas is
already planning on refactoring the method binder).
297839: (tomat) Ruby Singletons and Method Lookup
Adds support for singleton classes and fixes method lookup. A singleton
class is a class that adds additional members to object instances. In
the following example, “class << x†defines a singleton class for
instance x.
x = Object.new
class << x
def foo
end
end
x.foo
Object.new.foo # error
Singleton class is a first class object, you can store it to a variable
and even create another singleton class for it. The class itself is also
defining some methods. Those are implemented as extension methods in
SingletonOps.
There are many peculiarities involved. I’ve also found at least 2 bugs
in MRI during my investigation.
I’ll write a spec and present the internals of singletons and method
lookup (as I understand them) on Friday’s IronRuby meeting.
The shelveset also implementation rescue expression, rescue statement
and symbol constructors, which are handy for testing.
297943: (dinov) Remove references to built-in functions
This change removes the remaining references to BuiltinFunction’s (and
friends). Instead these objects now implement IDynamicObject and use
the CallBinderHelper to produce a call rule.
Supporting changes:
-
Needed to split RuntimeHelpers into 2 parts (RuntimeHelpers and
BinderOps). This is due to a FxCop error where RuntimeHelpers is
getting too complex and including things from too many classes.
BinderOps are just the methods that are called from generated code to do
binding (calls, create instance, get member, etc…)
2. Moved StrongBox helper functions to CompilerHelpers as these
need to be accessed outside the binder.
3. Added support for passing isBinaryOperator/isReversedOperator on
CallBinderHelper as well as ability to set Instance. The hope here is
that these flags go away from the MethodBinder (they’re awfully Python
specific) and that we have a more consumable way to do call binding in
the future but this is a reasonable intermediate step. The
CallBinderHelper also no longer stores the instance type instead getting
it from the _instance expression.
297991: (dinov)
This moves the Microsoft.Scripting.Types namespace to IronPython. No
namespaces are renamed during this remove (that touches over 100 files,
so it’s next). Appropriate resource strings also get moved to
IronPython, and much of the DynamicType machinery is marked as internal.
This reduces a release build of MS.Scripting to 868,352 bytes from
958,464 saving 90,112 bytes or 9.4%.
This reduces a debug build of MS.Scripting to 995,328 bytes from
1,118,208 saving 122,880 bytes or 10.9%.
298014: (jomes) Case Expressions in Ruby
This change implements case expressions in Ruby. Case expressions look
like this:
case x
when a0, … aN, *splat0:
when b0, … bN, *splat1:
…
else:
end
It translates roughly into:
if (a0 === x || … || aN === x || ) {
body0
} else if ( … ) {
body1
} else {
bodyN
}
Where is:
result = false;
foreach (object obj in splat0) {
if (obj === x) { result = true; break; }
}
And of course, === is Ruby’s case equality method ïŠ
It’s pretty straightforward—the only tricky part is cracking open the
splatted array, if present.