Code Review: HostingSeparation1

tfpt review /shelveset:HostingSeparation1;REDMOND\tomat

DLR, HAPI, Nessie and all languages.

A step towards full HAPI/DLR separation and fixes of some
inconsistencies with the spec.

Details:

Splits TokenCategorizer class into two classes: TokenizerService
(language API) and TokenCategorizer (HAPI). TokenizerService is what
languages implement, it could be either a base class for language
tokenizers or it could delegate to the language tokenizer - that’s up to
the language.

Moves TokenInfo, TokenCategory, TokenTriggers out of Hosting namespace -
these are shared concepts among HAPI, DLR and languages.
Moves and renames EngineTextContentProvider to
Microsoft.Scripting.LanguageBoundTextContentProvider (it was an internal
helper class in Hosting used from M.S.).
Removes Hosting.CodeSenseProvider - it’s not used.

Removes differences from specification:

  •      CompiledCode - renames, adds and remissing ones, removes 
    

CreateOptimizedScope. Adds parameter-less Execute() that executes in an
optimized scope. We need to revisit optimized scopes, but this is needed
for now.

  •      ScriptCode - adds parameter-less Run() that executes the code 
    

in an optimized scope. Consolidates optimized scope execution from
SourceUnit and CompiledCode.

  •      ScriptRuntime - adds CreateScope, Operations, 
    

CreateOperations. Removes Platform and DomainManager properties, they
are duplicating existing members.

  •      ScriptDomainManager - makes ExecuteSourceUnit private.
    
  •      SourceUnit, ScriptSource - removes ToString(). It was there 
    

just to display the path in debug watches, which is better done via
DebuggerDispay attribute.

Fixes bug in
ScriptDomainManager.LanguageRegistration.LoadLanguageContext - it wasn’t
thread-safe. Two different language contexts could have escaped the
method.

Adds missing methods to HostingHelpers. These are supposed to be used by
languages/frameworks that implement their hosting API on top of DLR
HAPI.

Tomas