tfpt review “/shelveset:TokenizerFixes;REDMOND\tomat”
Refactors string terminators in Ruby tokenizer a bit.
Removes unnecesary calls to newtok() method that allocated a new
StringBuilder per each token.
Fixes handling of \e in strings.
Removes uses of obsolete method TryStatementBuilder::SkipIf.
Removes TokenizerBuffer’s dependency on SourceUnitReader. Moves the
responsibility for line mapping to the tokenizer.
Tomas