# Code Review: TokenizerFixes

**URL:** <https://www.ruby-forum.com/t/code-review-tokenizerfixes/143161>\
**Category:** IronRuby\
**Created:** [July 15, 2008, 8:25pm UTC](https://www.ruby-forum.com/t/code-review-tokenizerfixes/143161 "2008-07-15T20:25:45Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![grujildo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/94ad74/32.png) [@grujildo](https://www.ruby-forum.com/u/grujildo)\
**Post date:** [July 15, 2008, 8:25pm UTC](https://www.ruby-forum.com/t/code-review-tokenizerfixes/143161/1 "2008-07-15T20:25:45Z")

</div>

tfpt review “/shelveset:TokenizerFixes;REDMOND\tomat”

Refactors string terminators in Ruby tokenizer a bit.  
Removes unnecesary calls to newtok() method that allocated a new  
StringBuilder per each token.  
Fixes handling of \e in strings.  
Removes uses of obsolete method TryStatementBuilder::SkipIf.  
Removes TokenizerBuffer’s dependency on SourceUnitReader. Moves the  
responsibility for line mapping to the tokenizer.

Tomas