# Detect non-ascii substrings in a file

**URL:** <https://www.ruby-forum.com/t/detect-non-ascii-substrings-in-a-file/140843>\
**Category:** Ruby\
**Created:** [June 19, 2008, 4:45am UTC](https://www.ruby-forum.com/t/detect-non-ascii-substrings-in-a-file/140843 "2008-06-19T04:45:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![killy971](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/13edae/32.png) [@killy971](https://www.ruby-forum.com/u/killy971)\
**Post date:** [June 19, 2008, 4:45am UTC](https://www.ruby-forum.com/t/detect-non-ascii-substrings-in-a-file/140843/1 "2008-06-19T04:45:22Z")

</div>

I have files encoded in Shift\_JIS, that mainly contains JSP source  
code (ascii), but sometimes also contains strings that are non-ascii  
(japanese words).

So, I would like to know if there is a way with ruby to :

- detect files containing something else than ascii,
- extract the non-ascii strings thare were found.

Thank you !

---

<div class="post-metadata">

**Author:** ![killy971](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/13edae/32.png) [@killy971](https://www.ruby-forum.com/u/killy971)\
**Post date:** [June 19, 2008, 12:35pm UTC](https://www.ruby-forum.com/t/detect-non-ascii-substrings-in-a-file/140843/2 "2008-06-19T12:35:40Z")

</div>

Any character that has the top bit clear is potentially valid ascii,  
though if you take away the non printing characters there’s an  
additional exlusion set.  
According to [Shift JIS - Wikipedia](http://en.wikipedia.org/wiki/Shift-JIS)

Testing for character codes with the top bit set should indicate  
either katakana or double byte characters. See the chart there for  
which ranges are double byte, which are single and which are not legal.

RF