# Unicode Character 'SUBSTITUTE' (U+001A) troubles

**URL:** <https://www.ruby-forum.com/t/unicode-character-substitute-u-001a-troubles/163137>\
**Category:** Ruby\
**Created:** [March 17, 2009, 5:09pm UTC](https://www.ruby-forum.com/t/unicode-character-substitute-u-001a-troubles/163137 "2009-03-17T17:09:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rlorimor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/96bed5/32.png) [@rlorimor](https://www.ruby-forum.com/u/rlorimor)\
**Post date:** [March 17, 2009, 5:09pm UTC](https://www.ruby-forum.com/t/unicode-character-substitute-u-001a-troubles/163137/1 "2009-03-17T17:09:08Z")

</div>

I’m implementing a version of the “Mini-AES” ([http://findarticles.com/](http://findarticles.com/)  
p/articles/mi\_qa3926/is\_200210/ai\_n9129484) in Ruby and have all of  
the bit shifts and stuff working…

The problem I’m having is with the file input. When ever the input  
file has the character U+001A ([Unicode](http://www.fileformat.info/info/unicode/)  
char/001a/index.htm), it stops reading in the line (and thus doesn’t  
complete the whole cipher text).

Basically, here is the code that I’m using to read the input:  
fin = File.open(ARGV[0], ‘r’)  
plainText = “”  
while (line = fin.gets)  
plainText += line  
end  
fin.close

Here is a sample line of cipher text:  
týH†a¤j{ÀìyMšyMÇÜ«àÃSÙ¿àé4þF&käj{Àì

if I read this line in, it would only read “týH†a¤j{ÀìyMšyM” and skip  
the rest.

---

<div class="post-metadata">

**Author:** ![rlorimor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/96bed5/32.png) [@rlorimor](https://www.ruby-forum.com/u/rlorimor)\
**Post date:** [March 17, 2009, 5:30pm UTC](https://www.ruby-forum.com/t/unicode-character-substitute-u-001a-troubles/163137/2 "2009-03-17T17:30:53Z")

</div>

On Mar 17, 12:01 pm, Robert L. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> fin = File.open(ARGV[0], ‘r’)  
> the rest.  
> I’m using Windows btw.

---

<div class="post-metadata">

**Author:** ![rlorimor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/96bed5/32.png) [@rlorimor](https://www.ruby-forum.com/u/rlorimor)\
**Post date:** [March 17, 2009, 6:36pm UTC](https://www.ruby-forum.com/t/unicode-character-substitute-u-001a-troubles/163137/3 "2009-03-17T18:36:20Z")

</div>

Robert L. wrote:

> On Mar 17, 12:01�pm, Robert L. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > fin = File.open(ARGV[0], ‘r’)  
> > the rest.  
> > I’m using Windows btw.

Open the file with “rb” instead of “r”.

---

<div class="post-metadata">

**Author:** ![rlorimor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/96bed5/32.png) [@rlorimor](https://www.ruby-forum.com/u/rlorimor)\
**Post date:** [March 17, 2009, 10:22pm UTC](https://www.ruby-forum.com/t/unicode-character-substitute-u-001a-troubles/163137/4 "2009-03-17T22:22:12Z")

</div>

On Mar 17, 1:33 pm, Tim H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Robert L. wrote:
> 
> > On Mar 17, 12:01 pm, Robert L. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> > 
> > > fin = File.open(ARGV[0], ‘r’)  
> > > the rest.  
> > > I’m using Windows btw.
> 
> ## Open the file with “rb” instead of “r”.
> 
> Posted viahttp://www.ruby-forum.com/.

I tried using rb and the output spits out 1/2 plain text and 1/2  
random Unicode.

Our project is only allowed to be run on Linux and it seems to work  
fine on there, so I’ll just leave the Windows problem alone unless  
someone suggests a good fix.