# Gsub: invalid byte sequence in US-ASCII **URL:** \ **Category:** Ruby\ **Created:** [June 15, 2010, 11:29am UTC](https://www.ruby-forum.com/t/gsub-invalid-byte-sequence-in-us-ascii/190107 "2010-06-15T11:29:01Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sentinel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/fbc32d/32.png) [@sentinel](https://www.ruby-forum.com/u/sentinel)\ **Post date:** [June 15, 2010, 11:29am UTC](https://www.ruby-forum.com/t/gsub-invalid-byte-sequence-in-us-ascii/190107/1 "2010-06-15T11:29:01Z")
I download the page [http://www.ruby-forum.com/forum/4](http://www.ruby-forum.com/forum/4) using wget. Then i cat the file and pipe to gsub. I get: -e:1:in `gsub’: invalid byte sequence in US-ASCII (ArgumentError) wget -q -k -O index11.html [http://www.ruby-forum.com/forum/4](http://www.ruby-forum.com/forum/4) cat index11.html | ruby -pe ‘gsub(/href=a/"/,"href="’${base}‘")’ \> ofile (The value of base is [http://www.ruby-forum.com/](http://www.ruby-forum.com/)) So what must i do so this command can run. It runs fine with another site. If i replace ruby with perl -pe ‘s|…|g’ that works fine. I actually run this in a loop with various URLS from cron. ---
**Author:** ![sentinel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/fbc32d/32.png) [@sentinel](https://www.ruby-forum.com/u/sentinel)\ **Post date:** [June 15, 2010, 5:56pm UTC](https://www.ruby-forum.com/t/gsub-invalid-byte-sequence-in-us-ascii/190107/2 "2010-06-15T17:56:19Z")
R.. Kumar wrote: > If i replace ruby with perl -pe ‘s|…|g’ that works fine. Replacing ruby 1.9.x with ruby 1.8.x is just as effective, and I would recommend this for maintaining your sanity. I can only guess that the external encoding picked up from your platform’s environment is US-ASCII (are you using cygwin by any chance?) You probably need to set the external encoding to UTF-8 or BINARY for your regexp not to crash. Try adding -Ku or -Kn to your ruby command line. If you want to attempt to understand String encoding in ruby 1.9, then good luck to you. I tried, documented what I found here: > and gave up after about 200 rules. There is no official documentation. ---
**Author:** ![sentinel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/fbc32d/32.png) [@sentinel](https://www.ruby-forum.com/u/sentinel)\ **Post date:** [June 16, 2010, 6:20am UTC](https://www.ruby-forum.com/t/gsub-invalid-byte-sequence-in-us-ascii/190107/3 "2010-06-16T06:20:14Z")
Brian C. wrote: > R.. Kumar wrote: > > > If i replace ruby with perl -pe ‘s|…|g’ that works fine. > > Replacing ruby 1.9.x with ruby 1.8.x is just as effective, and I would > recommend this for maintaining your sanity. > > I can only guess that the external encoding picked up from your > platform’s environment is US-ASCII (are you using cygwin by any chance?) > > You probably need to set the external encoding to UTF-8 or BINARY for > your regexp not to crash. Try adding -Ku or -Kn to your ruby command > line. > > If you want to attempt to understand String encoding in ruby 1.9, then > good luck to you. I tried, documented what I found here: > [string19/string19.rb at master · candlerb/string19 · GitHub](http://github.com/candlerb/string19/blob/master/string19.rb) > and gave up after about 200 rules. There is no official documentation. 1. I have moved to 1.9 long back. Don’t want to move back. 2. I am on OSX. I think I had _probably_ (?) solved this issue on my previous laptop (PPC) – now I;ve migrated my user to a new machine (Snow Leopard). All my settings, should have moved. (I say this since I had commented out the perl line). LC\_ALL=en\_US.UTF-8 LC\_CTYPE=en\_US.UTF-8 LANG=C 1. Thanks for the link, i will read it. But NO, i have already read up enough a few months back, and do not have the energy to do it again :-(. Thanks for the tip on -Ku / -Kn ---
**Author:** ![sentinel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/fbc32d/32.png) [@sentinel](https://www.ruby-forum.com/u/sentinel)\ **Post date:** [June 16, 2010, 6:31am UTC](https://www.ruby-forum.com/t/gsub-invalid-byte-sequence-in-us-ascii/190107/4 "2010-06-16T06:31:49Z")
Brian C. wrote: > I can only guess that the external encoding picked up from your > platform’s environment is US-ASCII (are you using cygwin by any chance?) > > You probably need to set the external encoding to UTF-8 or BINARY for > your regexp not to crash. Try adding -Ku or -Kn to your ruby command > line. Ok, I’ve got it. The problem occured when the program was run by cron. My user setting is UTF and it ran fine in terminal. So now in the program itself I have set LC\_CTYPE and LC\_ALL to en\_US.UTF-8. Hopefully, it should work fine now. Thanks. ---
**Author:** ![sentinel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/fbc32d/32.png) [@sentinel](https://www.ruby-forum.com/u/sentinel)\ **Post date:** [June 17, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/gsub-invalid-byte-sequence-in-us-ascii/190107/5 "2010-06-17T16:22:09Z")
On 6/15/10, R.. Kumar [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > (The value of base is [http://www.ruby-forum.com/](http://www.ruby-forum.com/)) > > So what must i do so this command can run. It runs fine with another > site. > If i replace ruby with perl -pe ‘s|…|g’ that works fine. > > I actually run this in a loop with various URLS from cron. Handling this kind of thing right means tracking encodings right… which means you’d have to extract the encoding from the http session and then mark the input as that encoding in your ruby script… and then deal with the inevitable incompatible encoding errors that would crop up. It sounds to me, tho, like in this case what you have a just some hacky little scripts and it would be acceptable for them to be imperfect. So, in that case, I suggest trying to set the encoding for your source file(s) to BINARY. That’s a hack, but it ought to be effective. Alternately, you could drop back to the 1.8 interpreter, like Brian suggests, which more or less uses BINARY as the default source encoding. ---
**Author:** ![sentinel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/fbc32d/32.png) [@sentinel](https://www.ruby-forum.com/u/sentinel)\ **Post date:** [June 17, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/gsub-invalid-byte-sequence-in-us-ascii/190107/6 "2010-06-17T16:22:14Z")
Caleb C. wrote: > your source file(s) to BINARY. That’s a hack, but it ought to be > effective. Additional info on the source, external, and internal encodings: [http://blog.grayproductions.net/articles/ruby\_19s\_three\_default\_encodings](http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings) For the OP, I’d expect `ruby -EBINARY ...` or `ruby -EASCII-8BIT ...` should work. Regards, Bill