gsub not works for me when replace ‘DBCS’(double byte character set)
character, using last version ruby 1.8.6
when “strºº×Öend”.gsub(/ºº×Ö/,“hanzi”),
output still is: strºº×Öend , but not strhanziend which I want to
get.
Searched web two whole night with no clue found.
Anyone can help are much appreciated, need got it work very urgent.
thank you!
Ryan S. wrote:
gsub not works for me when replace ‘DBCS’(double byte character set)
character, using last version ruby 1.8.6
Maybe try 1.9?
-r
On Thu, Jan 28, 2010 at 5:05 PM, Ryan S. [email protected]
wrote:
thank you!
Mixing encoding schemes is hell in almost any context, and Ruby is no
exception.
Until you have complete control in your program over all encoding inputs
you
are
going to fail.
If your input is coming from the shell environment or standard in the
text
can be
in the system encoding, regardless of what encoding you specify in Ruby.
It is preferable to use unicode (UTF-8) in any operation where you are
processing
multilingual text. Failing that there is the Iconv library which you can
use
to convert
between encoding schemes.
Note that ‘double-byte encoding scheme’ is an utterly useless term for
practical encoding
purposes. Its a gross simplification of what is going on, and especially
so
with Han
character sets. To do any practical work with non-unicode, multi-byte
character sets, you
have to know the encoding scheme.
With ruby 1.9.2dev (2010-01-14 trunk 26319) [x86_64-darwin10.2.0]
“strhanziend”
It works fine. You need to set encoding with “# encoding: utf-8” at
the top of the file.
In fact, it will complain if not in 1.9.2
Ruby 1.8.6 is kind of outdated, but at least I think it works with
“-Ku”.
2010/1/28 Ryan S. [email protected]:
Benoit D. wrote:
With ruby 1.9.2dev (2010-01-14 trunk 26319) [x86_64-darwin10.2.0]
“strhanziend”It works fine. You need to set encoding with “# encoding: utf-8” at
the top of the file.
In fact, it will complain if not in 1.9.2Ruby 1.8.6 is kind of outdated, but at least I think it works with
“-Ku”.
Ruby 1.8 isn’t outdated. It just doesn’t handle multibyte text that
well.
2010/1/28 Ryan S. [email protected]:
Marnen Laibow-Koser
http://www.marnen.org
[email protected]
On Thu, Jan 28, 2010 at 5:58 PM, Roger P. [email protected]
wrote:
Ryan S. wrote:
gsub not works for me when replace ‘DBCS’(double byte character set)
character, using last version ruby 1.8.6Maybe try 1.9?
-r
Its an option, but a better strategy is to outline what you are trying
to
achieve.
You should state where the Chinese text is coming from (database, text
file,
binary file (e.g. excel), shell, UI, stdin), what encoding it is in (or
what
encoding you
think it is in), and share the source of your ruby code (just the
relevant
bits, like
your KCODE attribute, Iconv usage etc.
On Thu, Jan 28, 2010 at 10:05 AM, Ryan S. [email protected]
wrote:
gsub not works for me when replace ‘DBCS’(double byte character set)
character, Â using last version ruby 1.8.6
The term ‘DBCS’ is not sufficient to determine what the encoding of
your input character set is. That could mean any of a large number of
mutually incompatible character encodings: UTF-16, Big5, Shift-JIS
(for Japanese), and EUC-CN are only a few of the possibilities. Before
you can even begin to process any text you get from anywhere, it’s
just a stream of bytes with no meaning until you know what its
encoding is. You can’t perform any kind of useful text processing
without knowing the encoding.
Once you do know the encoding, you should probably convert it to UTF-8
using the Iconv library, and you should probably be able to do
something useful with the text.
Also, make sure that $KCODE is ‘u’. Otherwise Unicode text processing
will not work.
On Thu, Jan 28, 2010 at 11:13 AM, Ryan S. [email protected]
wrote:
convert the string to utf-8 encoding, message is complain the char is
invalid.
GB2312? From a web page that probably means EUC-CN, so you have to
convert from that encoding into UTF-8.
I parse a webpage which encoded in gb2312, using Watir to get the
context of the page title, and want to replace the ‘chinese character’
in title with english words.
When puts title which watir get, the chinese character displaied as
corrupt code there (under windows cmd,code page using cp936, display
works normal when change code page to utf-8). But I think cmd’s code
page just display setting does not related with what I need (replace
chinese char). I did not know if string I get by Watir is also in
‘gb2312’ encoding or something others, the fact is fail happen when
convert the string to utf-8 encoding, message is complain the char is
invalid.
totally no idea what need to do.
Richard C. wrote:
On Thu, Jan 28, 2010 at 5:05 PM, Ryan S. [email protected]
wrote:thank you!
Mixing encoding schemes is hell in almost any context, and Ruby is no
exception.
Until you have complete control in your program over all encoding inputs
you
are
going to fail.If your input is coming from the shell environment or standard in the
text
can be
in the system encoding, regardless of what encoding you specify in Ruby.It is preferable to use unicode (UTF-8) in any operation where you are
processing
multilingual text. Failing that there is the Iconv library which you can
use
to convert
between encoding schemes.Note that ‘double-byte encoding scheme’ is an utterly useless term for
practical encoding
purposes. Its a gross simplification of what is going on, and especially
so
with Han
character sets. To do any practical work with non-unicode, multi-byte
character sets, you
have to know the encoding scheme.
On Thu, Jan 28, 2010 at 6:13 PM, Ryan S. [email protected]
wrote:
convert the string to utf-8 encoding, message is complain the char is
invalid.totally no idea what need to do.
I had a sneaking suspicion that your task was Watir related.
First off the solution to this is hard - I have done some Watir tasks
that
involved
processing international text (even in UTF-8) and there are some nasty
gotchas.
It is not Watir’s fault or anything, but there are 2 areas which really
annoy:
puts ‘International text’ in the windows cmd shell is almost useless as
a
way of
debugging the problem. Windows CMD shell takes perverse satisfaction in
ignoring
any encoding you might have set your ruby code to work in. The platform
codepage
is what Windows does all its work in.
Watir itself is implemented on top of Win32OLE which is yet another area
where
the platform encoding can interfere against your wishes.
You might get better luck posting these questions on the watir mailing
list.
Its been a
while for me, but useful workarounds to common and uncommon gotchas like
this
are discussed and answered there.
Also check the Watir wiki
http://wiki.openqa.org/display/WTR/Project+Home (odd
its currently down for me).
Testing of encoded text did come up a lot when I checked it last, and
they
heavily document their
workarounds.
Lastly check out the JRuby equivalent to Watir: Celerity. It is API
compatible with Watir so your script will
probably still work with only minor modifications. It runs over a java
http
library so you have good encoding
processing if you need it, and more importantly there are less entry
points
for the Windows platform
encoding to impose itself. Note: Celerity has no visual component.
FireWatir
(FF) or ChromeWatir may
also be useful to you for similar reasons.
Here is my code:
============File Main.rb ================
$KCODE = “U”
$LOAD_PATH << File.dirname(FILE).to_s+“/lib”
require ‘iv’
require ‘misc’
require ‘DBI’
require “watir”
url=“http://www.baidu.com”
ie = Watir::IE.new
ie.goto url
t=ie.title
p t
t=t.toutf8()
Misc.cn2en(t)
============File misc.rb ================
module Misc
require “rtranslate”
def Misc.cn2en(longMixTxt)
#s=“\344\270\255\345\233\275\344\270\255\345\233\275china,\344\270\255\345\233\275,china2.”
s=longMixTxt
cnarray=[]
enarray=[]
cnarray=s.scan(/[^[:alnum:][:punct:] ]+/); p cnarray
cnarray.each{ |x| puts x; enarray << Misc.googletranslatetoen(x) }
enarray.each{ |x| i=enarray.index(x); p i.to_s; p cnarray[i] ; p x; p
s.include?(cnarray[i]); s=s.gsub(/cnarray[i]/, x.to_s) ; p s ;}
return s
end #Misc.cn2en(longMixTxt)
==========output =============
C:\ruby_project>main.rb -b
“é§æƒ§å®³æ¶“€涓嬶ç´æµ£çŠ²æ°¨éラ亾”
[“é§æƒ§å®³æ¶“€涓嬶ç´æµ£çŠ²æ°¨éラ亾”]
é§æƒ§å®³æ¶“€涓嬶ç´æµ£çŠ²æ°¨éラ亾
start translate cn->en
“0”
“é§æƒ§å®³æ¶“€涓嬶ç´æµ£çŠ²æ°¨éラ亾”
“Baidu about, you know”
true
“é§æƒ§å®³æ¶“€涓嬶ç´æµ£çŠ²æ°¨éラ亾”
==== THANKS EVERYBODY WHO CARE THIS POST====
I found my mistake is using incorrect reg match, it works after change
to
s=s.gsub(/#{cnarray[i]}/, x.to_s)
thanks everyone!
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs