# Clipboard problem with utf-8 and ascii-8bit

**URL:** <https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291>\
**Category:** Ruby-Gnome 2\
**Created:** [February 11, 2010, 9:34am UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291 "2010-02-11T09:34:46Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 11, 2010, 9:34am UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/1 "2010-02-11T09:34:46Z")

</div>

Hi,

I need to count the number of characters entered, as example I used the  
“clipboard.rb” that comes in gtk-demo

If we introduce utf-8 characters (for example the character Â¿) in the  
first “Gtk::Entry”, to paste the contents in the second “Gtk::Entry” and  
look what the encoding always see that it is ASCII-8BIT, if we count the  
number of characters in the string, ruby us back in this case 2.  
Contrary if to paste content we make a “force\_encoding(“utf-8”)” and  
recounting again, ruby return 1.

The question: is there any parameter when we read the contents of  
“Gtk::Entry” so that by default will not detect the characters as  
ASCII-8BIT?

Thanks

Rafael

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 11, 2010, 11:24am UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/2 "2010-02-11T11:24:20Z")

</div>

If I remember correctly Ruby 1.8 stores strings as bytes. If you  
have a multibyte encoding like Unicode, the size isn’t counted  
as characters. It is counted as bytes. So when you store  
something like Â¿ it takes 2 bytes and the length is 2.

In order to count the characters you need to know the encoding  
of the string. There several different ways to find the length  
using different classes (all of which I have forgotten about – sorry).  
But usually you want to index into the string by character.  
You can’t really do this with Unicode strings. Instead I usually  
split the string into an array like this:

```
   TO_A_RE = Regexp.new('\s*',nil,'U')

   string = "ã“ã‚“ã«ã¡ã‚"
   a = string.split(TO_A_RE)
   size = a.size

```

The TO\_A\_RE regexp is set for unicode (the ‘U’ option). You  
can set it to other values for other encodings.

Now why is your GTK:Entry getting 8bit ASCII characters?  
My guess is that the locale you are using is not unicode,  
but actually 8bit ASCII. I don’t know what platform you  
are using, but if you are on Linux you could use  
es\_ES.utf8 (guessing that you are spanish???). That  
should fix the problem.

Note that Ruby 1.9 is completely different. If I understand  
correctly, the encoding is stored with the string and you  
can actually index characters in multi-byte strings. But  
I haven’t used it yet.

I hope that helps. Using different string encodings in  
Ruby 1.8 is considerably harder than it should be.

```
      MikeC
```

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 11, 2010, 11:46am UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/3 "2010-02-11T11:46:36Z")

</div>

## Sorry for not giving enough information about my environment:

## ruby 1.9.1p376 (2009-12-07 revision 26041) [i686-linux] LANG=es\_ES.UTF-8 LC\_COLLATE=es\_ES.UTF-8

The characters display correctly in GTK application, but the problem is  
to check with Ruby which is the encoding of the string:  
string = “Â¿”  
string.encoding -\> this always returns ASCII-8BIT instead of utf-8  
If we count the characters:  
string.length -\> 2 but should be 1

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 11, 2010, 2:00pm UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/4 "2010-02-11T14:00:38Z")

</div>

2010/2/11 Rg Rg [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> ## Sorry for not giving enough information about my environment:
> 
> ## Â ruby 1.9.1p376 (2009-12-07 revision 26041) [i686-linux] Â LANG=es\_ES.UTF-8 Â LC\_COLLATE=es\_ES.UTF-8

I don’t have this behavior.

#$ ruby -v  
ruby 1.9.1p376 (2009-12-07 revision 26041) [x86\_64-linux]  
#$ cat encoding.rb  
str = “Ã‰”  
puts str.encoding  
puts str.size  
puts str.length  
#$ ruby encoding.rb  
encoding.rb:1: invalid multibyte char (US-ASCII)  
encoding.rb:1: invalid multibyte char (US-ASCII)  
#$ ruby -Ku encoding.rb  
UTF-8  
1  
1  
#$

If you don’t want to specify -Ku, you can add encoding: utf-8 to the  
first two lines, see  
[http://blog.nuclearsquid.com/writings/ruby-1-9-encodings](http://blog.nuclearsquid.com/writings/ruby-1-9-encodings)

What you describe is the behavior of 1.8, not 1.9. Double check that.

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 11, 2010, 4:33pm UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/5 "2010-02-11T16:33:43Z")

</div>

Simon A. wrote:

> 2010/2/11 Rg Rg [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):
> 
> > ## Sorry for not giving enough information about my environment:
> > 
> > ## Â ruby 1.9.1p376 (2009-12-07 revision 26041) [i686-linux] Â LANG=es\_ES.UTF-8 Â LC\_COLLATE=es\_ES.UTF-8
> 
> I don’t have this behavior.
> 
> #$ ruby -v  
> ruby 1.9.1p376 (2009-12-07 revision 26041) [x86\_64-linux]  
> #$ cat encoding.rb  
> str = “Ã‰”  
> puts str.encoding  
> puts str.size  
> puts str.length  
> #$ ruby encoding.rb  
> encoding.rb:1: invalid multibyte char (US-ASCII)  
> encoding.rb:1: invalid multibyte char (US-ASCII)  
> #$ ruby -Ku encoding.rb  
> UTF-8  
> 1  
> 1  
> #$
> 
> If you don’t want to specify -Ku, you can add encoding: utf-8 to the  
> first two lines, see  
> [http://blog.nuclearsquid.com/writings/ruby-1-9-encodings](http://blog.nuclearsquid.com/writings/ruby-1-9-encodings)
> 
> What you describe is the behavior of 1.8, not 1.9. Double check that.

I think that I haven’t explained well, this behavior only happens to me  
when reading the chain of GTK, if I do the same example that you the  
behavior is the same as you.

Taking as example the “./Gtk-demo/clipboard.rb” in the following  
section:

```
 button.signal_connect('clicked', entry) do |w, e|
    clipboard = e.get_clipboard(Gdk::Selection::CLIPBOARD)
    clipboard.request_text do |board, text, data|
      e.text = text # The text utf-8 is displayed correctly
    end
 end

```

If we change the above code:  
“e.text = text” will exchange it for “e.text = text.encoding.to\_s”, this  
result in → ASCII-8BIT  
and if we show the length of the string the result is incorrect because  
it treats the string as utf-8

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 12, 2010, 10:46am UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/6 "2010-02-12T10:46:53Z")

</div>

2010/2/11 Rg Rg [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> I think that I haven’t explained well, this behavior only happens to me  
> when reading the chain of GTK, if I do the same example that you the  
> behavior is the same as you.

Ah, yes, I tested a little with a GTK::Entry and it gives back a  
string considered ASCII 8BIT.

Definately a bug somewhere, but dunno if it is ruby or ruby-gnome2.

I tried to look at the sources, but didn’t understand where the getter  
was defined.

Let’s hope Kou can have a look at it, or give directions where to look.

Simon

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 12, 2010, 1:51pm UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/7 "2010-02-12T13:51:49Z")

</div>

Hi,

In [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
“Re: [ruby-gnome2-devel-en] Clipboard problem with utf-8 and  
ascii-8bit” on Fri, 12 Feb 2010 10:40:10 +0100,  
Simon A. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> I tried to look at the sources, but didn’t understand where the getter  
> was defined.
> 
> Let’s hope Kou can have a look at it, or give directions where to look.

Gtk::Clipbord#request\_text passes UTF-8 encoding text to  
callback in trunk. We need more work around  
encoding. e.g. we should set UTF-8 encoding to a text  
returned by Gtk::Entry#text.

I’ve add RBG\_STRING\_SET\_UTF8\_ENCODING() macro. Could someone  
UTF-8 encoding set work in trunk? If someone sends a patch,  
please someone reviews and commit it into trunk.

## Thanks,

kou

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 11, 2010, 12:55pm UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/8 "2010-02-11T12:55:04Z")

</div>

On 11 February 2010 19:46, Rg Rg [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> The characters display correctly in GTK application, but the problem is  
> to check with Ruby which is the encoding of the string:  
> Â string = “Â¿”  
> Â string.encoding → this always returns ASCII-8BIT instead of utf-8  
> If we count the characters:  
> Â string.length → 2 but should be 1

Ah… It looks like the encoding coding is being detected wrong.  
This will be a Ruby 1.9 problem, not a GTK problem. I’m  
afraid I don’t know enough to help…

```
   MikeC

```

---

<div class="post-metadata">

**Author:** ![rafa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e495f1/32.png) [@rafa](https://www.ruby-forum.com/u/rafa)\
**Post date:** [February 15, 2010, 12:53pm UTC](https://www.ruby-forum.com/t/clipboard-problem-with-utf-8-and-ascii-8bit/183291/9 "2010-02-15T12:53:25Z")

</div>

Kouhei S. wrote:

> Hi,
> 
> In [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> “Re: [ruby-gnome2-devel-en] Clipboard problem with utf-8 and  
> ascii-8bit” on Fri, 12 Feb 2010 10:40:10 +0100,  
> Simon A. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > I tried to look at the sources, but didn’t understand where the getter  
> > was defined.
> > 
> > Let’s hope Kou can have a look at it, or give directions where to look.
> 
> Gtk::Clipbord#request\_text passes UTF-8 encoding text to  
> callback in trunk. We need more work around  
> encoding. e.g. we should set UTF-8 encoding to a text  
> returned by Gtk::Entry#text.
> 
> I’ve add RBG\_STRING\_SET\_UTF8\_ENCODING() macro. Could someone  
> UTF-8 encoding set work in trunk? If someone sends a patch,  
> please someone reviews and commit it into trunk.
> 
> ## Thanks,
> 
> kou

As I need to work in “GTK::Entry#text” I have patched the file  
rbgtkentry.c to use your macro.  
I haven’t read much code ruby-gnome, so the patch is probably not quite  
correct, but at least it work for me.

* * *

— gtk/src/rbgtkentry.c.old 2010-02-15 12:32:01.000000000 +0100  
+++ gtk/src/rbgtkentry.c 2010-02-15 12:31:29.000000000 +0100  
@@ -135,6 +135,18 @@  
}  
#endif

+static VALUE  
+entry\_request\_text(self)

- VALUE self;  
+{
- VALUE vtext = Qnil;
- const gchar \*text;
- text = gtk\_entry\_get\_text(\_SELF(self));
- vtext = CSTR2RVAL(text);
- RBG\_STRING\_SET\_UTF8\_ENCODING(vtext) ;
- return vtext;  
+}
- 

void  
Init\_gtk\_entry()  
{  
@@ -149,6 +161,7 @@  
#endif  
rb\_define\_method(gEntry, “layout\_index\_to\_text\_index”,  
entry\_layout\_index\_to\_text\_index, 1);  
rb\_define\_method(gEntry, “text\_index\_to\_layout\_index”,  
entry\_text\_index\_to\_layout\_index, 1);

- rb\_define\_method(gEntry, “text”, entry\_request\_text, 0);

#if GTK\_CHECK\_VERSION(2, 12, 0)  
rb\_define\_method(gEntry, “cursor\_hadjustment”,

* * *

Thanks
