# Libxml's SaxParser and UTF-8 problem

**URL:** <https://www.ruby-forum.com/t/libxmls-saxparser-and-utf-8-problem/87394>\
**Category:** Ruby\
**Created:** [March 2, 2007, 7:00am UTC](https://www.ruby-forum.com/t/libxmls-saxparser-and-utf-8-problem/87394 "2007-03-02T07:00:18Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tekhne](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/90ced4/32.png) [@tekhne](https://www.ruby-forum.com/u/tekhne)\
**Post date:** [March 2, 2007, 7:00am UTC](https://www.ruby-forum.com/t/libxmls-saxparser-and-utf-8-problem/87394/1 "2007-03-02T07:00:18Z")

</div>

I’ve written a small script to parse an xml doc with SaxParser and  
everything goes well until the parser encounters a Unicode character.  
For example, in the for the following snippet:

Name90’s Music

In case it doesn’t come through correctly, the “’” character above is an  
apostrophe, represented as \<80\>\<99\> when I view the xml with less.

When the on\_characters method is called for the string “90’s Music”, the  
buffer only contains “90”, with no error or warning being presented.  
After this is encountered parsing occurs normally; the first I saw of  
the bug was when I noticed some of my strings being truncated. Is there  
some setting of libxml or ruby that I’ve overlooked to cause this  
behavior?

---

<div class="post-metadata">

**Author:** ![tekhne](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/90ced4/32.png) [@tekhne](https://www.ruby-forum.com/u/tekhne)\
**Post date:** [March 5, 2007, 5:14am UTC](https://www.ruby-forum.com/t/libxmls-saxparser-and-utf-8-problem/87394/2 "2007-03-05T05:14:11Z")

</div>

Peter Higgins wrote:

> I’ve written a small script to parse an xml doc with SaxParser and  
> everything goes well until the parser encounters a Unicode character.  
> For example, in the for the following snippet:
> 
> Name90’s Music
> 
> In case it doesn’t come through correctly, the “’” character above is an  
> apostrophe, represented as \<80\>\<99\> when I view the xml with less.
> 
> When the on\_characters method is called for the string “90’s Music”, the  
> buffer only contains “90”, with no error or warning being presented.  
> After this is encountered parsing occurs normally; the first I saw of  
> the bug was when I noticed some of my strings being truncated. Is there  
> some setting of libxml or ruby that I’ve overlooked to cause this  
> behavior?

As part of researching the problem, I wrote a small test script with  
REXML looking for that particular string, and it returned the correct,  
full quote: “90’s Music”. It looks like this is a bug with libxml then,  
so I’ll post on their mailing list.

---

<div class="post-metadata">

**Author:** ![tekhne](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/90ced4/32.png) [@tekhne](https://www.ruby-forum.com/u/tekhne)\
**Post date:** [March 7, 2007, 3:06pm UTC](https://www.ruby-forum.com/t/libxmls-saxparser-and-utf-8-problem/87394/3 "2007-03-07T15:06:32Z")

</div>

Peter Higgins wrote:

> I’ve written a small script to parse an xml doc with SaxParser and  
> everything goes well until the parser encounters a Unicode character.  
> For example, in the for the following snippet:
> 
> Name90’s Music
> 
> In case it doesn’t come through correctly, the “’” character above is an  
> apostrophe, represented as \<80\>\<99\> when I view the xml with less.
> 
> When the on\_characters method is called for the string “90’s Music”, the  
> buffer only contains “90”, with no error or warning being presented.  
> After this is encountered parsing occurs normally; the first I saw of  
> the bug was when I noticed some of my strings being truncated. Is there  
> some setting of libxml or ruby that I’ve overlooked to cause this  
> behavior?

Any chance the quote is passed to another call to on\_characters? I do  
believe SAX does not always return all the content of a tag in one call  
to the handler, but sometimes calls the handler several times and you  
have to put it all together yourself.

Of course it could be Wuby unable to handle the UTF8.

Jenda