# Ruby, Unicode, and HTML Entities Problem **URL:** \ **Category:** Ruby\ **Created:** [September 26, 2010, 5:54pm UTC](https://www.ruby-forum.com/t/ruby-unicode-and-html-entities-problem/195601 "2010-09-26T17:54:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mrpeepers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/97f17d/32.png) [@mrpeepers](https://www.ruby-forum.com/u/mrpeepers)\ **Post date:** [September 26, 2010, 5:54pm UTC](https://www.ruby-forum.com/t/ruby-unicode-and-html-entities-problem/195601/1 "2010-09-26T17:54:07Z")
Hi all, Using Ruby (and REXML) to parse a directory full of HTML files mixed with Spanish but mostly English. For the most part, Ruby can correctly parse the HTML. That’s all fine and dandy, BUT, when there’s a unicode character NEAR an HTML entity, the parser bombs with: Missing end tag for ‘em’ (got “p”) Or Missing end tag for ‘em’ (got “em”) Or Missing end tag for ‘em’ (got “body”) etc, etc., etc… The errors are all related based on how/where the unicode characters are placed throughout the documents. Here’s an example of what the parser does NOT like and throws errors like what I posted above: _Ministerio Público de la Federación_ But just for fun, I changed the string to be this: _Ministerio Público de la Federaciónnn_ …and it loads/parses, no issue. What I have noticed is that when there’s a unicode character SO CLOSE to the begin/start tag =, i.e., _,_ , etc., etc… The parser bombs. But if I move the unicode character around FURTHER inside the tags, the parser loads the data without error. What’s going on? I can’t really post all the code due to company privacy. But here’s a listed of the requireds if it at all helps: require ‘rubygems’ require ‘sqlite3’ require ‘rexml/document’ require ‘rexml/streamlistener’ include REXML require ‘zlib’ require ‘CGI’ require ‘osx/cocoa’ include OSX And here’s some parsing if at all helps, I know, I know, probably not…: @title \<\< CGI::unescapeHTML(content.strip) title.gsub(’&’,’&’).gsub(’\<’,’\<’).gsub(’\>’,’\>’).gsub(’’’,’’’).gsub(’"’,’"’).gsub(’§’,‘§’) @body \<\< " #{attr\_name}="#{attr\_value}"" @body \<\< “\” ---
**Author:** ![mrpeepers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/97f17d/32.png) [@mrpeepers](https://www.ruby-forum.com/u/mrpeepers)\ **Post date:** [September 26, 2010, 8:52pm UTC](https://www.ruby-forum.com/t/ruby-unicode-and-html-entities-problem/195601/2 "2010-09-26T20:52:09Z")
On 26.09.2010 17:54, Mr Peepers wrote: > Here’s an example of what the parser does NOT like and throws errors > like what I posted above: > _Ministerio Público de la Federación_ > > But just for fun, I changed the string to be this: > _Ministerio Público de la Federaciónnn_ > > …and it loads/parses, no issue. > > I can’t really post all the code due to company privacy. No one wants your company data, but above you pasted a problematic snippet. Based on this, can’t you create a minimal test case from it? I was having similar problems with Nokogiri until I figured I need to set encoding manually because the automatic detection failed for some reason. But it’s premature to suggest anything without a smaller/real test case. - Markus ---
**Author:** ![mrpeepers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/97f17d/32.png) [@mrpeepers](https://www.ruby-forum.com/u/mrpeepers)\ **Post date:** [September 26, 2010, 9:08pm UTC](https://www.ruby-forum.com/t/ruby-unicode-and-html-entities-problem/195601/3 "2010-09-26T21:08:14Z")
Ok, get the Ruby and a sample of the data attempting to be loaded here, [www.khourys.com/Archive.rar](http://www.khourys.com/Archive.rar). Regarding 150340.html, as just an example. I’ll walk you through my troubleshooting quickly. 1. If I replace this text (where it fails): _Ministerio Público de la Federación_ with this: _Ministerio Público de la Federaciónnn_ FILE LOADS! 1. So I’m thinking, did this one o-acute get corrupted? Nope. But, if you remove this sentence: Whenever the Federal Public Ministry (_Ministerio Público de la Federación_) investigates the activities of organized crime members that deal with goods of illicit origin, the investigation is carried out with the assistance of the Secretariat of Finance and Public Credit (_Secretaría de la Hacienda y Crédito Público_). FILE LOADS! 1. I replaced this: _Ministerio Público de la Federación_ with this Ministerio Público de la Federación (removed the _'s)_ _FILE LOADS!_ _Now I’m thinking, hhhmmmm… (actually, WTF) Could the unicode character so close to the end HTML tag be causing the issue, potentially?_ _1. To test my theory, I’ll modify these lines together in the file:_ _Before: Ministerio Público de la Federación Secretaría de la Hacienda y Crédito Público_ _After Ministerio Público de la Federaciónnnn ← this to prove it’s not JUST this line Secretaría de la Hacienda y Crédito Pú ← this to prove that when a unicode character is so close to an HTML entity - we’re screwed._ _FILE BOMBED!_ _Think we have the culprit?_ ---
**Author:** ![mrpeepers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/97f17d/32.png) [@mrpeepers](https://www.ruby-forum.com/u/mrpeepers)\ **Post date:** [September 26, 2010, 10:11pm UTC](https://www.ruby-forum.com/t/ruby-unicode-and-html-entities-problem/195601/4 "2010-09-26T22:11:46Z")
Hi, On 26.09.2010 21:08, Mr Peepers wrote: > Ok, get the Ruby and a sample of the data attempting to be loaded here, > [www.khourys.com/Archive.rar](http://www.khourys.com/Archive.rar). > > Regarding 150340.html, as just an example. Not exactly a small test case, the 12kb load.rb . I was more talking along the line of you providing a simple script to show a reproducible error. Anyway, I simply fired up rexml to do basic thing but it bombs immediately. Disclaimer: I never used rexml before: $ ruby -rrexml/document -e ‘REXML::Document.new File.new(“150340.html”)’ /home/mfischer/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/rexml/parseexception.rb:31:in `gsub': invalid byte sequence in UTF-8 (ArgumentError) from /home/mfischer/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/rexml/parseexception.rb:31:in `to\_s’ from /home/mfischer/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:95:in `message' from /home/mfischer/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:95:in `rescue in parse’ from /home/mfischer/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:20:in `parse' from /home/mfischer/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/rexml/document.rb:230:in `build’ from /home/mfischer/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/rexml/document.rb:43:in `initialize' from -e:1:in `new’ from -e:1:in `’ Anyway, the same using Nokogiri works for me: $ ruby -rnokogiri -e ‘puts Nokogiri::HTML( File.open(“150340.html”) )/“//p/em[3]”’ _Ministerio Público de la Federación_ Does that help you? If not, you should provide your _small_ rexml test case which bombs. HTH, - Markus ---
**Author:** ![mrpeepers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/97f17d/32.png) [@mrpeepers](https://www.ruby-forum.com/u/mrpeepers)\ **Post date:** [September 26, 2010, 10:14pm UTC](https://www.ruby-forum.com/t/ruby-unicode-and-html-entities-problem/195601/5 "2010-09-26T22:14:09Z")
Not really… Think we figured out the problem. UTF-8 is a variable length encoding. We seem to have a character in there that’s indicating further encoding but it’s not so it pukes. Somewhere along the line the encoding of the actual file got messed up. Need to close this topic.