# Regexp for matching UTF-8 characters without close tag **URL:** \ **Category:** Ruby\ **Created:** [January 5, 2008, 10:25am UTC](https://www.ruby-forum.com/t/regexp-for-matching-utf-8-characters-without-close-tag/125093 "2008-01-05T10:25:30Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jesseproudman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/e274bd/32.png) [@jesseproudman](https://www.ruby-forum.com/u/jesseproudman)\ **Post date:** [January 5, 2008, 10:25am UTC](https://www.ruby-forum.com/t/regexp-for-matching-utf-8-characters-without-close-tag/125093/1 "2008-01-05T10:25:30Z")
Hi all, Im trying to solve this problem: string = “\302” TEXT\_PATTERN = /\A([^\<]\*)/um text\_data = string.match(TEXT\_PATTERN).to\_s =\> “\302” As you can see, the regular expression incorrectly captures not only the text part but also the closing tag, whereas what is supposed to be captured is just “\302”. This problem is actually part of the REXML::Source#match method ([http://www.germane-software.com/projects/rexml/browser/trunk/src/](http://www.germane-software.com/projects/rexml/browser/trunk/src/) rexml/source.rb?rev=1266#L104) and causes REXML to parse UTF-8 documents incorrectly sometimes. Any ideas why the pattern matching doesnt work? I dont see anything wrong with the regular expression. Although, Im not sure what the \A character class is for. Best regards, Jesse ---
**Author:** ![jesseproudman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/e274bd/32.png) [@jesseproudman](https://www.ruby-forum.com/u/jesseproudman)\ **Post date:** [January 5, 2008, 3:45pm UTC](https://www.ruby-forum.com/t/regexp-for-matching-utf-8-characters-without-close-tag/125093/2 "2008-01-05T15:45:43Z")
Jesse P. wrote: > Hi all, > > Im trying to solve this problem: > > string = “\302” > TEXT\_PATTERN = /\A([^\<]\*)/um > text\_data = string.match(TEXT\_PATTERN).to\_s > =\> “\302” A solution may be : require ‘iconv’ string = “\302” # string isn’t in utf-8 \302 in utf is \303\202 string = Iconv.conv(“UTF-8”,“ISO-8859-1”,string) TEXT\_PATTERN = /\A([^\<]\*)/um text\_data = string.match(TEXT\_PATTERN).to\_s text\_data = Iconv.conv(“ISO-8859-1”,“UTF-8”,text\_data) puts text\_data \A -\> beginig of line ---
**Author:** ![jesseproudman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/e274bd/32.png) [@jesseproudman](https://www.ruby-forum.com/u/jesseproudman)\ **Post date:** [January 5, 2008, 4:40pm UTC](https://www.ruby-forum.com/t/regexp-for-matching-utf-8-characters-without-close-tag/125093/3 "2008-01-05T16:40:25Z")
Hi Tiziano, My apologies. It seems that I have oversimplied the problem due to my lack of understanding for UTF-8. The actual string is an xml file I obtained from flickr at [http://api.flickr.com/services/rest/?method=flickr.people.getInfo&api\_key=44dfd94b104d544f8f80b521a70429e3&user\_id=55669962%40N00&api\_sig=6a39aab2fb665e24d2b6e1cef9d0be27:](http://api.flickr.com/services/rest/?method=flickr.people.getInfo&api_key=44dfd94b104d544f8f80b521a70429e3&user_id=55669962%40N00&api_sig=6a39aab2fb665e24d2b6e1cef9d0be27:) An excerpt is as follows: \ (\_.·´¯`·â"¢â(tm) â'ª Emirates Wizard â'ªâ(tm) â"¢Â·Â 0b88a178b28c40ff81d44c5ae475438abec2009c http://www.flickr.com/photos/emirates\_wizard/ http://www.flickr.com/people/emirates\_wizard/ http://m.flickr.com/photostream.gne?id=5467956 2006-07-16 15:22:42 1162548449 36 The part of the xml that is causing the problem is in the tag which if in ruby, is represented with octals as: “(\_.\302\267\302\264\302\257`\302\267\342\204\242\342\231 \342\202\252 Emirates Wizard \342\202\252\342\231 \342\204\242\302\267\302” Note that the XML says that the contents are in UTF-8. So when I use REXML to process this xml, after it processes the the tag “”, it is left with string = “(\_.\302\267\302\264\302\257`\302\267\342\204\242\342\231 \342\202\252 Emirates Wizard \342\202\252\342\231 \342\204\242\302\267\302” I just checked and if I match this string with TEXT\_PATTERN = / \A([^\<]_)/um, I get the text and also the close tag. TEXT\_PATTERN = /\A([^\<]_)/um text\_data = string.match(TEXT\_PATTERN).to\_s =\> “(\_.\302\267\302\264\302\257`\302\267\342\204\242\342\231 \342\202\252 Emirates Wizard \342\202\252\342\231 \342\204\242\302\267\302” Assuming that the XML has some malformed data (some are not actually UTF-8), is there anyway that I can process the xml as it is and only treat the malformed data differently? (e.g. you mentioned that the \302 character is not a UTF-8 character) Best regards, Jesse ---
**Author:** ![jesseproudman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/e274bd/32.png) [@jesseproudman](https://www.ruby-forum.com/u/jesseproudman)\ **Post date:** [January 5, 2008, 8:56pm UTC](https://www.ruby-forum.com/t/regexp-for-matching-utf-8-characters-without-close-tag/125093/4 "2008-01-05T20:56:31Z")
Jesse P. wrote: > Hi Tiziano, > > My apologies. It seems that I have oversimplied the problem due to my > lack of understanding for UTF-8. > > The actual string is an xml file I obtained from flickr at > [http://api.flickr.com/services/rest/?method=flickr.people.getInfo&api\_key=44dfd94b104d544f8f80b521a70429e3&user\_id=55669962%40N00&api\_sig=6a39aab2fb665e24d2b6e1cef9d0be27:](http://api.flickr.com/services/rest/?method=flickr.people.getInfo&api_key=44dfd94b104d544f8f80b521a70429e3&user_id=55669962%40N00&api_sig=6a39aab2fb665e24d2b6e1cef9d0be27:) You have a broken utf-8 docoment (a read the Matz reply) I see two solutions: a) require ‘rexml/document’ require ‘iconv’ data = your xml document data = Iconv.conv(“UTF-8”,“ISO-8859-1”,data) doc = REXML::Document.new(data) you must convert the data of doc from utf-8 to iso befor use it username = Iconv.conv(“ISO-8859-1”,“UTF-8”,doc. ) b) change the encoding of the xml require ‘rexml/document’ data = your xml document data = data.gsub(/encoding=“utf-8”/i, ‘encoding=“iso-8859-1”’) doc = REXML::Document.new(data) Anyway a) and b) don’t work if the document contains valid utf-8 chars not in ascii-7 (for example latin letters è, ò, etc.) ---
**Author:** ![jesseproudman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/e274bd/32.png) [@jesseproudman](https://www.ruby-forum.com/u/jesseproudman)\ **Post date:** [January 6, 2008, 2:30pm UTC](https://www.ruby-forum.com/t/regexp-for-matching-utf-8-characters-without-close-tag/125093/5 "2008-01-06T14:30:42Z")
Thanks Tiziano 🙂