# XML parser; maybe ruby is too slow?

**URL:** <https://www.ruby-forum.com/t/xml-parser-maybe-ruby-is-too-slow/114681>\
**Category:** Ruby\
**Created:** [September 17, 2007, 1:24pm UTC](https://www.ruby-forum.com/t/xml-parser-maybe-ruby-is-too-slow/114681 "2007-09-17T13:24:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nutsmuggler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/57b2e6/32.png) [@nutsmuggler](https://www.ruby-forum.com/u/nutsmuggler)\
**Post date:** [September 17, 2007, 1:24pm UTC](https://www.ruby-forum.com/t/xml-parser-maybe-ruby-is-too-slow/114681/1 "2007-09-17T13:24:05Z")

</div>

Hello folks.  
I managed to write a SGML parser with the hpricot library. As I  
explained in a previous thread, I just need to compare source and  
traget tags of translation memory files from IBM Translation manager.  
The script now runs effectively, but I realised that it cannot cope  
with large files; I tried to process TM file larger than 1MB and the  
script took ages to generate the output. Should I switch to a compiled  
language for this specific task?  
At any rate, here is the script, it’s very basic; please let me know  
if I did something wrong or if its slowness is a necessary drawback of  
ruby being interpreted. Cheers,  
Davide

#!/usr/local/bin/ruby  
require ‘rubygems’  
require ‘hpricot’

$pattern = “server”  
result = File.new(“result.html”, “w”)  
$stdout = result  
puts "\n

\n \n Ricerca di '#{$pattern}'\n body { } p { margin: 0px; } p.source { background: #FFFFCC; padding: 10px 5px 10px 5px; } p.target { background: #F8A271; padding: 10px 5px 10px 5px; } span.pattern { background: #B6B6B6; } \n \n" # per aprire lo stdin # doc = Hpricot.XML(STDIN)

doc = Hpricot.XML(open(“bch01aad006\_MEMORIA.EXP”))  
doc.search(“Source”).each do |item|  
if item.innerHTML =~ /#{$pattern}/  
highlightedSource = item.innerHTML.gsub(/#{$pattern}/, “#{$pattern}”)  
puts “

EN: #{highlightedSource}

\n”  
puts "

IT: #{item.next\_sibling.html}

\n  

* * *
"  
end  
end  
puts “”

---

<div class="post-metadata">

**Author:** ![nutsmuggler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/57b2e6/32.png) [@nutsmuggler](https://www.ruby-forum.com/u/nutsmuggler)\
**Post date:** [September 17, 2007, 1:24pm UTC](https://www.ruby-forum.com/t/xml-parser-maybe-ruby-is-too-slow/114681/2 "2007-09-17T13:24:37Z")

</div>

On 9/15/07, nutsmuggler [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> if I did something wrong or if its slowness is a necessary drawback of  
> ruby being interpreted. Cheers,  
> Davide

Davide,

This is not a result necessarily of ruby being slow. Hpricot is a DOM  
parser, and also (by default ) tries to fix up tags. This will parse  
the  
entire file to memory and build an internal tree structure out of it.  
The  
alternative is to use a SAX based or streaming parser. This is what  
happened in a part of Merb. The streaming parser, REXML, was much  
faster  
than Hpricot for the same job because it is, well a streaming parser  
instead  
of a DOM one.

It is my understanding that a streaming parser is best for large files,  
so  
if you can use one of these.

Cheers  
Daniel

#!/usr/local/bin/ruby

---

<div class="post-metadata">

**Author:** ![nutsmuggler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/57b2e6/32.png) [@nutsmuggler](https://www.ruby-forum.com/u/nutsmuggler)\
**Post date:** [September 17, 2007, 1:25pm UTC](https://www.ruby-forum.com/t/xml-parser-maybe-ruby-is-too-slow/114681/3 "2007-09-17T13:25:06Z")

</div>

On Sep 15, 7:49 am, nutsmuggler [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ruby being interpreted. Cheers,  
> ‘[http://www.w3.org/TR/html4/strict.dtd](http://www.w3.org/TR/html4/strict.dtd)’\>\n  
> background: #FFFFCC;
> 
> \n puts "
> 
> IT: #{item.next\_sibling.html}
> 
> \n 
> * * *
> " end end puts ""

I haven’t done any comparison testing, but if your \*.EXP files are  
truly XML, Ruby libxml might be a better choice as it’s just a wrapper  
around the libxml2 library (see [http://libxml.rubyforge.org/](http://libxml.rubyforge.org/)).

Jeremy

---

<div class="post-metadata">

**Author:** ![nutsmuggler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/57b2e6/32.png) [@nutsmuggler](https://www.ruby-forum.com/u/nutsmuggler)\
**Post date:** [September 25, 2007, 11:08pm UTC](https://www.ruby-forum.com/t/xml-parser-maybe-ruby-is-too-slow/114681/4 "2007-09-25T23:08:09Z")

</div>

On 16 Set, 05:56, “removed\_email\_address@domain.invalid” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > script took ages to generate the output. Should I switch to a compiled  
> > $pattern = “server”  
> > p {  
> > span.pattern {  
> > if item.innerHTML =~ /#{$pattern}/  
> > truly XML, Ruby libxml might be a better choice as it’s just a wrapper  
> > around the libxml2 library (seehttp://libxml.rubyforge.org/).
> 
> Jeremy

The problem is the EXP file are actually SGML; I could not parse them  
with REXML precisely because they are not well formed XML: they  
contains open tag, whoch are apparently valid in some SGML format, but  
not in XML. That is why I had to use hpricot, which is less picky.  
Cheers,  
Davide