I plan to parse a huge XML document (too big to fit into RAM) using a
stream parser. I can divide the stream into logical chunks which can be
processed individually. If a particular chunk fails, I want to append it
to an output XML file, which will contain all the failed chunks, and can
be patched up and retried.
To do this, I want to be able to regenerate the XML of the failed chunk,
preferably identical to how it was seen.
The options I can think of are:
A stream parser which gives me the raw XML alongside each parsed
item; I can concatenate the raw XML into a string.
A stream parser which gives me the byte pos of the current node, so I
can seek back within the file to fetch the XML again
A stream parser which gives me events to identify the different parts
of XML, together with an inverse process to which I can replay the
events and get the XML back again.
Playing with REXML StreamListener, I can get a series of method calls
like start_tag(…) and end_tag(…), and I can collect these into an
array; is there existing code which would let me squirt that array and
recreate the XML? Any other options I should be looking at?
Playing with REXML StreamListener, I can get a series of method calls
like start_tag(…) and end_tag(…), and I can collect these into an
array; is there existing code which would let me squirt that array and
recreate the XML? Any other options I should be looking at?
From my experience, REXML is far too wimpy to deal with data on this
scale. (Among other things, it was too slow.) I suggest using the
‘stream parser’ (a misnomer, this is really a lexer) in libxml
instead. I don’t know for sure if it can reconstruct the original text
the way you want, but that should be possible.
I plan to parse a huge XML document (too big to fit into RAM) using a
stream parser. I can divide the stream into logical chunks which can be
processed individually. If a particular chunk fails, I want to append it
to an output XML file, which will contain all the failed chunks, and can
be patched up and retried.
If you aren’t completely against Perl - XML-Twig [1] has a tool called
xml_split [2] which works rather well at splitting xml files. You might
wish
to split up your files into smaller files prior to even beginning the
processing and then if a file fails to process you just have the file in
hand. When finished you could smash the failed files back together using
xml_merge [3] from the same perl package.
If there is some ruby variant of this I couldn’t locate it but that
never
means much
Since this is a bit of disposable code, I’ve decided to cheat. I
pretty-print the XML, then I can read it line-at-a-time using gets into
a buffer, identify a range of lines which forms a chunk, then parse the
buffer. On error I write out the buffer again.
Would you care to use JRuby?
That would give you access to top XML Stream parsers IIRC
Just as an example: org.apache.xerces.parsers.SAXParser seems very
suited for your purpose, although it is a little bit of work to
construct your xml fragments it should be rather easy.
Actually, now I think about it, I wrote some code about 5 years ago for
storing XML docs as rows in a SQL database, and one component of that
was indeed a tag stream to XML converter.
I never fully completed or released it, but I’ve just pushed it out to a
git repo anyway.
Why?
And to add insult to injury, by interfacing J*** with JRuby you do
not even see Java, you see a Ruby API.
( Just wanted to be clear about this )
R.
not even see Java, you see a Ruby API.
( Just wanted to be clear about this )
Just to be clear, too: By interfacing Java with JRuby, you get a Ruby
API that feels like its written by a Java consultant struggling on his
first steps to learn Ruby.
While I am impressed how well the integration of JRuby into Java works,
Java libraries without a handwritten layer above them still feel very
alien. So, you do see Java - a lot, actually.
Why?
And to add insult to injury, by interfacing J*** with JRuby you do
not even see Java, you see a Ruby API.
( Just wanted to be clear about this )
Just to be clear, too: By interfacing Java with JRuby, you get a Ruby API that feels like its written by a Java consultant struggling on his first steps to learn Ruby.
While I am impressed how well the integration of JRuby into Java works, Java libraries without a handwritten layer above them still feel very alien. So, you do see Java - a lot, actually.
agreed, I was putting my bold statement to test, when calling into
Java you need to honor the java type checks and there are no block
parameters.
Thus there remains lots of work to be done to adapt a given API to be
“rubyish” my bad.
R.
Just to be clear, too: By interfacing Java with JRuby, you get a Ruby API that feels like its written by a Java consultant struggling on his first steps to learn Ruby.
While I am impressed how well the integration of JRuby into Java works, Java libraries without a handwritten layer above them still feel very alien.
Often true. However, the range of fast, reliable libraries is much
greater for Java than for Ruby.
Depending on how complicated the XML is, you may be able to use a
combination of self-parsing and XML libraries.
I’ve needed to handle arbitrarily large XML “streams” before in C,
Smalltalk, and Python. The “outer” XML was really a wrapper around (or
to connect) a bunch of XML fragments that were not large. We parsed the
“outer” XML until we located the fragment we were interested in, then
parsed it as though it were a complete XML document of its own. This
way we were able to handle XML files of infinite size by biting off
individual chunks.
In our case, the XML was coming across a network and there was no
knowing how big it would be. We also had the advantage that if the
/whole/ XML document was not well-formed (maybe a network error
interrupted it) we didn’t lose the fragments.
Just to be clear, too: By interfacing Java with JRuby, you get a Ruby API that feels like its written by a Java consultant struggling on his first steps to learn Ruby.
I don’t know a lot of struggling Java consultants that have released
Java libraries used on a wide scale. In fact, I don’t know any
struggling Java consultants that have released libraries, period.
Maybe the APIs would be better if they did.
I think you’re overstating the problem. Many Java libraries are
overdesigned, this is true. But JRuby does more than just provide a
means to call them; it provides a lot of other niceities like passing
a block or arbitrary object as the implementation of an interface and
not having to convert or cast values all over.
I also don’t think it’s a whole lot better when people write C
extensions that just wrap a raw C API. If anything, C APIs are usually underdesigned, and it becomes a mess just to fit them nicely into an
OO language. The truth is that just providing the ability to call from
Ruby a library written in C or Java isn’t always enough; but it’s a
hell of a lot easier to start with the Java library in JRuby, since
you don’t even have to compile anything.
Charlie
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
