Uniforma-0.0.1 - converter for text formats

Victor_SZverokSS_Sh · September 17, 2007, 1:24pm

Hi all.

I’m pleased to announce 0.0.1 (aka “early adopters only” release) of my
Uniforma library.

It’s here: http://rubyforge.org/projects/uniforma/

== What is it?

Library for parsing “simple text” formats (RD, Textile, Markdown, etc.)
and
generating output in various formats (including simple text, html/xml
and
more complex ones).

The heart of the library is two DSLs - for defining parsers and
generators.

== Why?

Preparing “one more serious library”'s documentation, I’ve found a
dillema: write it in RD? (for auto-generate all with RDoc) or Trac’s
wiki
format? (for uploading to Trac site) or Textile? (for once uploading to
stand-alone site) So I’ve decided to do conversion library/tool.
I’m using RedCloth (Textile) for all my works, and trying to patch it
for
my needs, I’ve found it’s a mess. I just need to have separate clear
description of “how is it parsed” and “how is it generated” aspects.
For my journalism, I need MS Word output (I have no fun to do text
editing in MS Word, but ability to generate it is a must). Now I use
“Textile=>(RedCloth)=>HTML=>winword mytext.html” scheme, which have
several flaws. I want be able to easy define MS Word generator (using
win32ole, of course, no hand-made heroism).

== Show. Me. The. Code.

Usage:

puts Uniforma::textile(‘some text “with
links”:http://google.com.’).to_html_string

output:

some text with links.

Defining parsers:

module Uniforma::Parsers
class Textile < LineParser
definition do
…
#how to parse some line
…
line /^h(\d+).\s+/ do para(:heading, :level => @_1.to_i) end
…
#how to parse inline formatting:
inline /(.+?)/, :italic
end
end
end

Defining generators

module Uniforma::Generators
class HtmlString < TextGenerator
definition do
…
#what to place around some “paragraph type”
around(:heading) {|p| i = p.level; [“<h#{i}>”, “</h#{i}>\n”]}
…

#what to place around some "inline markup type"
around(:italic)     {["<i>", "</i>"]}

end
end
end

Uniforma is smart enough to allow:

non-line based formats parsers (in fact, it also has one “toy” parser
for
HTML, which even works! on not-very-complex HTML documents)
non-text format generators (I’m working on PDF and MSWord generators.
It’s
not very hard to define with Uniforma)

== Important notes about current release

This release shamelessly includes htmlentities library by Paul
Battley[1],
without even notice it in license files. It is subject to change ASAP.
It’s really “early adopters” release. Almost no docs, and very, very
poor
tests. But it shows an idea and is a base for further work.
This release include parsers for: Textile, RD, HTML and generators
for:
BBcode, RD, HTML. All of them are incomplete but tend to work.
I’d want to hear opinions about whether DSLs for parser/generator
looks
“right” from point-of-view of a) native English speakers and b) real
Ruby
ninja. You can examine my parsers in lib/uniforma/parsers/ and
generators in
lib/uniforma/generators/

Again, the library is here: http://rubyforge.org/projects/uniforma/

Thanx.

Zverok.

1:http://rubyforge.org/projects/htmlentities/

Victor_SZverokSS_Sh · September 17, 2007, 1:25pm

Victor,

Great idea! It would be great to have one way to read and write all
these
formats.

See my embedded comments.

On 9/13/07, Victor Zverok S. [email protected] wrote:

Hi all.

I’m pleased to announce 0.0.1 (aka “early adopters only” release) of my
Uniforma library.

It’s here: http://rubyforge.org/projects/uniforma/

For my journalism, I need MS Word output (I have no fun to do text
editing in MS Word, but ability to generate it is a must). Now I use
“Textile=>(RedCloth)=>HTML=>winword mytext.html” scheme, which have
several flaws. I want be able to easy define MS Word generator (using
win32ole, of course, no hand-made heroism).

To generate msword docs - it might be easier (and more portable) to
simply
write out the new xml form of word rather than using win32ole. I believe
it
would still have all the same capabilities but just represented in xml
format.

And of course don’t forget to add open office xml to the list.

You might also take a look at deplate for some inspiration, it generates
latex, html, and docbook and can read from a few formats.

If you set up a mailing list, let me know as I would like to follow the
project since this could be very useful as I need to be able to generate
many formats.

Blessings,

Jeff

Victor_SZverokSS_Sh · September 17, 2007, 1:25pm

Your work is interesting and sure looks like good ruby to me.

I had to write a similar parser for Zena (to parse textile additions
and the zafu templates) and I thought that doing so many regex
evaluations on the full text (can be long) was too slow (please tell
if this is wrong). I thus chose to use regex anchored left /\A…/ and
eat through the text only once. This has the other advantage that you
enter different modes (tag parameters, comments, raw data, etc) along
the way. It makes it very easy to parse sub languages from within
these modes.

The parser is a two step operation: 1. parse, 2. render. This might be
overkill for the kind of transformations you need but it is very
interesting because the parsed elements can use some knowledge from
the context when they are rendered.

I intend to do a textile → Latex transformation so the users can
write zafu templates to generate PDF.

The current implementation of the parser is not as clean as yours but
works. You can have a look at the parser here :
http://dev.zenadmin.org/browser/trunk/lib/parser

Let me know what you think.

Gaspard

Victor_SZverokSS_Sh · September 25, 2007, 11:04pm

Library for parsing “simple text” formats (RD, Textile, Markdown, etc.) and
generating output in various formats (including simple text, html/xml and
more complex ones).

I wrote deplate[1], which has similar goals (well, with the exception
of
source quality maybe ;-).

The point here is of course that simple formats are easy to parse, so
the question is how simple do you mean with “simple”.

I want be able to easy define MS Word generator (using
win32ole, of course, no hand-made heroism).

If simple is really simple like rdoc-simple, why not simply import
HTML?
Although I like slightly more the way how OpenOffice uses HTML files.

If “simple” includes cross references, footnotes, endnotes, headers,
footers, table of contents/tables/figures etc., I think you’ll
probably
need:

- a general way to define counters and lists
- some notion of metadata (like index, footnotes, labels, section
  names etc.)
- make it possible to locate text at some random position in the
  output document (eg for headers & footers), e.g. move text to

the
top of the document, after packages are required but before the
start of the body etc. deplate defines “slots” for this which
allows users to place the element at any position they want.
- on the long (or intermediate-distance) run, you might also
think
of some plugin-mechanism (e.g. e-mail obfuscation that may be
loaded when converting the document without being hard-coded,
although this could also be done by post-processing the output).

non-line based formats parsers (in fact, it also has one “toy” parser for
HTML, which even works! on not-very-complex HTML documents)

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

Anyway, I’m eager to see how this develops.

Cheers,
Thomas.

[1] http://deplate.sf.net

Victor_SZverokSS_Sh · September 25, 2007, 11:04pm

On 9/16/07, Bira [email protected] wrote:

format.

http://en.wikipedia.org/wiki/Office_Open_XML

OOXML’s specification is over 6.000 pages long, and full of
idiosyncrasies - I don’t know how much of it he’d need for his
documents, but using OLE and Word is probably easier than trying to
build a OOXML-compliant document generator from scratch. Which I guess
is exactly why Microsoft made the spec that long in the first place,
but I digress… :).

Wow, that figures that they would make something too complicated to
implement.

However I was thinking one could take a simpler approach and just create
a
document in MSWord with everything you are supporting in the markup and
then
save it as open office xml. That should give you an example of what to
code
to, but if there are many idiosyncrasies then I guess the OLE way would
be
more straight forward. I was hoping rather that for simple markup that
there
wouldn’t be too much to learn, but I have not read the spec so maybe it
wouldn’t be as easy as I thought.

Boy I am glad that we have open source alternatives to most everything
these
days!

Victor_SZverokSS_Sh · September 25, 2007, 11:03pm

From: micathom [mailto:[email protected]]
Sent: Monday, September 17, 2007 10:50 AM

The point here is of course that simple formats are easy to parse, so
the question is how simple do you mean with “simple”.

I meant, those whose parsers can be defined with some easy common DSL

If “simple” includes cross references, footnotes, endnotes, headers,
footers, table of contents/tables/figures etc., I think you’ll
probably
need:

a general way to define counters and lists

yep, something already exists, something more will.

some notion of metadata (like index, footnotes, labels, section
names etc.)

yep, planned.

make it possible to locate text at some random position in the
output document (eg for headers & footers), e.g. move text to
the
top of the document, after packages are required but before the
start of the body etc. deplate defines “slots” for this which
allows users to place the element at any position they want.

If I understand you correctly, now Uniforma’s generators does this
“hack”
(placing fearst “heading” paragraph in generated html tag):

lib\uniforma\generators\html.rb (lines 6-17):

pre(:document) do |document|
title = document.find_first(:heading)
if title
%Q{

#{title.text}

}
else
%Q{\n}
end
end

So, I don’t think about “random” positions, but only about some
pre(:document) and post(:document) actions, which has access to overall
document.

This approach seems natural enough for me.

on the long (or intermediate-distance) run, you might also
think
of some plugin-mechanism (e.g. e-mail obfuscation that may be
loaded when converting the document without being hard-coded,
although this could also be done by post-processing the output).

yep. I’ve thought about syntax like (“plug-in” to rewrite some urls):

Uniforma.textile(‘mydocument.text’).to_html do
rewrite(:href, %r{http://somesite.com}) do |url|
url.gsub(/somesite/, ‘othersite’)
end
end

non-line based formats parsers (in fact, it also has one “toy” parser
for
HTML, which even works! on not-very-complex HTML documents)

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

From the pragmatic point of view, I’ve already had HTMLSax-related stuff
before started to work on Uniforma

I’ll think about converting this part to use Hpricot, but later, when
core
thing (parsers and generators) will work well.

V.

Victor_SZverokSS_Sh · September 25, 2007, 11:07pm

On 9/14/07, Jeff B. [email protected] wrote:

To generate msword docs - it might be easier (and more portable) to simply
write out the new xml form of word rather than using win32ole. I believe it
would still have all the same capabilities but just represented in xml
format.

http://en.wikipedia.org/wiki/Office_Open_XML

OOXML’s specification is over 6.000 pages long, and full of
idiosyncrasies - I don’t know how much of it he’d need for his
documents, but using OLE and Word is probably easier than trying to
build a OOXML-compliant document generator from scratch. Which I guess
is exactly why Microsoft made the spec that long in the first place,
but I digress… :).

Victor_SZverokSS_Sh · September 25, 2007, 11:05pm

From: Tom [mailto:[email protected]]
Sent: Monday, September 17, 2007 11:00 AM

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

BTW which makes me wonder, if you could your library into a parser and
a
formatter. A parser for wiki-like formats that generates something
like
hpricot does, would be something I would very much like to see.

Internally, it’s already here. Any parser creates Uniforma::Dom
structures,
which, though, lacks handy (Hpricot-like) navigation. Will think about
it.

V.

Victor_SZverokSS_Sh · September 25, 2007, 11:07pm

From: Jeff B. [mailto:[email protected]]
Sent: Friday, September 14, 2007 6:53 PM

To generate msword docs - it might be easier (and more portable) to simply
write out the new xml form of word rather than using win32ole. I believe it
would still have all the same capabilities but just represented in xml
format.

http://en.wikipedia.org/wiki/Office_Open_XML

And of course don’t forget to add open office xml to the list.

My primary goal is to create “right tool” for others to define their
own
parsers and generator
About complex formats, I’m planning to be pragmatic: win32ole-based
solution is enough for most cases. If somebody feels it’s not enough,
he/she
can do the “right” generator him/her-self.

You might also take a look at deplate for some inspiration, it generates
latex, html, and docbook and can read from a few formats.

Yeah, thanks.

If you set up a mailing list, let me know as I would like to follow the
project since this could be very useful as I need to be able to generate
many formats.

OK, I’l notify you in the case. I think, it will be soon.

V.

Victor_SZverokSS_Sh · September 25, 2007, 11:10pm

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

BTW which makes me wonder, if you could your library into a parser and
a
formatter. A parser for wiki-like formats that generates something
like
hpricot does, would be something I would very much like to see.

Regards,
Thomas.

Uniforma-0.0.1 - converter for text formats

module Uniforma::Parsers class Textile < LineParser definition do … #how to parse some line … line /^h(\d+).\s+/ do para(:heading, :level => @_1.to_i) end … #how to parse inline formatting: inline /(.+?)/, :italic end end end

end end end

pre(:document) do |document| title = document.find_first(:heading) if title %Q{ #{title.text} } else %Q{\n} end end

module Uniforma::Parsers
class Textile < LineParser
definition do
…
#how to parse some line
…
line /^h(\d+).\s+/ do para(:heading, :level => @_1.to_i) end
…
#how to parse inline formatting:
inline /(.+?)/, :italic
end
end
end

end
end
end

pre(:document) do |document|
title = document.find_first(:heading)
if title
%Q{

#{title.text}

}
else
%Q{\n}
end
end