Uniforma-0.0.1 - converter for text formats

Hi all.

I’m pleased to announce 0.0.1 (aka “early adopters only” release) of my
Uniforma library.

It’s here: http://rubyforge.org/projects/uniforma/

== What is it?

Library for parsing “simple text” formats (RD, Textile, Markdown, etc.)
and
generating output in various formats (including simple text, html/xml
and
more complex ones).

The heart of the library is two DSLs - for defining parsers and
generators.

== Why?

  1. Preparing “one more serious library”'s documentation, I’ve found a
    dillema: write it in RD? (for auto-generate all with RDoc) or Trac’s
    wiki
    format? (for uploading to Trac site) or Textile? (for once uploading to
    stand-alone site) So I’ve decided to do conversion library/tool.

  2. I’m using RedCloth (Textile) for all my works, and trying to patch it
    for
    my needs, I’ve found it’s a mess. I just need to have separate clear
    description of “how is it parsed” and “how is it generated” aspects.

  3. For my journalism, I need MS Word output (I have no fun to do text
    editing in MS Word, but ability to generate it is a must). Now I use
    “Textile=>(RedCloth)=>HTML=>winword mytext.html” scheme, which have
    several flaws. I want be able to easy define MS Word generator (using
    win32ole, of course, no hand-made heroism).

== Show. Me. The. Code.

Usage:

puts Uniforma::textile(‘some text “with
links”:http://google.com.’).to_html_string

output:

some text with links.

Defining parsers:


module Uniforma::Parsers
class Textile < LineParser
definition do

#how to parse some line

line /^h(\d+).\s+/ do para(:heading, :level => @_1.to_i) end

#how to parse inline formatting:
inline /(.+?)/, :italic
end
end
end

Defining generators


module Uniforma::Generators
class HtmlString < TextGenerator
definition do

#what to place around some “paragraph type”
around(:heading) {|p| i = p.level; ["<h#{i}>", “</h#{i}>\n”]}

#what to place around some "inline markup type"
around(:italic)     {["<i>", "</i>"]}

end
end
end

Uniforma is smart enough to allow:

  • non-line based formats parsers (in fact, it also has one “toy” parser
    for
    HTML, which even works! on not-very-complex HTML documents)
  • non-text format generators (I’m working on PDF and MSWord generators.
    It’s
    not very hard to define with Uniforma)

== Important notes about current release

  • This release shamelessly includes htmlentities library by Paul
    Battley[1],
    without even notice it in license files. It is subject to change ASAP.

  • It’s really “early adopters” release. Almost no docs, and very, very
    poor
    tests. But it shows an idea and is a base for further work.

  • This release include parsers for: Textile, RD, HTML and generators
    for:
    BBcode, RD, HTML. All of them are incomplete :slight_smile: but tend to work.

  • I’d want to hear opinions about whether DSLs for parser/generator
    looks
    “right” from point-of-view of a) native English speakers and b) real
    Ruby
    ninja. You can examine my parsers in lib/uniforma/parsers/ and
    generators in
    lib/uniforma/generators/

Again, the library is here: http://rubyforge.org/projects/uniforma/

Thanx.

Zverok.

1:http://rubyforge.org/projects/htmlentities/

Victor,

Great idea! It would be great to have one way to read and write all
these
formats.

See my embedded comments.

On 9/13/07, Victor Zverok S. [email protected] wrote:

Hi all.

I’m pleased to announce 0.0.1 (aka “early adopters only” release) of my
Uniforma library.

It’s here: http://rubyforge.org/projects/uniforma/

  1. For my journalism, I need MS Word output (I have no fun to do text
    editing in MS Word, but ability to generate it is a must). Now I use
    “Textile=>(RedCloth)=>HTML=>winword mytext.html” scheme, which have
    several flaws. I want be able to easy define MS Word generator (using
    win32ole, of course, no hand-made heroism).

To generate msword docs - it might be easier (and more portable) to
simply
write out the new xml form of word rather than using win32ole. I believe
it
would still have all the same capabilities but just represented in xml
format.

http://en.wikipedia.org/wiki/Office_Open_XML

And of course don’t forget to add open office xml to the list.

You might also take a look at deplate for some inspiration, it generates
latex, html, and docbook and can read from a few formats.

If you set up a mailing list, let me know as I would like to follow the
project since this could be very useful as I need to be able to generate
many formats.

Blessings,

Jeff

Your work is interesting and sure looks like good ruby to me.

I had to write a similar parser for Zena (to parse textile additions
and the zafu templates) and I thought that doing so many regex
evaluations on the full text (can be long) was too slow (please tell
if this is wrong). I thus chose to use regex anchored left /\A…/ and
eat through the text only once. This has the other advantage that you
enter different modes (tag parameters, comments, raw data, etc) along
the way. It makes it very easy to parse sub languages from within
these modes.

The parser is a two step operation: 1. parse, 2. render. This might be
overkill for the kind of transformations you need but it is very
interesting because the parsed elements can use some knowledge from
the context when they are rendered.

I intend to do a textile -> Latex transformation so the users can
write zafu templates to generate PDF.

The current implementation of the parser is not as clean as yours but
works. You can have a look at the parser here :
http://dev.zenadmin.org/browser/trunk/lib/parser

Let me know what you think.

Gaspard

Library for parsing “simple text” formats (RD, Textile, Markdown, etc.) and
generating output in various formats (including simple text, html/xml and
more complex ones).

I wrote deplate[1], which has similar goals (well, with the exception
of
source quality maybe ;-).

The point here is of course that simple formats are easy to parse, so
the question is how simple do you mean with “simple”.

I want be able to easy define MS Word generator (using
win32ole, of course, no hand-made heroism).

If simple is really simple like rdoc-simple, why not simply import
HTML?
Although I like slightly more the way how OpenOffice uses HTML files.

If “simple” includes cross references, footnotes, endnotes, headers,
footers, table of contents/tables/figures etc., I think you’ll
probably
need:

- a general way to define counters and lists
- some notion of metadata (like index, footnotes, labels, section
  names etc.)
- make it possible to locate text at some random position in the
  output document (eg for headers & footers), e.g. move text to

the
top of the document, after packages are required but before the
start of the body etc. deplate defines “slots” for this which
allows users to place the element at any position they want.
- on the long (or intermediate-distance) run, you might also
think
of some plugin-mechanism (e.g. e-mail obfuscation that may be
loaded when converting the document without being hard-coded,
although this could also be done by post-processing the output).

  • non-line based formats parsers (in fact, it also has one “toy” parser for
    HTML, which even works! on not-very-complex HTML documents)

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

Anyway, I’m eager to see how this develops.

Cheers,
Thomas.

[1] http://deplate.sf.net

On 9/16/07, Bira [email protected] wrote:

format.

http://en.wikipedia.org/wiki/Office_Open_XML

OOXML’s specification is over 6.000 pages long, and full of
idiosyncrasies - I don’t know how much of it he’d need for his
documents, but using OLE and Word is probably easier than trying to
build a OOXML-compliant document generator from scratch. Which I guess
is exactly why Microsoft made the spec that long in the first place,
but I digress… :).

Wow, that figures that they would make something too complicated to
implement.

However I was thinking one could take a simpler approach and just create
a
document in MSWord with everything you are supporting in the markup and
then
save it as open office xml. That should give you an example of what to
code
to, but if there are many idiosyncrasies then I guess the OLE way would
be
more straight forward. I was hoping rather that for simple markup that
there
wouldn’t be too much to learn, but I have not read the spec so maybe it
wouldn’t be as easy as I thought.

Boy I am glad that we have open source alternatives to most everything
these
days!

From: micathom [mailto:[email protected]]
Sent: Monday, September 17, 2007 10:50 AM

The point here is of course that simple formats are easy to parse, so
the question is how simple do you mean with “simple”.

I meant, those whose parsers can be defined with some easy common DSL :slight_smile:

If “simple” includes cross references, footnotes, endnotes, headers,
footers, table of contents/tables/figures etc., I think you’ll
probably
need:

  • a general way to define counters and lists

yep, something already exists, something more will.

  • some notion of metadata (like index, footnotes, labels, section
    names etc.)

yep, planned.

  • make it possible to locate text at some random position in the
    output document (eg for headers & footers), e.g. move text to
    the
    top of the document, after packages are required but before the
    start of the body etc. deplate defines “slots” for this which
    allows users to place the element at any position they want.

If I understand you correctly, now Uniforma’s generators does this
“hack”
(placing fearst “heading” paragraph in generated html tag):

lib\uniforma\generators\html.rb (lines 6-17):


pre(:document) do |document|
title = document.find_first(:heading)
if title
%Q{

#{title.text}

}
else
%Q{\n}
end
end

So, I don’t think about “random” positions, but only about some
pre(:document) and post(:document) actions, which has access to overall
document.

This approach seems natural enough for me.

  • on the long (or intermediate-distance) run, you might also
    think
    of some plugin-mechanism (e.g. e-mail obfuscation that may be
    loaded when converting the document without being hard-coded,
    although this could also be done by post-processing the output).

yep. I’ve thought about syntax like (“plug-in” to rewrite some urls):

Uniforma.textile(‘mydocument.text’).to_html do
rewrite(:href, %r{http://somesite.com}) do |url|
url.gsub(/somesite/, ‘othersite’)
end
end

  • non-line based formats parsers (in fact, it also has one “toy” parser
    for
    HTML, which even works! on not-very-complex HTML documents)

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

From the pragmatic point of view, I’ve already had HTMLSax-related stuff
before started to work on Uniforma :slight_smile:

I’ll think about converting this part to use Hpricot, but later, when
core
thing (parsers and generators) will work well.

V.

On 9/14/07, Jeff B. [email protected] wrote:

To generate msword docs - it might be easier (and more portable) to simply
write out the new xml form of word rather than using win32ole. I believe it
would still have all the same capabilities but just represented in xml
format.

http://en.wikipedia.org/wiki/Office_Open_XML

OOXML’s specification is over 6.000 pages long, and full of
idiosyncrasies - I don’t know how much of it he’d need for his
documents, but using OLE and Word is probably easier than trying to
build a OOXML-compliant document generator from scratch. Which I guess
is exactly why Microsoft made the spec that long in the first place,
but I digress… :).

From: Tom [mailto:[email protected]]
Sent: Monday, September 17, 2007 11:00 AM

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

BTW which makes me wonder, if you could your library into a parser and
a
formatter. A parser for wiki-like formats that generates something
like
hpricot does, would be something I would very much like to see.

Internally, it’s already here. Any parser creates Uniforma::Dom
structures,
which, though, lacks handy (Hpricot-like) navigation. Will think about
it.

V.

From: Jeff B. [mailto:[email protected]]
Sent: Friday, September 14, 2007 6:53 PM

To generate msword docs - it might be easier (and more portable) to simply
write out the new xml form of word rather than using win32ole. I believe it
would still have all the same capabilities but just represented in xml
format.

http://en.wikipedia.org/wiki/Office_Open_XML

And of course don’t forget to add open office xml to the list.

  1. My primary goal is to create “right tool” for others to define their
    own
    parsers and generator
  2. About complex formats, I’m planning to be pragmatic: win32ole-based
    solution is enough for most cases. If somebody feels it’s not enough,
    he/she
    can do the “right” generator him/her-self.

You might also take a look at deplate for some inspiration, it generates
latex, html, and docbook and can read from a few formats.

Yeah, thanks.

If you set up a mailing list, let me know as I would like to follow the
project since this could be very useful as I need to be able to generate
many formats.

OK, I’l notify you in the case. I think, it will be soon.

V.

From a pragmatic point of view, using hpricot and writing and map
classes on its output could be the better strategy.

BTW which makes me wonder, if you could your library into a parser and
a
formatter. A parser for wiki-like formats that generates something
like
hpricot does, would be something I would very much like to see.

Regards,
Thomas.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs