# RedCloth, JRuby and national characters

**URL:** <https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120>\
**Category:** Redcloth\
**Created:** [February 25, 2010, 2:00pm UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120 "2010-02-25T14:00:25Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [February 25, 2010, 2:00pm UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/1 "2010-02-25T14:00:25Z")

</div>

Am I doing something wrong or have I hit a bug in the JRuby version of  
RedCloth?

RedCloth version 4.2.2

Program:  
require ‘rubygems’  
require ‘redcloth’  
str = ‘blÃ¥bærgrød’  
puts 'String : ’ + str  
puts 'HTML : ’ + RedCloth.new(str).to\_html()

In “ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]” the output is

String : blåbærgrød  
HTML :

blåbærgrød

In “jruby 1.4.0 (ruby 1.8.7 patchlevel 174) (2009-11-02 69fbfa3) (Java  
HotSpot™ Client VM 1.6.0\_17) [x86-java]” the output is

String : blåbærgrød  
HTML :

bl

As you can see, when running i JRuby, parsing of the string seems to  
have stopped, when the first Danish national character, “Ã¥”, was met.

Claus

---

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [February 26, 2010, 7:03am UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/2 "2010-02-26T07:03:10Z")

</div>

Claus Folke B. wrote:

> Am I doing something wrong or have I hit a bug in the JRuby version of  
> RedCloth?
> 
> RedCloth version 4.2.2

I should add that I am on Windows XP.

Claus

---

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [October 18, 2010, 11:12am UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/3 "2010-10-18T11:12:29Z")

</div>

Hi Claus,

I hit the same issue. Linux, Redcloth 4.2.3, jruby 1.5.1 (ruby 1.8.7  
patchlevel 249).

My workaround was to html-escape non-standard characters before passing  
them to redcloth. It works but my solution is a little bit fragile. Did  
you find a proper solution?

Best, Georg

---

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [November 12, 2010, 11:20am UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/4 "2010-11-12T11:20:00Z")

</div>

Great work, Marek! I pulled your work into a branch: jruby-mbc

The problem, as you pointed out, is extra whitespace. I’m hoping you or  
someone else can help me get it figured out so I can release it!

It seems to be just when there’s HTML in the input. (At least that’s all  
I’ve found so far.) When it’s a standalone HTML tag (just a block tag  
on a line), it puts two BRs after. When it’s an HTML block (start tag,  
contents, end tag), it puts the BR inside the beginning of the next  
block. When just one newline ends the document, it puts a BR inside the  
end of the last block; two newlines before EOF behave fine though.

> test
> 
> Another p.

Results in:

> Another p.

Weird, huh? I’d greatly appreciate anyone who can help this Java dunce  
(me). Here’s the fast way to get it checked out and set up:

> git clone removed\_email\_address@domain.invalid:jgarber/redcloth.git  
> cd redcloth  
> git checkout jruby-mbc  
> rvm use jruby-1.5.3@redcloth # assuming you’re using rvm and you’ve done ‘rvm  
> install jruby’  
> bundle  
> rake compile

Thanks!  
Jason

---

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [November 12, 2010, 2:59pm UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/5 "2010-11-12T14:59:48Z")

</div>

Hey,  
I’m glad I could contribute a little 🙂 I spent a lot of time trying  
to figure out the reason for extra whitespace and failed. I came to  
the conclusion that the newline coding for ragel had to be updated so  
that it matches the 32bit characters. I know nothing about Ragel so I  
gave up. Nowadays I’m doing python, so I cannot help any further.  
Still I guess that the whitespace problem should be far easier to spot  
and fix. Last but not least it could be just ignored, this is what I  
did, I just deployed the code as it is to the webapp I’ve been working  
on.

Regards and good luck!  
Marek Kowalski

2010/11/12 Jason G. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

---

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [October 18, 2010, 2:02pm UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/6 "2010-10-18T14:02:25Z")

</div>

Hi Georg,  
I went the same went. Eventually I managed to kind of solve the  
problem, I have the proper jar generated which handles UTF characters  
properly. What basicly what needs to be done is to use char (16-bit)  
datatype in Ragel code instead of byte (8-bit). You can take a look  
here at my work:

> **[GitHub - kowalski/redcloth: RedCloth is a Ruby library for converting Textile...](https://github.com/kowalski/redcloth)**
>
> RedCloth is a Ruby library for converting Textile into HTML. - GitHub - kowalski/redcloth: RedCloth is a Ruby library for converting Textile into HTML.

Problem is, that when you run rspec now on the new code, there is a  
number of tests that fails. The difference is the extra whitespaces  
added to the resulting html code. For me it is no harm so I have this  
jar working on production for a few months now.

Cheers,  
Marek Kowalski

2010/10/18 Georg M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

---

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [November 27, 2010, 2:07pm UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/7 "2010-11-27T14:07:50Z")

</div>

I’ve posted this task to oDesk with a $100 budget. Let’s hope someone  
takes  
the job!

[http://www.odesk.com/jobs/JRuby-fix-for-RedCloth\_~~273ff7a15a938782](http://www.odesk.com/jobs/JRuby-fix-for-RedCloth_%7E%7E273ff7a15a938782)

---

<div class="post-metadata">

**Author:** ![Claus\_Folke\_B](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/e36b37/32.png) [@Claus\_Folke\_B](https://www.ruby-forum.com/u/Claus_Folke_B)\
**Post date:** [December 3, 2010, 1:35am UTC](https://www.ruby-forum.com/t/redcloth-jruby-and-national-characters/184120/8 "2010-12-03T01:35:45Z")

</div>

Mhm, thanks for keeping me informed! I don’t agree with the task  
description though. As far as I remember the problem was with ragel  
code not html\_esc function. It should be easy to figured out for  
someone with the ragel experience… You might want to update the  
description, to lure people with correct profile. Well we will see,  
hope someone figures it out.

Cheers!  
MK

2010/11/27 Jason G. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):