Syntax Highlighter

Would you agree that Ruby shouldn’t lack for a syntax highlighting
library
as good as any that exists for other dynamic languages? I do. But as it
stands Ruby lags well behind Python’s Pygments (http://pygments.org/).
This
should not stand!

Ruby has a pretty good library in CodeRay (http://coderay.rubychan.de/)
but
to catch up it needs to support many more formats.

So I want to encourage all Ruby developers to carve out a bit of time to
create a CodeRay scanner script for an as-of-yet unsupported format.
Have a
look at http://odd-eyed-code.org/versions/show/15 for some suggestions
and
have a look at
https://github.com/rubychan/coderay/tree/master/lib/coderay/scanners for
reference on how it’s done.

(NOTE: I have no direct affiliation with the CodeRay project. I just
want
to see Ruby have a top-notch syntax highlighting library.)

On Thu, Jun 14, 2012 at 9:07 AM, Intransition [email protected]
wrote:

look at http://odd-eyed-code.org/versions/show/15 for some suggestions
and have a look at
coderay/lib/coderay/scanners at master · rubychan/coderay · GitHub for
reference on how it’s done.

(NOTE: I have no direct affiliation with the CodeRay project. I just want
to see Ruby have a top-notch syntax highlighting library.)

We should suggest this to http://codebrawl.com/ there hasn’t been a good
code brawl in a long time.

-Josh

On 06/14/2012 04:07 PM, Intransition wrote:

Would you agree that Ruby shouldn’t lack for a syntax highlighting library as
good as any that exists for
other dynamic languages? I do. But as it stands Ruby lags well behind Python’s
Pygments
(http://pygments.org/). This should not stand!

Ruby has a pretty good library in CodeRay (http://coderay.rubychan.de/) but to
catch up it needs to support
many more formats.

If you have a look, on what is scheduled for 1.1, there is a lot to
come. But the problem is, when it will
code. I added fortran support to coderay more than 2 years ago, but it
still hasn’t made it into any release.
Dont get me wrong, no criticism here. A lot of people seem to have
written valid scanners. Might be good to
have a place to share them.

Am 14.06.2012 16:07, schrieb Intransition:

So I want to encourage all Ruby developers to carve out a bit of time to
create a CodeRay scanner script for an as-of-yet unsupported format.
Have a look at http://odd-eyed-code.org/versions/show/15 for some
suggestions and have a look at
coderay/lib/coderay/scanners at master · rubychan/coderay · GitHub for
reference on how it’s done.

I’ve written a Lua scanner for CodeRay, but the pull request still isn’t
approved. Lua scanner -- Meta-commit. by Quintus · Pull Request #22 · rubychan/coderay · GitHub

Vale,
Marvin


Blog: http://pegasus-alpha.eu/blog

ASCII-Ribbon-Kampagne () | ASCII Ribbon Campaign ()

On Thursday, June 14, 2012 10:39:55 AM UTC-4, Ralf M. wrote:

many more formats.

If you have a look, on what is scheduled for 1.1, there is a lot to come.
But the problem is, when it will
code. I added fortran support to coderay more than 2 years ago, but it
still hasn’t made it into any release.
Dont get me wrong, no criticism here. A lot of people seem to have written
valid scanners. Might be good to
have a place to share them.

Oh, where is this scanner? Could you submit as a pull request with it?

On 06/14/2012 07:12 PM, Intransition wrote:

support

On Jun 14, 2012, at 07:07 , Intransition wrote:

So I want to encourage all Ruby developers to carve out a bit of time to create
a CodeRay scanner script for an as-of-yet unsupported format. Have a look at
http://odd-eyed-code.org/versions/show/15 for some suggestions and have a look
athttps://github.com/rubychan/coderay/tree/master/lib/coderay/scanners for
reference on how it’s done.

Without a commitment from the developer to take on these contributions,
this call to arms seems like a waste of effort. It seems pretty clear
that the author has been sitting on a lot of scanner contributions for
~2 years (many of which I could use right now) and hasn’t done anything
with them.

On Thu, Jun 14, 2012 at 3:39 PM, Henry M. [email protected]
wrote:

On 15/06/2012, at 7:07 AM, Ryan D. [email protected] wrote:

Without a commitment from the developer to take on these contributions,
this call to arms seems like a waste of effort. It seems pretty clear that
the author has been sitting on a lot of scanner contributions for ~2 years
(many of which I could use right now) and hasn’t done anything with them.

Fork it!

And maintain it! And get everyone to use your version!

Marvin Gülker wrote in post #1064597:

I’ve written a Lua scanner for CodeRay, but the pull request still isn’t
approved. Lua scanner -- Meta-commit. by Quintus · Pull Request #22 · rubychan/coderay · GitHub

I submitted scanners for AVR Assembler and VHDL a few years ago, but
they haven’t been included. And unfortunately they aren’t compatible
with the current version anymore, so they probably have to be rewritten.

On 15/06/2012, at 7:07 AM, Ryan D. [email protected] wrote:

Without a commitment from the developer to take on these contributions, this
call to arms seems like a waste of effort. It seems pretty clear that the author
has been sitting on a lot of scanner contributions for ~2 years (many of which I
could use right now) and hasn’t done anything with them.

Fork it!

Henry

On Jun 14, 2012, at 12:39 , Henry M. wrote:

On 15/06/2012, at 7:07 AM, Ryan D. [email protected] wrote:

Without a commitment from the developer to take on these contributions, this
call to arms seems like a waste of effort. It seems pretty clear that the author
has been sitting on a lot of scanner contributions for ~2 years (many of which I
could use right now) and hasn’t done anything with them.

Fork it!

I’m not against the idea… but as Yossef implied, it is clearly less
effort to either get the original developer to recommit to the project
or to delegate the responsibilities out to others.

On Jun 15, 2012, at 00:38 , Ralf M. wrote:

Coderay seems to be well prepared for a plugin structure. The additions for the
fortran scanner are very little: only one additional file + the extension of a
single Hash (which is responsible for choosing a parser for files with a given
extension).

It would be, if it used this:

def list
  Gem.find_files("coderay/scanners/*.rb")
end

instead of:

def list
  Dir[path_to('*')].select do |file|
    File.basename(file)[/^(?!_)\w+\.rb$/]
  end.map do |file|
    File.basename(file, '.rb').to_sym
  end
end

Ryan D. wrote in post #1064682:

On Jun 14, 2012, at 12:39 , Henry M. wrote:

On 15/06/2012, at 7:07 AM, Ryan D. [email protected] wrote:

Without a commitment from the developer to take on these contributions, this
call to arms seems like a waste of effort. It seems pretty clear that
the author
has been sitting on a lot of scanner contributions for ~2 years (many of
which I
could use right now) and hasn’t done anything with them.

Fork it!

I’m not against the idea… but as Yossef implied, it is clearly less
effort to either get the original developer to recommit to the project
or to delegate the responsibilities out to others.

Has anyone privately contacted the maintainer with a specific plan?

Something short-term and targeted like…“In the next 30 days
I’d like to help you push out a point release that adds support for 3-10
new language scanners; no internal refactorings. I’m guessing you’re
swamped, so if you’ll add me as a GH contributor, I’ll do all the
integration and testing legwork on a new branch. Once I think things
look solid, I’ll sync back up with you to review for the point release.”

Jon

On 06/15/2012 04:11 AM, Ryan D. wrote:

On Jun 14, 2012, at 12:39 , Henry M. wrote:

On 15/06/2012, at 7:07 AM, Ryan D. [email protected] wrote:

Without a commitment from the developer to take on these contributions, this
call to arms seems like a waste of effort. It seems pretty clear that the author
has been sitting on a lot of scanner contributions for ~2 years (many of which I
could use right now) and hasn’t done anything with them.
Fork it!
I’m not against the idea… but as Yossef implied, it is clearly less effort to
either get the original developer to recommit to the project or to delegate the
responsibilities out to others.

Coderay seems to be well prepared for a plugin structure. The additions
for the fortran scanner are very
little: only one additional file + the extension of a single Hash (which
is responsible for choosing a parser
for files with a given extension).

Without a commitment from the developer to take on these
contributions, this call to arms seems like a waste of effort.
It seems pretty clear that the author has been sitting on a
lot of scanner contributions for ~2 years (many of which I
could use right now) and hasn’t done anything with them.

I think the author is murphy. I don’t know how it was for you,
but I wrote to him some months ago about the choice of colours
and had some email exchanges, which were productive. So I would
not say that he is inactive at all.

He can however reply here when he finds the time and someone
shows him the link to it. :slight_smile:

Has anyone privately contacted the maintainer with a specific plan?

That should be the cheerleader’s job.

Well. I’d say the first thing to do is get to point out this
thread to murphy. He’ll surely be able to find 2-4 minutes to
just write a short reply whether he plans to merge in syntax
changes like the lua parser, and if not why not, and then these
problems can eventually be fixed.

So I think, the first thing that should be done is to find murphy.

Murphy, where are you!

On 06/15/2012 10:59 PM, Marc H. wrote:

He can however reply here when he finds the time and someone
So I think, the first thing that should be done is to find murphy.

Murphy, where are you!

contacted via mail with link to ruby-forum, 'cause I wasn’t sure murphy
has subscribed to ruby-talk

On Jun 15, 2012, at 08:38 , Jon F. wrote:

Has anyone privately contacted the maintainer with a specific plan?

That should be the cheerleader’s job.

Hi, I’m murphy, the developer of CodeRay. I have been busy with work and
private life in the last months, so I couldn’t answer to all emails and
tickets. But I’m still interested and willing to work on CodeRay!

The easiest way to get a syntax highlighting library with lots of
languages is to port the awesome Pygments library to Ruby. CodeRay
scanners are hand-crafted methods, while Pygments uses a clever DSL. So,
CodeRay is faster, but Pygments is easier to extend.

If we want to extend CodeRay instead of porting Pygments, I need your
help :slight_smile:

It’s true that I received about 20 scanner contribution over the last
years. Some of them have been included, but others were not up to the
quality standard I want to set with CodeRay. I mean no offence to the
people who want to contribute; their help is invaluable. But the
scanners just need more work. They lack:

  1. Example code

To be tested, a scanner needs a lot of example code. CodeRay has an
extensive test suite build around this idea:
http://svn.rubychan.de/coderay-scanner-tests/trunk/ (still under private
SVN because of open copyright questions; you can download and run the
suite simply from the git checkout with rake test:scanners.) The more
complex the language, the more example code is needed. For Lua, a dozen
files from different projects can be enough. For a language like Perl, I
think that even the PLEAC
(http://pleac.sourceforge.net/pleac_perl/index.html) is only scratching
the surface.

  1. Completeness

I don’t want to release a scanner that covers only 80% of a languages’
features, and call it “supported”. It’s possible to beta-test new
scanners as standalone Gems, like coderay_bash. When they reach
stability and a very good level of language coverage (see 1), I promise
to include them quickly.

  1. Testing in general

Some of the code I received was just buggy. That’s my own fault, because
the testing framework you need to write a good scanner is still not well
documented. (If you need any help, please contact me.)

However, stableness is critical, since some use CodeRay to highlight
user input live - no scanner should ever run into an infinite loop or
eat obscene amounts of memory because of a typo or an overlooked edge
case in the Scanner.

Thomas: The 1.1 roadmap you linked to is outdated and not realistic; it
never got updated after the 1.0 release. How about a new roadmap that
focusses on additional language support? I suggest to give:

  • CoffeeScript, Sass (both used often in the Ruby community)
  • Objective-C (rising star)
  • Bash, Lua (often requested)

highest priority. Please comment if I’m missing anything here.

Josh: A codebrawl might be a good idea, but I think it requires better
documentation for scanner implementation. I may not find the time do
that right now.

Ryan:

Gem.find_files(“coderay/scanners/*.rb”)

CodeRay was written for Ruby 1.8, and I didn’t want to
depend on RubyGems at that time. A future version can be 1.9-only, so
maybe we can ditch the Plugin system and use RubyGems.

I will go through all unanswered mails and tickets in the next days, and
of course follow this thread. Thanks to everybody who contacted me; I
was not subsribed to ruby-talk.

  • [murphy]