Literate Ruby (#102)

The three rules of Ruby Q.:

  1. Please do not post any solutions or spoiler discussion for this quiz
    until
    48 hours have passed from the time on this message.

  2. Support Ruby Q. by submitting ideas as often as you can:

http://www.rubyquiz.com/

  1. Enjoy!

Suggestion: A [QUIZ] in the subject of emails about the problem helps
everyone
on Ruby T. follow the discussion. Please reply to the original quiz
message,
if you can.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

by Justin B.

“Literate Programming”[1] is an idea popularized by Donald Knuth, where
the
traditional order of code and comments in a source file is switched.
Instead of
using special delimiters to mark comments, special delimiters are used
to mark
code.

Innocuous as it sounds, this style of programming makes for a great way
to post
code snippets, tutorials, or even whole libraries to mailing lists,
blogs, and
web pages. It’s also an excellent way to develop your CS homework :wink:

There are, of course, a variety of ways to make a source file
“literate”. One
popular method is called “bird notation”. Code is delimited by lines
starting
with “>”:

> puts "The first line of literate Ruby you may have ever seen"

Another method, used in the Haskell language, is borrowed from LateX and
makes
it very easy to embed working code into longer papers:

\begin{code}
puts "And here, we have"
puts "the second and third lines of literate Ruby to be produced."
\end{code}

Beyond how to represent literate code, a host of issues present
themselves.
Can a class, method, or even string span multiple code sections? Can the
different styles of code demarcation be mixed in one file? How do you
“escape”
code demarcation? What about inserting the output of code lines into the
same
literate file?

Your task is to enable literate Ruby. What that means is up to you. Is
literate
programming only available at the file level (e.g. only files ending in
“.lrb”
are considered literate)? Or is literate programming supported with
eval/class_eval/module_eval? Would this enable embedded literated here
(i.e. <<)
docs?

At the minimum, this quiz should be seen as a literate program, and your
code
should be able to run it! [Editor’s Note: The indention added to Ruby
blocks in
this quiz are a side effect of the Ruby Q. software. Feel free to
remove them
when treating this quiz as literate code. --JEG2]

Justin

> puts "Here's to hoping you enjoyed the quiz!"

[1] Literate programming - Wikipedia

On Fri, 17 Nov 2006 22:56:55 +0900, Ruby Q. wrote:

At the minimum, this quiz should be seen as a literate program, and your code
should be able to run it! [Editor’s Note: The indention added to Ruby blocks in
this quiz are a side effect of the Ruby Q. software. Feel free to remove them
when treating this quiz as literate code. --JEG2]

Does this mean I need to accept both the LaTeX syntax and the
email-quoted
syntax to make this quiz run as a literate program? Or may I reorganize
the quiz to use my preferred code demarcation?

–Ken

On Nov 17, 2006, at 10:25 AM, Ken B. wrote:

quoted
syntax to make this quiz run as a literate program? Or may I
reorganize
the quiz to use my preferred code demarcation?

I doubt I would do both. Would be great to have it configurable though.

James Edward G. II

On Fri, 17 Nov 2006 22:56:55 +0900, Ruby Q. wrote:

Another method, used in the Haskell language, is borrowed from LateX and makes
it very easy to embed working code into longer papers:

\begin{code}
puts “And here, we have”
puts “the second and third lines of literate Ruby to be produced.”
\end{code}

Suddenly, I’m reminded of this winner of the 2000 International
Obfuscated
C Code Contest:

http://www0.us.ioccc.org/2000/tomx.c
http://www0.us.ioccc.org/2000/tomx.hint

–Ken

Ruby Q. wrote:

using special delimiters to mark comments, special delimiters are used to mark
code.

Here is my solution. It features basically both methods to mark that
were proposed (with > end \begin{code} \end{code}). It should be able
to run as is irb output (practical to test directly from an email) and
it features a small hack to require literate ruby files.

There are five attached files:

  • rweb.rb, the actual interpreter;
  • small_test.lrb and required.lrb, a test file and the file included
    from it (to demonstrate the require feature);
  • rweb2tex.lrb, a literate program converting a literate program into
    “appropriately” formatted LaTeX file (that definitely could be improved)
  • rweb2tex.tex,the result of rweb2tex.lrb ran on itself. (I personally
    don’t like its look so much, but, well, I don’t like literate
    programming so much anyway ;-)…)

Hope you appreciate it !

Vince

#!/usr/bin/env ruby

RLit (0.1)

Author: Louis J. Scoras [email protected]

Date: Sat Nov 18 23:29:29 EST 2006

RLit – a simple script for enabling literate programming in Ruby

using erb

and BlueCloth.

It is licensed under the same terms as ruby.

I think it’s proper form to send the code for solutions in the body

of the mail, so

I’m just sending the regular ruby code to the list. If you want to

see the literate

version check out it out here:

http://www.ljstech.net/articles/rlit.html

This solution is a pretty simple one, but by leveraging erb and

BlueCloth it

makes getting an html version of the document really easy.

require ‘rubygems’
require ‘bluecloth’
require ‘cgi’
require ‘erb’
require ‘optparse’

module RLit
class Processor
attr_reader :html, :code

def initialize(io, which_chunk)
  corpus  = io.read
  @prog   = ERB.new(corpus, nil, '%<>')
  @chunks = Hash.new {|h,k| h[k] = ''}
  @html, @code = do_html, do_code(which_chunk)
end

def chunk c
  r,ch = nil,nil
  c.each do |chunk, code|
    @chunks[chunk] << code
    r = code
    ch = chunk
  end
  "<div 

class="caption">:#{ch}\n

#{CGI.escapeHTML(r)}
"
end

def ref c
  t = ERB.new(@chunks[c])
  t.result(binding)
end

def do_html
  doc = @prog.result(binding)
  '<link rel="stylesheet" href="style.css" type="text/css" />' +
  BlueCloth.new(doc).to_html
end

def do_code(which_chunk)
  return unless which_chunk
  t = ERB.new(@chunks[which_chunk])
  t.result(binding)
end

private :do_html, :do_code

end
end

def usage(opts)
puts opts; exit
end

method, arg = nil, nil

opts = OptionParser.new do |o|
o.banner = “Usage: #{File.basename $0} output_method FILE(s)”
o.separator ‘’
o.separator “output_method can be either”
o.on(‘-d’,‘–documentation’, ‘Output the document’) {
method = :html
}
o.on(‘-c’,‘–code [CHUNK_NAME]’, ‘Output the code for the
interpreter’) {|chunk|
arg = chunk
method = :code
}
o.separator ‘’
o.separator “Other options”
o.on(‘-h’,‘–help’, ‘Print this help message’) {
usage(o)
}
end

opts.parse!(ARGV)

unless method
puts “Invalid Arguments: Must specify an output method”
usage(opts)
end

p = RLit::Processor.new(ARGF, arg && arg.to_sym) #chunk)
puts p.send(method)

On Fri, 17 Nov 2006 22:56:55 +0900, Ruby Q. wrote:

“Literate Programming”[1] is an idea popularized by Donald Knuth, where the
traditional order of code and comments in a source file is switched. Instead of
using special delimiters to mark comments, special delimiters are used to mark
code.

Here’s my solution #1 which uses email style quoting. In my quest for
perfection, I figured out how to do this without polluting the execution
environment (from the perspective of the literate script) with any new
methods, constants, or (more amazingly) local variables. FILE and
LINE work as one would expect, returning the name and location in
the
original literate file.

#!/usr/bin/env ruby

#the literate interpreter cannot be implemented as literate code itself
#for obvious reasons. A literate compiler could.

#I chose to use the line prefix “> " because many email clients have
#automatic “Add Quote Chars” functions which can add this to the
#beginning of each line, without affecting wrapping.
line_prefix=/^> /
#Read the code, and get the file name right
if ARGV[0]
filename=ARGV.shift
code=open(filename).readlines
else
code=STDIN.readlines
#this is how ruby itself identifies stdin when that’s the source
#of its code
filename=”-"
end
#process the code to strip the documentation, and the line prefix
code.map! do |line|
if line=~line_prefix
line.sub(line_prefix,"")
else
#we want LINE to return the correct line number in the
#literate file when we evaluate the file so we don’t delete
#documentation lines – we just replace them with blank lines
“\n”
end
end

#the goal here is to have NO local variables or special
#methods introduced into the execution environment
def __ken_binding
self.class.class_eval {remove_method :__ken_binding}
binding
end

#evaluate, setting FILE appropriately
eval code.join, __ken_binding , filename

(Apologies if this turns out to be a dupe - I posted this about an hour
ago on the newsgroup and it doesn’t seem to be showing up anywhere).

Here’s my solution to this quiz. It’s basically a non-too-efficient
regex-based solution wrapped in a combination command-line script and
library. The main file (lrb.rb) contains both the library and script
code, while the other three files make up some tests of the
functionality in lrb.rb.

To run the tests, use the lrb script:

$ ./lrb.rb test.lrb

If you like, lrb will output the code it’s about to run before running
it if you enable ruby debugging:

$ ./lrb.rb test.lrb -d

The first argument has to be the filename. If the second argument is a
number, it’s taken as the index of the code block to run. Following
arguments are passed to the ruby interpreter that replaces the one
running lrb. You can run individual tests using this, but you need to
manually require test/unit. E.g:

$ ./lrb.rb test.lrb 1 -rtest/unit
$ ./lrb.rb test.lrb 2 -rtest/unit

(the numbers are 0-based, but the tests have a ‘require test/unit’ line
at block 0 which does nothing on it’s own). If you need to pass a number
as the first argument to your script, give the block number as – .

You can also get a dump of the extracted code, rather than running it:

$ ./lrb.rb test.lrb --dump

When used as a library, lrb provides a literate version of eval
(lrb_eval) and an lrb-enabled require (lrb_require) that won’t work with
rubygems :(. The library is not auto-required into the interpreter that
runs the lrb code.

It has it’s warts (especially, no handling of piped-in code or data, no
nested \begin…\end blocks, and debugging can be a PITA) but generally
it seems to work and is pretty flexible.

Thanks for this quiz - literate programming is something I’m still
finding out about (mostly via Haskell, which I’ve just started looking
at properly) so this was an interesting quiz topic for me :slight_smile:

On Fri, 17 Nov 2006 22:56:55 +0900, Ruby Q. wrote:

“Literate Programming”[1] is an idea popularized by Donald Knuth, where
the traditional order of code and comments in a source file is switched.
Instead of using special delimiters to mark comments, special delimiters
are used to mark code.

Here is my solution #2.

#!/usr/bin/env ruby

#This is a variation on my other solution. The same basic mechanisms
#are used for evaluation, but different demarcations are used.

#Code begins at \begin{ruby} or \begin{ruby}[codeword]
#Code ends at \end{ruby} or \end{ruby}[codeword], but only matching
#the original pattern. If a codeword was used to start the block, then
#the same codeword is required to end the block. If no codeword was used
#to start the block, then no codeword may be used at the end of the
#block.

#Of course, if literate blocks don’t nest properly in LaTeX, that’s
#beyond the scope of the Ruby Q. ;-).

#Read the code, and get the file name right
if ARGV[0]
filename=ARGV.shift
code=open(filename).readlines
else
code=STDIN.readlines
#this is how ruby itself identifies stdin when that’s the source
#of its code
filename="-"
end

#process the code to strip the documentation, and the demarcations
inblock=nil
code.map! do |line|

inblock=nil if inblock and
line=~/^\end{ruby}#{Regexp.escape(inblock)}$/

l=line
l="\n" if not inblock

if not inblock and line=~/^\begin{ruby}([\w+])?$/
if Regexp.last_match[1]
inblock=Regexp.last_match[1]
else
inblock=""
end
end

l
end

#the goal here is to have NO local variables or special
#methods introduced into the execution environment
def __ken_binding
self.class.class_eval {remove_method :__ken_binding}
binding
end

#evaluate, setting FILE appropriately
eval code.join, __ken_binding , filename

It’s amazing how complete some of these solutions are – it’s clear to
me that you could have a fully functional literate programming
environment in Ruby that supported HTML documentation, re-ordering of
code and fully functional require and evals in a few hundred lines of
code.

My solution doesn’t do most of that, but it does generate html and text
documentation using BlueCloth. The code is posted below, the literate
version is here:

http://www.theaboutbox.com/code/lrb/lrb.html

Cheers!
-Cameron

----- lrb.rb
require ‘rubygems’
require ‘bluecloth’
class LRB
def parse(io, &block)
current_state = :in_text
io.each_line do |line|
if current_state == :in_text
case line
when /^>\s?(.)/: yield :code, $1 + “\n” if block_given?
when /\begin{.
}\s*./: current_state = :in_code
else yield :text, line if block_given?
end
else
case line
when /\end{.
}\s*.*/: current_state = :in_text
else yield :code, line if block_given?
end
end
end
end
def self.to_code(io)
code = String.new
LRB.new.parse(io) do |type, line|
code << line if type == :code
end
return code
end
def self.to_markdown(io)
doc = String.new
LRB.new.parse(io) do |type, line|
case type
when :code: doc << " " << line
when :text: doc << line
end
end
return doc
end
def self.to_html(io)
markdown = self.to_markdown io
doc = BlueCloth::new markdown
doc.to_html
end
end # class LRB
if $0 == FILE
opt = ARGV.shift
file = ARGV.shift
case opt
when ‘-c’: puts LRB::to_code(File.new(file))
when ‘-t’: puts LRB::to_markdown(File.new(file))
when ‘-h’: puts LRB::to_html(File.new(file))
when ‘-e’: eval LRB::to_code(File.new(file))
else
usage = <<“ENDING”
Usage:
lrb.rb [option] [file]

Options:
-c: extract code
-t: extract text documentation
-h: extract html documentation
-e: evaluate as Ruby program
ENDING
puts usage
end
end

On 11/17/06, Ruby Q. [email protected] wrote:

Your task is to enable literate Ruby. What that means is up to you.

So I enabled literate programming in “Junebug”:www.junebugwiki.com, a
small wiki based on
“Camping”:http://code.whytheluckystiff.net/camping/wiki.

Redcloth provides all the markup capability, and any code on a page
can be executed in place, or generated to a plain .rb source file.

The modifications were minimal and are described here as a literate
program:

First, add 2 buttons to the normal page view. In module
Junebug::Views, modify the method #show. Existing code:

module Junebug::Views
def show
#…
_markup @version.body

+Add+ These two lines:

    _button 'raw source', R(Raw, @page.title, @version.version),

{:style=>‘float: left; margin: 5px 0 0 5px;’}
_button ‘exec’, R(Exec, @page.title, @version.version),
{:style=>‘float: right; margin: 5px 0 0 5px;’} if logged_in?
#…
end
end

To implement exec, the next step is a controller. The controller
fetches the page data, sets the appropriate version, and calls the
render function. It handles older versions of the page, allowing you
to go back and run prior versions of the code.

module Junebug::Controllers
class Exec < R ‘/([\w ]+)/exec’, ‘/([\w ]+)/(\d+)/exec’
def get page_name, version = nil
redirect(“#{Junebug.config[‘url’]}/login”) and return unless
logged_in?
@page_title = “Exec #{page_name}: Results”
@page = Page.find_by_title(page_name)
@version = (version.nil? or version == @page.version.to_s) ?
@page : @page.versions.find_by_version(version)
render :exec
end
end

The raw source button links to a controller for the raw view. It
starts out the same, except there is no redirect to the login page,
since there is no login required.

class Raw < R '/([\w ]+)\raw', '/([\w ]+)/(\d+)/raw'
  def get page_name, version = nil
    @page_title = "#{page_name}.rb"
    @page = Page.find_by_title(page_name)
    @version = (version.nil? or version == @page.version.to_s) ?

@page : @page.versions.find_by_version(version)

But the end is different - instead of calling render, which would wrap
the results in HTML tags, it just returns the raw code directly.

@headers['Content-Type'] =  "text/plain"
get_code(@version.body)
  end
end

end #module Junebug::Controllers

Next, the exec render function, back among the views:

module Junebug::Views
def exec
_header :show, @page.title
_body {
_markup @version.body
div.formbox {
form do
p {
label ‘Execution Results’

Here is the only interesting line. ‘execute’ is a new helper function
that will extract the code blocks from the markup, and execute the
code.

          textarea

_execute(get_code(@version.body,[@page.title])), :rows => 10, :cols =>
80
}
br
end
br :clear=>‘all’
}
_button ‘edit’, R(Edit, @page.title, @version.version),
{:style=>‘float: right; margin: 5px 0 0 5px;’} if logged_in? &&
(@version.version == @page.version && (! @page.readonly || is_admin?))
_button ‘view’, R(Show, @page.title, @version.version),
{:style=>‘float: right; margin: 5px 0 0 5px;’}
br
}
_footer {
text '[readonly] ’ if @page.readonly
span.actions {
text "Version #{@version.version} "
text "(current) " if @version.version == @page.version
a ‘«older’, :href => R(Exec, @page.title,
@version.version-1) unless @version.version == 1
a ‘newer»’, :href => R(Exec, @page.title,
@version.version+1) unless @version.version == @page.version
a ‘current’, :href => R(Exec, @page.title) unless
@version.version == @page.version
a ‘versions’, :href => R(Versions, @page.title)
}
}
end

Here’s the function that actually runs the code. It is a wrapper for
the FreakyFreaky Sandbox, which should protect our system from
evildoers. Note: This part is untested, since Sandbox does not
currently compile on Windows. For internal testing, the body can be
replaced with “eval(code)”.

def _execute code
  begin
    Sandbox.safe.eval(code)
  rescue Sandbox::Exception => show_ex
  end
end

end #module Junebug::Views

The only thing left is one helper function to take the page source and
extract anything in a code block.

module Junebug::Helpers
def get_code html,found_list

Here’s the extractor. Instead of reparsing the page, let RedCloth do
the work of identifying code:

  html = RedCloth.new(source).to_html
  code = 

html.scan(/$lt;code>.*?</code>/m).join(“\n”).gsub(/</?code/,‘’)

And here’s a tricky bit. If we find a require statement, first search
the wiki for a page matching that name…

  code.gsub!(/require\s*(["'])(.*?)\1/){|match|
    name = $+
    codepage = Junebug::Models::Page.find_by_title(name)
    if !found_list.include?(name) and codepage

and if we find it, replace the statement with the code from that page.
This allows us to put our project over multiple wiki pages. The
found_list tracks the replacements, to prevent us from getting into
require loops. If there is no matching page, the require statement is
left in place for ruby to handle normally.

      get_code codepage.body, found_list<< name
    end

In a poor use of overloading, a nil found_list argument is a signal
not to do the require expansion. The raw view uses this -trick-
feature. The last step is to unescape any HTML in the code, so that
Ruby gets ‘>’ instead of ==‘>’==

  } if found_list
  GGI.unescapeHTML code
end

end

And that’s it.

I ran into a few small issues with RedCloth, which might be a matter
of getting the right settings. My main problem was that some of the
markup gets inserted before the code sections are identified, so
various code symbols, like *'s and +'s are replaced by html tags.

-Adam

On Nov 22, 2006, at 9:23 AM, Adam S. wrote:

On 11/17/06, Ruby Q. [email protected] wrote:

Your task is to enable literate Ruby. What that means is up to you.

So I enabled literate programming in “Junebug”:www.junebugwiki.com, a
small wiki based on
“Camping”:http://code.whytheluckystiff.net/camping/wiki.

Very clever interpretation of the problem. Too cool.

James Edward G. II