Pythonic indentation (or: beating a dead horse)

Ruby
61

On May 20, 2009, at 15:28 , Tony A. wrote:

And I guess I just can’t shut up about this: your solution would
require a
scanner with some degree of grammatical awareness for every
expression which
contains an indent block. When you start talking about a solution
like
that, I really think you’re going down a rabbit hole.

To be fair, if you looked at ruby’s lexer and parser you’d know that
we’re already down a rabbit hole. Your statement completely ignores
the complexity already inherent in the language.

Don’t chalk me up as a proponent of this idea as I’ve written more
than my fair share of python and hate the language (and especially the
libraries).

62

On May 21, 1:12 pm, James B. [email protected] wrote:

(Side rant: interesting that so many Rubyists go ga-ga over magic
indentation in Yaml and Haml, but find the idea repulsive in Ruby.)

YAML and Haml are fundamentally different from Ruby, or do you
consider markup to be programming?
It’s true that Haml has the same any-Ruby-is-allowed rules as ERB, but
you naturally restrict yourself because you’re working with a template
as opposed to an actual program. (Or at least you should.)

And for the record, I get kind of annoyed with the significant
whitespace in Haml when it comes to Ruby code, but I love how easy and
clean HTML becomes.

63

On Thu, May 21, 2009 at 4:29 PM, Ryan D.
[email protected]wrote:

To be fair, if you looked at ruby’s lexer and parser you’d know that we’re
already down a rabbit hole.

Oh yes, things like Ruby’s lexer/parser feedback involved in
interpolated
strings makes my eyes bleed. That’s sure begging for a PEG.

That said, as messy as lexer/parser feedback is we’re talking about
something worse here: including grammatical knowledge in the scanner
which
cannot be expressed as a CFG at all. As bad as Ruby already is, the
solution is even messier than anything which already exists.

It would involve some ad hoc grammatical analysis by the lexer for the
purposes of having a pushdown deeper in the stack terminate one which is
higher up by sneakily injecting “end” tokens. Aiee!

64

On May 21, 2009, at 10:24 PM, Tony A. wrote:

Quite frequently I do something like:

str = [a, b, c].map do |x|
thing1 = do_something_to x
thing2 = do_something_else_to x
“#{thing1}:#{thing2}”
end.join(",")

Me too. It’s interesting how seemingly small changes can generate a
lot of unexpected corner cases and unforeseen consequences. Of
course, one could argue that using indentation does not preclude using
explicit blocks in the same file. I doubt, for example, that anyone
wishing to remove the ‘end’ keyword would wish to disallow something
like:

 array_of_strings_that_are_numbers.sort! {|a,b| a.to_i <=> b.to_i}

…even though the {} brackets are the same as do…end.

If explicit blocks were allowed in the same file where blocks were, by
default, determined by indentation, then you could have your cake and
eat it too in the example above.

Nevertheless, I think your gut feel is correct…this would be a
tricky thing to do right.

steven

65

On Thu, May 21, 2009 at 5:20 PM, Tony A. [email protected] wrote:

It would involve some ad hoc grammatical analysis by the lexer for the
purposes of having a pushdown deeper in the stack terminate one which is
higher up by sneakily injecting “end” tokens. Aiee!

Perhaps a minimalist example of the “expression problem” is in order.

Take this example (in normal Ruby):

result = 2 * [1,2,3].inject(0) do |a, b|
a + b
end
puts result

prints “12”

Here the 2 * takes precedence over the method invocation (with a block).
Furthermore, nothing precludes us from sticking additional expressions
at
the end of the block:

result = 2 * [1,2,3].inject(0) do |a, b|
a + b
end + 1
puts result

Here the 2 * [method_with_block] is lower precedence than the plus
operand,
so we get a parse tree ala:

(+ (* 2 [method_with_block]) 1)

And that doesn’t even include setting “result”!

But through funky lexer tricks, we want indent blocks to terminate the
entire expression stack somehow. We want [method_with_block] when it
hits a
dedent to pop all the way up the pushdown stack. It also means that a
method with a block must always be the rightmost operand in any
pushdown.

That is not an easy problem to solve, and afaict cannot be done in a
context
free grammar. It also limits some of the expressivity that Ruby
normally
provides. Quite frequently I do something like:

str = [a, b, c].map do |x|
thing1 = do_something_to x
thing2 = do_something_else_to x
“#{thing1}:#{thing2}”
end.join(“,”)

An ident-sensitive grammar would preclude that sort of thing.

But hey, if anyone’s still about trying to make an indent-sensitive
Ruby,
here’s a nice simple example based on the one above to cut your teeth
on:

result = 2 * [1,2,3].inject(0) do |a, b|
a + b
puts result

Parse that. I dare ya.

66

On May 20, 1:45 pm, Tony A. [email protected] wrote:

when Array
result
end
p result

The output of this program is:

[16, 22, 28]

Show me how you would parse the equivalent program in an
indentation-sensitive Ruby, e.g.:

Okay. But just to recap what’s gone before: you’ve said all along that
Pythonic indentation in Ruby was impossible. It’s impossible, because
it’s incompatible with blocks. It’s impossible, because everything’s
an expression. It’s impossible, because Guido said so. And I’ve asked
you over and over for an example demonstrating its impossibility.
You’ve had time to think it over, you were under no constraints, and
this is what you’ve come up with.

I found the old preprocessor script from awhile ago that I mentioned
earlier in this thread. I’ve pastebinned it at
http://pastebin.com/m5fee1e92.
It’s very short, and pretty simple. I have no doubt that it would need
a lot of work to be robust. (In particular, I think things need to be
added to the enumeration at line 68.)

But Tony, it blasted through your “impossible” example without
skipping a beat, working perfectly on the first try.

jhaas@littlefinger:~/pyruby$ cat pyrbtest.rb
BEGIN

Look, ma! No ends!

foo = [1,2,3]
x = foo.map do |n|:
n += 1
n *= 2

result = case x:
when Array
x.map! do |n|:
n *= 3
n += 4
when NilClass
x

result = if result.size > 10:
result[0…2]
else:
result

p result
jhaas@littlefinger:~/pyruby$ irb
irb(main):001:0> require ‘pyruby’
=> true
irb(main):002:0> require ‘pyrbtest’
[16, 22, 28]
=> true

As I’ve said repeatedly, I don’t follow your reasons for claiming that
this is impossible. Can you come up with a better demonstration?

67

On May 21, 9:16 am, J Haas [email protected] wrote:

I found the old preprocessor script from awhile ago that I mentioned
earlier in this thread. I’ve pastebinned it athttp://pastebin.com/m5fee1e92.

Sorry for the repeated posts, but I forgot to mention: I did not write
this, although I did make minor modifications. I don’t recall where I
found it, and Google was no help. I regret that I cannot credit the
author; should he happen to be reading this, speak up!

68

On Thu, May 21, 2009 at 10:27 PM, J Haas [email protected] wrote:

Okay. But just to recap what’s gone before: you’ve said all along that
Pythonic indentation in Ruby was impossible. It’s impossible, because
it’s incompatible with blocks. It’s impossible, because everything’s
an expression. It’s impossible, because Guido said so. And I’ve asked
you over and over for an example demonstrating its impossibility.
You’ve had time to think it over, you were under no constraints, and
this is what you’ve come up with.

My claims were it’s impossible with a Pythonic lexer and a backing
context
free grammar. I certainly didn’t claim that if you throw enough regexes
at
the problem it won’t go away.

I found the old preprocessor script from awhile ago that I mentioned
earlier in this thread. I’ve pastebinned it at
http://pastebin.com/m5fee1e92.
It’s very short, and pretty simple. I have no doubt that it would need
a lot of work to be robust. (In particular, I think things need to be
added to the enumeration at line 68.)

But Tony, it blasted through your “impossible” example without
skipping a beat, working perfectly on the first try.

Well great! I’m not really sure how that script works, but cool.
However,
what you have there is a marked departure from how Python actually
works.
But if you’re happy with it, great. Go for it and see how popular you
can
make it.

69

On May 20, 3:18 pm, Tony A. [email protected] wrote:

n = 2 * -[1,2,3,4,5].inject(0) do |a, b|
c = a * 2
a * c + b
end
result = 8 * -if n > 0
-n += 10000
else
n + 500000
end
puts result

BEGIN

n = 2 * -[1,2,3,4,5].inject(0) do |a, b|:
c = a * 2
a * c + b
result = 8 * -if n > 0:
-n += 10000
else:
n + 500000
puts result

Once again, worked perfectly on its first time through the
preprocessor.

70

On Thu, May 21, 2009 at 10:27 PM, J Haas [email protected] wrote:

Okay. But just to recap what’s gone before: you’ve said all along that
Pythonic indentation in Ruby was impossible. It’s impossible, because
it’s incompatible with blocks. It’s impossible, because everything’s
an expression. It’s impossible, because Guido said so. And I’ve asked
you over and over for an example demonstrating its impossibility.
You’ve had time to think it over, you were under no constraints, and
this is what you’ve come up with.

And sorry to be frank about this: I’ve been cordial and fact oriented in
this discussion, trying to explain my opinion as best as possible. Not
only
are you strawmanning me here, you’re being a right cunt. Perhaps you
could
try to express your ideas without making it personal?

71

Juan Z. wrote:

does and more.
JDD actually not only regrets using XML for Ant, he eclusively uses
Rake now to build his Java projects. At least that’s what I read,
anyway.

jwm

72

On May 22, 2009, at 12:27 AM, J Haas wrote:

As I’ve said repeatedly, I don’t follow your reasons for claiming that
this is impossible. Can you come up with a better demonstration?

First, let me say thank you for the code. This community thrives on
code, not on bickering. More code is always a good thing. That said…

cat tough.rb

“end” can make things clearer, sometimes…

result = case ARGV[0]
when /^\d*$/
“That’s an integer!”
when /^[A-Za-z]*$/
“That’s a word!”
else
“That’s something that’s not an integer or a word…”
end if ARGV[0]

puts result

73

On Fri, 2009-05-22 at 14:33 +0900, Tony A. wrote:

Well great! I’m not really sure how that script works, but cool.
However,
what you have there is a marked departure from how Python actually
works.
But if you’re happy with it, great. Go for it and see how popular you
can
make it.

It doesn’t really. Or at least only simplistically; by re-writing the
input code, attempting to place ‘end’ in the proper place based on
indentation. It breaks silently if you happen to misplace your
whitespace incorrectly, placing an ‘end’ in the wrong place.

74

n = 2 * -[1,2,3,4,5].inject(0) do |a, b|:
c = a * 2
a * c + b
result = 8 * -if n > 0:
-n += 10000
else:
n + 500000
puts result

Once again, worked perfectly on its first time through the
preprocessor.

It would appear that “end’s” actually help with readability in a few
cases :slight_smile:

I’m glad to see you have some code. Now put it on github, make a gem of
it, and advocate it :slight_smile:

Here are a few other thoughts on it. (Keeping things friendly and
civil…)

  1. maybe it could become more of a generic preprocessor? (i.e. you give
    it arbitrary rules, like “see this, change it to this”)?

  2. maybe it could be even smarter about its extrapolation, i.e. allow

4.times {|n|:
puts n

[accomodate for braces–one fewer keystroke!]

or even

4.times |n|:
puts n

[put in its own do’s and end’s if not there]

or who even needs :'s?

4.times |n|
puts n

[whenever it sees an ending | without preceding do or {, count it as an
indent based block]

That would be interesting, and much more whitespacey, though slightly
less ruby-y .

To clarify Tony’s point from above, you can’t really do case statements
using only indentation and :'s. Other constructs would work though
(except for the ambiguities mentioned earlier might prevent anythin but
a preprocessor being able to, for ruby’s case).

-=r

75

On May 21, 10:33 pm, Tony A. [email protected] wrote:

My claims were it’s impossible with a Pythonic lexer and a backing context
free grammar. I certainly didn’t claim that if you throw enough regexes at
the problem it won’t go away.

Tony, while I may be a newb to Ruby, I assure you that I’m no stranger
to mailing lists and Usenet, and I’m aware of these things called
“archives”. Perhaps you are too. If not, allow me to demonstrate:

On May 19, 3:23 pm, Tony A. [email protected] wrote:

On Tue, May 19, 2009 at 3:40 PM, J Haas [email protected] wrote:

I think code blocks are cool, and I love Ruby’s very flexible
expressiveness. I dig the way every statement is an expression
These are both incompatible with a Python-style indentation sensitive
syntax. You can have the Pythonic indent syntax or a purely expression
based grammar with multi-line blocks. You can’t have both.

And yet here I’ve demonstrated having both, while repeatedly asking
you for examples to exhibit the “incompatibility”, and repeatedly
shooting down every one you’ve so proudly produced. In the quoted text
above, please note the absence of words like “lexer” and “backing
context free grammar”. In fact, you were quite explicit about Pythonic-
style indentation syntax being the problem. You stood behind that
claim several times. Now tell me, Tony, was that claim correct, or was
it wrong?

Now getting back to your amended claim, that it’s impossible with a
Python lexer and a backing context free grammar, well, let’s not
forget that Ruby-style blocks themselves are impossible. Yes,
completely impossible! Can’t be done! You just can’t have Ruby-style
blocks, there’s no way to make them work… with the Commodore BASIC
interpreter, a ham sandwich, and half a ton of elephant dung. I
realize that this is a bold claim to make, but I’m gonna stand by it
without fear.

As I’ve said before, this isn’t Python. And whether or not my proposal
would be possible in a Python parser is of absolutely no interest to
me. Clearly it is possible in Ruby’s parser, it’s even possible in
Ruby itself.

Well great! I’m not really sure how that script works, but cool. However,
what you have there is a marked departure from how Python actually works.

Oh noes!! You mean my Ruby script is a marked departure from Python?
Heaven forfend!

And sorry to be frank about this:

No, you’re not. Not even a tiny bit.

I’ve been cordial and fact oriented in
this discussion,

Actually, Tony, I’ve found you to be arrogant and condescending
throughout. I’ve merely responded in kind.

Not only are you strawmanning me here, you’re being a right c–t.

I’m strawmanning? I’m strawmanning?? Thanks for the tip, Mr. Your-
Proposal-Is-Impossible-But-Only-If-I-Set-Ridiculous-Conditions-Such-As-
Requiring-Python’s-Parser-For-Some-Reason. And have you forgotten
this?

And a bit of supplemental information: I conducted a poll of what Rubyists’
favorite features are in the language. Blocks were #1 by a wide margin.

Clearly, this one deserves at least honorable mention on the all-time
list of most flagrant strawmen. I think you’re projecting.

And by the way, I’ve heard that in other English-speaking countries,
the profanity you used is not considered as shockingly offensive as it
is in the United States. Perhaps you’re from one of those countries
and are unaware that in America it’s among the most vile of curse
words. I know that you, with your emphasis on cordiality and fact-
orientation, will appreciate me bringing this to your attention.

That being said I will plead guilty to being a terrible spokesman for
my ideas (and this extends well beyond the current discussion) because
I lack tact, empathy, and sensitivity… in short, I’m an asshole (and
not, as you suggested, someplace a couple of inches anterior.) This
has bitten me more times than I can say, but in the end, I gotta be me.

76

On May 22, 6:32 am, Reid T. [email protected] wrote:

It doesn’t really. Or at least only simplistically; by re-writing the
input code, attempting to place ‘end’ in the proper place based on
indentation. It breaks silently if you happen to misplace your
whitespace incorrectly, placing an ‘end’ in the wrong place.

As I said when I posted it:

It’s very short, and pretty simple. I have no doubt that it
would need a lot of work to be robust.

In its present form, this is not something I want to do in production
code. It does however serve at least one valuable purpose: it
demonstrates that the code snippet that Tony posted, that he
specifically contrived as an example of a piece of Ruby code that
would be utterly impossible to parse without ‘end’ statements, was in
fact quite easily parsable.

77

Tony, while I may be a newb to Ruby, I assure you that I’m no stranger
to mailing lists and Usenet, and I’m aware of these things called
“archives”. Perhaps you are too. If not, allow me to demonstrate:

Tony’s point was that certain constructs, like case statements, won’t be
transformable into indentation only blocks. Does that make sense?
If you feel frustrated then maybe take a walk (to cool down) before
posting :slight_smile:
Take care.
-=r

78

“J Haas” [email protected] schrieb im Newsbeitrag
news:[email protected]…

This sort of structuring pieces of code stopped me from trying Python
seriously.
And I specifically addressed this in my OP. A lot of engineers who
haven’t done work in Python think that the idea of giving syntactical
significance to whitespace is a ridiculous idea. I used to be one of
them. Now I know better. Don’t knock it until you’ve tried it.

I’ve used many languages in team projects, for example Assembler,
Fortran,
Modula, C, C++, Pascal, Ruby, Batch, bash, TCL.
The only thing common to these projects: Nobody seems to be able to use
formatting consistently on different platforms.

Isn’t there an Syntax-Highlighting Editor out there which allows
assigning
1-Point size to End-Only-Lines?
Aside from the fact that this wouldn’t solve the problem that I’d
still have to write the damned things, this is possibly the
kludgiest solution imaginable.
So just use a macro which replaces CTRL-BS by end.

Wouldnt this be the easiest way to solve your problem?
The easiest way to solve my problem would be for me to use one of
the preprocessing scripts that impose Python-like indentation syntax
on Ruby.
You will additionally need preprocessing for importing code.

But that wouldn’t solve the larger problem, which is that
Ruby could be better than it is.
Then it would not be your problem anymore, but the problems of tons of
other
Rubyists. Which, at current time, probably have absolutely no problem
using
end or “}”.
(BTW: For at least 10 years, I’ve never read or heard about this
suggestion
for Ruby.)

Are you really sure you are no FUD-guy paid by MS which tries to
spoil other scripting languages in favor of F#?

79

Reid T. wrote:

It doesn’t really. Or at least only simplistically; by re-writing the
input code, attempting to place ‘end’ in the proper place based on
indentation. It breaks silently if you happen to misplace your
whitespace incorrectly, placing an ‘end’ in the wrong place.

This is an interesting aspect of indentation-sensitive syntax.

I’ve had Yaml files that, technically, were valid Yaml, but I had
mistakenly added some extra indentation. I then had to puzzle over some
odd errors when running my program.

It seems like an easy enough mistake to make, with the unfortunate risk
of creating proper Yaml, so you don’t get any errors from the parser.

I’ve wondered if this sort thing is a problem in Python; perhaps it is
less likely there that any accidental indent or out-dent will still give
you well-formed code.

–
James B.

www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff
www.neurogami.com - Smart application development

80

J Haas wrote:

On May 21, 10:33 pm, Tony A. [email protected] wrote:

My claims were it’s impossible with a Pythonic lexer and a backing context
free grammar. I certainly didn’t claim that if you throw enough regexes at
the problem it won’t go away.

Tony, while I may be a newb to Ruby, I assure you that I’m no stranger
to mailing lists and Usenet, and I’m aware of these things called
“archives”. Perhaps you are too. If not, allow me to demonstrate:

rampant snideness elided

However justified you may feel, you’re now presenting yourself really
poorly here.

You may disagree that Tony was being civil (he certainly struck me as
civil), but replying to perceived offense with even more rudeness is
tacky. Worse, it seems a good way to kill what could continue to be a
good thread.

Tacking on self-deprecating comments doesn’t change that, either.

–
James B.

www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff
www.neurogami.com - Smart application development