Strange whitespace parsing behavior on Ruby 1.8.7 (patchlevel 249/302)

An issue pops up when I use a method’s integer return value in a
function argument while performing arithmetic on that value. Then
missing whitespace between the arithmetical operator and the second
argument can cause random errors, but only if there is whitespace
preceding the operator. I don’t want to rush to judgement and call this
a bug, but it certainly seems like one, hidden till now because you have
have formatting that’s slightly ugly for the error to turn up. Here’s an
irb session that demonstrates the issue:
irb(main):062:0* x = ‘123456’=> "123456"irb(main):063:0> x.slice(1,
x.size -2)SyntaxError: compile error(irb):63: syntax error, unexpected
‘)’, expecting kDO_BLOCK from (irb):63 from
:0irb(main):064:0> x.slice(1, x.size - 2)=> "2345"irb(main):065:0>
x.slice(1, x.size-2)=> "2345"irb(main):066:0> x.slice(1, x.size
-2)SyntaxError: compile error(irb):66: syntax error, unexpected ‘)’,
expecting kDO_BLOCK from (irb):66 from :0irb(main):067:0>
x[x.size -3,1]ArgumentError: wrong number of arguments (2 for 0)
from (irb):67:in size' from (irb):67 from :0irb(main):068:0> x[x.size - 3,1]=> "4"irb(main):069:0> def ok(a,b)irb(main):070:1> end=> nilirb(main):071:0> ok(x.size +3,4)ArgumentError: wrong number of arguments (2 for 0) from (irb):71:insize’ from (irb):71 from :0irb(main):072:0>
ok(4,x.size +3)SyntaxError: compile error(irb):72: syntax error,
unexpected ‘)’, expecting kDO_BLOCK from (irb):72 from
:0irb(main):074:0> ok(4,x.size + 3)=> nil

On Tue, Nov 2, 2010 at 5:49 AM, Ehsanul H. [email protected]
wrote:

An issue pops up when I use a method’s integer return value in a function
argument while performing arithmetic on that value. Then missing whitespace
between the arithmetical operator and the second argument can cause random errors,
but only if there is whitespace preceding the operator. I don’t want to rush to
judgement and call this a bug, but it certainly seems like one, hidden till now
because you have have formatting that’s slightly ugly for the error to turn up.
Here’s an irb session that demonstrates the issue:
irb(main):062:0* x = ‘123456’=> "123456"irb(main):063:0> x.slice(1, x.size
-2)SyntaxError: compile error(irb):63: syntax error, unexpected ‘)’, expecting
kDO_BLOCK from (irb):63 from :0irb(main):064:0> x.slice(1, x.size - 2)=>
"2345"irb(main):065:0> x.slice(1, x.size-2)=> "2345"irb(main):066:0> x.slice(1,
x.size -2)SyntaxError: compile error(irb):66: syntax error, unexpected ‘)’,
expecting kDO_BLOCK from (irb):66 from :0irb(main):067:0> x[x.size
-3,1]ArgumentError: wrong number of arguments (2 for 0) from (irb):67:in size' from (irb):67 from :0irb(main):068:0> x[x.size - 3,1]=> "4"irb(main):069:0> def ok(a,b)irb(main):070:1> end=> nilirb(main):071:0> ok(x.size +3,4)ArgumentError: wrong number of arguments (2 for 0) from (irb):71:insize’ from (irb):71
from :0irb(main):072:0> ok(4,x.size +3)SyntaxError: compile error(irb):72: syntax
error, unexpected ‘)’, expecting kDO_BLOCK from (irb):72 from
:0irb(main):074:0> ok(4,x.size + 3)=> nil

Can you repost with proper indentation and line wrapping? Guessing
from your text I assume you might have fallen into a unary / binary
operator ambiguity.

Kind regards

robert

On Tue, Nov 2, 2010 at 12:19 PM, Colin B.
[email protected] wrote:

For the OP:

          # from (irb):63 from :0

Again, the “-” in " -2" is parsed on a unary on 2 giving integer value

-2.

Why the ‘)’ is unexpected I leave to Robert!

Very generous. :slight_smile:

Concluding from systematic experimentation I believe both errors you
left for me are related to the fact how Ruby parses (or rather: tries
to parse) in absence of brackets:

14:53:27 $ ruby19 -ce ‘a(b.c +1)’
Syntax OK
14:53:45 $ ruby19 -ce ‘a(d,b.c +1)’
-e:1: syntax error, unexpected tINTEGER, expecting ‘)’
a(d,b.c +1)
^
14:53:50 $ ruby19 -ce ‘a(b.c +1,e)’
Syntax OK
14:53:55 $ ruby19 -ce ‘a(d,b.c(+1))’
Syntax OK
14:54:08 $ ruby19 -ce ‘a((d,b.c)+1)’
-e:1: syntax error, unexpected ‘)’, expecting ‘=’
a((d,b.c)+1)
^
-e:1: syntax error, unexpected $end, expecting ‘)’
14:59:30 $ ruby19 -ce ‘a(d,(b.c) +1)’
Syntax OK

It seems that 14:53:45 is parsed like 14:54:08 which is apparently is
illegal because you also cannot do this:

14:54:16 $ ruby19 -ce ‘(a,b)’
-e:1: syntax error, unexpected ‘\n’, expecting ‘=’

14:58:18 $ ruby19 -ce ‘(d,b.c +1)’
-e:1: syntax error, unexpected tINTEGER, expecting tCOLON2 or ‘[’ or ‘.’
(d,b.c +1)
^

Hope that sheds some light.

Kind regards

robert

robert


remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Sorry about that, Hotmail’s “rich text” formatting messed it up I
believe. Here it goes again with plain text:
irb(main):062:0* x = ‘123456’=> "123456"irb(main):063:0> x.slice(1,
x.size -2)SyntaxError: compile error(irb):63: syntax error, unexpected
‘)’, expecting kDO_BLOCK from (irb):63 from :0 irb(main):064:0>
x.slice(1, x.size - 2)=> "2345"irb(main):065:0> x.slice(1, x.size-2)=>
"2345"irb(main):066:0> x.slice(1, x.size -2)SyntaxError: compile
error(irb):66: syntax error, unexpected ‘)’, expecting kDO_BLOCK from
(irb):66 from :0 irb(main):067:0> x[x.size -3,1]ArgumentError: wrong
number of arguments (2 for 0) from (irb):67:in size' from (irb):67 from :0 irb(main):068:0> x[x.size - 3,1]=> "4"irb(main):069:0> def ok(a,b)irb(main):070:1> end=> nilirb(main):071:0> ok(x.size +3,4)ArgumentError: wrong number of arguments (2 for 0) from (irb):71:insize’ from (irb):71 from :0 irb(main):072:0>
ok(4,x.size +3)SyntaxError: compile error(irb):72: syntax error,
unexpected ‘)’, expecting kDO_BLOCK from (irb):72 from :0
irb(main):074:0> ok(4,x.size + 3)=> nil
Insert mode

On 02.11.2010 17:15, Ehsanul H. wrote:

An issue pops up when I use a method’s integer return value in a function
argument while performing arithmetic on that value. Then missing whitespace
between the arithmetical operator and the second argument can cause random errors,
but only if there is whitespace preceding the operator. I don’t want to rush to
judgement and call this a bug, but it certainly seems like one, hidden till now
because you have have formatting that’s slightly ugly for the error to turn up.
Here’s an irb session that demonstrates the issue:

irb(main):062:0* x = ‘123456’=> "123456"irb(main):063:0> x.slice(1, x.size
-2)SyntaxError: compile error(irb):63: syntax error, unexpected ‘)’, expecting
kDO_BLOCK from (irb):63 from :0irb(main):064:0> x.slice(1, x.size -
2)=> "2345"irb(main):065:0> xslice(1, x.size-2)=> "2345"irb(main):066:0>
x.slice(1, x.size -2)SyntaxError: compile error(irb):66: syntax error, unexpected
‘)’, expecting kDO_BLOCK from (irb):66 from :0irb(main):067:0>
x[x.size -3,1]ArgumentError: wrong number of arguments (2 for 0) from
(irb):67:in size' from (irb):67 from :0irb(main):068:0> x[x.size - 3,1]=> "4"irb(main):069:0> def ok(a,b)irb(main):070:1> end=> nilirb(main):071:0> ok(x.size +3,4)ArgumentError: wrong number of arguments (2 for 0) from (irb):71:insize’ from (irb):71 from
:0irb(main):072:0> ok(4,x.size +3)SyntaxError: compile error(irb):72: syntax
error, unexpected ‘)’, expecting kDO_BLOCK from
(irb):72 from :0irb(main):074:0> ok(4,x.size + 3)=> nil

Can you repost with proper indentation and line wrapping? Guessing
from your text I assume you might have fallen into a unary / binary
operator ambiguity.

Sorry about that, Hotmail’s “rich text” formatting messed it up I believe Here
it goes again with plain text:
irb(main):062:0* x = ‘123456’=> "123456"irb(main):063:0> x.slice(1, xsize
-2)SyntaxError: compile error(irb):63: syntax error, unexpected ‘)’, expecting
kDO_BLOCK from (irb):63 from :0 irb(main):064:0> x.slice(1, x.size

  • 2)=> "2345"irb(main):065:0> x.slice(1, x.size-2)=> "2345"irb(main):066:0>
    x.slice(1, x.size -2)SyntaxError: compile error(irb):66: syntax error, unexpected
    ‘)’, expecting kDO_BLOCK from (irb):66 from :0 irb(main):067:0>
    x[x.size -3,1]ArgumentError: wrong number of arguments (2 for 0) from
    (irb):67:in `size’ from (irb):67 from :0 irb(main):068:0> x[x.size
  • 3,1]=> "4"irb(main):069:0> def ok(a,b)irb(main):070:1> end=>
    nilirb(main):071:0> ok(x.size +3,4)ArgumentError: wrong number of arguments (2
    for 0) from (irb):71:in `size’ from (irb):71 from :0
    irb(main):072:0> ok(4,x.size +3)SyntaxError: compile error(irb):72: syntax error,
    unexpected ‘)’, expecting kDO_BLOCK
    from (irb):72 from :0 irb(main):074:0> ok(4,x.size + 3)=> nil

Insert mode

Not really… :slight_smile:

robert

irb(main):062:0* x = ‘123456’=> "123456"irb(main):063:0> x.slice(1, xsize
-2)SyntaxError: compile error(irb):63: syntax error, unexpected ‘)’, expecting
kDO_BLOCK from (irb):63 from :0 irb(main):064:0> x.slice(1, x.size - 2)=>
"2345"irb(main):065:0> x.slice(1, x.size-2)=> "2345"irb(main):066:0> x.slice(1,
x.size -2)SyntaxError: compile error(irb):66: syntax error, unexpected ‘)’,
expecting kDO_BLOCK from (irb):66 from :0 irb(main):067:0> x[x.size
-3,1]ArgumentError: wrong number of arguments (2 for 0) from (irb):67:in size' from (irb):67 from :0 irb(main):068:0> x[x.size - 3,1]=> "4"irb(main):069:0> def ok(a,b)irb(main):070:1> end=> nilirb(main):071:0> ok(x.size +3,4)ArgumentError: wrong number of arguments (2 for 0) from (irb):71:insize’ from (irb):71 from :0
irb(main):072:0> ok(4,x.size +3)SyntaxError: compile error(irb):72: syntax error,
unexpected ‘)’, expecting kDO_BLOCK
from (irb):72 from :0 irb(main):074:0> ok(4,x.size + 3)=> nil

Insert mode

Not really… :slight_smile:

robert
You know what, I should probably just use pastie. Here you
go:http://pastie.org/1267606

Insert mode

On Tue, Nov 2, 2010 at 7:17 AM, Robert K.
[email protected]wrote:

Can you repost with proper indentation and line wrapping?
Guessing from your text I assume you might have fallen
into a unary / binary operator ambiguity.

That seems to be a correct diagnosis. See below for the OP’s irb session
with formatting.
For the OP:

7.abs + 3 #=> 10
7.abs+3 #=> 10

7.abs +3 #=> ArgumentError: wrong number of arguments(1 for 0)

Ruby parses the “+” in " +3" as a unary + on 3 giving integer 3,

which means that integer 3 is an argument to abs, hence the error

message.

def okp( a, b ); a + b; end #=> nil

okp(4, 7.abs + 3) #=> 14
okp(4, 7.abs+3) #=> 14
okp(4, 7.abs +3) #=> SyntaxError: (irb):23: syntax error,
# unexpected tInteger, expecting “)”

I leave that - and the next - one to Robert to explain!

x = ‘123456’ #=> “123456”
x.slice(1, x.size -2) #=> SyntaxError: compile error(irb):63:
# syntax error, unexpected ‘)’,
# expecting kDO_BLOCK
# from (irb):63 from :0

Again, the “-” in " -2" is parsed on a unary on 2 giving integer

value
-2.

Why the ‘)’ is unexpected I leave to Robert!

*** the OP’s irb session

irb(main):062:0* x = ‘123456’=> “123456”

irb(main):063:0> x.slice(1, x.size -2)
SyntaxError: compile error(irb):63: syntax error, unexpected ‘)’,
expecting kDO_BLOCK
from (irb):63 from :0

irb(main):064:0> x.slice(1, x.size - 2)=> “2345”

irb(main):065:0> x.slice(1, x.size-2)=> “2345”

irb(main):066:0> x.slice(1, x.size -2)
SyntaxError: compile error(irb):66: syntax error, unexpected ‘)’,
expecting kDO_BLOCK
from (irb):66 from :0

irb(main):067:0> x[x.size -3,1]
ArgumentError: wrong number of arguments (2 for 0)
from (irb):67:in `size’ from (irb):67 from :0

irb(main):068:0> x[x.size - 3,1]=> “4”

irb(main):069:0> def ok(a,b)
irb(main):070:1> end=> nil

irb(main):071:0> ok(x.size +3,4)
ArgumentError: wrong number of arguments (2 for 0)
from (irb):71:in `size’ from (irb):71 from :0

irb(main):072:0> ok(4,x.size +3)
SyntaxError: compile error(irb):72: syntax error, unexpected ‘)’,
expecting kDO_BLOCK
from (irb):72 from :0

irb(main):074:0> ok(4,x.size + 3)=> nil


(irb):72 from :0irb(main):074:0> ok(4,x.size + 3)=> nil

Insert mode

while you seem to be expecting

ok((x.size + 3), 4))

Similarly with all your other examples.

-Justin

That makes sense. But the inconsistency between the different behaviors
given just a small difference in whitespace is still quite off-putting.

Also how is this line parsed?:

x.slice(1, x.size -2)

  • It seems unreasonable to parse it any way other than that
    intended.Insert mode

Ehsanul H. wrote:

Sorry about that, Hotmail’s “rich text” formatting messed it up I believe Here
it goes again with plain text:
You know what, I should probably just use pastie. Here you go:
http://pastie.org/1267606

Insert mode

Robert was right.

ok(x.size +3,4)

is parsed as

ok(x.size(+3, 4))

while you seem to be expecting

ok((x.size + 3), 4))

Similarly with all your other examples.

-Justin

On 11/02/2010 10:45 PM, Ehsanul H. wrote:

operator ambiguity.

Insert mode
ok(x.size(+3, 4))
-Justin

That makes sense. But the inconsistency between the different behaviors given
just a small difference in whitespace is still quite off-putting.

Also how is this line parsed?:

x.slice(1, x.size -2)

  • It seems unreasonable to parse it any way other than that intended.Insert mode

Poorly.

Also, I somehow missed Colin and Robert’s answers, so I’ll refer you to
them.

-Justin

On Wed, Nov 3, 2010 at 5:45 AM, Ehsanul H.
[email protected]wrote:

Subject to correction by Robert - or anyone else! - to summarise I think
the
basic problem for Ruby here is when to parse “-2” (without any space
between
the “-” and the “2”) as a unary operator on 2 and as a binary operator
on a
previous “token” with 2 as the argument.

At one extreme, Ruby could raise an error if there is any possible
ambiguity. Ruby, I think, tries to do the best it can with limited
knowledge of the methods being interpreted.

For example:
7 -2 => 5
this works for a similar reason to the reason this works:
x = 13
x -2 #=> 11
Ruby knows (or anyway interprets) x as a variable, not a method, so no
ambiguity: the “-” is either a mistake or is a binary operator.

x.size #=> 4
x.size-3 #=> 1
I’m guessing that here Ruby “thinks” no whitespace, so bind the “-” to
the
first possible token, which is (x.size), so “-” gets treated as a binary
operator.

But in:
x.size -2
this could mean
((x.size) - 2) or (x.size( -2 ))
In this case size doesn’t take any arguments, so with some more work the
interpreter could parse it as being likely to be ((x.size)-2), but that
won’t work for methods which can take zero or one (or more) arguments.
So (again a guess) the idea is that the interpreter won’t even try to to
some extra work which might, or might not, resolve the ambiguity.

The result is what seems to me to be a reasonable compromise, but I
ought to
add that some other possibilities might also be a reasonable (but
different)
compromise!

On Wed, Nov 3, 2010 at 6:45 AM, Ehsanul H. [email protected]
wrote:

That makes sense. But the inconsistency between the different behaviors given
just a small difference in whitespace is still quite off-putting.

One whitespace or an infinite number is rather irrelevant. A small
difference to you is a world of difference for a computer. :wink:

Also how is this line parsed?:

x.slice(1, x.size -2)

  • It seems unreasonable to parse it any way other than that intended.

Computers can’t read your mind. They do what you tell them, not what
you mean. That results in you having to do the work of expressing your
thoughts in terms the computer can actually understand and that are
also logically correct.


Phillip G.

Though the folk I have met,
(Ah, how soon!) they forget
When I’ve moved on to some other place,
There may be one or two,
When I’ve played and passed through,
Who’ll remember my song or my face.

On Wed, Nov 3, 2010 at 4:07 PM, Colin B. [email protected]
wrote:

  • It seems unreasonable to parse it any way other than that intended.Insert
    knowledge of the methods being interpreted.

For example:
7 -2 => 5
this works for a similar reason to the reason this works:
x = 13
x -2 #=> 11
Ruby knows (or anyway interprets) x as a variable, not a method, so no
ambiguity: the “-” is either a mistake or is a binary operator.

I think the Ruby interpreter knows it because during parsing it
recognizes “x = 13” as a local variable assignment which makes x a
local variable and will shadow method x (if it exists). That
knowledge might also be used to disambiguate “x -2” which does not
really make sens with a local variable if not interpreted as an
expression with a binary “-”.

In this case size doesn’t take any arguments, so with some more work the
interpreter could parse it as being likely to be ((x.size)-2), but that
won’t work for methods which can take zero or one (or more) arguments.
So (again a guess) the idea is that the interpreter won’t even try to to
some extra work which might, or might not, resolve the ambiguity.

The result is what seems to me to be a reasonable compromise, but I ought to
add that some other possibilities might also be a reasonable (but different)
compromise!

Absolutely agree. I’d also like to add that this has never been an
issue for me in practice. So we are probably putting something under
a microscope which is actually really small. :slight_smile:

Cheers

robert

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs