Ruby Forum Ruby-core > Collaborative Ruby Language Specification

Hi Everyone,

I thought I'd throw out some suggestions about creating a formal 
specification for the Ruby Language. I'm also new to Microsoft, so I'm 
learning a lot about the rules of engagement here, so please forgive me 
if you find some of my proposals to be a big ... strange.

I think it would be really useful if we could create a central 
repository for an official language verification suite. Like a lot of 
folks, I favor reading code over reading prose, wherever possible. I've 
been working on a little side project where I'm defining language 
behavior using RSpec. Over the next week or so, I anticipate being able 
to spend a significant chunk of time working on fleshing out my spec. 
I'd like to contribute it back to the community, and use it as a 
starting point for some more serious discussions about the definition of 
the language.

Here are my ideas:


1)      I think that RubyForge would be a natural place to host the 
specification project.

2)      I think that the license for the specification project would 
need to be very open - something like MIT would rock.

3)      It would be great to give commit rights to representatives from 
each of the Ruby implementation projects by default, and to any 
interested members from the Ruby community.

4)      Would it be possible to have RubyCentral act as the owner of the 
project? Or some other neutral party? Suggestions welcome.

5)      We should focus our energy on documenting existing behavior of 
Ruby - do folks object to 1.8.4 as the baseline?

Thanks!
-John

On 1/28/07, John Lam (CLR) <jflam@microsoft.com> wrote:

> I'm also new to Microsoft, so I'm
> learning a lot about the rules of engagement here, so please forgive me if
> you find some of my proposals to be a big … strange.

?  Ruby's not being developed by Microsoft, so why would their "rules
of engagement" matter here?

[Collaborative Ruby Language Specification]

Please see

http://www.rubyforge.org/projects/rubygrammar/

  nikolai

On 1/28/07, Nikolai Weibull <now@bitwi.se> wrote:
> On 1/28/07, John Lam (CLR) <jflam@microsoft.com> wrote:
>
> > I'm also new to Microsoft, so I'm
> > learning a lot about the rules of engagement here, so please forgive me if
> > you find some of my proposals to be a big … strange.
>
> ?  Ruby's not being developed by Microsoft, so why would their "rules
> of engagement" matter here?

If Microsoft wants to pay a smart (and nice) guy like John to work on
a Ruby spec, and that is the list of conditions (which sounds
completely reasonable to me). I am all for it.

pth

Nikolai Weibull wrote:
> On 1/28/07, John Lam (CLR) <jflam@microsoft.com> wrote:
> 
>> I'm also new to Microsoft, so I'm
>> learning a lot about the rules of engagement here, so please forgive 
>> me if
>> you find some of my proposals to be a big … strange.
> 
> ?  Ruby's not being developed by Microsoft, so why would their "rules
> of engagement" matter here?

Of course, they won't matter for Ruby, but they do matter for John Lam,
which is what he's saying.

> http://www.rubyforge.org/projects/rubygrammar/

That's only one part of it. The rubygrammar project won't ever say
anything about Marshalling, for example, and that would be a very
important part of a Ruby Language Specification.

I dare say that we in the JRuby project thinks this is a great idea, and
RSpec is a very nice platform to build such a spec on.

Take a look at
http://headius.com/rubyspec/index.php/Main_Page

Where we have begun work against a specification for Ruby.

Regards
--
  Ola Bini (http://ola-bini.blogspot.com)
  JvYAML, RbYAML, JRuby and Jatha contributor
  System Developer, Karolinska Institutet (http://www.ki.se)
  OLogix Consulting (http://www.ologix.com)

  "Yields falsehood when quined" yields falsehood when quined.

>>That's only one part of it. The rubygrammar project won't ever say
>> anything about Marshalling, for example, and that would be a very
>> important part of a Ruby Language Specification.

Exactly. There's also the dynamic <-> static language interop questions 
to consider, which directly affects projects like JRuby. For example - 
when do you provide a Java string vs. when do you provide a string that 
maps to Ruby semantics?

I envision a RSpec spec being useful to folks that would be writing a 
scanner/parser for Ruby. Other things which dive into more runtime 
semantics that will only run on a subset of Ruby's in the wild will be 
better served by wiki-style docs.

I think that the RSpec suite will be useful for language conformance 
testing, and for resolving questions like whether there's a bug in the 
language implementation vs. a bug in the test suite.

Thanks,
-John



 -----Original Message-----
From: Ola Bini [mailto:ola.bini@ki.se]
Sent: Sunday, January 28, 2007 10:05 AM
To: ruby-core@ruby-lang.org
Subject: Re: Collaborative Ruby Language Specification

Nikolai Weibull wrote:
> On 1/28/07, John Lam (CLR) <jflam@microsoft.com> wrote:
>
>> I'm also new to Microsoft, so I'm
>> learning a lot about the rules of engagement here, so please forgive
>> me if
>> you find some of my proposals to be a big ... strange.
>
> ?  Ruby's not being developed by Microsoft, so why would their "rules
> of engagement" matter here?

Of course, they won't matter for Ruby, but they do matter for John Lam,
which is what he's saying.

> http://www.rubyforge.org/projects/rubygrammar/

That's only one part of it. The rubygrammar project won't ever say
anything about Marshalling, for example, and that would be a very
important part of a Ruby Language Specification.

I dare say that we in the JRuby project thinks this is a great idea, and
RSpec is a very nice platform to build such a spec on.

Take a look at
http://headius.com/rubyspec/index.php/Main_Page

Where we have begun work against a specification for Ruby.

Regards
--
  Ola Bini (http://ola-bini.blogspot.com)
  JvYAML, RbYAML, JRuby and Jatha contributor
  System Developer, Karolinska Institutet (http://www.ki.se)
  OLogix Consulting (http://www.ologix.com)

  "Yields falsehood when quined" yields falsehood when quined.

On 1/28/07, Patrick Hurley <phurley@gmail.com> wrote:
> If Microsoft wants to pay a smart (and nice) guy like John to work on
> a Ruby spec, and that is the list of conditions (which sounds
> completely reasonable to me). I am all for it.

I'm obviously a stupid (and unpleasant) guy unlike John, but what list
of conditions?

I was just wondering what he was trying to say.

  nikolai

On 1/28/07, Ola Bini <ola.bini@ki.se> wrote:
>
> Of course, they won't matter for Ruby, but they do matter for John Lam,
> which is what he's saying.

I guess the "here" threw me, but that shouldn't come as a surprise
because, as I've already stated earlier, I'm an ignorant asshole, so I
really need very clear formulations to be able to understand other
people.

> > http://www.rubyforge.org/projects/rubygrammar/

> That's only one part of it. The rubygrammar project won't ever say
> anything about Marshalling, for example, and that would be a very
> important part of a Ruby Language Specification.

Then how about

http://rubytests.rubyforge.org/

?

I figured John might want some pointers to already existing projects
that seem to be doing roughly the same thing that he's doing (or at
least some part of it), so that everyone can get together and work on
some unified "Ruby Language Specification", "collaboratively".
Perhaps he already knew about these two projects, but I was trying to
be helpful, for a change - considering that I'm usually not a nice
guy, so I figured I'd post a link I felt relevant to this discussion.
I promise that this will be my last response, as I'm obviously doing
more harm than good.

   nikolai (awaiting your responses to tell me what I did wrong this 
time)

John Lam (CLR) wrote:
> folks, I favor reading code over reading prose, wherever possible. 
> specification project.
>
> 5) We should focus our energy on documenting existing behavior of Ruby 
> – do folks object to 1.8.4 as the baseline?
>
> Thanks!
>
> -John
>
Pat Eyler, feel free to jump in here and correct me if I mis-speak:

1. At RubyConf 2006, there was an implementers' summit, at which the
YARV, Rubinius, jRuby and (I think) CLR people were all represented.
There was not to my knowledge a Cardinal or Carbone representative.

2. The agreement as I recall it was that aside from YARV/Ruby 1.9/2.0,
the standard was Ruby 1.8.5, not Ruby 1.8.4.

3. Part of the agreement was that the BFTS would come out of the back
room, appear on RubyForge, and be "the standard test suite for Ruby
1.8.x implementations". I believe that has been done.

4. Other folks volunteered to work on documenting the syntax and
semantics, although I don't recall who they were or what the status of
these efforts is.

5. There is to be another implementers' summit at the MountainWest Ruby
Conference in Salt Lake City March 16 -17, 2007.

6. Ruby 1.8.6 is due for release somewhere in the same time frame as the
MountainWest Ruby Conference, driving another nail in the 1.8.4 coffin.

So I'm guessing a lot of these ideas are either in progress in some form
or another already. But there is one thing in your proposal that
intrigues me -- using RSpec to define language behavior. I don't know
much about BFTS, but I imagine it isn't written in RSpec, since it was
initially done before RSpec had any significant mind share in the Ruby
community. In any event, I find RSpec code a heck of a lot easier to
read and write than Test::Unit code or "raw Ruby", so I'd welcome a
"port" of BFTS to RSpec. But it's not something I'd personally spend
time on. I'd much rather see the concurrency primitives made as elegant,
clean and efficient as is humanly possible in all of the
implementations, for example.

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given 
rabbits fire.

Hi --

On Mon, 29 Jan 2007, John Lam (CLR) wrote:

> a scanner/parser for Ruby. Other things which dive into more runtime
> semantics that will only run on a subset of Ruby's in the wild will
> be better served by wiki-style docs.

All the Rubys out there should run the same things with the same
results, though.  At least I assume that ensuring that would be the
main goal of a test-based specification.  Or am I misunderstanding
what you mean?

(There may be Ruby-like languages that aren't Ruby-compatible, but I'm
thinking of things that describe themselves as Ruby interpreters or
VMs or whatever.)


David

Hi --

On Mon, 29 Jan 2007, John Lam (CLR) wrote:

> 4)      Would it be possible to have RubyCentral act as the owner of
> the project? Or some other neutral party? Suggestions welcome.

I'm not sure what there is to be non-neutral about :-)


David

>> I'm not sure what there is to be non-neutral about :-)

Here's the problem: there are going to be multiple implementations of 
Ruby in the wild. And for those who run in other VMs, there will be 
compatibility problems. It's up to the spec to make determinations about 
what are 'important' incompatibilities vs. 'unimportant' 
incompatibilities. For example, which Ruby C libraries will be deemed to 
be 'unimportant' and not something that must be ported to a 3rd party VM 
in order for that language to be called 'Ruby'.

So, it's in the best interests of the community to have a neutral 3rd 
party be the 'owner' of the spec, otherwise there may be the perception 
of, let's say, some large company trying to steer the specification to 
run Ruby better on its own VM. These are issues that I'd like to get out 
in the open and have a resolution that everyone is comfortable with, and 
as early as possible in the process.

Thanks,
-John

Hi --

On Mon, 29 Jan 2007, John Lam (CLR) wrote:

>
> So, it's in the best interests of the community to have a neutral
> 3rd party be the 'owner' of the spec, otherwise there may be the
> perception of, let's say, some large company trying to steer the
> specification to run Ruby better on its own VM. These are issues
> that I'd like to get out in the open and have a resolution that
> everyone is comfortable with, and as early as possible in the
> process.

If it's a matter of the applicability of the name Ruby, then Matz is
the first and last arbitrator.


David

On Mon, 29 Jan 2007, dblack@wobblini.net wrote:

>> 'unimportant' incompatibilities. For example, which Ruby C libraries
>> process.
>
> If it's a matter of the applicability of the name Ruby, then Matz is
> the first and last arbitrator.

I should add: I'm not unwilling for Ruby Central to get involved in
some way, but I'd want to be clear that it wasn't at the level of
actually making decisions about what was or was not Ruby, since that's
Matz's prerogative (unless he delegates it, of course).

(And I probably meant "arbiter" :-)


David

>> At RubyConf 2006, there was an implementers' summit, at which the
YARV, Rubinius, jRuby and (I think) CLR people were all represented.
There was not to my knowledge a Cardinal or Carbone representative.

I was at that summit (and will be at the MountainWest summit). However, 
I've been spending all of my time over the past couple of months moving 
my family to a new country and starting a new job at Microsoft - so this 
is a way for me to restart the conversations around this - at least from 
my end. I'm really happy that I finally have cycles to start devoting to 
Ruby again!

By the end of next week, I should have a reasonable core set of stuff in 
RSpec form, and I'll be looking for a place to put it.

One thing that I'm learning at Microsoft is the influence of the legal 
department on things that I can and cannot do. I'm still discovering 
what those things are, so please be patient as I figure it out.

>> I'd much rather see the concurrency primitives made as elegant, clean and efficient as is humanly possible in all of the implementations, for example.

Amen.

Strange question of the day: under what terms is BFTS licensed? I didn't 
see a description of it in Rubyforge.

Thanks
-John

I'm sorry - I didn't mean that Ruby Central would make final technical 
decisions - that's clearly Matz's job. However, there are lots of 
administrative and sponsorship issues where it makes sense for a neutral 
organization to drive it. Witness the Python Software Foundation and how 
they drive their process.

There are also issues about IP rights assignment of contributors to the 
*specification*. This is why there are long, formal processes around any 
real standardization, with lots of scary legal terms and agreements 
thrown into the mix. I suspect that such a large effort would be beyond 
the scope of what we're all trying to do here.

Back to the original point: rather than creating a 'Ruby Software 
Foundation', might it make better sense to drive spec work through Ruby 
Central?

Thanks
-John

John Lam (CLR) wrote:
> One thing that I'm learning at Microsoft is the influence of the legal department on things that I can and cannot do. I'm still discovering what those things are, so please be patient as I figure it out.
>   
This is by no means unique to Microsoft. I have never worked in *any*
organization that didn't have such restrictions, and the saddest part
about them is that in many cases you only find out what they are by
violating them and getting a severe -- and temporarily career-limiting
-- reprimand. I don't know how it is in other nations that call
themselves "democratic" and "capitalistic", but here in the USA, lawyers
and accountants must be reckoned with constantly. I understand ... I
will be patient ... and I hope *you* have the patience, because you'll
need it more than I do. :)
> Strange question of the day: under what terms is BFTS licensed? I didn't see a description of it in Rubyforge.
>   
If the project is there, its front page should in fact specify a
license. When you open a project, that's one of the things you fill out
in the form.

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given 
rabbits fire.

John Lam (CLR) wrote:
> By the end of next week, I should have a reasonable core set of stuff in RSpec form, and I'll be looking for a place to put it.

Take a look at the work Rubinius have done for RSpec:ing their core
before starting to hack yourself.

--
  Ola Bini (http://ola-bini.blogspot.com)
  JvYAML, RbYAML, JRuby and Jatha contributor
  System Developer, Karolinska Institutet (http://www.ki.se)
  OLogix Consulting (http://www.ologix.com)

  "Yields falsehood when quined" yields falsehood when quined.

John Lam (CLR) wrote:
> Hi Everyone,

Hello John! You're alive!

> I think it would be really useful if we could create a central 
> repository for an official language verification suite. Like a lot of 
> folks, I favor reading code over reading prose, wherever possible. I’ve 
> been working on a little side project where I’m defining language 
> behavior using RSpec. Over the next week or so, I anticipate being able 
> to spend a significant chunk of time working on fleshing out my spec. 
> I’d like to contribute it back to the community, and use it as a 
> starting point for some more serious discussions about the definition of 
> the language.

I hope such a spec would be developed "in the open" from the beginning,
rather than being developed behind closed doors and only opened much
later. And perhaps that's what you're getting at with your points 
below...

> 1)      I think that RubyForge would be a natural place to host the 
> specification project.

That seems reasonable to me, and there's already the RubyTests project
which has been collecting test suites from multiple other projects. I've
long wanted that to "reawaken" as the source for a complete test/spec
suite for Ruby, since it's such a nicely named project and there's
already a bunch of folks interested in it.

See: RubyTests project on RubyForge

> 2)      I think that the license for the specification project would 
> need to be very open - something like MIT would rock.

Well, of course Microsoft would be interested in something they could
take behind closed doors :) Seriously though, for the spec/test suite,
just about anything is fine; I'd be very surprised if anyone were able
to take a test suite to closed source and do anything evil with it.

> 3)      It would be great to give commit rights to representatives from 
> each of the Ruby implementation projects by default, and to any 
> interested members from the Ruby community.

RubyTests already has committers from almost all the major projects
(except the CLR-based ones, though they're welcome to come too). And
we've already contributed our JRuby tests back, eventually to use
RubyTests as our primary test repository.

You will find it's difficult to get people to work on tests, but with
the rising interest in JRuby and Rubinius, there seems to be growing
desire to collaborate.

> 4)      Would it be possible to have RubyCentral act as the owner of the 
> project? Or some other neutral party? Suggestions welcome.

Owner?

> 5)      We should focus our energy on documenting existing behavior of 
> Ruby – do folks object to 1.8.4 as the baseline?

1.8.4 at a minimum, and we've generally just been going with 1.8.5 for
test and spec work.

...

And Ola brought it up, but I'll mention it again. The RubySpec project
is a wiki for community-driven spec/documentation of Ruby. So far it's
been helpful for our efforts implementing Marshal behavior, and we try
to update it whenever we can. However, again, this is a tough area to
get people to contribute given the prevalence of books that answer
peoples' questions and a general lack of desire to spend time
documenting Ruby's nooks and crannies.

See: http://www.headius.com/rubyspec
(soon to be moved to a real host on a nice machine)

I believe that a documented spec, even a free-form, community-driven
spec, is a necessary complement to a test suite. Yes, rspec creates nice
output. It's not nearly human-readable enough for an implementer to flip
through it and find some implementation detail they might have missed. A
test/spec suite also assumes a number of things: namely, that the parser
and core interpreter are already functional, and that a number of basic
builtin classes and libraries already work correctly. But how do you get
up to that point without a grammar, documentation on the interpreter,
and so on?

See: RubyGrammar project on RubyForge

And as far as compliance with a spec goes, I think almost any
implementation will be able to comply at an extremely high level. JRuby
obviously is a completely different implementation on a completely
different VM, but we can run Rails and Gems and Camping and almost all
the weirdest and wildest pure-Ruby libraries. And ideally, any Ruby spec
would include a reasonable level of details, features, libraries, and so
on that make up Ruby, so there's little chance of someone creating a
"Ruby-like" language that alters something crucial.

I'd love for JRuby to be taken as the model of how to implement
Ruby...we have the deepest respect for "matz's Ruby" and consider
compliance with that implementation of paramount importance. We have no
desire to extend or alter the language in incompatible ways, and our
primary focus has not just been "implementing Ruby" but actually being
able to run real Ruby apps. To this end, I believe a complete
specification should also include a list of applications that should
reasonably be expected to work on a given implementation. Is a Ruby
implementation complete without support for RubyGems or Rake? Or without
Rails? RSpec? These are staples of the Ruby world.

See: Legion, Pat Eyler's collection of app tests, and JRuby's own
similar collection of tests

In closing, I'd say I'm 100% behind any effort to form a complete spec,
and I've been trying to push such efforts in many different places. It
won't be an easy thing to create, but the value of such a spec would be
immeasurable.

- Charlie

John Lam (CLR) wrote:
>>> I'd much rather see the concurrency primitives made as elegant, clean and efficient as is humanly possible in all of the implementations, for example.
> 
> Amen.
> 
> Strange question of the day: under what terms is BFTS licensed? I didn't see a description of it in Rubyforge.

Knowing Ryan it's probably a pretty liberal license. Also see the tests
under RubyTests, which are not as complete for individual classes but
which cover a much wider range of language and library features.

And of course JRuby's tests are quite large too, and complete enough
that during massive redesign and refactoring changes all we generally
need to do is get the test suite green again...and the apps are all back
to working.

I've been trying to get implementers to all use RubyTests for their test
repo, but everyone seems to want to have things locally. C'est la vie.

- Charlie

Charles Oliver Nutter wrote:
> You will find it's difficult to get people to work on tests, but with 
> the rising interest in JRuby and Rubinius, there seems to be growing 
> desire to collaborate.
Well, I for one don't mind working on tests in RSpec, but I'd balk at
using the older tools.
> I'd love for JRuby to be taken as the model of how to implement 
> Ruby...we have the deepest respect for "matz's Ruby" and consider 
> compliance with that implementation of paramount importance. We have 
> no desire to extend or alter the language in incompatible ways, and 
> our primary focus has not just been "implementing Ruby" but actually 
> being able to run real Ruby apps. To this end, I believe a complete 
> specification should also include a list of applications that should 
> reasonably be expected to work on a given implementation. Is a Ruby 
> implementation complete without support for RubyGems or Rake? Or 
> without Rails? RSpec? These are staples of the Ruby world.
How does jRuby deal with extensions written in C or C++? Sure, there are
lots of "pure Ruby" applications and I assume jRuby will run them all.

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given 
rabbits fire.

Hi --

On Mon, 29 Jan 2007, John Lam (CLR) wrote:

>>> be compatibility problems. It's up to the spec to make
>>> that I'd like to get out in the open and have a resolution that
>
> around any real standardization, with lots of scary legal terms and
> agreements thrown into the mix. I suspect that such a large effort
> would be beyond the scope of what we're all trying to do here.

I think that's right, if you mean things like ISO certification (or
whatever it's called).

> Back to the original point: rather than creating a 'Ruby Software
> Foundation', might it make better sense to drive spec work through
> Ruby Central?

If a non-profit Ruby foundation is to be involved, then I'd say it
makes sense to use our existing one, though there are a number of
questions about both whether it's necessary, and about resources.
And, as Charles says, "Owner?" :-)  So it's not time to decide
anything, but I'll certainly lend a Ruby Central ear, along with my
Ruby developer ear, to the discussion as it proceeds.


David

M. Edward (Ed) Borasky wrote:
> How does jRuby deal with extensions written in C or C++? Sure, there are 
> lots of "pure Ruby" applications and I assume jRuby will run them all.

Porting...delicious porting.

Actually in most cases the C portion of a library is usually
well-represented in the Java world, and it's just a matter of wrapping
it. There's already an early/partial RMagick port using a similar
library in Java, there's a version of Ragel that can generate Java-based
parsers for Mongrel, there's zlib, socket, and bigdecimal libraries for
Java, and so on. At this point, we're starting to close in on some level
of "true compatibility" with C Ruby as measured by our own and other
applications' unit tests, and after that I suppose more effort will be
paid to porting C libraries people really need.

Generally, we (Tom and Charlie) have had to do almost no extension
porting ourselves; community members have decided an extension was worth
writing and have done so. We've also seen that the original C extension
writers have been very receptive to including our completed Java
versions as they mature. So I don't expect that in the long term the
lack of C extensions will hurt us much. And it's pretty easy to find
Java developers, as you might guess.

There's also at least one effort to provide a pseudo-C-Ruby API via JNI
that actually just uses JRuby behind the scenes, but due to the
monumental complexity of such a task, I don't expect it to bear fruit
any time soon.

- Charlie

M. Edward (Ed) Borasky wrote:
> How does jRuby deal with extensions written in C or C++? Sure, there are 
> lots of "pure Ruby" applications and I assume jRuby will run them all.

It's also worth saying two other things:

- Yes, ideally all "pure Ruby" apps should run, provided they don't use
platform-specific features heavily (non-portable file or POSIX APIs we
can't support in Java, for example) or continuations (we don't support
continuations, and have seen almost no demand for us to support
them...plus, it would be rather difficult to support both continuations
and Java bytecode compilation).
- With the recent compiler and optimization work, it's my hope that
people won't *have to* write extensions when running under JRuby,
because the compiled code will run at a comfortable speed. We also
leverage Java's excellent reflective capabilities to allow Ruby code to
call Java code directly, further reducing the requirement to write
physical Java extensions (since you can just manipulate the given
library from Ruby).

- Charlie

Dear Core Colleagues,

As a programmer with a Long Memory, does anyone remember the
Microsoft Credo for dealing with successful products IT DOES NOT
CONTROL:

****  Embrace, Extend and Corrupt!!!!

My advice is to have NOTHING to do with Microsoft.  They have
NEVER Helped anyone but themselves.

I you think they do not have plan to PRY RUBY Away from Matz,
you are naive!!!

As you might guess, I do not use windows except the most extreme
conditions.  Look at what is going on with Linux --- Microsoft is
promoting the idea that Linux uses Microsoft  IP!!!

Ruby is quickly becoming a major language, with the prospect of
replacing may scripting languages.  Microsoft has a vested interest
in slowing  and marginalizing any language that is does not control.

Sorry for this Long Flame guys, but this is not a minor issue.  Right
out of the starting gate Microsoft suggests setting an independent
laywer run "Standards" Organization for Ruby!  PLEASE!!!!!!

                 Charles E. Thornton

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I was checking some CLR opinions and - correct me please if I'm wrong - 
seems
that there were some trouble with continuations. Is just a happy 
coincidence the
Matz's plans to remove continuations from 1.9.x?

- --
Eustáquio "TaQ" Rangel
http://eustaquiorangel.com

"Programs must be written for people to read, and only incidentally for 
machines
to execute."
H. Abelson and G. Sussman (in "The Structure and Interpretation of 
Computer
Programs)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFFvzEub6UiZnhJiLsRAtclAKCDW/3BGPKIz85bJ1vJlTqxt5t34QCgodBf
vg+cRBNXDRAhfHRkWO56nc0=
=BCHC
-----END PGP SIGNATURE-----

On 1/30/07, Eustaquio Rangel de Oliveira Jr. <eustaquiorangel@yahoo.com> 
wrote:

> I was checking some CLR opinions and - correct me please if I'm wrong - seems
> that there were some trouble with continuations. Is just a happy coincidence the
> Matz's plans to remove continuations from 1.9.x?

As far as I understood it, a big reason for removing contiuations is
because they were hard to implement on (certain?) virtual machines.
It's sad, and wrong in my opinion, that they're being removed.  They
don't enjoy much use, but they do allow for some funky stuff.

  nikolai

> I hope such a spec would be developed "in the open" from the beginning,
> rather than being developed behind closed doors and only opened much
> later. And perhaps that's what you're getting at with your points below...

It's absolutely my intent to make sure that any spec is developed in the 
open. Believe me when I say that I'm seriously constrained in terms of 
what I can do outside of Microsoft vs. inside. It's *much* easier to get 
stuff done inside the company vs. getting stuff done outside. So for me, 
it's a lot more interesting writing code vs. meeting with lawyers :)

>> See: RubyTests project on RubyForge

Thanks - will look at that stuff.

> Well, of course Microsoft would be interested in something they could
> take behind closed doors :) Seriously though, for the spec/test suite,
> just about anything is fine; I'd be very surprised if anyone were able
> to take a test suite to closed source and do anything evil with it.

The bottom line on license agreements (and this is nothing that I can 
change) is that projects with copyleft style licenses are projects that 
I cannot work on. That's why the selection of the license agreement is 
important to me personally. I want to work in an open fashion, but I 
also need to respect the needs of my employer. While I actually 
personally agree that copyleft licenses are good for open source 
projects because of the requirement for giving back, there are 
significant legal risks that (now that I understand what they really 
mean) I would not be willing to undertake in certain software projects.

> RubyTests already has committers from almost all the major projects
> (except the CLR-based ones, though they're welcome to come too). And
> we've already contributed our JRuby tests back, eventually to use
> RubyTests as our primary test repository.

Will do. Good to see that it's the Ruby license too :)

> Owner?

The reason for this is roughly as follows. Typical disclaimer about 
IANAL here as well:

In 'real' standards organizations like ISO or ECMA, there are a set of 
by-laws that govern the behavior of individuals and companies that 
participate in the standardization effort. This may include statements 
like participants agree to license at reasonable or zero cost (RAND[Z]) 
any patents etc that they may have that pertain to the technologies 
being standardized or to agree not to sue over those patents. This is 
done to avoid individuals or companies subverting the standardization 
process such that the only way that something can be implemented 
according to spec is to infringe on patent(s) that they own. Clearly 
this is a bad thing, and clearly such things have been attempted 
(successfully?) in the past.

For more details and discussion: http://xml.coverpages.org/patents.html

> Yes, rspec creates nice
> output. It's not nearly human-readable enough for an implementer to flip
> through it and find some implementation detail they might have missed. A
> test/spec suite also assumes a number of things: namely, that the parser
> and core interpreter are already functional, and that a number of basic
> builtin classes and libraries already work correctly. But how do you get
> up to that point without a grammar, documentation on the interpreter,
> and so on?

+1. But the value of rspec is defining the behavior of an existing Ruby 
interpreter, especially with respect to the 'language nooks and 
crannies' :)

> To this end, I believe a complete
> specification should also include a list of applications that should
> reasonably be expected to work on a given implementation. Is a Ruby
> implementation complete without support for RubyGems or Rake? Or without
> Rails? RSpec? These are staples of the Ruby world.

+1. This is the issue that I was trying to (and obviously failing to) 
bring up when talking about how the community should define Ruby 
behavior. It's unreasonable to expect that a non C Ruby implementation 
be able to run *any* Ruby program. What's acceptable vs. non-acceptable 
is not really a language issue at all - it's an issue of what defines 
reasonable expectations about what programs can or cannot run.

Thanks,
-John

On 1/30/07, Nikolai Weibull <now@bitwi.se> wrote:
> On 1/30/07, Eustaquio Rangel de Oliveira Jr. <eustaquiorangel@yahoo.com> wrote:
>
> > I was checking some CLR opinions and - correct me please if I'm wrong - seems
> > that there were some trouble with continuations. Is just a happy coincidence the
> > Matz's plans to remove continuations from 1.9.x?
>
> As far as I understood it, a big reason for removing contiuations is
> because they were hard to implement on (certain?) virtual machines.
> It's sad, and wrong in my opinion, that they're being removed.  They
> don't enjoy much use, but they do allow for some funky stuff.

Continuations are not being "removed." At least the removal hasn't
been announced if there is going to be one. The only thing people keep
mistaking this for is that YARV did not support continuations at merge
time. They might be added in if code is volunteered. Considering this
context, they weren't removed as much as they were _never_added_.

I think the position Matz has taken is that they would allow
continuations to be implemented again but they are not going to hold
up the release of 1.9 at the end of the year for the feature (my
interpretation could be wrong). This means that later releases might
include them. For now, most uses of continuations in the stdlib have
been replaced by working alternative methods IIRC (i.e. generators).

Honestly, I hate to see the lack of callcc but I find this trade off
very pragmatic at this point.

Brian.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Brian.

> Continuations are not being "removed." At least the removal hasn't
> been announced if there is going to be one. The only thing people keep
> mistaking this for is that YARV did not support continuations at merge
> time. They might be added in if code is volunteered. Considering this
> context, they weren't removed as much as they were _never_added_.

On http://www.ruby-forum.com/topic/86862#161750, Matz says:

* Ruby 1.9.1 will not have continuation, since they have lowest
  priority and are difficult to implement efficiently.

* no promise (neither positive nor negative) on continuation for
  1.9.2 or 2.0, for exact same reason above.

Don't know if my interpretation is wrong also, but seems more a negative 
than a
positive thing about continuations on Ruby 1.9.x >, if, as you said, no
volunteered code is added.

- --
Eustáquio "TaQ" Rangel
http://eustaquiorangel.com

"Live as if you will die tomorrow, and learn as if you will live for 
ever."
Mahatma Gandhi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFFv3hMb6UiZnhJiLsRAu2dAKCGc4tGgF+cp6YS0tKU9oJtFJ6k8ACfdtU6
xQTaPJ4Xf086l2vlV51GWrc=
=VZ3G
-----END PGP SIGNATURE-----

John Lam (CLR) wrote:
> The reason for this is roughly as follows. Typical disclaimer about IANAL here as well:
>
> In 'real' standards organizations like ISO or ECMA, there are a set of by-laws that govern the behavior of individuals and companies that participate in the standardization effort. This may include statements like participants agree to license at reasonable or zero cost (RAND[Z]) any patents etc that they may have that pertain to the technologies being standardized or to agree not to sue over those patents. This is done to avoid individuals or companies subverting the standardization process such that the only way that something can be implemented according to spec is to infringe on patent(s) that they own. Clearly this is a bad thing, and clearly such things have been attempted (successfully?) in the past.
>
> For more details and discussion: http://xml.coverpages.org/patents.html
>   
Been there -- done that. The very good news is that *every* language
that I'm aware of that underwent the "crucible" of standardization in
such a fashion emerged alive and kicking butt in its niche and active
and growing, etc. FORTRAN, COBOL, C, C++, Lisp and Forth are cases in
point. Interestingly enough, Java, Perl and PHP are also kicking butt in
their niches with only "de facto" standards so far. So there are
examples of success in committee-standardized languages,
vendor-standardized languages (Java) and community-standardized
languages (Perl and PHP) to use as models.

That said, I wonder if Perl and PHP *ought* to become
committee-standardized, and I also wonder if Ruby is ready, and if
seeking a committee-standardized Ruby would give it an edge over Perl,
PHP or Python? It's really the *users* of the languages that decide
that, not the creators. :)

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given 
rabbits fire.

On 1/30/07, John Lam (CLR) <jflam@microsoft.com> wrote:

> > > See: RubyTests project on RubyForge

> Thanks - will look at that stuff.

I just want to know: Am I in everyones kill-file by now?  I mentioned
this project in [ruby-core 10091], but I guess I'd already burned my
bridges by then...

I'm not posting this to claim some sort of reward for posting about
the project mentioned above first.  It just saddens me that it seems
that I've managed to make myself a reputation for being a troll or
something.

Peace.

  nikolai

[ruby-core 10091]
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/10091

Nikolai Weibull schrieb:
> I just want to know: Am I in everyones kill-file by now?  I mentioned
> this project in [ruby-core 10091], but I guess I'd already burned my
> bridges by then...

Nikolai, you're still in my heart^h^h^h^h^hinbox :-)
I sometimes get the same feeling. Strange.

Regards,
Pit

On 1/30/07, Pit Capitain <pit@capitain.de> wrote:
> Nikolai Weibull schrieb:
> > I just want to know: Am I in everyones kill-file by now?  I mentioned
> > this project in [ruby-core 10091], but I guess I'd already burned my
> > bridges by then...
>
> Nikolai, you're still in my heart^h^h^h^h^hinbox :-)

Good; thanks!

> I sometimes get the same feeling. Strange.

I hope I'll prove that feeling wrong with this response ;-).

  nikolai

Nikolai Weibull wrote:
> I'm not posting this to claim some sort of reward for posting about
> the project mentioned above first.  It just saddens me that it seems
> that I've managed to make myself a reputation for being a troll or
> something.

You can have all the credit for pointing it out first. However since
it's like pulling teeth to get people to write tests, it's probably
worth repeating.

And of course I am the primary remaining admin on the project, so I've
got a duty to mention it either way :)

- Charlie

On Jan 31, 2007, at 8:38 PM, Charles Oliver Nutter wrote:
> The latter. The JVM and CLR do not provide any way to manipulate  
> the call stack, which is the typical and probably most efficient  
> way to implement continuations (aside from providing your own  
> machine and call stack implementation, which would essentially be  
> an interpreter-on-VM in both cases).

Interesting.  I was going to suggest that maybe you could use threads
to implement continuations but a little research and intuition tells
me that continuations are actually more 'primitive' than threads so
you could build threads (and exceptions and co-routines and ...) on
top of continuations but that you can't really implement continuation
semantics on top of threads.

> Ruby developers will have to decide if inability to run on JVM or  
> CLR-based Ruby implementations is worth all the continuation-based  
> "funky stuff". Honestly, I don't think it is.

I find it somewhat disconcerting that Ruby might be hobbled by VMs
designed for other languages.  I've been playing around with
continuations to simplify web programming (like Iowa) and it is a
really nice solution.

Gary Wright

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi.

> The bottom line on license agreements (and this is nothing that I can change) is that projects with copyleft style licenses are projects that I cannot work on. 
> That's why the selection of the license agreement is important to me personally. I want to work in an open fashion, but I also need to respect the needs of my employer. 

And what is the opinion of your employer about the Ruby license? I'm 
curious
about that. :-)

> While I actually personally agree that copyleft licenses are good for open source projects because of the requirement for giving back, 
> there are significant legal risks that (now that I understand what they really
mean) I would not be willing to
> undertake in certain software projects.

Some legal "mambo-jumbo" can be really a pain for companies, but I think 
that if
the developers pay much attention to this kind of details, things can 
run slow
than usual. Its a new and strange element on the development equation, 
made
specially for the big companies.

- --
Eustáquio "TaQ" Rangel
http://eustaquiorangel.com

"The most important scientific revolutions all include, as their only 
common
feature, the dethronement of human arrogance from one pedestal after 
another of
previous convictions about our centrality in the cosmos."
Stephen Jay Gould
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFFv3Rnb6UiZnhJiLsRApgkAJ0b6fKL9DWSKeFOsC8bTmh9ru5IiACfSvpJ
YoCRpiL+8MkrbdnkWLniFkc=
=5327
-----END PGP SIGNATURE-----

Hi --

On Wed, 31 Jan 2007, John Lam (CLR) wrote:

> This is one of those things that we'll need to solve as a community
> for defining what Ruby is and isn't.

Well, as a community of people giving advice and feedback to the
person who defines what Ruby is and isn't :-)  (I'm feeling very old
school and mom-and-pop-like in my attitudes these days, but I think
it's important.)


David

Charles Oliver Nutter wrote:
> Ruby developers will have to decide if inability to run on JVM or 
> CLR-based Ruby implementations is worth all the continuation-based 
> "funky stuff". Honestly, I don't think it is.
By "Ruby developers", do you mean anyone who develops in Ruby? The
reason I ask is that not everyone who develops in Ruby will have a hard
requirement to run on the JVM or CLR, and I also doubt very seriously if
they'll have a hard requirement to use continuations either.
Requirements are *what* must be done -- using a JVM or CLR platform or
using continuations are *hows*, not whats.

The question is then, "if there is to be one Ruby 1.8.x, and JVM and CLR
Ruby 1.8.x don't have continuations, but stock Ruby in C and Rubinius
Ruby 1.8.x *do* have continuations, do we take continuations out of the
ones that do have it?"
>
> - Charlie
>
>


--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given 
rabbits fire.

gwtmp01@mac.com wrote:
> code) that elaborate on the difficulties of implementing continuations?  
> Is it difficult in the theoretical sense or simply difficult to 
> implement on top of VMs that were designed without continuations in mind?

The latter. The JVM and CLR do not provide any way to manipulate the
call stack, which is the typical and probably most efficient way to
implement continuations (aside from providing your own machine and call
stack implementation, which would essentially be an interpreter-on-VM in
both cases).

So yes, they're difficult to do on the JVM and CLR. Maybe impossible
without severe performance degradation, and entirely impossible to do
when third-party libraries are involved that don't play nice with even
the cleverest tricks we could come up with.

For this reason, it's highly likely that even if continuations did
survive into later 1.9 and 2.0 releases, they may never be available on
the CLR or JVM.

Ruby developers will have to decide if inability to run on JVM or
CLR-based Ruby implementations is worth all the continuation-based
"funky stuff". Honestly, I don't think it is.

- Charlie

> > I was checking some CLR opinions and - correct me please if I'm wrong - seems
> > that there were some trouble with continuations.

On the CLR front (and I'm pretty sure on the JVM front as well) 
continuations pose a very major problem for implementers. It's a 
difficult problem to solve while preserving decent performance 
characteristics. And the approaches that have been attempted in the past 
(like exploiting the exception facilities of the runtime) don't hold 
forth the promise of future performance improvements.

This is one of those things that we'll need to solve as a community for 
defining what Ruby is and isn't.

-John

On Jan 30, 2007, at 10:38 AM, John Lam (CLR) wrote:
> On the CLR front (and I'm pretty sure on the JVM front as well)  
> continuations pose a very major problem for implementers. It's a  
> difficult problem to solve while preserving decent performance  
> characteristics. And the approaches that have been attempted in the  
> past (like exploiting the exception facilities of the runtime)  
> don't hold forth the promise of future performance improvements.

Do you have any pointers to resources (papers, blog entries, source
code) that elaborate on the difficulties of implementing
continuations?  Is it difficult in the theoretical sense or simply
difficult to implement on top of VMs that were designed without
continuations in mind?


Gary Wright

gwtmp01@mac.com wrote:
> I find it somewhat disconcerting that Ruby might be hobbled by VMs 
> designed for other languages.  I've been playing around with 
> continuations to simplify web programming (like Iowa) and it is a really 
> nice solution.

Well, I'm not suggesting that Ruby itself should be held back by other
VMs that can't support certain features, but any developers using those
features will have to weigh their utility against the fact that they
won't be able to run on some of the other implementations. If you don't
care about the other implementations and the benefits they might bring,
that ought to work out fine. If you do care about the other
implementations, you'll choose not to use those features.

Already people have to make this choice anyway...symlinks or not, fork
or not, check platform and alter behavior accordingly or not. The
alternative implementations will just add a few more items to consider.
It may also start to work the other way too...rely on native threading
(JRuby) or not, rely on native unicode (JRuby) or not, rely on C or Java
extensions or not. Choices always.

- Charlie

M. Edward (Ed) Borasky wrote:
> 
> The question is then, "if there is to be one Ruby 1.8.x, and JVM and CLR 
> Ruby 1.8.x don't have continuations, but stock Ruby in C and Rubinius 
> Ruby 1.8.x *do* have continuations, do we take continuations out of the 
> ones that do have it?"

I would never suggest such a thing. There are already features in C Ruby
that we'll never be able to support, and features in JRuby that C Ruby
won't support for some time (pure native threading, native unicode,
highly advanced garbage collection). People will weigh the pros and cons
of the various implementations and make their decisions. If they
absolutely must have continuations or fork, they probably won't be able
to use JRuby. If they absolutely must have state-of-the-art garbage
collection or native threads, they probably won't be able to use Ruby 
1.8.

- Charlie

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Well, as a community of people giving advice and feedback to the
> person who defines what Ruby is and isn't :-)

I fully agree with you.

- --
Eustáquio "TaQ" Rangel
http://eustaquiorangel.com

"The most important scientific revolutions all include, as their only 
common
feature, the dethronement of human arrogance from one pedestal after 
another of
previous convictions about our centrality in the cosmos."
Stephen Jay Gould
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFFv3JIb6UiZnhJiLsRAvUUAJ9yGvqyCxH4hRjeOsw+axDen82xNgCfeXdq
J+jt7NBfygxOUDJoGEOxHco=
=zY32
-----END PGP SIGNATURE-----