Ruby Forum Ruby-core > A plea for a release process

Hi all,

I'd like to open a discussion about the possibilities of formalizing a
release process for MatzRuby.

First, here are several points that contribute to the problems I see:

1. There are presently at least 4 highly viable alternative
implementations for Ruby. We agreed at a recent ruby-design meeting
that we would all follow MatzRuby's RUBY_VERSION and RUBY_PATCHLEVEL
to indicate to user code precisely what features and behavior that
code can expect of the language and libraries.
2. The RubySpec project includes "guards" (mechanisms to control the
execution of particular specs) for version_is/version_is_not and
ruby_bug that take a string like "1.8.6.114" to indicate version and
patchlevel. The guards make it possible to run the specs sanely across
many operating systems, versions, and implementations, but using the
guards also complicates the specs.
3. There are multiple currently-supported versions of MatzRuby (at
least 1.8.5.x, 1.8.6.x, 1.8.7.x, 1.9.x if I'm not mistaken).

The RubySpec project attempts to accommodate every single alternative
implementation and every supported version of MatzRuby at least from
1.8.6 onward. This is definitely a work in progress, but it has
accomplished a great deal toward the goal of ensuring that everyone
who writes Ruby code can do so with confidence that their code will
run correctly on any compatible version of the *Ruby language*, not
just on an implementation.

Recently, there has been an explosion of patchlevel commits in
MatzRuby. There have been over 200 since 4 June 2008, and there were
92 in a single day, on 15 June 2008. At least that's what I see with
these two commands:

svn log --limit 200 http://svn.ruby-lang.org/repos/ruby/tags
svn log --limit 200 http://svn.ruby-lang.org/repos/ruby/tags | grep
'15 Jun 2008' | wc -l

There are at least two significant problems that arise from the
combination of everything above.

First, whenever there are bugs in one of these released patchlevels,
the RubySpec sources must have ruby_bug and possibly version_is guards
added. The ruby_bug guard ensures that the spec for the correct
behavior will run on alternative implementations, but *not* run on the
MatzRuby version with the bug. This is done to prevent a bunch of
errors when running the RubySpecs on an already released version of
MatzRuby (since nothing can be done about the errors until a new
version or patchlevel). With so many patchlevels being released, the
number of these guards is increasing. While the guards are useful,
they complicate the specs and should ideally be avoided.

Second, since alternative implementations do not get any notice of
what features will be released, they are continually playing catch-up.
What this means for a user is that code that ran yesterday on JRuby,
for example, might break today on the released version of MatzRuby. If
we are all writing code for a particular implementation, this won't be
much of an issue. But one of the biggest goals of the RubySpec project
is to ensure that a user _does not have to worry about where their
code will run_.

So, I'd like to suggest that the community and the MatzRuby core
developers come to some agreement on a process for MatzRuby releases.
Here are several features I hope the process will include:

1. Security fixes should be highest priority. A patchlevel across
versions that incorporates security fixes should be released as soon
as possible. It would be great to have the issues communicated to key
folks across all alternative implementations, but that might not be
possible in every situation.
2. A regular schedule of patchlevel releases on some reasonable
timetable (one month, two months?). For each of these scheduled
releases:
2.a. A wiki page on the new Redmine tracker that lists the features to
be rolled in from the particular version's development branch.
2.b. An opportunity for folks to run the specs against these proposed
changes to catch errors before the release is done.
2.c. Along with 2.b. this would give alternative implementations an
opportunity to plan to release the same fixes/features with the
corresponding RUBY_VERSION and RUBY_PATCHLEVEL synchronized to match
the MatzRuby releases at a time closer to when the MatzRuby releases
occur.
2.d. An opportunity for the community to then be aware of what is
planned to be released and comment on it so that we don't have
situations like what recently happened here (http://groups.google.com/
group/ruby-core-google/browse_thread/thread/1b116e4bbaeca3d2).

If the process is formalized a bit, I am sure there will be folks that
will help out with things needed at various stages. The RubySpec
project and the Rubinius project are working toward having a single
command that can be used to run the specs against all official
versions of MatzRuby based on Ryan Davis' multiruby tool. We hope to
make this process as simple and pain free as possible.

The Ruby community is presently growing in complex ways with the
appearance of multiple, useful alternative implementations of the Ruby
language. My hope (and I believe the hope of everyone who contributes
to or uses the RubySpec project) is to ensure people continue to find
the Ruby language great to use regardless of the implementation. Since
we are all following the "standard" implementation (MatzRuby), it
would help tremendously if we could, somehow, formalize (a bit) the
release process for MatzRuby.

Cheers,
Brian

On 2008-06-19 04:08 +0900 (Thu), Brian Ford wrote:

> ...toward the goal of ensuring that everyone who writes Ruby code
> can do so with confidence that their code will run correctly on
> any compatible version of the *Ruby language*, not just on an
> implementation.

Is it just me, or does there seem to be some confusion here between the
Ruby language and the Ruby libraries?

cjs

Hello,

In message "[ruby-core:17327] A plea for a release process"
    on Jun.19,2008 04:08:35, <brixen@gmail.com> wrote:
> First, whenever there are bugs in one of these released patchlevels,
> the RubySpec sources must have ruby_bug and possibly version_is guards
> added. The ruby_bug guard ensures that the spec for the correct
> behavior will run on alternative implementations, but *not* run on the
> MatzRuby version with the bug. This is done to prevent a bunch of
> errors when running the RubySpecs on an already released version of
> MatzRuby (since nothing can be done about the errors until a new
> version or patchlevel). With so many patchlevels being released, the
> number of these guards is increasing. While the guards are useful,
> they complicate the specs and should ideally be avoided.

Only these patchleves has been released.
 1.8.5       Aug 25 2006
 1.8.5-p2    Dec  4 2006 (security fix)
 1.8.5-p11   Dec 18 2006
 1.8.5-p12   Dec 25 2006
 1.8.5-p35   Mar 13 2007
 1.8.5-p52   Jun  9 2007
 1.8.5-p113  Sep 23 2007
 1.8.5-p114  Oct  4 2007 (security fix)
 1.8.5-p115  Mar  3 2008 (security fix)
(1.8.5-p2xx  maybe soon? / last release of 1.8.5)

 1.8.6       Mar 13 2007
 1.8.6-p36   Jun  9 2007
 1.8.6-p110  Sep 23 2007
 1.8.6-p111  Oct  4 2007 (security fix)
 1.8.6-p114  Mar  3 2008 (security fix)
(1.8.6-p2xx  maybe soon?)

 1.8.7       Jun  1 2008
 1.8.7-p17   Jun  9 2008

Tagged versions on svn are not released.
I think that RubySpec doesn't need to take care of them.


Regards,

On Jun 18, 12:43 pm, Curt Sampson <c...@cynic.net> wrote:
> On 2008-06-19 04:08 +0900 (Thu), Brian Ford wrote:
>
> > ...toward the goal of ensuring that everyone who writes Ruby code
> > can do so with confidence that their code will run correctly on
> > any compatible version of the *Ruby language*, not just on an
> > implementation.
>
> Is it just me, or does there seem to be some confusion here between the
> Ruby language and the Ruby libraries?

Is there a useful distinction you'd like to offer in this context?

The point is that I want some assurance that my code will run
correctly, for example, on JRuby or Rubinius (1.8.6 p114 compatible)
when it runs on MatzRuby with that version, and vice versa.
Restricting "language" to mean "syntax and grammar" is useless in this
context. And "language" (i.e. syntax and grammar) changes occur in
versions (e.g. 1.8 vs 1.9) as well. So, in this context, I mean "Ruby
language" to be the syntax and semantics of code running under a
particular version, regardless of implementation.

On Jun 18, 1:12 pm, "U.Nakamura" <u...@garbagecollect.jp> wrote:
> > errors when running the RubySpecs on an already released version of
>  1.8.5-p35   Mar 13 2007
>  1.8.6-p114  Mar  3 2008 (security fix)
> (1.8.6-p2xx  maybe soon?)
>
>  1.8.7       Jun  1 2008
>  1.8.7-p17   Jun  9 2008
>
> Tagged versions on svn are not released.
> I think that RubySpec doesn't need to take care of them.

I've used the word "released" too imprecisely, but I don't think that
RubySpec can completely ignore these tags.

On the one hand, if the specs are only run against officially released
versions, then potential bugs (regressions) introduced in the
development branch won't be caught. On the other hand, if the specs
are used with the development branch "trunk" or "head" for each
version (e.g. 1.8.6, 1.8.7, etc.), then the spec guards need to be
able to accurately identify under which version the specs are being
run (which means checking patchlevel too), otherwise there will be
false positives for failures or some specs that should run would not.

Why are all the tags used?

Thanks,
Brian

On Jun 18, 2:19 pm, Brian Ford <bri...@gmail.com> wrote:
> > > the RubySpec sources must have ruby_bug and possibly version_is guards
> >  1.8.5       Aug 25 2006
> >  1.8.6       Mar 13 2007
> > I think that RubySpec doesn't need to take care of them.
>
> I've used the word "released" too imprecisely, but I don't think that
> RubySpec can completely ignore these tags.

Sorry, I meant *cannot* completely ignore these tags.

On Wed, Jun 18, 2008 at 5:24 PM, Brian Ford <brixen@gmail.com> wrote:

> >
> > I've used the word "released" too imprecisely, but I don't think that
> > RubySpec can completely ignore these tags.
>
> Sorry, I meant *cannot* completely ignore these tags.
>

Are you sure you weren't right the first time.

If you don't think that RubySpec cannot completely ignore these tags, 
then I
think you think that RubySpec CAN completely ignore these tags, and I 
dont'
think that's what you thought you meant.

I think! <G>

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

On Jun 18, 3:23 pm, "Rick DeNatale" <rick.denat...@gmail.com> wrote:
> > > > I think that RubySpec doesn't need to take care of them.
> think that's what you thought you meant.
>
> I think! <G>

Oh man, I'm just asking for it, huh?

In case there's any remaining confusion, I think that RubySpec cannot
just ignore the tags. :)

In article 
<51e4ec23-9189-48fe-b797-3dee85b6cc83@w5g2000prd.googlegroups.com>,
  Brian Ford <brixen@gmail.com> writes:

> On the one hand, if the specs are only run against officially released
> versions, then potential bugs (regressions) introduced in the
> development branch won't be caught. On the other hand, if the specs
> are used with the development branch "trunk" or "head" for each
> version (e.g. 1.8.6, 1.8.7, etc.), then the spec guards need to be
> able to accurately identify under which version the specs are being
> run (which means checking patchlevel too), otherwise there will be
> false positives for failures or some specs that should run would not.

A released patchlevel can be used to distinguish "head" from
released versions.

RUBY_PATCHLEVEL <= the-last-released-patchlevel checks
released versions.

For 1.8.6, RUBY_PATCHLEVEL <= 115 means a released version,
now.

I think I was called :)

The reasons why I committed so much was:

1) I wanted to fix as much bugs in 1.8.5 as possible, before it expires.

2) Sadly I was busy for a while so there were many bugs stacked, and at
last I got time to touch them in 3 June.  This was my fault.  I should
have done this more frequently, little by little.

Brian Ford wrote:
> So, I'd like to suggest that the community and the MatzRuby core
> developers come to some agreement on a process for MatzRuby releases.
> Here are several features I hope the process will include:
>
> 1. Security fixes should be highest priority. A patchlevel across
> versions that incorporates security fixes should be released as soon
> as possible. It would be great to have the issues communicated to key
> folks across all alternative implementations, but that might not be
> possible in every situation.
>   

They are already.  But please also note that we are not always possible
to disclose then immediately.  Security incidents are treated based on
international concord;  not fully controlled by us.

> 2. A regular schedule of patchlevel releases on some reasonable
> timetable (one month, two months?). For each of these scheduled
> releases:
>   

Maybe I've not said this?  Patchlevel "releases" (apart from those tags)
are released every 3 months normally in Mar, Jun, Sep, Dec, except for
security fixes of course. We did not released 1.8 patchlevels on last
Dec because we released 1.9.0 instead.

> 2.a. A wiki page on the new Redmine tracker that lists the features to
> be rolled in from the particular version's development branch.
>   

We should have this.  I'm currently managing that kind of list by hand
(accessible via
http://coderepos.org/share/browser/docs/shyouhei/ruby%20development/revisions.txt).

> 2.b. An opportunity for folks to run the specs against these proposed
> changes to catch errors before the release is done.
> 2.c. Along with 2.b. this would give alternative implementations an
> opportunity to plan to release the same fixes/features with the
> corresponding RUBY_VERSION and RUBY_PATCHLEVEL synchronized to match
> the MatzRuby releases at a time closer to when the MatzRuby releases
> occur.

That is a nice idea.

> 2.d. An opportunity for the community to then be aware of what is
> planned to be released and comment on it so that we don't have
> situations like what recently happened here (http://groups.google.com/
> group/ruby-core-google/browse_thread/thread/1b116e4bbaeca3d2).
>   

What recently happened there was actually "the community to be aware of
what is planned to be released."  No?

On 2008-06-19 06:10 +0900 (Thu), Brian Ford wrote:

> On Jun 18, 12:43$B".(Bpm, Curt Sampson <c...@cynic.net> wrote:
> > On 2008-06-19 04:08 +0900 (Thu), Brian Ford wrote:
> >
> > > ...any compatible version of the *Ruby language*, not just on an
> >
> > Is it just me, or does there seem to be some confusion here between the
> > Ruby language and the Ruby libraries?
> ...
> Restricting "language" to mean "syntax and grammar" is useless in this
> context.

Well, you did say "language" and put it in boldface. I took that to mean
you were particularly distinguishing the language from the libraries. If
you just say, "Ruby", people will get what you mean just fine, I'm sure.
I would have.

Anyway, if you're depending on particular buggy behaviour, take
advantage of Ruby's amazing ease of modifying the standard libraries and
include a version with the bug in your program. I do this all the time,
though usually the other way around (including a non-buggy routine to
let a program run on an older, buggier set of libraries.)

cjs

Hello,

On Jun 18, 8:16 pm, Urabe Shyouhei <shyou...@ruby-lang.org> wrote:
> I think I was called :)

Thank you for the input. :)

> > So, I'd like to suggest that the community and the MatzRuby core
> to disclose then immediately.  Security incidents are treated based on
>
> > 2.a. A wiki page on the new Redmine tracker that lists the features to
> > be rolled in from the particular version's development branch.
>
> We should have this.  I'm currently managing that kind of list by hand
> (accessible viahttp://coderepos.org/share/browser/docs/shyouhei/ruby%20development/r...).

Ah, the release schedule is good to know. I'm sure some of the
concerns here will be addressed by having a single good source of
information about the process for all the non-Japanese speakers.
Having this wiki page would be awesome.

>
> > 2.d. An opportunity for the community to then be aware of what is
> > planned to be released and comment on it so that we don't have
> > situations like what recently happened here (http://groups.google.com/
> > group/ruby-core-google/browse_thread/thread/1b116e4bbaeca3d2).
>
> What recently happened there was actually "the community to be aware of
> what is planned to be released."  No?

Yes, I think this situation worked out well. My concern is that we
continue to catch these issues before releases. Which means figuring
out how to run the RubySpecs against the development branch "head" for
each version

1) without causing a lot of churn in the specs (i.e. a bunch of specs
had to be modified to not fail after the convert_type change), and
2) without causing confusion from failing specs for issues that have
been fixed.

In the first case, I think we should only add ruby_bug guards for
failures in a released patchlevel of a particular version, but never
for issues that exist in the development branch for that version. Does
that sound reasonable?

In the second case, if we only add ruby_bug guards for released
versions, then any specs that fail in the development branch might be
bugs? In that case, we should file tickets on the new tracker? Will
this lead to a lot of work for everyone? Or is there a better way to
recommend that we handle this?

Thanks,
Brian

Hi folks,

Sorry, I'm joining this discussion a bit late, since I tried to
restrain myself and
not be negative, since the whole subject is sensitive.

Obviously, the MRI release process should be own and defined by 
ruby-core folks
themselves, but it is also clear that some processes are better for
other Ruby implementors,
and some are more problematic.

On Thu, Jun 19, 2008 at 5:16 AM, Urabe Shyouhei <shyouhei@ruby-lang.org> 
wrote:
> I think I was called :)
>
> The reasons why I committed so much was:
> 2) Sadly I was busy for a while so there were many bugs stacked, and at
> last I got time to touch them in 3 June.  This was my fault.  I should
> have done this more frequently, little by little.

I'm not really sure :) For me, the 1.8.6 line is a golden standard at
the moment,
and we try to follow its behavior when writing new RubySpec tests. So,
essentially,
MRI 1.8.6 is also a *REFERENCE* implementation. It is not good when
the reference implementation changes its behavior  from patchlevel to
patchlevel, hard to write specs, hard to understand what's the correct
behavior, lots of version guards in the specs, additional maintenance
to keep those version spec guards up-to-date.

And since the documentation for core/stdlib classes is not very
strict/precise, all we really
have is the MRI 1.8.6 behavior. Again, since the docs are not precise,
it is hard to distinguish between "bugfix" and "feature". Some
bugfixes are ending up changing behavior and introducing backward
incompatible changes.

Ideally (from my side of RubySpec contributor), the 1.8.6 line should
be really frozen and only highest-priority fixes should go there, like
security fixes.

>> 1. Security fixes should be highest priority. A patchlevel across
>> versions that incorporates security fixes should be released as soon
>> as possible. It would be great to have the issues communicated to key
>> folks across all alternative implementations, but that might not be
>> possible in every situation.

I fully agree with this one.

>> 2. A regular schedule of patchlevel releases on some reasonable
>> timetable (one month, two months?). For each of these scheduled
>> releases:
>>
>
> Maybe I've not said this?  Patchlevel "releases" (apart from those tags)
> are released every 3 months normally in Mar, Jun, Sep, Dec, except for
> security fixes of course.

Here's where I have a problem. The current patch levels are ordered, and 
you say
that the ones we see in the subversion repository tags, are not
"really released".

But what happens when a security issue found? Right, a new fix on top
of all previous patchlevels. And the security patch-level effectively
is a public release (which also gets LOTS of attention due to
sensitivity of the security problem), and that's where we have lots of
problems, since in the patchlevel changes we have: bugfixes (important
and critical and sometimes not important), some backward incompatible
changes (due to fixes, etc), some security fixes, some even minor
features.

The end result is that the patchlevel with security fix is way
unstable than needed.

Take a look at the comment here:
http://weblog.rubyonrails.com/2008/6/21/multiple-ruby-security-vulnerabilities

It clearly shows that the released security fix patchlevel is really
unstable/incompatible/non-useable.

Maybe that sounds radical, but how about using the patchlevel only for
critical/security fixes *ONLY*, no other non-critical changes at all?
And the regural fixes/features and even some controlled backward
incompatible changes should go to the next minor release?

Also, since MRI is de-facto reference implementation for Ruby
language, it would be really good to not have incompatible changes and
new features in the same version number. Why not use the proper
versioning mechanisms and release dot-dot releases when needed
instead?

Thanks,
  --Vladimir

I'll sum up my feelings in as few words as possible:

- JRuby has not started to move away from 1.8.6p114 compatibility,
because we keep hearing reports about app breakage in later patchlevels
and in 1.8.7. This is extremely worrisome.
- I think we need to take a step back and examine all behavioral changes
post 1.8.6p114 and justify them individually. If behavioral changes
can't be justified and are backward-incompatible, they must be reverted.

We have too many people using Ruby now to break things in patchlevels
and minor releases, and JRuby for one will not leave users with an
option of not upgrading or upgrading and breaking their apps.

I have to agree with Charles. We have no plans to upgrade to
ruby1.8.7 as it is broken. Ruby1.8.6p111/p114 is what I and most
others consider to be ruby. Any new features in the 1.8.* branch
should have only gone in 1.9 or been justified and tested by the
community before being released and reeking havok on the ruby community.

  Ruby has seen far too much adoption for the release management to be
so completely broken. We have the rails test suite and the ruby specs
that can help manage releases and let us know that things have not
broken. but for some reason releases are still made without running
either of these spec suites.

  How can we the ruby community get the point across to ruby-core that
the current process is completely broken and needs to be revamped from
the bottom up?

  Rubinius is currently targeting 1.8.6p114 as well since it seems to
be the last sane release of the ruby 1.8 line.

Thanks-
-Ezra


On Jun 27, 2008, at 7:01 PM, Charles Oliver Nutter wrote:

> We have too many people using Ruby now to break things in  
>> 1. There are presently at least 4 highly viable alternative
>> guards also complicates the specs.
>> MatzRuby. There have been over 200 since 4 June 2008, and there were
>> added. The ruby_bug guard ensures that the spec for the correct
>> up.
>> developers come to some agreement on a process for MatzRuby releases.
>> to
>> situations like what recently happened here (http:// 
>> appearance of multiple, useful alternative implementations of the  
>
>

- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra@engineyard.com
-- EngineYard.com