Forum: Ruby-core [ANN] meeting log of RubyDeveloperKaigi20090622

Hi,

Matz, ko1, shyouhei, akr, nobu, naruse and I held a meeting yesterday.
We decided a plan for Ruby 1.9.2.

== Schedule
17 Jul (RubyKaigi2009)
  * releases a patch level release of the 1.9.1.
  * releases 1.9.2 preview release 1.
25 Aug
  * 1.9.2 preview release 2
25 Sep
  * 1.9.2 preview release 3
  * we aim to freeze the feature of 1.9.2 at the time.
25 Oct
  * 1.9.2 Release candidate 1
  * 1.9.2 will be in maintenance mode as 1.9.1 is now in.
25 Nov
  * 1.9.2 Release candidate 2
25 Dec
  * 1.9.2

== New/Changed Features
Some libraries were improved/reimplemented.
* Socket
* Time
* pipe, pty and open3

see http://svn.ruby-lang.org/repos/ruby/trunk/NEWS for more detail.

=== TODO
Some features need more discussions. We must decide detail of the
features by 25 Sep.
* Enumerable#gather
  * [ruby-dev:38392]
* Time#+ can accept a String as a rhs.
  * Do we need #to_rational in addition to #to_r ?
  * [ruby-core:23729]
  * assigned to akr and matz.
* Should the encoding for ENV be the locale encoding?
  * assigned to naruse.
* we need an option for IO methods which makes the methods skip BOMs.
  * [ruby-dev:37075]
  * assigned to naruse.
* Should we increase the version of Marshal format?
  * for "\u"
  * [ruby-dev:36750]
  * assigned to shyouhei and matz.

=== If possible
I hope to include the following features into the 1.9.2. But they might
take too many time for deciding their specs and implementing them.
* SQLite as a standard library
  * [ruby-dev:38463]
  * assigned to naruse
* improvement of YARV opcode set.
  * assigned to ko1.
* debugger support
  * assigned to ko1
* profiler support
  * assigned to ko1
* dtrace support on FreeBSD, OpenSolaris and OSX.
  * assigned to yugui and ko1.

The features will be included in the 1.9.2 if they will be in time for
the feature freeze.

-- Yugui (Yuki Sonoda)  <yugui@yugui.jp>

Good!

Any hope for 1.9.2 of fixing the matrix library which currently has at 
least
8 open bugs? I have not received any comment on the much improved 
library I
posted three weeks ago (http://redmine.ruby-lang.org/issues/show/1532 ), 
nor
did I get any reply from my email to Keiju Ishitsuka.

I'd like to add that I would gladly help maintaining that library.

2009/6/23 Yugui (Yuki Sonoda) <yugui@yugui.jp>

Thanks for the update. :-)

> * Enumerable#gather
>  * [ruby-dev:38392]

For the benefit of those who do not speak Japanese, the following
example was given in that thread:.

    (1..10).chunk_by {|n| n & 2 }.each {|a| p a }
    #=> [1]           # 1 & 2 = 0
    #   [2, 3]        # 2 & 2 = 3 & 2 = 1
    #   [4, 5]        # 4 & 2 = 5 & 2 = 0
    #   [6, 7]        # 6 & 2 = 7 & 2 = 1
    #   [8, 9]        # 8 & 2 = 9 & 2 = 0
    #   [10]          # 10 & 2 = 1

It's a kind of amalgamation of boolean flip-flops and
Enumerbale#partition. The name appears to be in doubt. IMO, 'gather'
is too similar to 'collect', and to a slightly lesser extent,
'select'. 'chunk' or 'chunk_by' seems to capture the intent of the
method better. (I disagree with the suggestion that the block could
return certain underscore-prefixed symbols to control the method's
operation; it seems inelegant).

> * Should the encoding for ENV be the locale encoding?
>  * assigned to naruse.

Could you clarify this proposal? That Encoding.find('locale') should
be influenced by Encoding.locale_charmap?

> * SQLite as a standard library
>  * [ruby-dev:38463]

So the plan is to re-implement a mostly-Ruby library in C and bundle
it with Ruby? I don't understand the motivation of these kind of
changes. They increase the size of the core codebase, burdening the
core team further. They do so using a language that most Ruby
programmers don't know, thus limiting severely outside contributions.
And, further, they increase the complexity of the alternative Ruby
implementations, who each have to write their own version of the
library in question. What's the goal? :-)

I support the inclusion of Marc-Andre's Matrix patch.

On Jun 23, 2009, at 4:23 AM, Run Paint Run Run wrote:

>> * SQLite as a standard library
>>  * [ruby-dev:38463]
>
> So the plan is to re-implement a mostly-Ruby library in C and bundle
> it with Ruby?

I don't read Japanese well enough to understand what is being
discussed in that thread, but here are my thoughts:

* Having SQLite available from a standard Ruby library would be just
excellent
* Building the library from scratch ourselves is a horrible idea

It looks like Jamis abandoning development of the SQLite library is
being discussed in the thread, but that's really a non-issue to me.
There's a far superior SQLite interface for Ruby called Amalgalite:

   http://copiousfreetime.rubyforge.org/amalgalite/

Amalgalite has a lot of things going for it:

* It embeds SQLite in the extension
* It supports the backup/replication features (which it seems is also
discussed in the thread)
* It supports taps
* It has pretty much everything the old SQLite library did
* It's actively maintained

I can't think of any good reason for us to build a library when such
an excellent candidate already exists.  Please consider bundling
Amalgalite instead!

James Edward Gray II

> There's a far superior SQLite interface for Ruby called Amalgalite:
> 
>    http://copiousfreetime.rubyforge.org/amalgalite/
> 
> Amalgalite has a lot of things going for it:

....including:

I've just started looking at this on my different Win32 systems using 
Luis's 1.9.1p129 binary and devkit from 
http://www.rubyinstaller.org/downloads/

How great is it for "gem install amalgalite" to just work (and build 
native extensions!!!) correctly on the first Win32 system I tried things 
on!?

Jon

James Gray wrote:
> Amalgalite has a lot of things going for it:

+ RTree support compiled in (not a default for sqlite)

+ Jeremy Evans added an amalgalite adapter to sequel (though it is
apparently not yet as fast as the sqlite adapter)

2009/6/23 Yugui (Yuki Sonoda) <yugui@yugui.jp>

> == New/Changed Features
> Some libraries were improved/reimplemented.
> * Socket
> * Time
> * pipe, pty and open3


Since Time is no longer bound by epoch and supports timezones, is there
future need for DateTime? Could it be reimplemented in terms of Time?


=== If possible
> I hope to include the following features into the 1.9.2. But they might
> take too many time for deciding their specs and implementing them.
> [...]
> * debugger support
> * profiler support
> * dtrace support on FreeBSD, OpenSolaris and OSX.


Yes! These are crucial, extremely useful features. I hope to see 
ruby-debug
and ruby-prof become part of Ruby itself.

Thanks for the update. I look forward to Christmas!

jeremy

On Tue, Jun 23, 2009 at 2:30 PM, Joel 
VanderWerf<vjoel@path.berkeley.edu> wrote:
> James Gray wrote:
>>
>> Amalgalite has a lot of things going for it:
>
> + RTree support compiled in (not a default for sqlite)
>
> + Jeremy Evans added an amalgalite adapter to sequel (though it is
> apparently not yet as fast as the sqlite adapter)

Amalgalite is great, but it does *embed* SQLite in Ruby. So when you
upgrade Ruby, you're also upgrading SQLite. What if we embedded other
libraries we offer bindings for? I think it's a practice to be
discouraged.

Also, in general, I prefer to see stdlib shrink instead of grow.
Bundling with Ruby is a good way to retard independent library
development.

jeremy

On Jun 23, 2009, at 4:19 PM, Jeremy Kemper wrote:

>
> Amalgalite is great, but it does *embed* SQLite in Ruby. So when you
> upgrade Ruby, you're also upgrading SQLite. What if we embedded other
> libraries we offer bindings for? I think it's a practice to be
> discouraged.

Obviously, there are plusses and minuses to both sides.  I don't want
to get into a huge debate.  However, I do think we should at least
list some of the advantages for going embedded, since we've listed the
minuses.

I have a big SQLite project and I had nothing but trouble when we used
an extension that dynamically linked to the library.  I frequently
used SQL functions that were only in certain versions thus my code
would work fine on some boxes, but not others.  Also, it would be
super hard to reliably support features like the discussed hot backups
because those require very current versions.

I just think it's really great to know if Amalgalite compiles, you
know it works (bugs excepted, of course) and you know exactly what it
can do.

> Also, in general, I prefer to see stdlib shrink instead of grow.
> Bundling with Ruby is a good way to retard independent library
> development.

I think deciding to ship a simple database with Ruby is excellent.
Rails uses SQLite by default and how great would it be for new users
to have reasonably high chances that just works?  Also, it's great to
know you can count on having a decent database for any Ruby
application.  I feel it's a really big win.

We're also nowhere near the first language to include a database in
the standard library, or even SQLite specifically.

Anyway, I've given my opinions.  They are just that and I really don't
wish to fight.  I'll be quiet now, Scout's honor.

James Edward Gray II

Good work
Would there be chance to add a nonblocking version (a default parameter 
may
be) to constructors of TCPSocket, UDPSocket and UNIXSocket ?

something like: TCPSocket.connect_nonblock(xxxx, yyyy) # as a class 
method

Most libraries use TCPSocket or UNIXSocket rather than Socket. So unless 
I
patch the class inheritance chain (ugly! and lacks when it comes to
UNIXSocket) I have no hope of getting those to connect in a nonblocking
manner seamlessly.

Also, what is going on with the generational GC? I have seen a lot of
longlife_heap references and a write barrier function. The GC profiler 
is
reporting the same old behavior though. Any word if this would be
finished/enabled before 1.9.2?. Is there something that I need to do to
enable it in the current trunk?

regards,

oldmoe

2009/6/23 Yugui (Yuki Sonoda) <yugui@yugui.jp>

On Jun 23, 2009, at 02:23, Run Paint Run Run wrote:
>> * Should the encoding for ENV be the locale encoding?
>>  * assigned to naruse.
>
> Could you clarify this proposal? That Encoding.find('locale') should
> be influenced by Encoding.locale_charmap?

I think, what should:

   puts ENV['PATH'].encoding

output?

> It's a kind of amalgamation of boolean flip-flops and
> Enumerbale#partition. The name appears to be in doubt. IMO, 'gather'
> is too similar to 'collect', and to a slightly lesser extent,
> 'select'. 'chunk' or 'chunk_by' seems to capture the intent of the
> method better. (I disagree with the suggestion that the block could
> return certain underscore-prefixed symbols to control the method's
> operation; it seems inelegant).

partition_by perhaps?

A few related attempts have been:

rails' group_by [1]
>> (1..10).group_by{|n| n & 2}
=> {0=>[1, 4, 5, 8, 9], 2=>[2, 3, 6, 7, 10]}

and

enumerable_mapper gem's:

>> (1..10).grouped_by{|n| n & 2}
=> [[0, [1, 4, 5, 8, 9]], [2, [2, 3, 6, 7, 10]]]

The output of the latter is a little more of what I had expected the
"gather" method to do, too, something that includes the values

(1..10).chunk_by {|n| n & 2 }
   #=> [0, [1]]           # 1 & 2 = 0
   #   [1, [[2, 3]]        # 2 & 2 = 3 & 2 = 1
   #   [0, [[4, 5]]        # 4 & 2 = 5 & 2 = 0
   #   [1, [[6, 7]]        # 6 & 2 = 7 & 2 = 1
   #   [0, [[8, 9]]        # 8 & 2 = 9 & 2 = 0
   #   [1, [[10]]          # 10 & 2 = 1

otherwise you lose your gathered values which might not be as useful.

>> * SQLite as a standard library
>>  * [ruby-dev:38463]

That might be quite useful, but why is the question.  To make it more
available to users, since it comes by default [i.e. the equivalent of
bundling the sqlite headers along with ruby and including a sqlite
gem]?


I have a few suggestions for core change too, that might be
considered...guess I'll throw them all out here now since we're in the
Ruby RCR mood...

1) allow for better multi line comments

history: currently there is not easy way to do multi line comments
except by adding many #'s to the beginning of each line, or adding a
somewhat [to me] jarring
                 begin code
=begin
                 some code
=end
                 continue code

So the request is for prettier multi line comments.  Most people on
ruby talk call me crazy for wanting this, but I just cringe every time
I have to do it in Ruby.

The possible options would be to introduce a new multi-line comment
delineator [like /* */] or perhaps  modify the =begin syntax to allow
for it to not be at the beginning of a new line
  =begin
  some code
  =end
  more code.

I'm not one of those with a cool enough editor to do the multi line
comments for me [and many aren't], so such an addition would result in
cleaner looking multi line comments--at least to me.

2) default BasicSocket.do_not_reverse_lookup to true.

background: Currently ruby socket code defaults to always doing a
reverse DNS lookup
when it can [ex: once per incoming UDP packet].
This causes "surprising" pauses when the lookup fails, because the DNS
waits 15s to timeout, before passing the packet back to the
program--the (possibly random) pause is unexpected.

to see this in action: http://betterlogic.com/roger/?p=1646
comment out the do_not_reverse_lookup on the receiver and change the
sender to send to IP127.0.0.255 and watch the "slow for now reason"
packets come in.

The example works great [no pauses] if you set
BasicSocket.do_not_reverse_lookup to
true, and is much less surprising.  This isn't the first time I've been
bit by ruby's internal DNS lookups, and others have similarly been bit
by this, and it always causes confusion.  It also decreases the
network traffic overhead, etc.

A followup suggestion would be to deprecate do_not_reverse_lookup and
in its place create reverse_lookup so that you don't have to use as
many double negatives. [2]

3) with ruby in "-d" or "-v" mode (debug, verbose) don't swallow  any
lines from the exception backtrace given when a program exits because
of uncaught exception.  Those swallowed lines are sometimes useful,
and it is inconvenient to not be able to get at them easily.  So I'd
suggest that if ruby's in verbose mode, they not be swallowed.
Thoughts?

4) (the controversial one). Have load 'rubyscriptname' behave the same
as require 'libraryname' in terms of extension guessing.  It's
surprising to me when require doesn't require extension but load does,
because require and load typically both end up being used against .rb
files, so when I go from one to the other, I'm forced to include
extension.  It's confusing for newbies (and surprising for me).

Just thought I'd get those off my chest.

Thoughts?
-=roger

[1] 
http://allgems.faithpromotingstories.org/gems/doc/activesupport/2.3.2/classes/Enumerable.html#M000948

[2] http://www.ruby-forum.com/topic/184011#new

Hi,

In message "Re: [ruby-core:23995] Re: [ANN] meeting log of 
RubyDeveloperKaigi20090622"
    on Wed, 24 Jun 2009 06:19:42 +0900, Jeremy Kemper 
<jeremy@bitsweat.net> writes:

|Since Time is no longer bound by epoch and supports timezones, is there
|future need for DateTime? Could it be reimplemented in terms of Time?

Date and DateTime has more advanced calendar features based of Julian
date, so they are still needed and the implementation must be
independent from Time.  But casual usage of DateTime will be replaced
by Time gradually.

              matz.

On Thu, Jun 25, 2009 at 6:20 PM, Yukihiro Matsumoto<matz@ruby-lang.org> 
wrote:
> independent from Time.  But casual usage of DateTime will be replaced
> by Time gradually.

Hi Matz.

Would it make sense to consider making DateTime a subclass of Time
rather than Date with the new implementation of Time?

The current DateTime implementation is *really* slow due to all of the
rational arithmetic involved. In my icalendar gem (RiCal) I was using
DateTime to do computations for recurring events, and by far the
biggest consumer of cpu cycles was the reduction of rationals.  I
finally implemented a FastDateTime class for internal use only in the
gem.

Looking at the new code it looks like it goes to the extent of using
separate ints for year, month, day, hour, min, sec and NOT a rational,
is that right?

--
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale
WWR: http://www.workingwithrails.com/person/9021-rick-denatale
LinkedIn: http://www.linkedin.com/in/rickdenatale

Hi,

In message "Re: [ruby-core:24041] Re: [ANN] meeting log of 
RubyDeveloperKaigi20090622"
    on Fri, 26 Jun 2009 08:50:22 +0900, Rick DeNatale 
<rick.denatale@gmail.com> writes:

|Would it make sense to consider making DateTime a subclass of Time
|rather than Date with the new implementation of Time?

The definition of DateTime is a Date class with time attributes added,
so making DateTime a subclass of Time is not reasonable, I think.  I'd
rather recommend to replace DateTime by Time gradually.

|Looking at the new code it looks like it goes to the extent of using
|separate ints for year, month, day, hour, min, sec and NOT a rational,
|is that right?

Correct.

              matz.

On Wed, Jun 24, 2009 at 8:28 AM, Eric Hodel<drbrain@segment7.net> wrote:
>  puts ENV['PATH'].encoding
>
> output?

Ah, thanks! I see that it currently returns ASCII-8BIT, which doesn't
seem to make sense. It's certainly reasonable to expect that a user's
locale encoding will mirror that of his environment variables.

On Wed, Jun 24, 2009 at 8:28 AM, Eric Hodel<drbrain@segment7.net> wrote:
>  puts ENV['PATH'].encoding
>
> output?

Ah, thanks! I see that it currently returns ASCII-8BIT, which doesn't
seem to make sense. It's certainly reasonable to expect that a user's
locale encoding will mirror that of his environment variables.

2009/6/23 Yugui (Yuki Sonoda) <yugui@yugui.jp>:
> === If possible
> I hope to include the following features into the 1.9.2. But they might
> take too many time for deciding their specs and implementing them.
> * SQLite as a standard library
>  * [ruby-dev:38463]
>  * assigned to naruse

Ok, an obvious concern here: implementations that can't support native
extensions will have another new core library they have to support
somehow. This may mean having to build an FFI version of sqlite
interface or simply failing to support it. I'm confused why a library
that works well as a gem (and has newer maintained versions like
Amalgite) needs to be folded into core. Didn't we want to shrink the
core set of libraries, rather than grow them?

> * improvement of YARV opcode set.
>  * assigned to ko1.

Will this include freezing the opcode set at some point?

> * debugger support
>  * assigned to ko1
> * profiler support
>  * assigned to ko1

I'd like to collaborate on these so we can support the same
profiling/debugging interfaces in JRuby. ruby-debug works now with
JRuby but ruby-prof does not.

- Charlie

On Wed, Jul 1, 2009 at 3:20 PM, Charles Oliver
Nutter<headius@headius.com> wrote:
> somehow. This may mean having to build an FFI version of sqlite
> interface or simply failing to support it. I'm confused why a library
> that works well as a gem (and has newer maintained versions like
> Amalgite) needs to be folded into core. Didn't we want to shrink the
> core set of libraries, rather than grow them?
>

I would have to agree with Charles on this point.

Right now is hard to fix already existing bugs at Ruby for Windows,
adding another extension and keep compatibility with it, provide
binaries during the build process and everything is going to
complicate our existence (RubyInstaller Team).

Also, evolution of code and acceptance of patches to ruby-core takes
times and sometimes got forgotten. We cannot depend on that for every
release of these components.

On the contrary, the gem approach help us update it more often and
deliver alternate versions for Windows users.

On Wed, Jul 1, 2009 at 1:34 PM, Luis Lavena<luislavena@gmail.com> wrote:
> On the contrary, the gem approach help us update it more often and
> deliver alternate versions for Windows users.

In JRuby we've started moving more and more of our JRuby-specific
libraries out to gems for exactly this reason. We can simply spin new
versions a lot faster.

I think it would be unfortunate if sqlite were included in Ruby 1.9.2
and those interfaces then became subject to Ruby release cycles,
rather than much faster gem release cycles. In fact, I think as much
as possible should be moved out of Ruby into gems, like the vast
majority of stdlib, and then have specific versions preinstalled when
you install Ruby. But at the very least don't start including new
external library dependencies into stdlib.

- Charlie

>
> === TODO
> Some features need more discussions. We must decide detail of the
> features by 25 Sep.

Any chance of us gettting
http://redmine.ruby-lang.org/issues/show/578
"add method to disassemble Proc objects"

as well?
That would be nice.
Thanks.
=r

It's possible adding an instruction-sequence method (iseq) into Proc 
would
suffice? Off of the instruction sequence, one could use the methods like
disasm or to_a.

The actual additional code (put in the right place) for this seems 
pretty
small:

VALUE
proc_iseq(VALUE self)
{
    rb_proc_t *proc;
    rb_iseq_t *iseq;
    VALUE rb_iseq;
    GetProcPtr(self, proc);
    iseq = proc->block.iseq;
    if (!iseq) return Qnil;
    rb_iseq = iseq_alloc_shared(rb_cISeq);
    RDATA(rb_iseq)->data = iseq;
    return rb_iseq;
}

 rb_define_method(rb_cProc, "iseq", proc_iseq, 0);

Warning: the above may have flaws in it.

I'll attach the above as a comment to redmine.

Sorry for late response,

2009/06/23 18:23, Run Paint Run Run wrote:
>> * SQLite as a standard library
>>   * [ruby-dev:38463]

First of all, I think Ruby should include a RDB.

You know Ruby included some databases: dbm, gdbm and sdbm and pstore.
However, there is no RDBs.
RDBs are widely spread and many people use them.
So Ruby should include RDBs if there is a suitable RDB.

I think SQLite is the suitable RDB.
PHP and Python includes it as stdlib.

So I want to bundle SQLite with Ruby.
But sqlite3-ruby is not maintained now.
Amalgalite is too big.
Amalgalite::SQLite has suitable scope but its style is not Ruby way.
(for example the usage of methods with !.)

Anyway as you can find by this late response,
this plan is for 1.9.3.
And other implementation and plan,
or another candidate maintainers are welcome.

On Tue, Jul 7, 2009 at 12:12 AM, NARUSE, Yui<naruse@airemix.jp> wrote:
> Sorry for late response,
>
> [...]
>
> So I want to bundle SQLite with Ruby.
> But sqlite3-ruby is not maintained now.
> Amalgalite is too big.
> Amalgalite::SQLite has suitable scope but its style is not Ruby way.
> (for example the usage of methods with !.)
>

I've worked on a spinoff version of sqlite3-ruby, to offer binary
versions for Windows using a concept called "fat binaries" to allow
Ruby 1.8 and 1.9 coexist in the same gem.

But I still think that bundling sqlite3-ruby into Ruby itself is going
to slowdown the progress and release cycle of the extension, not
taking in consideration it adds another layer of complexity for the
ones that build Ruby from source (me).

This also will impose a issue for implementations like JRuby, since
this put more work on the team to be able to replicate everything
under the sun of what MRI does.

On Jul 6, 2009, at 10:12 PM, NARUSE, Yui wrote:

> So I want to bundle SQLite with Ruby.
> But sqlite3-ruby is not maintained now.
> Amalgalite is too big.
> Amalgalite::SQLite has suitable scope but its style is not Ruby way.
> (for example the usage of methods with !.)

First, thank you for explaining your concerns.  They are good points.

However, do you not think there could be some value in asking the
Amalgalite maintainer if he is willing to accept some changes?  That
seems easier than recreating all of his efforts.

James Edward Gray II

On Mon, Jul 6, 2009 at 10:18 PM, Luis Lavena<luislavena@gmail.com> 
wrote:
> But I still think that bundling sqlite3-ruby into Ruby itself is going
> to slowdown the progress and release cycle of the extension, not
> taking in consideration it adds another layer of complexity for the
> ones that build Ruby from source (me).
>
> This also will impose a issue for implementations like JRuby, since
> this put more work on the team to be able to replicate everything
> under the sun of what MRI does.

Agreed on both counts. What I'd find a lot more useful is if an
FFI-based sqlite wrapper were written, and then everyone can just use
the *exact same code*. Please, for Ruby's sake, no more C extensions.

- Charlie

> I hope to include the following features into the 1.9.2. But they might
> take too many time for deciding their specs and implementing them.
> * SQLite as a standard library

One possibility would be to include the sqlite gem, but as a gem, then
people could use gem update to get the latest version.

> * improvement of YARV opcode set.
>  * assigned to ko1.
> * debugger support
>  * assigned to ko1

Debugger support would be grand.
Thanks!
=r

Charles Oliver Nutter wrote:
> FFI-based sqlite wrapper were written, and then everyone can just use
> the *exact same code*.

Except when they can't. It is true that much more Ruby
Implementations can support FFI than MRI C Extensions, but there
are still some implementations that can't support either of
those. I'm thinking of browser-based implementations like HotRuby
or Red Sun (or that V8-based one that constantly keeps popping up
on wishlists, but noone wants to actually write), implementations
running in limited environments like JRuby/GAE or JRuby/JME-CDC
and implementations running in "strange" environments like
BlueRuby.

Also, there are some implementations which run in environments,
where there already *is* a relational database. BlueRuby can only
run inside the ABAP VM; the ABAP VM itself is actually not just a
VM, it is more like an application server, or even an entire
application stack. And one of the things it includes, is a
relational database. In fact, the database is so tightly
integrated, that the BlueRuby Interpreter cannot even read .rb
text files. If you want to run Ruby code, you have to store it in
the database as a semi-preparsed structured syntax tree, and the
BlueRuby Interpreter will load *that* from the database and
interpret it. In fact, in ABAP there is no filesystem, all I/O
goes through the database. So, if you add SQLite to Ruby, you
would have to store the SQLite database file inside a BLOB column
in the relational database, just so you can get a relational
database. That's crazy!

However, that is not even the *real* problem.

If you add SQLite to Ruby, whether that be as an MRI C Extension,
an FFI Extension, a Gem or some other solution, there will still
be a fundamental problem remaining: the C sourcecode of SQLite
becomes part of the Ruby Language Specification, at a time when
the Ruby Community is working very hard on getting *away* from C
as the specification language for Ruby.

I remember you saying at a talk about JRuby something like "the
good news is, there is a specification for Ruby; the bad news is,
it is written in C." In this case it is even worse news, because
not only is the specification written in C, but it is written in
C *by someone else*, which means that the Ruby Community has
*no control* over its *own* specification!

Some examples for what it means to make random third-party
sourcecode part of the Ruby Language Specification: the only way
for JRuby to become fully compliant with the OpenSSL
specification was to lock Ola Bini in a dark room with a box of
Pizza, a crate of Red Bull, a printout of the OpenSSL sourcecode
and an empty Emacs buffer, and have him transliterate the entire
codebase line-by-line from C to Java. The only way to become
compliant with the Ruby 1.9 Regexp specification was for Marcin
Mielżyński to do the same with Oniguruma. What's next, having
John Lam hand-translate SQLite to C#? Don't these guys have more
important work to do?

> Please, for Ruby's sake, no more C extensions.

Yes! But also, no more C in the Ruby Specification.

What would be a better approach is to add a "Ruby Interface For A
Lightweight Relational Datastore Specification" (maybe similar to
CoreData, but less, how shall I say this, "NeXTStep-ish") to the
Ruby Language Specification (plus corresponding Examples in the
RubySpec suite, of course) and then allow each individual
implementation to fulfill this spec in the way they see fit: MRI,
YARV, Rubinius and tinyrb could use an FFI version of the SQLite
Ruby wrapper, JRuby and XRuby could use Derby, H2 or HSQLDB (or
also SQLite-FFI), IronRuby could use, I don't know, Jet maybe,
HotRuby and Red Sun could use the new HTML5 Offline Storage
features, MacRuby could use CoreData, MagLev could use a
relational wrapper around their distributed object store, and
Blue Ruby wouldn't have to do anything, because it *already* sits
on top of a relational database.

Or, we could just do nothing, since SQLite seems to have worked
quite fine without being included in the stdlib. (I thought the
general consensus was that we need to *remove* stuff from the
stdlib, not add even more crap?)

Anyway, I'm not a Ruby Implementer, nor a Ruby Designer, nor a
Ruby Specification Author, so this is just my 2 cents.

(I *am*, however, a MRI-on-Windows, YARV-on-Windows, JRuby and
IronRuby user and as such I am painfully aware of the enormous
problems all that C crap causes.)

Cheers,
    jwm.

On Thu, Jul 9, 2009 at 11:36 PM, Roger Pack<rogerdpack@gmail.com> wrote:
>> I hope to include the following features into the 1.9.2. But they might
>> take too many time for deciding their specs and implementing them.
>> * SQLite as a standard library
>
> One possibility would be to include the sqlite gem, but as a gem, then
> people could use gem update to get the latest version.

Or just leave it a gem and find someone from ruby-core to help
maintain it, since that would have to happen anyway.

- Charlie

I agree pretty much across the board. I was actually hoping that
instead of effort going into a new baked-in sqlite wrapper, that
effort could go into new gem-based FFI sqlite wrapper, which would
benefit many more people than baking it into Ruby 1.9.*. It is true,
it would not benefit everyone, and even where FFI works for JRuby we
still encourage people to use the platform-native options. But if the
question is how to ensure sqlite remains well supported, then the
answer is most definitely not to ship it with Ruby 1.9, since that
just makes Ruby harder to implement on other platforms and runtimes.
An FFI version is at least a better answer.

To be honest, I'd prefer shipping a pure-Ruby SQL implementation that
*everyone* can run on any platform, performance be damned. I think
that would be the Ruby way.

2009/7/10 Jörg W Mittag <joergwmittag@googlemail.com>:

Luis Lavena wrote:
> But I still think that bundling sqlite3-ruby into Ruby itself is going
> to slowdown the progress and release cycle of the extension, not
> taking in consideration it adds another layer of complexity for the
> ones that build Ruby from source (me).

If the library's progress slowdown, bundling it is too early.
This is why the next generation cgi.rb is still not bundled.

This is the reason why Amalgalite is too big, I said.
Some people also think SQLite wrapper is early too, even if I think not.

> This also will impose a issue for implementations like JRuby, since
> this put more work on the team to be able to replicate everything
> under the sun of what MRI does.

If I am JRuby people, I will think so.
But on MRI a RDB library should be bundled.

Charles Oliver Nutter wrote:
> I agree pretty much across the board. I was actually hoping that
> instead of effort going into a new baked-in sqlite wrapper, that
> effort could go into new gem-based FFI sqlite wrapper, which would
> benefit many more people than baking it into Ruby 1.9.*. It is true,
> it would not benefit everyone, and even where FFI works for JRuby we
> still encourage people to use the platform-native options. But if the
> question is how to ensure sqlite remains well supported, then the
> answer is most definitely not to ship it with Ruby 1.9, since that
> just makes Ruby harder to implement on other platforms and runtimes.
> An FFI version is at least a better answer.

Some people in ruby-dev said bundle Ruby-FFI and write with it.

> To be honest, I'd prefer shipping a pure-Ruby SQL implementation that
> *everyone* can run on any platform, performance be damned. I think
> that would be the Ruby way.

If exits, this sounds the best answer :)

On Jul 11, 2009, at 3:06 PM, NARUSE, Yui wrote:

> Charles Oliver Nutter wrote:
>
>> To be honest, I'd prefer shipping a pure-Ruby SQL implementation that
>> *everyone* can run on any platform, performance be damned. I think
>> that would be the Ruby way.
>
> If exits, this sounds the best answer :)

So would we be using an existing SQLite library?  Or building our own?

James Edward Gray II

On Sat, Jul 11, 2009 at 5:06 PM, NARUSE, Yui<naruse@airemix.jp> wrote:
> Charles Oliver Nutter wrote:
>>
>> [...]
>>
>> An FFI version is at least a better answer.
>
> Some people in ruby-dev said bundle Ruby-FFI and write with it.
>

That would be the best answer. FFI would open the access and adoption
to other libraries besides SQLite bindings.

James Gray wrote:
> So would we be using an existing SQLite library?  Or building our own?
Is there a pure ruby sqlite library, or for that matter _any_ pure ruby
sql rdbms library? Google doesn't find anything in the first few pages
of hits (not counting pure ruby clients to sql dbs, and pure ruby sql
generators).

If there were, would anyone use it? Seems doubtful, due to the
"performance be damned" comment. Supporting a significant fraction of
SQL and RDB seems like a huge effort for marginal value.

But I'm not in favor of native sqlite in stdlib, either, after listening
to the convincing arguments from Charles and others.

Tell me again, why does out-of-the-box ruby need a sql database? For
most simple purposes (config files, saving program state) aren't yaml,
marshal, and pstore enough? For more complex purposes, there are gems,
and there is the unavoidable need to make choices. Sqlite isn't even
always the best choice for lightweight db (if you value thread/process
concurrency and don't care about sql, then IMO fsdb is a better choice).

James Gray wrote:
> However, do you not think there could be some value in asking the 
> Amalgalite maintainer if he is willing to accept some changes?  That 
> seems easier than recreating all of his efforts.

Ruby's culture and Rails' one are different.
In Rails' culture APIs have no problem without my favorites.

So before we decide bundle SQLite asking him seems too early.

Of cource if the library we needed is convered by Amalgalite,
using it is reasonable.
But current our status is do we need RDB? or Is SQLite enough one?
native or using FFI and so on.

James Gray wrote:
> So would we be using an existing SQLite library?  Or building our own?

Current subject is "Should Ruby 1.9 bundle SQLite library?".
What would we use is next.

Jörg W Mittag wrote:
> implementations running in "strange" environments like
> BlueRuby.

Ruby is optimized for text processing on POSIX.
Many APIs arround IO reflect this.

So inconsistency on "strange" environments cannot be avoided.
Implementations on such environments should adjust it.

> If you add SQLite to Ruby, whether that be as an MRI C Extension,
> an FFI Extension, a Gem or some other solution, there will still
> be a fundamental problem remaining: the C sourcecode of SQLite
> becomes part of the Ruby Language Specification, at a time when
> the Ruby Community is working very hard on getting *away* from C
> as the specification language for Ruby.

Yes, dependency to SQLite's spec and C source is fundamental problem.

> What would be a better approach is to add a "Ruby Interface For A
> Lightweight Relational Datastore Specification" (maybe similar to
> CoreData, but less, how shall I say this, "NeXTStep-ish") to the
> Ruby Language Specification (plus corresponding Examples in the
> RubySpec suite, of course)

Like HTML5 Structured storage, deciding abstract spec may be good 
choice.

> Or, we could just do nothing, since SQLite seems to have worked
> quite fine without being included in the stdlib. (I thought the
> general consensus was that we need to *remove* stuff from the
> stdlib, not add even more crap?)

Removing some libraries from bundled libraries is done Ruby 1.9.0.
It is true.

This do nothing option has following problem
* environments where users can't install their own gems
* codes which uses native gems doesn't work on other implementations

Charles Oliver Nutter wrote:
> I think it would be unfortunate if sqlite were included in Ruby 1.9.2
> and those interfaces then became subject to Ruby release cycles,
> rather than much faster gem release cycles. In fact, I think as much
> as possible should be moved out of Ruby into gems, like the vast
> majority of stdlib, and then have specific versions preinstalled when
> you install Ruby. But at the very least don't start including new
> external library dependencies into stdlib.

If Ruby release cycle is slower than the library's cycle, bundling it is 
too early.

Moving into gems and preinstall tactics seem to have version dependency 
problem.

On Jul 11, 2009, at 5:28 PM, NARUSE, Yui wrote:

> James Gray wrote:
>> So would we be using an existing SQLite library?  Or building our  
>> own?
>
> Current subject is "Should Ruby 1.9 bundle SQLite library?".
> What would we use is next.

I see.  I understand what we are discussing much better now.  Thanks
for explaining.

I was mainly just worried that we were about to spend a bunch of
effort building an all new library so we could bundle it in.  That
really feels wasteful to me.  We're not to worrying about that yet
though, so I'll stop worrying.  :)

As for including a database, I vote yes.  I would love to know that if
I have a working Ruby install, I also have a ready to use database.

James Edward Gray II

Joel VanderWerf wrote:
> Is there a pure ruby sqlite library, or for that matter _any_ pure ruby 
> sql rdbms library? Google doesn't find anything in the first few pages 
> of hits (not counting pure ruby clients to sql dbs, and pure ruby sql 
> generators).

I thought this is fiction at present.

> If there were, would anyone use it? Seems doubtful, due to the 
> "performance be damned" comment. Supporting a significant fraction of 
> SQL and RDB seems like a huge effort for marginal value.

If truely damned, anyone doesn't use it.
This is matter of degree.

> Tell me again, why does out-of-the-box ruby need a sql database? For 
> most simple purposes (config files, saving program state) aren't yaml, 
> marshal, and pstore enough? For more complex purposes, there are gems, 
> and there is the unavoidable need to make choices. Sqlite isn't even 
> always the best choice for lightweight db (if you value thread/process 
> concurrency and don't care about sql, then IMO fsdb is a better choice).

In bundled libraries we have some persistent libraries:
dbm series and Marshal series.

DBM series are key-value store. So structured data are hard to store in.

Marshal series can be used by some ways.
If store objects as-is, data is weak at changing models.
If store objects as array-hash structure, selecting a certain object
will be hard work.
RDBMS seems better than them.

For very complex and huge purposes, using MySQL or some gems is right.
Recently however people use RDBMS to store settings.
Firefox and Google Gears may be good example.