Forum: Ruby-core [ruby-trunk - Feature #6047][Assigned] read_all: Grow buffer exponentially in generic case

Issue #6047 has been reported by Martin Bosslet.

----------------------------------------
Feature #6047: read_all: Grow buffer exponentially in generic case
https://bugs.ruby-lang.org/issues/6047

Author: Martin Bosslet
Status: Assigned
Priority: Normal
Assignee: Martin Bosslet
Category: core
Target version: 2.0.0


In the general case, read_all grows its buffer linearly by just the 
amount that is currently read from the underlying source. This results 
in a linear number of reallocs, It might turn out beneficial if the 
buffer were grown exponentially by multiplying with a constant factor 
(e.g. 1.5 or 2), thus resulting in only a logarithmic numver of 
reallocs.

I will provide a patch and benchmarks, but I'm already opening this 
issue so I won't forget.

See also https://bugs.ruby-lang.org/issues/5353 for more details.

Issue #6047 has been updated by ko1 (Koichi Sasada).


ping. status?
Do you need helps or comments?

----------------------------------------
Feature #6047: read_all: Grow buffer exponentially in generic case
https://bugs.ruby-lang.org/issues/6047#change-31677

Author: MartinBosslet (Martin Bosslet)
Status: Assigned
Priority: Normal
Assignee: MartinBosslet (Martin Bosslet)
Category: core
Target version: 2.0.0


In the general case, read_all grows its buffer linearly by just the 
amount that is currently read from the underlying source. This results 
in a linear number of reallocs, It might turn out beneficial if the 
buffer were grown exponentially by multiplying with a constant factor 
(e.g. 1.5 or 2), thus resulting in only a logarithmic numver of 
reallocs.

I will provide a patch and benchmarks, but I'm already opening this 
issue so I won't forget.

See also https://bugs.ruby-lang.org/issues/5353 for more details.

Issue #6047 has been updated by MartinBosslet (Martin Bosslet).


ko1 (Koichi Sasada) wrote:
> ping. status?
> Do you need helps or comments?

Thanks for your help, to be honest, I haven't tried so far. Can we leave 
it at 2.0.0 target for now? If I run into problems, I'll ask here!

----------------------------------------
Feature #6047: read_all: Grow buffer exponentially in generic case
https://bugs.ruby-lang.org/issues/6047#change-33374

Author: MartinBosslet (Martin Bosslet)
Status: Assigned
Priority: Normal
Assignee: MartinBosslet (Martin Bosslet)
Category: core
Target version: 2.0.0


In the general case, read_all grows its buffer linearly by just the 
amount that is currently read from the underlying source. This results 
in a linear number of reallocs, It might turn out beneficial if the 
buffer were grown exponentially by multiplying with a constant factor 
(e.g. 1.5 or 2), thus resulting in only a logarithmic numver of 
reallocs.

I will provide a patch and benchmarks, but I'm already opening this 
issue so I won't forget.

See also https://bugs.ruby-lang.org/issues/5353 for more details.

Martin Bosslet <Martin.Bosslet@googlemail.com> wrote:
> In the general case, read_all grows its buffer linearly by just the
> amount that is currently read from the underlying source. This results
> in a linear number of reallocs, It might turn out beneficial if the
> buffer were grown exponentially by multiplying with a constant factor
> (e.g. 1.5 or 2), thus resulting in only a logarithmic numver of
> reallocs.

I think growing the buffer exponentially makes sense.

I would enforce a hard limit (probably <= 8 MB) for each growth,
to:

1) discourage read_all() for large files, it's very wasteful and
   usually hurts performance

2) prevent memory exhaustion for edge cases (especially on 32-bit)

Issue #6047 has been updated by mame (Yusuke Endoh).

Target version changed from 2.0.0 to next minor

My experience also shows that it is useless to open a ticket for a 
reminder to myself :-)

I'm setting to next minor tentatively, but if it is really just a 
performance improvement (i.e., it affects no external modules), you can 
commit it to 2.0.0 before code freeze.

--
Yusuke Endoh <mame@tsg.ne.jp>
----------------------------------------
Feature #6047: read_all: Grow buffer exponentially in generic case
https://bugs.ruby-lang.org/issues/6047#change-33749

Author: MartinBosslet (Martin Bosslet)
Status: Assigned
Priority: Normal
Assignee: MartinBosslet (Martin Bosslet)
Category: core
Target version: next minor


In the general case, read_all grows its buffer linearly by just the 
amount that is currently read from the underlying source. This results 
in a linear number of reallocs, It might turn out beneficial if the 
buffer were grown exponentially by multiplying with a constant factor 
(e.g. 1.5 or 2), thus resulting in only a logarithmic numver of 
reallocs.

I will provide a patch and benchmarks, but I'm already opening this 
issue so I won't forget.

See also https://bugs.ruby-lang.org/issues/5353 for more details.