# Read efficiency?

**URL:** <https://www.ruby-forum.com/t/read-efficiency/183827>\
**Category:** Ruby\
**Created:** [February 21, 2010, 8:16am UTC](https://www.ruby-forum.com/t/read-efficiency/183827 "2010-02-21T08:16:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![guyminuslife](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/a3d4f5/32.png) [@guyminuslife](https://www.ruby-forum.com/u/guyminuslife)\
**Post date:** [February 21, 2010, 8:16am UTC](https://www.ruby-forum.com/t/read-efficiency/183827/1 "2010-02-21T08:16:30Z")

</div>

I’m wondering if anyone knows much about Ruby’s efficiency with IO#read.  
Specifically, I’m wondering about libraries I might use to speed up disk  
reads.

To see what I mean, here’s some test code that iterates over an  
11-megabyte file. All it does is call IO#read on a number of bytes (set  
on the command-line) over the entire file, and times it.

#!/usr/bin/env ruby

# readspeed.rb

buf\_size = ARGV[0].to\_i  
fd = File.open(“some.txt”)

start = Time.now  
while (fd.read(buf\_size))  
end  
stop = Time.now

puts (stop - start).to\_s + " seconds"

#— EOF

Running this on my system yields:

$ ruby readspeed.rb 4096  
0.014 seconds

$ ruby readspeed.rb 1  
7.547 seconds

Obviously a big difference! This is a simplified version of the test I  
was actually running, which tried to account for the increased amount of  
overhead when calling with 1 byte at a time. There’s still an  
order-of-magnitude difference between the two…reading one byte at a  
time is _slow_, slow enough to bog down an entire program.

I know this is supposed to be the case with unbuffered input, such as  
the C standard library “read”, but isn’t IO#read supposed to be  
buffered? What’s causing this slowdown? I’m writing a class that will  
hopefully speed up smaller reads from binary files by explicitly caching  
data in memory, but I’m wondering if there are any pre-built (i.e.,  
tested) solutions that Ruby programmers might be using.

---

<div class="post-metadata">

**Author:** ![guyminuslife](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/a3d4f5/32.png) [@guyminuslife](https://www.ruby-forum.com/u/guyminuslife)\
**Post date:** [February 21, 2010, 9:08am UTC](https://www.ruby-forum.com/t/read-efficiency/183827/2 "2010-02-21T09:08:33Z")

</div>

I hate it when I put my foot in my mouth. After further testing, I’m  
almost entirely sure this is just due to overhead, not a problem with  
disk access.

Who would have thought looping 11\*2\*\*20 times would incur a performance  
hit?

---

<div class="post-metadata">

**Author:** ![guyminuslife](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/a3d4f5/32.png) [@guyminuslife](https://www.ruby-forum.com/u/guyminuslife)\
**Post date:** [February 21, 2010, 2:30pm UTC](https://www.ruby-forum.com/t/read-efficiency/183827/3 "2010-02-21T14:30:34Z")

</div>

On 02/21/2010 09:08 AM, Notmy Realname wrote:

> I hate it when I put my foot in my mouth. After further testing, I’m  
> almost entirely sure this is just due to overhead, not a problem with  
> disk access.

Good that you did not use your real name. 😉

> Who would have thought looping 11\*2\*\*20 times would incur a performance  
> hit?

The overhead of a simple method call is significant already:

robert@fussel:~$ ruby19 -r benchmark \<\<XXX  
def f;end # function without effect  
R1=11_1024_1024  
R2=R1/4096  
Benchmark.bmbm 20 do |r|  
r.report(“rare”) { R2.times { f } }  
r.report(“often”) { R1.times { f } }  
end  
XXX

Rehearsal -------------------------------------------------------  
rare 0.000000 0.000000 0.000000 ( 0.000643)  
often 2.620000 0.000000 2.620000 ( 2.663366)  
---------------------------------------------- total: 2.620000sec

```
                       user system total real

```

rare 0.000000 0.000000 0.000000 ( 0.000643)  
often 2.620000 0.000000 2.620000 ( 2.665057)  
robert@fussel:~$

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![guyminuslife](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/a3d4f5/32.png) [@guyminuslife](https://www.ruby-forum.com/u/guyminuslife)\
**Post date:** [February 21, 2010, 10:44pm UTC](https://www.ruby-forum.com/t/read-efficiency/183827/4 "2010-02-21T22:44:00Z")

</div>

On 02/21/2010 10:26 PM, Notmy Realname wrote:

> > Rehearsal -------------------------------------------------------
> 
> disappeared. Note to self: Ruby is not C.  
> Just an additional hint: you can dramatically reduce the String creation  
> overhead by using the second argument to IO#write:

robert@fussel:~$ dd bs=1024 count=10240 if=/dev/zero of=x  
10240+0 records in  
10240+0 records out  
10485760 bytes (10 MB) copied, 0.135645 s, 77.3 MB/s  
robert@fussel:~$ ruby19 bm  
Rehearsal -------------------------------------------------------  
simple 0.210000 0.240000 0.450000 ( 0.450689)  
smart 0.100000 0.160000 0.260000 ( 0.262826)  
---------------------------------------------- total: 0.710000sec

```
                       user system total real

```

simple 0.210000 0.240000 0.450000 ( 0.463716)  
smart 0.130000 0.130000 0.260000 ( 0.260381)  
robert@fussel:~$ cat bm  
require ‘benchmark’  
REP = 10  
BS = 1024  
Benchmark.bmbm 20 do |r|  
r.report ‘simple’ do  
REP.times do  
File.open “x”, “rb” do |io|  
while b = io.read(BS)  
end  
end  
end  
end  
r.report ‘smart’ do  
REP.times do  
File.open “x”, “rb” do |io|  
b = “”  
while io.read(BS, b)  
end  
end  
end  
end  
end  
robert@fussel:~$

Note: Ruby is not C but sometimes there is room for improvement. 🙂

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![guyminuslife](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/a3d4f5/32.png) [@guyminuslife](https://www.ruby-forum.com/u/guyminuslife)\
**Post date:** [February 21, 2010, 10:26pm UTC](https://www.ruby-forum.com/t/read-efficiency/183827/5 "2010-02-21T22:26:00Z")

</div>

Robert K. wrote:

> On 02/21/2010 09:08 AM, Notmy Realname wrote:
> 
> > I hate it when I put my foot in my mouth. After further testing, I’m  
> > almost entirely sure this is just due to overhead, not a problem with  
> > disk access.
> 
> Good that you did not use your real name. 😉
> 
> > Who would have thought looping 11\*2\*\*20 times would incur a performance  
> > hit?
> 
> The overhead of a simple method call is significant already:
> 
> robert@fussel:~$ ruby19 -r benchmark \<\<XXX  
> def f;end # function without effect  
> R1=11_1024_1024  
> R2=R1/4096  
> Benchmark.bmbm 20 do |r|  
> r.report(“rare”) { R2.times { f } }  
> r.report(“often”) { R1.times { f } }  
> end  
> XXX
> 
> Rehearsal -------------------------------------------------------  
> rare 0.000000 0.000000 0.000000 ( 0.000643)  
> often 2.620000 0.000000 2.620000 ( 2.663366)  
> ---------------------------------------------- total: 2.620000sec
> 
> ```
> user system total real
> 
> ```
> 
> rare 0.000000 0.000000 0.000000 ( 0.000643)  
> often 2.620000 0.000000 2.620000 ( 2.665057)  
> robert@fussel:~$
> 
> Kind regards
> 
> robert

I know, right? I’m embarrassed I use this username elsewhere. And I  
spent like an hour trying to figure this out.

In my defense, I did do more or less exactly what you did and found the  
function call overhead to be significant, but not enough to account for  
the difference. Then I remembered that Ruby isn’t just doing a disk  
read—it’s also doing stuff like packaging the bytes it reads into a  
string. Once I accounted for _that_, the difference magically  
disappeared. Note to self: Ruby is not C.

---

<div class="post-metadata">

**Author:** ![guyminuslife](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/a3d4f5/32.png) [@guyminuslife](https://www.ruby-forum.com/u/guyminuslife)\
**Post date:** [February 22, 2010, 9:47am UTC](https://www.ruby-forum.com/t/read-efficiency/183827/6 "2010-02-22T09:47:05Z")

</div>

2010/2/21 Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> Note: Ruby is not C but sometimes there is room for improvement. 🙂

## 09:40:20 Temp$ ./bmx.rb Rehearsal

f-100.bin simple 0.000000 0.000000 0.000000 (  
0.004000)  
f-100.bin smart 0.016000 0.000000 0.016000 (  
0.005000)  
f-100.bin complete 0.000000 0.000000 0.000000 (  
0.008000)  
f-1048576.bin simple 0.094000 0.046000 0.140000 (  
0.133000)  
f-1048576.bin smart 0.078000 0.016000 0.094000 (  
0.100000)  
f-1048576.bin complete 0.015000 0.031000 0.046000 (  
0.054000)  
f-104857600.bin simple 9.031000 3.875000 12.906000 (  
12.913000)  
f-104857600.bin smart 5.766000 4.047000 9.813000 (  
9.820000)  
f-104857600.bin complete 0.609000 3.156000 3.765000 (  
3.807000)  
------------------------------------------------------- total:  
26.780000sec

```
                                user system total 

```

real  
f-100.bin simple 0.016000 0.000000 0.016000 (  
0.007000)  
f-100.bin smart 0.000000 0.000000 0.000000 (  
0.008000)  
f-100.bin complete 0.000000 0.000000 0.000000 (  
0.007000)  
f-1048576.bin simple 0.093000 0.109000 0.202000 (  
0.207000)  
f-1048576.bin smart 0.078000 0.016000 0.094000 (  
0.103000)  
f-1048576.bin complete 0.000000 0.046000 0.046000 (  
0.037000)  
f-104857600.bin simple 8.594000 4.125000 12.719000 (  
12.760000)  
f-104857600.bin smart 6.219000 3.500000 9.719000 (  
9.721000)  
f-104857600.bin complete 0.593000 3.125000 3.718000 (  
3.778000)  
09:42:46 Temp$

Code is here [File reading benchmark · GitHub](http://gist.github.com/310932#file_bmx.rb)

Cheers

robert

---

<div class="post-metadata">

**Author:** ![guyminuslife](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/a3d4f5/32.png) [@guyminuslife](https://www.ruby-forum.com/u/guyminuslife)\
**Post date:** [February 22, 2010, 4:40pm UTC](https://www.ruby-forum.com/t/read-efficiency/183827/7 "2010-02-22T16:40:46Z")

</div>

> I hate it when I put my foot in my mouth. After further testing, I’m  
> almost entirely sure this is just due to overhead, not a problem with  
> disk access.
> 
> Who would have thought looping 11\*2\*\*20 times would incur a performance  
> hit?

I believe that Rubinius has less overhead “hit time” – as long as you  
define everything as methods so it can JIT them.  
GL!  
-rp