# Scattered I/O (on MS Windows)

**URL:** <https://www.ruby-forum.com/t/scattered-i-o-on-ms-windows/118066>\
**Category:** Ruby\
**Created:** [October 23, 2007, 6:07pm UTC](https://www.ruby-forum.com/t/scattered-i-o-on-ms-windows/118066 "2007-10-23T18:07:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\
**Post date:** [October 23, 2007, 6:07pm UTC](https://www.ruby-forum.com/t/scattered-i-o-on-ms-windows/118066/1 "2007-10-23T18:07:05Z")

</div>

Hi all,

Park H. and I came up with a custom implementation for  
IO.readlines using scattered I/O I thought would be fun to share. I  
think I’m seeing a 2x performance increase, but page caching is making  
it difficult to tell. Also, it looks like the main profiling issue is  
the call to ‘split’ at the end, so you can remove that last bit of  
logic if you want to see the speed without it.

What do folks think? Are you seeing a performance increase? You  
probably won’t see any noticeable difference unless the file is  
greater than 25mb or so, btw.

Link to ReadFileScatter() function definition:

> **[ReadFileScatter function (fileapi.h) - Win32 apps](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-readfilescatter?redirectedfrom=MSDN)**
>
> Reads data from a file and stores it in an array of buffers.

# nio.rb - requires the latest windows-pr gem

require ‘windows/file’  
require ‘windows/handle’  
require ‘windows/error’  
require ‘windows/memory’  
require ‘windows/nio’  
require ‘windows/synchronize’  
require ‘windows/system\_info’  
require ‘windows/msvcrt/io’  
require ‘windows/msvcrt/buffer’  
require ‘win32/event’

module Win32  
class NIO  
include Windows::File  
include Windows::Handle  
include Windows::Error  
include Windows::Synchronize  
include Windows::MSVCRT::IO  
include Windows::MSVCRT::Buffer  
include Windows::SystemInfo  
include Windows::Memory  
include Windows::NIO  
extend Windows::File  
extend Windows::Handle  
extend Windows::Error  
extend Windows::Synchronize  
extend Windows::MSVCRT::IO  
extend Windows::MSVCRT::Buffer  
extend Windows::SystemInfo  
extend Windows::Memory  
extend Windows::NIO

```
  class Error < StandardError; end
  # Reads the entire file specified by portname as individual

```

lines, and  
# returns those lines in an array. Lines are separated by +sep+.  
#–  
# The semantics are the same as the MRI version but the  
implementation  
# is drastically different. We use a scattered IO read, which is  
about  
# as fast as the MRI version for small files, but much faster  
for very  
# large files.  
#  
def self.readlines(file, sep = $/)  
handle = CreateFile(  
file,  
GENERIC\_READ,  
FILE\_SHARE\_READ,  
nil,  
OPEN\_EXISTING,  
FILE\_FLAG\_OVERLAPPED | FILE\_FLAG\_NO\_BUFFERING,  
nil  
)

```
     if handle == INVALID_HANDLE_VALUE
        raise Error, get_last_error
     end

     # Get your system's page size, probably 4k
     sysbuf = 0.chr * 40
     GetSystemInfo(sysbuf)
     page_size = sysbuf[4,4].unpack('L')[0]
     num_pages = (File.size(file).to_f / page_size).ceil

     base_address = VirtualAlloc(
        nil,
        page_size * num_pages,
        MEM_COMMIT,
        PAGE_READWRITE
     )

     buf_list = []

     for i in 0...num_pages
        buf_list.push(base_address + page_size * i)
     end

     seg_array = buf_list.pack('Q*') + 0.chr * 8
     olap = 0.chr * 20
     olap[16,4] = [CreateEvent(nil, 1, 0, nil)].pack('L')

     bool = ReadFileScatter(
        handle,
        seg_array,
        page_size * num_pages,
        nil,
        olap
     )

     unless bool
        raise Error, get_last_error
     end

     WaitForSingleObject(olap[16,4].unpack('L')[0], INFINITE)

     # MRI's File.size cannot be trusted for files larger than

```

2gb.  
file\_size = [0].pack(‘Q’)  
GetFileSizeEx(handle, file\_size)  
file\_size = file\_size.unpack(‘Q’)[0]

```
     unless CloseHandle(handle)
        raise Error, get_last_error
     end

     buffer = 0.chr * file_size
     memcpy(buffer, buf_list[0], file_size)
     VirtualFree(base_address, 0, MEM_RELEASE)

     # TODO: Fix line ending issue (?)
     unless sep.nil?
        if sep.empty?
           buffer = buffer.split("\r\n\r\n")
        else
           buffer = buffer.split(sep)
        end
     end

     buffer
  end

```

end  
end

---

<div class="post-metadata">

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\
**Post date:** [October 24, 2007, 10:35am UTC](https://www.ruby-forum.com/t/scattered-i-o-on-ms-windows/118066/2 "2007-10-24T10:35:00Z")

</div>

2007/10/23, Daniel B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> probably won’t see any noticeable difference unless the file is  
> greater than 25mb or so, btw.

Can’t test it at the moment. But I wonder how a scattered read can  
help with #readlines which reads a file sequentially. AFAIK scattered  
reading is for reading different portions of the same file.

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\
**Post date:** [October 24, 2007, 4:30pm UTC](https://www.ruby-forum.com/t/scattered-i-o-on-ms-windows/118066/3 "2007-10-24T16:30:26Z")

</div>

On Oct 24, 2:34 am, “Robert K.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> > What do folks think? Are you seeing a performance increase? You  
> > probably won’t see any noticeable difference unless the file is  
> > greater than 25mb or so, btw.
> 
> Can’t test it at the moment. But I wonder how a scattered read can  
> help with #readlines which reads a file sequentially. AFAIK scattered  
> reading is for reading different portions of the same file.

No, it’s just a different technique for reading files. You pre-  
allocate the line buffers up front and then it then reads the lines  
into those buffers asynchronously. But the resulting array still ends  
up “in order”. At least, that’s how it seems to work on MS Windows. 🙂

Regards,

Dan

---

<div class="post-metadata">

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\
**Post date:** [October 24, 2007, 4:49pm UTC](https://www.ruby-forum.com/t/scattered-i-o-on-ms-windows/118066/4 "2007-10-24T16:49:11Z")

</div>

2007/10/24, Daniel B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> > > think I’m seeing a 2x performance increase, but page caching is making  
> > > reading is for reading different portions of the same file.
> 
> No, it’s just a different technique for reading files. You pre-  
> allocate the line buffers up front and then it then reads the lines  
> into those buffers asynchronously. But the resulting array still ends  
> up “in order”. At least, that’s how it seems to work on MS Windows. 🙂

Ah, I see! So the “scatter” does not refer to the user request but to  
the fact that blocks of a file are likely scattered on the disk and  
the OS attempts to do an optimized read. Thanks for clarifying!

Kind regards

robert