Win32 File.size slow?

Ruby
1

Last week I posted a question about incorrect file sizes coming back
incorrectly from File.size? for really large files. I changed my script
to use the win32 version, which returns the correct sizes, and now the
script is unacceptably slow. The bottleneck is the File.size call. My
scripts startup went from about a minute to, well I don’t know because
it’s never finished. I’m walking a file path that is somewhat indirect,
but the other file operations perform well. I’m reading files, using
UNC paths, through samba on linux boxes that mount a giant file system
through fiber.

So anyway, does anyone have any alternatives?

Carey Nation mailto:[email protected]

Lead Software Engineer

CNN BEST

Broadcast Production Systems

Video Solutions Group

(404)827-2935 (wk)

(404)824-0033 (cell)

2

On Mar 18, 12:56 pm, “Nation, Carey” [email protected] wrote:

Last week I posted a question about incorrect file sizes coming back
incorrectly from File.size? for really large files. I changed my script
to use the win32 version, which returns the correct sizes, and now the
script is unacceptably slow. The bottleneck is the File.size call. My
scripts startup went from about a minute to, well I don’t know because
it’s never finished. I’m walking a file path that is somewhat indirect,
but the other file operations perform well. I’m reading files, using
UNC paths, through samba on linux boxes that mount a giant file system
through fiber.

That’s interesting. The File.size method is really a wrapper for
File::Stat#size, which uses stat64() to stat the file in question. I
wonder if it’s choking on the Samba/UNC path. Wouldn’t be the first
app to do so.

What happens if you replace File.size in lib/win32/file.rb with this
version:

Untested

class << self
def size(file)
handle = CreateFile(
file,
GENERIC_READ,
FILE_SHARE_READ,
nil,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING,
nil
)

  if handle == INVALID_HANDLE_VALUE
     raise Error, get_last_error
  end

  file_size = [0].pack('Q')
  GetFileSizeEx(handle, file_size)
  file_size = file_size.unpack('Q')[0]

  unless CloseHandle(handle)
     raise Error, get_last_error
  end

  file_size

end
end

3

I’ll try this. I don’t think that it’s the path because it does work.
It’s just not in any hurry. I’ll let you know in a bit about the call
below…

4

That works and the performance is good. Two caveats. First, in the
CreateFile, it wants 0 rather than nil. Second, CloseHandle isn’t
implemented in windows/file.rb, so I added it to my local copy. This is
actually a really big oversight since leaving handles of any kind open
in win32 is a big leak problem.

Thanks!

5

On Mar 19, 12:23 pm, “Nation, Carey” [email protected] wrote:

That works and the performance is good. Two caveats. First, in the CreateFile, it wants 0 rather than nil.

It should work. I’ll look into that.

Second, CloseHandle isn’t implemented in windows/file.rb, so I added it to my local copy. This is actually a really big oversight since leaving handles of any kind open in win32 is a big leak problem.

Ah, ok. Not to panic, since win32-file uses stat64(), so there are no
open handles to begin with.

Regards,

Dan