Forum: Ruby SparseFile

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Erik V. (Guest)
on 2007-01-06 20:30
(Received via mailing list)
I had to send huge files over a network to another machine.
Most of these files were image files for QEMU: typically 4 GB,
of which only a small portion (~ 400 MB) was used. Both client
and server were Ruby programs on Linux boxes, communicating via
FTP.

I thought it was a good idea to use sparse files [1], so I
searched for a SparseFile class, couldn't find one and wrote
one myself:

 http://www.erikveen.dds.nl/rubycodesnippets/index.html#4.0.0

It seems to work pretty well... ;]

Any thoughts? Ideas? Comments? Do you want it to be available
as a library/gem? Should it become part of Ruby F.s? Trans?

gegroet,
Erik V. - http://www.erikveen.dds.nl/

 [1] http://en.wikipedia.org/wiki/Sparse_file
Eric H. (Guest)
on 2007-01-06 20:30
(Received via mailing list)
On Jan 5, 2007, at 16:26, Erik V. wrote:

>  http://www.erikveen.dds.nl/rubycodesnippets/index.html#4.0.0
>
> It seems to work pretty well... ;]
>
> Any thoughts? Ideas? Comments? Do you want it to be available
> as a library/gem? Should it become part of Ruby F.s? Trans?

You can have a gem hosted on Rubyforge in about 20 minutes with Hoe.
(Provided you have a project to store it in.)

--
Eric H. - removed_email_address@domain.invalid - http://blog.segment7.net

I LIT YOUR GEM ON FIRE!
Olivier (Guest)
on 2007-01-07 20:10
(Received via mailing list)
Hi,

I didn't know about this kind of files, so I googled around for some
explanations. Then, I looked at your code, and I have a question :

What's the purpose of this strange String in the following line ?

SPARSE_BLOCKS[length] ||= "%SCRIPT_SPARSEFILE%00" * length

Is it juste a fake string, or does it have a particular meaning ?

Thanks for any answer !
Erik V. (Guest)
on 2007-01-07 23:08
(Received via mailing list)
> What's the purpose of this strange String in the following
> line ?
>
> SPARSE_BLOCKS[length] ||= "%SCRIPT_SPARSEFILE%00" * length

Oops... Should have been:

 SPARSE_BLOCKS[length] ||= "\000" * length

The Ruby code which generates my site, does a gsub with a
string as the replacement. That means that "\0" has a special
meaning... ;]

Using gsub in the block form seems to work..

Thanks.

gegroet,
Erik V. - http://www.erikveen.dds.nl/
Olivier (Guest)
on 2007-01-08 01:07
(Received via mailing list)
Le dimanche 07 janvier 2007 22:07, Erik V. a écrit :
> string as the replacement. That means that "\0" has a special
> meaning... ;]
>
> Using gsub in the block form seems to work..
>
> Thanks.
>
> gegroet,
> Erik V. - http://www.erikveen.dds.nl/

Oh yes, I really thought I was missing something :)
Thanks !
Erik V. (Guest)
on 2007-01-08 02:45
(Received via mailing list)
I updated the implementation [1] of SparseFile [2] and added
some unit tests [3].

Any thoughts? Ideas? Comments? I welcome any feedback.

gegroet,
Erik V. - http://www.erikveen.dds.nl/

 [1] http://localhost:1234/rubycodesnippets/index.html#4.1.0
 [2] http://localhost:1234/rubycodesnippets/index.html#4.0.0
 [3] http://localhost:1234/rubycodesnippets/index.html#4.3.0
Mauricio F. (Guest)
on 2007-01-08 03:52
(Received via mailing list)
On Mon, Jan 08, 2007 at 09:44:33AM +0900, Erik V. wrote:
> I updated the implementation [1] of SparseFile [2] and added
> some unit tests [3].
>
> Any thoughts? Ideas? Comments? I welcome any feedback.

Instead of
    SPARSE_BLOCKS[length] ||= "\000" * length

    unless data == SPARSE_BLOCKS[length]
      @file.pos = @pos
      @file.write(data)
    end

What about

  unless data.count("\0") == data.size
    @file.pos = @pos
    @file.write(data)
  end

? Saves some mem if you have lots of "null blocks" of different sizes
(or use
huge blocks), and the loop shouldn't be much slower than a strcmpish
one:

    while (s < send) {
  if (table[*s++ & 0xff]) {
      i++;
  }
    }

But a C implementation that detects leading \0s and skips them would be
best.
In Ruby it'd be
  idx = data.index(/[^\0]/)
  if idx
      @file.pos += idx
      @file.write(data[idx..-1])
  else
      @file.write(data)
  end
but the regexp matching and the aref would make it way too slow.
I'll post it tomorrow if I don't forget.
Erik V. (Guest)
on 2007-01-08 12:01
(Received via mailing list)
> unless data.count("\0") == data.size

OK.

I experimented with this as well:

 i = data.index(/[^\000]/)
 if i
   @file.pos = @pos + i
   @file.write(data.gsub(/^\000*|\000*$/, ""))
 end

In theory, it could give better results. But, of course, it is
definitely slower: 10.6 seconds instead of 9.5 seconds for one
particular test.

I'll stick to "unless data.count("\0") == data.size".

gegroet,
Erik V. - http://www.erikveen.dds.nl/
Robert K. (Guest)
on 2007-01-19 17:29
(Received via mailing list)
On 06.01.2007 01:26, Erik V. wrote:
>  http://www.erikveen.dds.nl/rubycodesnippets/index.html#4.0.0
>
> It seems to work pretty well... ;]
>
> Any thoughts? Ideas? Comments? Do you want it to be available
> as a library/gem? Should it become part of Ruby F.s? Trans?

Looks good!  Do you think it's feasible and reasonable to include it
into the std lib's File class?  Could be a flag in open mode, i.e.

File.open("foo", "wsb") do |io|
...

Maybe functionality can go into a Module and which will be included if
the flag is present?

Kind regards

  robert
unknown (Guest)
on 2007-01-19 17:29
(Received via mailing list)
On Sat, 6 Jan 2007, Erik V. wrote:

> http://www.erikveen.dds.nl/rubycodesnippets/index.html#4.0.0
>
> It seems to work pretty well... ;]
>
> Any thoughts? Ideas? Comments? Do you want it to be available
> as a library/gem? Should it become part of Ruby F.s? Trans?
>
> gegroet,
> Erik V. - http://www.erikveen.dds.nl/
>
> [1] http://en.wikipedia.org/wiki/Sparse_file

why not simply zip them?  of course one can also read/write directly to
zipped
files.  the reason i ask is that we do this alot.  what would you
consider
trade-offs to be with SparseFile?

cheers.

-a
Erik V. (Guest)
on 2007-01-19 17:30
(Received via mailing list)
> I had to send huge files over a network to another machine.

(That's a bit misleading. SparseFile is about saving disk space
on the server; not about saving bandwidth....)

A quick test on Windows2000/NTFS: Cygwin works as expected,
including the savings. On plain Windows, the full size of the
file is allocated, although the checksums of the files are
correct.

For now, it looks safe to use SparseFile in your
platform-agnostic application. What about OS/X? BSD?

gegroet,
Erik V. - http://www.erikveen.dds.nl/
Erik V. (Guest)
on 2007-01-19 17:30
(Received via mailing list)
> Looks good! Do you think it's feasible and reasonable to
> include it into the std lib's File class? Could be a flag in
> open mode, i.e.

Reasonable? Well, I think so. It's just a special kind of file
handling. However, it's not too special.

Feasible? I don't know. Somebody who understandes the code in
file.c could answer that. That's not me... ;]

Is somebody able and willing to patch file.c? I would do it
myself, if I was a C programmer...

> File.open("foo", "wsb") do |io|

That sounds reasonable.

gegroet,
Erik V. - http://www.erikveen.dds.nl/
Erik V. (Guest)
on 2007-01-19 17:31
(Received via mailing list)
> why not simply zip them? of course one can also read/write
> directly to zipped files. the reason i ask is that we do this
> alot.

* Availability: A zipped file is not readily available (for
  QEMU).

* Speed: It takes an awful lot of time to zip and unzip
  (typically 4 GB per file!)

* Size: The sparse file is smaller than the zipped file (using
  gzip). (Although I didn't expect that... ;])

* Code: Better readable code:

 io = TCPServer.new("0.0.0.0", 4444).accept
 SparseFile.open("file") do |f|
   while (not io.closed? and data = io.gets)
     f.write(data)
   end
 end

> what would you consider trade-offs to be with SparseFile?

* It might me just a little bit slower than writing to ordinary
  files, because of the extra check and the housekeeping. (But
  it is much, much faster than using ZIP-files...)

* Over time, after thousands or millions of random-access
  writes to the same file, the holes are gone. ZIP-files might
  stay small. (But ZIP-files are very, very slow...) Notice
  this is not a disadvantage compared to ordinary files.

gegroet,
Erik V. - http://www.erikveen.dds.nl/

----------------------------------------------------------------

 $ ls -lh winxp.img
 -rw-r--r-- 1 erik erik 4.0G 2007-01-05 17:29 winxp.img

 $ du -h winxp.img
 997M    winxp.img

 $ gzip < winxp.img | wc -c
 1467534047             # That's more than the sparse file!

 $ time gzip < winxp.img > /dev/null
 real    5m55.910s      # 5 Minutes!
 user    4m51.922s
 sys     0m6.816s
This topic is locked and can not be replied to.