Pure Ruby Zlib::GzipWriter


#1

Hi,

There’s a project out there called “zliby” [1] which partially
implements the zlib library. Unfortunately, only the Zlib::GzipReader
class has been implemented so far. I’d like to have a pure Ruby
implementation of the Zlib::GzipWriter class.

I’ve put in a feature request [2] but I’d like to see it expedited. In
order to expedite this, I’m offering $100 to anyone who can implement
it. In order to qualify for the bounty, your Zlib::GzipWriter class
must meet the following criteria:

  • It must be pure Ruby. No extensions. No FFI.
  • It must have tests (and pass them)
  • It must match the current Zlib::GzipWriter interface
  • It must be cross platform
  • It must be able to generate a proper gem file from a gemspec (the
    acid test)
  • It must be contributed back to the zliby project
  • It must be done within 60 days of this post

If these conditions are met, you will receive your bounty via PayPal.

What’s my motivation? Mainly MS Windows. I run a hand built version of
Ruby, and getting the zlib wrapper to build with MS VC++ is such a
colossal PITA that I’d like to avoid it completely. Beyond that,
however, is that it eliminates a third party dependency (zlib) and
gives me the ability to create gems. At the moment I can only read
them.

Any takers? Any matchers?

Regards,

Dan

[1] http:/www.rubyforge.org/projects/zliby
[2]
http://rubyforge.org/tracker/index.php?func=detail&aid=23239&group_id=5769&atid=22371


#2

I’ve put in a feature request [2] but I’d like to see it expedited. In
order to expedite this, I’m offering $100 to anyone who can implement
it. In order to qualify for the bounty, your Zlib::GzipWriter class
must meet the following criteria:
That would indeed be convenient.
Would be nice to someday no longer have the dependency of rubygems on
zlib.
http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/a6966f6fc1424f61
Then there’d be one less setup hurdle for ruby.
I’m a watcher for now though :slight_smile:
-=R


#3

rogerdpack wrote:

I’ve put in a feature request [2] but I’d like to see it expedited. In
order to expedite this, I’m offering $100 to anyone who can implement
it. In order to qualify for the bounty, your Zlib::GzipWriter class
must meet the following criteria:
That would indeed be convenient.
Would be nice to someday no longer have the dependency of rubygems on
zlib.
http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/a6966f6fc1424f61
Then there’d be one less setup hurdle for ruby.
I’m a watcher for now though :slight_smile:

Of course you could use JRuby, which doesn’t have such issues with zlib
and Windows…

  • Charlie

#4

On Dec 21, 1:08 pm, Charles Oliver N. removed_email_address@domain.invalid
wrote:

I’m a watcher for now though :slight_smile:

Of course you could use JRuby, which doesn’t have such issues with zlib
and Windows…

I thought JRuby used FFI. No? If not, it should just be matter of
porting the code. But if so, you still have two third party
dependencies - FFI and zlib.dll.

Regards,

Dan


#5

Daniel B. wrote:

I thought JRuby used FFI. No? If not, it should just be matter of
porting the code. But if so, you still have two third party
dependencies - FFI and zlib.dll.

The JVM includes zlib and obviously has a very nice version for Windows.
We just wrap the JVM’s zlib-based compression classes. But that doesn’t
really solve your original problem since you want to run a handbuilt C
Ruby…

  • Charlie

#6

On Dec 21, 5:16 pm, The Higgs bozo removed_email_address@domain.invalid wrote:

file, decode it, then call it with DL or FFI?
I’m not really sure how to do that, and it’s not really what I’m
looking for I’m afraid.

Regards,

Dan


#7

Daniel B. wrote:

On Dec 21, 5:16�pm, The Higgs bozo removed_email_address@domain.invalid wrote:

file, decode it, then call it with DL or FFI?
I’m not really sure how to do that, and it’s not really what I’m
looking for I’m afraid.

You didn’t know how, or didn’t understand? DL is part of the standard
library. You could carry around the zlib dll right inside an .rb file,
and use DL to call the zlib function(s) you needed.

I understand you were looking for a general-purpose solution, but in the
meantime this does what you need, albeit gymnastically. Namely, zlib on
Windows without the requirement of building a ruby extension.


#8

Daniel B. wrote:

What’s my motivation? Mainly MS Windows. I run a hand built version of
Ruby, and getting the zlib wrapper to build with MS VC++ is such a
colossal PITA that I’d like to avoid it completely. Beyond that,
however, is that it eliminates a third party dependency (zlib) and
gives me the ability to create gems. At the moment I can only read
them.

As a cheap shortcut, could you store a base64-encoded zlib dll in a .rb
file, decode it, then call it with DL or FFI?


#9

The Higgs bozo wrote:

library. You could carry around the zlib dll right inside an .rb file,
and use DL to call the zlib function(s) you needed.

Out of curiosity, how do you suggest doing this? Put the dll in the DATA
section, write to a tmp file, and load? Or is there some magic?


#10

On Dec 21, 10:59 pm, The Higgs bozo removed_email_address@domain.invalid wrote:

Daniel B. wrote:

On Dec 21, 5:16 pm, The Higgs bozo removed_email_address@domain.invalid wrote:

file, decode it, then call it with DL or FFI?
I’m not really sure how to do that, and it’s not really what I’m
looking for I’m afraid.

You didn’t know how, or didn’t understand? DL is part of the standard
library. You could carry around the zlib dll right inside an .rb file,
and use DL to call the zlib function(s) you needed.

Just to clarify for purposes of the bounty, no DL, and no distributed
dll’s.

I understand you were looking for a general-purpose solution, but in the
meantime this does what you need, albeit gymnastically. Namely, zlib on
Windows without the requirement of building a ruby extension.

I really do appreciate the suggestion but it’s not what I’m looking
for.

Regards,

Dan


#11

What’s my motivation? Mainly MS Windows. I run a hand built version of
Ruby, and getting the zlib wrapper to build with MS VC++ is such a
colossal PITA that I’d like to avoid it completely. Beyond that,
however, is that it eliminates a third party dependency (zlib) and
gives me the ability to create gems. At the moment I can only read
them.

One thing I use to help me compile on doze is Lavena’s rubyinstaller
[I’m sure you’ve seen it…].
Anyway I’ll match $50 :slight_smile:
-=r


#12

On Dec 26, 8:05 pm, Roger P. removed_email_address@domain.invalid wrote:

What’s my motivation? Mainly MS Windows. I run a hand built version of
Ruby, and getting the zlib wrapper to build with MS VC++ is such a
colossal PITA that I’d like to avoid it completely. Beyond that,
however, is that it eliminates a third party dependency (zlib) and
gives me the ability to create gems. At the moment I can only read
them.

One thing I use to help me compile on doze is Lavena’s rubyinstaller
[I’m sure you’ve seen it…].
Anyway I’ll match $50 :slight_smile:

Excellent Roger, thank you.

By the way, MS Windows is not the only reason removing the external
dependency on zlib would be a good thing:

http://rubyforge.org/tracker/index.php?func=detail&aid=23354&group_id=126&atid=575

Regards,

Dan


#13

On Dec 15 2008, 9:32 am, Daniel B. removed_email_address@domain.invalid wrote:

must meet the following criteria:
If these conditions are met, you will receive your bounty via PayPal.

What’s my motivation? Mainly MS Windows. I run a hand built version of
Ruby, and getting the zlib wrapper to build with MS VC++ is such a
colossal PITA that I’d like to avoid it completely. Beyond that,
however, is that it eliminates a third party dependency (zlib) and
gives me the ability to create gems. At the moment I can only read
them.

Yet more ammunition:

http://www.misuse.org/science/2007/09/07/ruby-zlib-buffer-error-on-windows/

Regards,

Dan


#14

Yet more ammunition:

http://www.misuse.org/science/2007/09/07/ruby-zlib-buffer-error-on-windows/

Interesting. I wonder if it’s somehow related to
http://redmine.ruby-lang.org/issues/show/776
-=r


#15

On Jan 3, 8:21 pm, Roger P. removed_email_address@domain.invalid wrote:

Yet more ammunition:

http://www.misuse.org/science/2007/09/07/ruby-zlib-buffer-error-on-wi

Interesting. I wonder if it’s somehow related to http://redmine.ruby-lang.org/issues/show/776

It probably is, but there’s nothing we can do about it since zlib
isn’t developed separately as a gem, and it would still require that
someone build a binary.

If we had a pure Ruby version available as a gem on the other hand…

:slight_smile:

Regards,

Dan


#16

Hi,

2009/1/6 Daniel B. removed_email_address@domain.invalid:

If we had a pure Ruby version available as a gem on the other hand…

:slight_smile:

I finished translation and testing of the pure ruby version.
The ruby version is exact same api interface and functionality with
the zlib c source code.
It is also compatible with ruby 1.8.x and 1.9.x and includes
examble.rb and minizip.rb for testing.

You can download at http://121.78.227.9/rbzlib/

I am working on translation of ruby library zlib.c to pure ruby code.

Regards,

Park H.


#17

Daniel B. wrote:

If we had a pure Ruby version available as a gem on the other hand…

:slight_smile:

Regards,

Dan

I’m most of the way done, just missing dynamic huffman trees, and more
interface similarity with zlib. Need to change the specs from zliby
though as the algorithm I’m using is not identical to zlib and gives
different (but still valid) results.

For example, zlib’s lz77 phase doesn’t seem to behave optimally wrt
repeats:

Zlib::Deflate.deflate(0.chr * 10, 9).length
=> 10

CZlib::Deflate.deflate(0.chr * 10, 9).length
=> 11

CZlib::Inflate.inflate(Zlib::Deflate.deflate(0.chr * 10, 9)) == 0.chr * 10
=> true

CZlib::Inflate.inflate(CZlib::Deflate.deflate(0.chr * 10, 9)) == 0.chr * 10
=> true

(using CZlib for inflation there as I’ve not integrated it with the
zliby code yet).


#18

Heesob P. wrote:

It is also compatible with ruby 1.8.x and 1.9.x and includes
examble.rb and minizip.rb for testing.

You can download at http://121.78.227.9/rbzlib/

Awesome!

I am working on translation of ruby library zlib.c to pure ruby code.

Looks like you and Charles L. are in a race. :slight_smile:

Regards,

Dan


#19

Daniel B. wrote:

Sounds like you’re on track. But, you’re in a race with Park H… :slight_smile:

Dan

Heh. :slight_smile: Somehow I think he’s got it covered. That reimplementation of
zlib in ruby is actually pretty cool. I’m curious Park, as to how much
you did by hand and how much you were able to automate it…?

I’ve put the deflate code i wrote on pastie - http://pastie.org/355367

It’s about 600 lines, but needs cleaning up. Will probably be ~500 when
I’ve added in dynamic trees and cleaned it up a bit.

Prelim benchmarks show the current bottleneck to be the LZ77
implementation which is interesting. I think for it to perform
acceptably, a lot more of the code needs to become more idiomatic,
higher-level ruby.


#20

Charles L. wrote:

interface similarity with zlib. Need to change the specs from zliby

CZlib::Inflate.inflate(Zlib::Deflate.deflate(0.chr * 10, 9)) == 0.chr * 10
=> true

CZlib::Inflate.inflate(CZlib::Deflate.deflate(0.chr * 10, 9)) == 0.chr * 10
=> true

(using CZlib for inflation there as I’ve not integrated it with the
zliby code yet).

Sounds like you’re on track. But, you’re in a race with Park H… :slight_smile:

Dan