Forum: Ruby Rubyzip - `dup': can't dup NilClass (TypeError)

Hello.  I've trying to figure out rubyzip.  Here's the code I had:

require 'rubygems'
require 'zip/zip'

zf = Zip::ZipFile.open('616910.zip')

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup': can't dup NilClass (TypeError)
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`map'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1399:in
`initialize'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`new'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open'
        from zip.rb:5

I'm pretty sure the zip file is there and is not empty.  What might be
causing it?  I googled the problem, but couldn't find a definitive
answer.

Thank you,
Luka

Luka Stolyarov wrote:
> require 'rubygems'
> require 'zip/zip'
> 
> zf = Zip::ZipFile.open('616910.zip')
> 
> However, when I run it, it throws an error
> 
> /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
> `dup': can't dup NilClass (TypeError)

Works fine for me with rubyzip-0.9.1 + "ruby 1.8.6 (2007-09-24 
patchlevel 111) [i486-linux]", and I just upgraded to rubyzip-0.9.4 with 
the same results.

What platform are you on?

The offending code is here:

    # deep clone
    def dup
      newZipEntrySet = ZipEntrySet.new(@entrySet.values.map { |e| e.dup 
})
    end

which suggests to me that the zipfile is corrupt or an unsupported 
format, since @entrySet must contain a {value=>nil} pair.

Try modifying this code (line 657):

    def ZipEntry.read_c_dir_entry(io)  #:nodoc:all
      entry = new(io.path)
      entry.read_c_dir_entry(io)
      return entry
    rescue ZipError
      return nil
    end

For example, comment out the rescue ZipError // return nil pair.

It seems that this error handling is bad. Either an exception should be 
raised here, or the bad entry should be skipped (not saved as a nil 
value in @entrySet which causes the dup error you saw)

Regards,

Brian.

I'll give it a try, thank you!

Luka Stolyarov wrote:
> I'll give it a try, thank you!

Just to be clear: I'd expect the program to crash still, but this time 
at an earlier point which will give a much more useful error about what 
went wrong when parsing the zip directory entry.

The 1.57 zipfile on this page fails for me: 
http://www.vim.org/scripts/script.php?script_id=2441

The Unix unzip command unzips it just fine.

Here's the direct link to the failing zip: 
http://www.vim.org/scripts/download_script.php?src_id=11978

To test it in irb:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src..., 
'rb') { |f| Zip::ZipFile.open(f) }

this dies with:

TypeError: can't dup NilClass
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `dup'
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `block in dup'
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `map'
        ...etc


the zipfiles before 1.57 are fine.  For instance:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src..., 
'rb') { |f| Zip::ZipFile.open(f) }

works just fine.


The nil entry is actually being added by read_central_directory_entries 
around line 1250 (I added the raise):

    @entrySet = ZipEntrySet.new
    @size.times {
        @entrySet << ZipEntry.read_c_dir_entry(io) || raise("nil 
entry!")
    }

For some reason, read_c_dir_entry is returning nil.  I haven't tried to 
figure out why since I'm not familiar with the internals of a zipfile.

Scott Bronson wrote:
> require 'open-uri'
> require 'zip/zipfilesystem'
> open('http://www.vim.org/scripts/download_script.php?src..., 
> 'rb') { |f| Zip::ZipFile.open(f) }

For me (with ruby 1.8.7 and rubyzip-0.9.4) that dies with "cannot 
convert Tempfile to String", but if I download the zip locally and then 
do Zip::ZipFile.open("ert.zip") then I get the same error as you.

> For some reason, read_c_dir_entry is returning nil.  I haven't tried to 
> figure out why since I'm not familiar with the internals of a zipfile.

I suggest you apply the following patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969 
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:34:19.673351372 +0100
@@ -658,8 +658,8 @@
       entry = new(io.path)
       entry.read_c_dir_entry(io)
       return entry
-    rescue ZipError
-      return nil
+    #rescue ZipError
+    #  return nil
     end

     def file_stat(path)  # :nodoc:

Then you get a more helpful error:

/var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:645:in 
`read_c_dir_entry': unknown file type 00 (Zip::ZipInternalError)
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:659:in 
`read_c_dir_entry'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1255:in 
`read_central_directory_entries'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in 
`times'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in 
`read_central_directory_entries'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1261:in 
`read_from_stream'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in 
`initialize'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in 
`open'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in 
`initialize'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in `new'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in 
`open'

To be honest, I have no idea why rubyzip is catching these errors and 
returning nil, instead of letting them propagate upwards. All it does is 
make a more obscure error later on ("cannot dup nil")

With another patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969 
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:38:07.475854345 +0100
@@ -642,7 +642,7 @@
         when 012
           @ftype = :symlink
         else
-          raise ZipInternalError, "unknown file type #{'0%o' % 
(@externalFileAttributes >> 28)}"
+          raise ZipInternalError, "unknown file type #{'0%o' % 
(@externalFileAttributes >> 28)} for entry #{@name.inspect}"
         end
       else
         if name_is_directory?

you can see that the affected entry is .DS_Store. That unpacks as a 
regular file from unix unzip.

I don't know why this entry happens to have a type of 0, but you can 
allow it like this:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969 
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
         case (@externalFileAttributes >> 28)
         when 04
           @ftype = :directory
-        when 010
+        when 010, 00
           @ftype = :file
         when 012
           @ftype = :symlink

Anyway, since you have a good test case for this, I suggest you post it 
to the rubyzip mailing list or tracker, if there is one.

Regards,

Brian.

Brian Candler wrote:
> I don't know why this entry happens to have a type of 0, but you can 
> allow it like this:
> 
> --- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969 
> +0100
> +++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:41:14.853352187 +0100
> @@ -637,12 +637,12 @@
>          case (@externalFileAttributes >> 28)
>          when 04
>            @ftype = :directory
> -        when 010
> +        when 010, 00
>            @ftype = :file
>          when 012
>            @ftype = :symlink
> 

The encoding of the external file attributes appears to be illegal. To 
verify this with another tool try running "zipinfo -v" on the zip-file 
and notice that it prints a '?' as the file type in the external file 
attributes.

rubyzip should not throw if the value is unknown - it should simply 
translate it to :unknown.

My familiarity with both ruby and rubyzip are rusting, but perhaps 
something like this:

diff -u -r1.47 zip.rb
--- lib/zip/zip.rb  14 May 2010 20:19:08 -0000  1.47
+++ lib/zip/zip.rb  10 Sep 2010 19:51:32 -0000
@@ -643,7 +643,7 @@
         when 012
           @ftype = :symlink
         else
-          raise ZipInternalError, "unknown file type #{'0%o' % 
(@externalFileAttributes >> 28)}"
+          @ftype = :unknown
         end
       else
         if name_is_directory?
@@ -709,11 +709,11 @@
         when :symlink
           ft = 012
           @unix_perms ||= 0755
-        else
-          raise ZipInternalError, "unknown file type #{self.inspect}"
         end

-        @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) << 
16
+        if (!ft.nil?)
+          @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) 
<< 16
+        end
       end

       io <<

The problem should also be reported to the author of the tool generating 
the zip archives.

Regards,

Thomas

Thomas Sondergaard wrote:
> The encoding of the external file attributes appears to be illegal. To 
> verify this with another tool try running "zipinfo -v" on the zip-file 
> and notice that it prints a '?' as the file type in the external file 
> attributes.

Very strange.  My zipinfo -v doesn't show anything amiss.  I'm using:

   ZipInfo 3.00 of 20 April 2009, by Greg Roelofs and the Info-ZIP 
group.


> The problem should also be reported to the author of the tool generating 
> the zip archives.

Whatever tool is generating them, it's already in widespread use.  :( 
For instance, most zipfiles in the following packages are valid, but 
rubyzip can't handle them.

http://www.vim.org/scripts/script.php?script_id=3114
http://www.vim.org/scripts/script.php?script_id=3123
http://www.vim.org/scripts/script.php?script_id=3148
http://www.vim.org/scripts/script.php?script_id=3150
http://www.vim.org/scripts/script.php?script_id=3169

Thanks Thomas.  For now I've worked around it by shelling out to the 
unzip command.  If I have time I'll try your patch next week.

    - Scott

Scott Bronson wrote:
> Whatever tool is generating them, it's already in widespread use.  :( 

That might not actually be true...  Most failing zipfiles were upped by 
the same author.  I'll ask Peter Odding what he used to zip them.

    - Scott

Hi Scott,

>> Whatever tool is generating them, it's already in widespread use.  :( 
> 
> That might not actually be true...  Most failing zipfiles were upped by 
> the same author.  I'll ask Peter Odding what he used to zip them.

I publish my Vim plug-ins with a Python script that uses the zipfile [1] 
module to generate ZIP files. All of the archives uploaded in the last 
few months were generated on Ubuntu 9.10 (Karmic) with Python 2.6, 
although I just yesterday upgraded my laptop to Ubuntu 10.4 (Lucid) so I 
can't check the exact version. The ZIP file generation comes down to the 
following code:

  from zipfile import ZipFile, ZIP_DEFLATED
  archive = ZipFile('easytags.zip', 'w', ZIP_DEFLATED)
  for filename in ['plugin/easytags.vim', 'autoload/easytags.vim', ...]:
    # This is where "git cat-file" is used, see below.
    with open(filename) as handle:
      archive.writestr(filename, handle.read())
  archive.close()

One peculiarity about the Python code is that it uses ZipFile.writestr() 
[2] instead of ZipFile.write() [3] so that it will only package 
committed versions of my scripts (using "git cat-file -p 
HEAD:some/file").

Let me know if you need any more information. If it helps I might be 
able to resurrect my old environment in a virtual machine, but I haven't 
tried yet.

 - Peter Odding

[1] http://docs.python.org/library/zipfile.html
[2] http://docs.python.org/library/zipfile.html#zipfil...
[3] http://docs.python.org/library/zipfile.html#zipfil...

I've filed the following issue against the python ZipFile module:

http://bugs.python.org/issue9835