Forum: Ruby Rubyzip - `dup': can't dup NilClass (TypeError)

6f1a640dc3970f4db848cb46a8f0edc3?d=identicon&s=25 Luka Stolyarov (lukich)
on 2010-06-08 07:14
Hello.  I've trying to figure out rubyzip.  Here's the code I had:

require 'rubygems'
require 'zip/zip'

zf = Zip::ZipFile.open('616910.zip')

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup': can't dup NilClass (TypeError)
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`map'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1399:in
`initialize'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`new'
        from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open'
        from zip.rb:5

I'm pretty sure the zip file is there and is not empty.  What might be
causing it?  I googled the problem, but couldn't find a definitive
answer.

Thank you,
Luka
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2010-06-08 10:11
Luka Stolyarov wrote:
> require 'rubygems'
> require 'zip/zip'
>
> zf = Zip::ZipFile.open('616910.zip')
>
> However, when I run it, it throws an error
>
> /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
> `dup': can't dup NilClass (TypeError)

Works fine for me with rubyzip-0.9.1 + "ruby 1.8.6 (2007-09-24
patchlevel 111) [i486-linux]", and I just upgraded to rubyzip-0.9.4 with
the same results.

What platform are you on?

The offending code is here:

    # deep clone
    def dup
      newZipEntrySet = ZipEntrySet.new(@entrySet.values.map { |e| e.dup
})
    end

which suggests to me that the zipfile is corrupt or an unsupported
format, since @entrySet must contain a {value=>nil} pair.

Try modifying this code (line 657):

    def ZipEntry.read_c_dir_entry(io)  #:nodoc:all
      entry = new(io.path)
      entry.read_c_dir_entry(io)
      return entry
    rescue ZipError
      return nil
    end

For example, comment out the rescue ZipError // return nil pair.

It seems that this error handling is bad. Either an exception should be
raised here, or the bad entry should be skipped (not saved as a nil
value in @entrySet which causes the dup error you saw)

Regards,

Brian.
6f1a640dc3970f4db848cb46a8f0edc3?d=identicon&s=25 Luka Stolyarov (lukich)
on 2010-06-08 18:41
I'll give it a try, thank you!
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2010-06-08 19:30
Luka Stolyarov wrote:
> I'll give it a try, thank you!

Just to be clear: I'd expect the program to crash still, but this time
at an earlier point which will give a much more useful error about what
went wrong when parsing the zip directory entry.
89cd14259a097cd0d6644f4a88bdb90d?d=identicon&s=25 Scott Bronson (sabr)
on 2010-08-27 10:17
The 1.57 zipfile on this page fails for me:
http://www.vim.org/scripts/script.php?script_id=2441

The Unix unzip command unzips it just fine.

Here's the direct link to the failing zip:
http://www.vim.org/scripts/download_script.php?src_id=11978

To test it in irb:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src...,
'rb') { |f| Zip::ZipFile.open(f) }

this dies with:

TypeError: can't dup NilClass
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `dup'
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `block in dup'
  from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `map'
        ...etc


the zipfiles before 1.57 are fine.  For instance:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src...,
'rb') { |f| Zip::ZipFile.open(f) }

works just fine.


The nil entry is actually being added by read_central_directory_entries
around line 1250 (I added the raise):

    @entrySet = ZipEntrySet.new
    @size.times {
        @entrySet << ZipEntry.read_c_dir_entry(io) || raise("nil
entry!")
    }

For some reason, read_c_dir_entry is returning nil.  I haven't tried to
figure out why since I'm not familiar with the internals of a zipfile.
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2010-08-27 10:43
Scott Bronson wrote:
> require 'open-uri'
> require 'zip/zipfilesystem'
> open('http://www.vim.org/scripts/download_script.php?src...,
> 'rb') { |f| Zip::ZipFile.open(f) }

For me (with ruby 1.8.7 and rubyzip-0.9.4) that dies with "cannot
convert Tempfile to String", but if I download the zip locally and then
do Zip::ZipFile.open("ert.zip") then I get the same error as you.

> For some reason, read_c_dir_entry is returning nil.  I haven't tried to
> figure out why since I'm not familiar with the internals of a zipfile.

I suggest you apply the following patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:34:19.673351372 +0100
@@ -658,8 +658,8 @@
       entry = new(io.path)
       entry.read_c_dir_entry(io)
       return entry
-    rescue ZipError
-      return nil
+    #rescue ZipError
+    #  return nil
     end

     def file_stat(path)  # :nodoc:

Then you get a more helpful error:

/var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:645:in
`read_c_dir_entry': unknown file type 00 (Zip::ZipInternalError)
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:659:in
`read_c_dir_entry'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1255:in
`read_central_directory_entries'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
`times'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
`read_central_directory_entries'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1261:in
`read_from_stream'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`initialize'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`open'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`initialize'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in `new'
  from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open'

To be honest, I have no idea why rubyzip is catching these errors and
returning nil, instead of letting them propagate upwards. All it does is
make a more obscure error later on ("cannot dup nil")

With another patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:38:07.475854345 +0100
@@ -642,7 +642,7 @@
         when 012
           @ftype = :symlink
         else
-          raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)}"
+          raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)} for entry #{@name.inspect}"
         end
       else
         if name_is_directory?

you can see that the affected entry is .DS_Store. That unpacks as a
regular file from unix unzip.

I don't know why this entry happens to have a type of 0, but you can
allow it like this:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
         case (@externalFileAttributes >> 28)
         when 04
           @ftype = :directory
-        when 010
+        when 010, 00
           @ftype = :file
         when 012
           @ftype = :symlink

Anyway, since you have a good test case for this, I suggest you post it
to the rubyzip mailing list or tracker, if there is one.

Regards,

Brian.
4f190e0b6a6ac75c5f0838b018d64323?d=identicon&s=25 Thomas Sondergaard (pong42)
on 2010-09-10 21:54
Brian Candler wrote:
> I don't know why this entry happens to have a type of 0, but you can
> allow it like this:
>
> --- rubyzip-0.9.4/lib/zip/zip.rb.orig  2010-06-16 21:38:16.755077969
> +0100
> +++ rubyzip-0.9.4/lib/zip/zip.rb  2010-08-27 09:41:14.853352187 +0100
> @@ -637,12 +637,12 @@
>          case (@externalFileAttributes >> 28)
>          when 04
>            @ftype = :directory
> -        when 010
> +        when 010, 00
>            @ftype = :file
>          when 012
>            @ftype = :symlink
>

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running "zipinfo -v" on the zip-file
and notice that it prints a '?' as the file type in the external file
attributes.

rubyzip should not throw if the value is unknown - it should simply
translate it to :unknown.

My familiarity with both ruby and rubyzip are rusting, but perhaps
something like this:

diff -u -r1.47 zip.rb
--- lib/zip/zip.rb  14 May 2010 20:19:08 -0000  1.47
+++ lib/zip/zip.rb  10 Sep 2010 19:51:32 -0000
@@ -643,7 +643,7 @@
         when 012
           @ftype = :symlink
         else
-          raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)}"
+          @ftype = :unknown
         end
       else
         if name_is_directory?
@@ -709,11 +709,11 @@
         when :symlink
           ft = 012
           @unix_perms ||= 0755
-        else
-          raise ZipInternalError, "unknown file type #{self.inspect}"
         end

-        @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) <<
16
+        if (!ft.nil?)
+          @externalFileAttributes = (ft << 12 | (@unix_perms & 07777))
<< 16
+        end
       end

       io <<

The problem should also be reported to the author of the tool generating
the zip archives.

Regards,

Thomas
89cd14259a097cd0d6644f4a88bdb90d?d=identicon&s=25 Scott Bronson (sabr)
on 2010-09-10 22:14
Thomas Sondergaard wrote:
> The encoding of the external file attributes appears to be illegal. To
> verify this with another tool try running "zipinfo -v" on the zip-file
> and notice that it prints a '?' as the file type in the external file
> attributes.

Very strange.  My zipinfo -v doesn't show anything amiss.  I'm using:

   ZipInfo 3.00 of 20 April 2009, by Greg Roelofs and the Info-ZIP
group.


> The problem should also be reported to the author of the tool generating
> the zip archives.

Whatever tool is generating them, it's already in widespread use.  :(
For instance, most zipfiles in the following packages are valid, but
rubyzip can't handle them.

http://www.vim.org/scripts/script.php?script_id=3114
http://www.vim.org/scripts/script.php?script_id=3123
http://www.vim.org/scripts/script.php?script_id=3148
http://www.vim.org/scripts/script.php?script_id=3150
http://www.vim.org/scripts/script.php?script_id=3169

Thanks Thomas.  For now I've worked around it by shelling out to the
unzip command.  If I have time I'll try your patch next week.

    - Scott
89cd14259a097cd0d6644f4a88bdb90d?d=identicon&s=25 Scott Bronson (sabr)
on 2010-09-10 22:16
Scott Bronson wrote:
> Whatever tool is generating them, it's already in widespread use.  :(

That might not actually be true...  Most failing zipfiles were upped by
the same author.  I'll ask Peter Odding what he used to zip them.

    - Scott
F8d3b4213f6ac612d8d14f097defcbc3?d=identicon&s=25 Peter Odding (xolox)
on 2010-09-11 12:25
Hi Scott,

>> Whatever tool is generating them, it's already in widespread use.  :(
>
> That might not actually be true...  Most failing zipfiles were upped by
> the same author.  I'll ask Peter Odding what he used to zip them.

I publish my Vim plug-ins with a Python script that uses the zipfile [1]
module to generate ZIP files. All of the archives uploaded in the last
few months were generated on Ubuntu 9.10 (Karmic) with Python 2.6,
although I just yesterday upgraded my laptop to Ubuntu 10.4 (Lucid) so I
can't check the exact version. The ZIP file generation comes down to the
following code:

  from zipfile import ZipFile, ZIP_DEFLATED
  archive = ZipFile('easytags.zip', 'w', ZIP_DEFLATED)
  for filename in ['plugin/easytags.vim', 'autoload/easytags.vim', ...]:
    # This is where "git cat-file" is used, see below.
    with open(filename) as handle:
      archive.writestr(filename, handle.read())
  archive.close()

One peculiarity about the Python code is that it uses ZipFile.writestr()
[2] instead of ZipFile.write() [3] so that it will only package
committed versions of my scripts (using "git cat-file -p
HEAD:some/file").

Let me know if you need any more information. If it helps I might be
able to resurrect my old environment in a virtual machine, but I haven't
tried yet.

 - Peter Odding

[1] http://docs.python.org/library/zipfile.html
[2] http://docs.python.org/library/zipfile.html#zipfil...
[3] http://docs.python.org/library/zipfile.html#zipfil...
4f190e0b6a6ac75c5f0838b018d64323?d=identicon&s=25 Thomas Sondergaard (pong42)
on 2010-09-11 14:13
I've filed the following issue against the python ZipFile module:

http://bugs.python.org/issue9835
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.