Rubyzip - `dup': can't dup NilClass (TypeError)


#1

Hello. I’ve trying to figure out rubyzip. Here’s the code I had:

require ‘rubygems’
require ‘zip/zip’

zf = Zip::ZipFile.open(‘616910.zip’)

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
dup': can't dup NilClass (TypeError) from /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:indup’
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
map' from /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:indup’
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1399:in
initialize' from /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:innew’
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open’
from zip.rb:5

I’m pretty sure the zip file is there and is not empty. What might be
causing it? I googled the problem, but couldn’t find a definitive
answer.

Thank you,
Luka


#2

Luka S. wrote:

require ‘rubygems’
require ‘zip/zip’

zf = Zip::ZipFile.open(‘616910.zip’)

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup’: can’t dup NilClass (TypeError)

Works fine for me with rubyzip-0.9.1 + “ruby 1.8.6 (2007-09-24
patchlevel 111) [i486-linux]”, and I just upgraded to rubyzip-0.9.4 with
the same results.

What platform are you on?

The offending code is here:

# deep clone
def dup
  newZipEntrySet = ZipEntrySet.new(@entrySet.values.map { |e| e.dup 

})
end

which suggests to me that the zipfile is corrupt or an unsupported
format, since @entrySet must contain a {value=>nil} pair.

Try modifying this code (line 657):

def ZipEntry.read_c_dir_entry(io)  #:nodoc:all
  entry = new(io.path)
  entry.read_c_dir_entry(io)
  return entry
rescue ZipError
  return nil
end

For example, comment out the rescue ZipError // return nil pair.

It seems that this error handling is bad. Either an exception should be
raised here, or the bad entry should be skipped (not saved as a nil
value in @entrySet which causes the dup error you saw)

Regards,

Brian.


#3

I’ll give it a try, thank you!


#4

Luka S. wrote:

I’ll give it a try, thank you!

Just to be clear: I’d expect the program to crash still, but this time
at an earlier point which will give a much more useful error about what
went wrong when parsing the zip directory entry.


#5

The 1.57 zipfile on this page fails for me:
http://www.vim.org/scripts/script.php?script_id=2441

The Unix unzip command unzips it just fine.

Here’s the direct link to the failing zip:
http://www.vim.org/scripts/download_script.php?src_id=11978

To test it in irb:

require ‘open-uri’
require ‘zip/zipfilesystem’
open(‘http://www.vim.org/scripts/download_script.php?src_id=11978’,
‘rb’) { |f| Zip::ZipFile.open(f) }

this dies with:

TypeError: can’t dup NilClass
from rubyzip-0.9.4/lib/zip/zip.rb:1163:in dup' from rubyzip-0.9.4/lib/zip/zip.rb:1163:inblock in dup’
from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `map’
…etc

the zipfiles before 1.57 are fine. For instance:

require ‘open-uri’
require ‘zip/zipfilesystem’
open(‘http://www.vim.org/scripts/download_script.php?src_id=11524’,
‘rb’) { |f| Zip::ZipFile.open(f) }

works just fine.

The nil entry is actually being added by read_central_directory_entries
around line 1250 (I added the raise):

@entrySet = ZipEntrySet.new
@size.times {
    @entrySet << ZipEntry.read_c_dir_entry(io) || raise("nil 

entry!")
}

For some reason, read_c_dir_entry is returning nil. I haven’t tried to
figure out why since I’m not familiar with the internals of a zipfile.


#6

Scott B. wrote:

require ‘open-uri’
require ‘zip/zipfilesystem’
open(‘http://www.vim.org/scripts/download_script.php?src_id=11978’,
‘rb’) { |f| Zip::ZipFile.open(f) }

For me (with ruby 1.8.7 and rubyzip-0.9.4) that dies with “cannot
convert Tempfile to String”, but if I download the zip locally and then
do Zip::ZipFile.open(“ert.zip”) then I get the same error as you.

For some reason, read_c_dir_entry is returning nil. I haven’t tried to
figure out why since I’m not familiar with the internals of a zipfile.

I suggest you apply the following patch:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:34:19.673351372 +0100
@@ -658,8 +658,8 @@
entry = new(io.path)
entry.read_c_dir_entry(io)
return entry

  • rescue ZipError
  •  return nil
    
  • #rescue ZipError

  • return nil

    end

    def file_stat(path) # :nodoc:

Then you get a more helpful error:

/var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:645:in
read_c_dir_entry': unknown file type 00 (Zip::ZipInternalError) from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:659:inread_c_dir_entry’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1255:in
read_central_directory_entries' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:intimes’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
read_central_directory_entries' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1261:inread_from_stream’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
initialize' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:inopen’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
initialize' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:innew’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open’

To be honest, I have no idea why rubyzip is catching these errors and
returning nil, instead of letting them propagate upwards. All it does is
make a more obscure error later on (“cannot dup nil”)

With another patch:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:38:07.475854345 +0100
@@ -642,7 +642,7 @@
when 012
@ftype = :symlink
else

  •      raise ZipInternalError, "unknown file type #{'0%o' % 
    

(@externalFileAttributes >> 28)}"

  •      raise ZipInternalError, "unknown file type #{'0%o' % 
    

(@externalFileAttributes >> 28)} for entry #{@name.inspect}"
end
else
if name_is_directory?

you can see that the affected entry is .DS_Store. That unpacks as a
regular file from unix unzip.

I don’t know why this entry happens to have a type of 0, but you can
allow it like this:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
case (@externalFileAttributes >> 28)
when 04
@ftype = :directory

  •    when 010
    
  •    when 010, 00
         @ftype = :file
       when 012
         @ftype = :symlink
    

Anyway, since you have a good test case for this, I suggest you post it
to the rubyzip mailing list or tracker, if there is one.

Regards,

Brian.


#7

Thomas Sondergaard wrote:

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running “zipinfo -v” on the zip-file
and notice that it prints a ‘?’ as the file type in the external file
attributes.

Very strange. My zipinfo -v doesn’t show anything amiss. I’m using:

ZipInfo 3.00 of 20 April 2009, by Greg Roelofs and the Info-ZIP
group.

The problem should also be reported to the author of the tool generating
the zip archives.

Whatever tool is generating them, it’s already in widespread use. :frowning:
For instance, most zipfiles in the following packages are valid, but
rubyzip can’t handle them.

http://www.vim.org/scripts/script.php?script_id=3114
http://www.vim.org/scripts/script.php?script_id=3123
http://www.vim.org/scripts/script.php?script_id=3148
http://www.vim.org/scripts/script.php?script_id=3150
http://www.vim.org/scripts/script.php?script_id=3169

Thanks Thomas. For now I’ve worked around it by shelling out to the
unzip command. If I have time I’ll try your patch next week.

- Scott

#8

Scott B. wrote:

Whatever tool is generating them, it’s already in widespread use. :frowning:

That might not actually be true… Most failing zipfiles were upped by
the same author. I’ll ask Peter Odding what he used to zip them.

- Scott

#9

Brian C. wrote:

I don’t know why this entry happens to have a type of 0, but you can
allow it like this:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
case (@externalFileAttributes >> 28)
when 04
@ftype = :directory

  •    when 010
    
  •    when 010, 00
         @ftype = :file
       when 012
         @ftype = :symlink
    

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running “zipinfo -v” on the zip-file
and notice that it prints a ‘?’ as the file type in the external file
attributes.

rubyzip should not throw if the value is unknown - it should simply
translate it to :unknown.

My familiarity with both ruby and rubyzip are rusting, but perhaps
something like this:

diff -u -r1.47 zip.rb
— lib/zip/zip.rb 14 May 2010 20:19:08 -0000 1.47
+++ lib/zip/zip.rb 10 Sep 2010 19:51:32 -0000
@@ -643,7 +643,7 @@
when 012
@ftype = :symlink
else

  •      raise ZipInternalError, "unknown file type #{'0%o' % 
    

(@externalFileAttributes >> 28)}"

  •      @ftype = :unknown
       end
     else
       if name_is_directory?
    

@@ -709,11 +709,11 @@
when :symlink
ft = 012
@unix_perms ||= 0755

  •    else
    
  •      raise ZipInternalError, "unknown file type #{self.inspect}"
       end
    
  •    @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) << 
    

16

  •    if (!ft.nil?)
    
  •      @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) 
    

<< 16

  •    end
     end
    
     io <<
    

The problem should also be reported to the author of the tool generating
the zip archives.

Regards,

Thomas


#10

Hi Scott,

Whatever tool is generating them, it’s already in widespread use. :frowning:

That might not actually be true… Most failing zipfiles were upped by
the same author. I’ll ask Peter Odding what he used to zip them.

I publish my Vim plug-ins with a Python script that uses the zipfile [1]
module to generate ZIP files. All of the archives uploaded in the last
few months were generated on Ubuntu 9.10 (Karmic) with Python 2.6,
although I just yesterday upgraded my laptop to Ubuntu 10.4 (Lucid) so I
can’t check the exact version. The ZIP file generation comes down to the
following code:

from zipfile import ZipFile, ZIP_DEFLATED
archive = ZipFile(‘easytags.zip’, ‘w’, ZIP_DEFLATED)
for filename in [‘plugin/easytags.vim’, ‘autoload/easytags.vim’, …]:
# This is where “git cat-file” is used, see below.
with open(filename) as handle:
archive.writestr(filename, handle.read())
archive.close()

One peculiarity about the Python code is that it uses ZipFile.writestr()
[2] instead of ZipFile.write() [3] so that it will only package
committed versions of my scripts (using “git cat-file -p
HEAD:some/file”).

Let me know if you need any more information. If it helps I might be
able to resurrect my old environment in a virtual machine, but I haven’t
tried yet.

  • Peter Odding

[1] http://docs.python.org/library/zipfile.html
[2] http://docs.python.org/library/zipfile.html#zipfile.ZipFile.writestr
[3] http://docs.python.org/library/zipfile.html#zipfile.ZipFile.write


#11

I’ve filed the following issue against the python ZipFile module:

http://bugs.python.org/issue9835