# OT: Why are .zip files so much bigger than tar.gz files? **URL:** \ **Category:** Ruby\ **Created:** [February 19, 2008, 12:54am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489 "2008-02-19T00:54:25Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 12:54am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/1 "2008-02-19T00:54:25Z")
So I has started to offer .zip packages of my projects to make life a little easier for Windows folks --seeing as it’s no sweat off the backs of Linux folks either (unzip) --but then I noticed that zip files are huge. I have a 1MB tar.gz that’s over 3MB as a .zip. Can that really be right? T. ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 1:11am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/2 "2008-02-19T01:11:59Z")
On Feb 19, 2008 10:53 AM, Trans [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > So I has started to offer .zip packages of my projects to make life a > little easier for Windows folks --seeing as it’s no sweat off the > backs of Linux folks either (unzip) --but then I noticed that zip > files are huge. I have a 1MB tar.gz that’s over 3MB as a .zip. Can > that really be right? Certainly can. The ZIP algorithm isn’t as good as compressing things as the gzip algorithm. Have a look at bz2 – it’s typically even better than gzip. > T. Arlen ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 1:23am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/3 "2008-02-19T01:23:07Z")
On Feb 18, 2008 6:53 PM, Trans [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > So I has started to offer .zip packages of my projects to make life a > little easier for Windows folks --seeing as it’s no sweat off the > backs of Linux folks either (unzip) --but then I noticed that zip > files are huge. I have a 1MB tar.gz that’s over 3MB as a .zip. Can > that really be right? Yes, it sounds like it can really be right. A zip archive compresses each file indivually and then adds them to the archive, this makes it easy to extract indivdual files later [1]. A tar.gz archive adds all the files to the tar and then gzips everything at once, this takes advantage of cross-file redundancy for a better overall compression ratio [2]. [1] [ZIP (file format) - Wikipedia](http://en.wikipedia.org/wiki/.zip) [2] [tar (computing) - Wikipedia](http://en.wikipedia.org/wiki/.tar)[gzip - Wikipedia](http://en.wikipedia.org/wiki/.gz) ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 1:50am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/4 "2008-02-19T01:50:31Z")
Personally I am using mostly .tar.bz2 these days, simply because it compresses better than gz, even though gz is faster and nicer to your CPU. I have around 15 GIG of archivable material, if I would move this to gz I assume I would come to around 18 GIG, and when transferring on USB (via a ruby script automatically) every byte that is not transferred matters - some computers only have USB 1.1 and transfer is boringly slow already. But its mostly archival reasons here. 🙂 PS: There recently was a comment about 7zip compressing even better than bzip. Just unfortunately, 7zip is hardly known and I will stick to bzip for now since its much more supported, known and also easy to handle (and doesnt look as attached to company-development as is 7zip) ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 2:49am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/5 "2008-02-19T02:49:51Z")
On Feb 18, 7:22 pm, removed\_email\_address@domain.invalid wrote: > easy to extract indivdual files later [1]. A tar.gz archive adds all > the files to the tar and then gzips everything at once, this takes > advantage of cross-file redundancy for a better overall compression > ratio [2]. That’s interesting. I created a utility a while back called rtar (recursive tar). It drills-down to the bottom of each directory and tars & compresses each directory and compressed each file, working it’s way back up to the top. You end up with a compressed archive similar to your explanation of zip in accessibility, but still with the overall compression of a single tar. I thought it was pretty cool, but basically trivial to implement. So I emailed the GNU maintainers of tar asking them if it might make a nice option to add to tar itself. Of course, they never responded ☹ T. ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 5:24am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/6 "2008-02-19T05:24:07Z")
I saw this on the front page of the programming section of reddit tonight. Thought it fit with the conversation. Seems someone benchmarked the 3 algorithms being discussed. [http://blogs.reucon.com/srt/2008/02/18/compression\_gzip\_vs\_bzip2\_vs\_7\_zip.html](http://blogs.reucon.com/srt/2008/02/18/compression_gzip_vs_bzip2_vs_7_zip.html) ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 2:30am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/7 "2008-02-19T02:30:31Z")
- Trans, 2008-02-19, 08:53: > So I has started to offer .zip packages of my projects to make life a > little easier for Windows folks --seeing as it’s no sweat off the > backs of Linux folks either (unzip) --but then I noticed that zip > files are huge. I have a 1MB tar.gz that’s over 3MB as a .zip. Can > that really be right? Not if done correctly. Hack is as follows: 1. Create zip without using compression 2. Zip that one into a new zip using maximum compression This does not deal with the issue that zip has an inferior compression rate compared to bz2 but at least allows to make use of cross-file redundancy (zip packs each file individually). 7zip and rar are superior to zip but less commonly supported. HTH Josef ‘Jupp’ Schugt ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 5:46am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/8 "2008-02-19T05:46:25Z")
Trans wrote: > > A zip archive compresses > > each file indivually and then adds them to the archive, this makes it > > easy to extract indivdual files later [1]. A tar.gz archive adds all > > the files to the tar and then gzips everything at once, this takes > > advantage of cross-file redundancy for a better overall compression > > ratio [2]. > > That’s interesting. I created a utility a while back called rtar > (recursive tar). Microsoft’s CAB file format has a mixed approach, using a proprietry LZ compression. Basically they compress small files together in groups, which gives you most of the advantages of zip and tar in one. Another approach that could be taken is to flush the compressor for every 32kB of input (since deflate can’t repeat input from further ago anyhow), then append a manifest recording the output byte offset of the flush point that preceeds each file. To grab a file in the middle, seek to the offset that preceeds that file by _two_ blocks, decompress the 32kB to use as history, then continue decompressing until you get to your file. The nice thing is that flushing the compressor leaves the output as a valid deflate() stream, even for decompressors that don’t know why you’ve flushed. If the manifest looks like a normal file, a standard tar utility could still extract the whole archive. I’ve done these sort of games with the zlib compressor - it’s easier than you might think. With a bit of cunning, you can make the resultant file rsync-able even in the face of localized changes, a fact I discussed with Andrew Tridgell some years back. I also wanted to add a long-range predictor to zlib so deflate could repeat blocks from far back… you can use an rsync-style approach to do the prediction rather than an LZ suffix tree. Clifford H… ---
**Author:** ![Trans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/dfb087/32.png) [@Trans](https://www.ruby-forum.com/u/Trans)\ **Post date:** [February 19, 2008, 11:37am UTC](https://www.ruby-forum.com/t/ot-why-are-zip-files-so-much-bigger-than-tar-gz-files/129489/9 "2008-02-19T11:37:33Z")
2008/2/19, Trans [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > So I has started to offer .zip packages of my projects to make life a > little easier for Windows folks --seeing as it’s no sweat off the > backs of Linux folks either (unzip) --but then I noticed that zip > files are huge. I have a 1MB tar.gz that’s over 3MB as a .zip. Can > that really be right? The reason is that in a ZIP all entries are compressed individually while in a TGZ or TBZ the whole stream is compressed. The effect shows especially when there are many small files with similar content as is typical for source code. However, I do have seen ZIP files that were similarly sized - certainly not as much difference as you have observed. This may also depend on the compression algorithm used in a ZIP (I believe IZArc for example supports three different compression algorithms for ZIP). Kind regards robert