ZIP

ZIP

A zip archive icon from The Unarchiver's icon set.
Filename extension	.zip .zipx (newer compression algorithms)
Internet media type	application/zip
Uniform Type Identifier	com.pkware.zip-archive
Magic number	PK\003\004 or PK\005\006 (empty archive) (unless a bootstrap script is present)
Developed by	Phil Katz, PKWARE
Type of format	Data compression
Extended to	JAR (EAR, RAR (Java), WAR) Office Open XML (Microsoft) OpenDocument (ODF) XPI (Mozilla extensions)
Standard(s)	APPNOTE from PKWARE

The ZIP file format is a data compression and archive format. A ZIP file contains one or more files that have been compressed to reduce file size, or stored as-is. The ZIP file format permits a number of compression algorithms but, as of 2009^[update], the Deflate method continues to be dominant.

The format was originally created in 1986 by Phil Katz for PKZIP,^[1] and evolved from the previous ARC compression format by Thom Henderson. The PKZIP format is now supported by many software utilities other than PKZIP (see List of file archivers). Microsoft has included built-in ZIP support (under the name "compressed folders") in versions of its Windows operating system since 1998. Apple has included built-in ZIP support in Mac OS X 10.3 and later.

ZIP files generally use the file extensions ".zip" or ".ZIP" and the MIME media type application/zip, although the ZIP file format has also been used by many programs, usually under a different name. Examples of this usage are Java JAR files, Mozilla Firefox Add-ons (.xpi), id Software .pk3/.pk4 files, Activity Bundles (.xo) for the Sugar learning platform, Winamp and Windows Media Player skins, OpenDocument format and Office Open XML.

History

The name "zip" (meaning "speed") was suggested by Katz's friend, Robert Mahoney. They wanted to imply that their product would be faster than ARC and other compression formats of the time.

The earliest known version of .ZIP File Format Specification was first published as part of PKZIP 0.9 package under the file APPNOTE.TXT.

Katz publicly released technical documentation on the ZIP file format making it an open format. The specification can also be found in the PKWARE site.

Similarly named formats

There are numerous other standards and formats using "zip" as part of their name. Phil Katz stated that he wanted to allow the "zip" name for any archive type.^{[citation needed]} For example, ZIP is distinct from gzip, and the latter is defined in an IETF RFC (1952). Both ZIP and GZIP primarily use the DEFLATE algorithm for compression. Likewise, the ZLIB format (IETF RFC 1950) also uses the DEFLATE compression algorithm, but specifies different headers for error and consistency checking. Other common, similarly-named formats and programs with different native formats include 7-Zip, bzip2, and rzip.

Version history

The .ZIP File Format Specification has its own version number, which does not necessarily correspond to the version numbers for the PKZIP tool, especially with PKZIP 6 or later. At various times, PKWARE adds preliminary features that allows PKZIP products to extract archives using advanced features, but PKZIP products that create such archives won't be available until the next major release. Other companies or organizations support the PKWARE specifications at their own pace.

A summary of key advances in various versions of the PKWARE spec:

• 2.0: File entries can be compressed with DEFLATE.
• 4.5: Documented 64-bit ZIP format.
• 5.0: DES, 3DES, RC2, RC4 supported for encryption
• 5.2: RC2-64 supported for Encryption.
• 6.1: Documented certificate storage.
• 6.2.0: Documented Central Directory Encryption.
• 6.3.0: Documented Unicode (UTF-8) filename storage. Expanded list of supported hash, compression, encryption algorithms.
• 6.3.1: Corrected standard hash values for SHA-256/384/512.
• 6.3.2: Documented compression method 97 (WavPack).

WinZip, starting with version 12.1, uses the extension .zipx for ZIP files that use compression methods newer than DEFLATE.^[2]

Technical information

ZIP is a simple archive format that compresses every file separately. Compressing files separately allows for individual files to be retrieved without reading through other data; in theory, it may allow better compression by using different algorithms for different files. A caveat to this is that archives containing a large number of small files end up significantly larger than if they were compressed as a single file, due to the fact that the data structures which store information on each individual file are stored uncompressed (the classic example of the latter is the common tar.gz archive which consists of a TAR archive compressed using gzip).

The original ZIP format had a 4GB limit on various things (uncompressed size of a file, compressed size of a file and total size of the archive), as well as a limit of 65535 entries in a zip archive. In version 4.5 of the specification (which is not the same as v4.5 of any particular tool), PKWARE introduced the "ZIP64" format extensions to get around these limitations. Zip64 support is emerging. For example, the File Explorer in Windows XP does not support ZIP64, but the Explorer in Windows Vista does. Likewise - some libraries, such as IO::Compress::Zip in Perl, have new support for ZIP64, while others, such as Java's built-in java.util.zip, still lack it.

The ZIP spec also supports spreading archives across multiple filesystem files. Originally intended for storage of large zip files across multiple 1.44mb floppy disks, this feature is now used for sending zip archives in parts over email, or over other transports or removable media. The FAT filesystem of DOS only has a timestamp resolution of two seconds; ZIP file records mimic this. As a result, the built-in timestamp resolution of files in a ZIP archive is only two seconds, though extra fields can be used to store more accurate timestamps.

Since September 2007, the ZIP specification (APPNOTE.TXT) contains a provision to store file names using UTF-8, finally adding Unicode compatibility to ZIP^{[citation needed]}.

Not all of the zip features are implemented by all of the various libraries and zip toolkits.

Compression methods

Current .ZIP File Format Specification documented the details of following methods: stored (no compression), Shrunk, Reduced (methods 1-4), Imploded, Tokenizing, Deflated, Deflate64, BZIP2, LZMA (EFS), WavPack, PPMd.

Mentioned, but not documented in detail methods include PKWARE Data Compression Library Imploding (old IBM TERSE), IBM TERSE (new), IBM LZ77 z Architecture (PFS).

Methods introduced with WinZip include Jpeg compression, AES encryption.^[3] AES encryption is technically not a compression method, but was introduced in WinZip 9.0 public beta when PKWARE's specification omitted Strong Encryption Specification.

Encryption

ZIP supports a simple password-based symmetric encryption system which is documented in the ZIP spec, and known to be seriously flawed. In particular it is vulnerable to known-plaintext attacks which are in some cases made worse by poor implementations of random number generators.^[4]

New features including new compression and encryption (e.g. AES) methods have been documented in the .ZIP File Format Specification since version 5.2. A WinZip-developed AES-based standard is used also by 7-Zip, XCeed, and DotNetZip, but some vendors use other formats.^[5] PKWARE SecureZIP also supports RC2, RC4, DES, 3DES encryption methods, Digital Certificate-based encryption and authentication (X.509), and archive header encryption.^[6]

The structure of a ZIP file

The ZIP file contents are files and directories which are stored in arbitrary order. The location of a file is indicated in the so called central directory which is located at the end of the ZIP file. The files and directories are represented by file entries.

Each file entry is introduced by a local header with information about the file such as the comment, file size and file name, followed by optional "Extra" data fields, and then the possibly compressed, possibly encrypted file data. The "Extra" data fields are the key to the extensibility of the ZIP format. It is the "Extra" fields that are exploited to support ZIP64 formats, WinZip-compatible AES encryption, and NTFS file timestamps. In theory there are many other extensions possible via this coded "extra" field.

The central directory consists of file headers holding, among other metadata, the file names and the relative offset in the archive of the local headers for each file entry.

Each file entry is marked by a specific 4-byte "signature"; each entry in the central directory is likewise marked with a different particular 4-byte signature. ZIP file parsers typically look for the appropriate signatures when parsing a ZIP file.

Due to the fact that the order of the file entries in the directory need not conform to the order of file entries in the archive, the format is non-sequential.

There is no BOF or EOF marker in the ZIP spec. Instead, ZIP tools scan for the signatures of the various fields.

Combining ZIP with other file formats

The ZIP file format allows for a comment containing any data to occur at the end of the file after the central directory.^[7] Also, because the central directory specifies the offset of each file in the archive with respect to the start, it is possible in practice for the first file entry to start at an offset other than zero.

This allows arbitrary data to occur in the file both before and after the ZIP archive data, and for the archive to still be read by a ZIP application. A side-effect of this is that it is possible to author a file that is both a working ZIP archive and another format, provided that the other format tolerates arbitrary data at its end, beginning, or middle. Self-extracting archives (SFX), of the form supported by WinZip and DotNetZip, take advantage of this - they are .exe files that conform to the PKZIP AppNote.txt specification and can be read by compliant zip tools or libraries.

This property of the ZIP format, and of the JAR format which is a variant of ZIP, can be exploited to hide harmful Java classes inside a seemingly harmless file, such as a GIF image uploaded to the web. This so-called GIFAR exploit has been demonstrated as an effective attack against web applications such as Facebook.^[8]

Implementing a ZIP application

There are numerous ZIP tools available, and numerous ZIP libraries for various programming environments. Some of the libraries are commercial, some are not. Some are open source, some are not. WinZip is perhaps the most popular and famous ZIP tool - it runs primarily on Windows based PC and is a user tool for creating or extracing ZIP files. For Windows based Servers where VSS-aware, multi-threading and support for NTFS data stream and attributes are important, BackupAssist Zip Engine is an alternative tool to compress and archive a group of files or Microsoft Exchange Server or Microsoft SQL Server database to a variety of storage devices - from [tape drive] to NAS (Network attached storage). WinRAR, IZarc, Info-zip, 7-zip are other tools, available on various platforms. Some of those tools have library or programmatic interfaces.

There are some useful development libraries which are available as Free Software contributions such as the GNU gzip project and Info-ZIP. For Java, there are a few options: Java Platform, Standard Edition contains the package "java.util.zip" to handle standard zip files; the Zip64File library specifically supports large files (larger than 4GB) and treats ZIP files using random access; and the Apache Ant tool contains a more complete implementation released under the Apache Software License.

For .NET applications, there is a no-cost library called DotNetZip^[9] available in source and binary form under the Microsoft Public License.^[10] It does passwords for symmetric ZIP encryption, Unicode, ZIP64, and WinZip-compatible AES encryption. The Microsoft .NET 3.5 runtime library includes a class System.IO.Packaging.Package^[11] that supports the ZIP format. It is primarily designed for document formats using the ISO/IEC international standard Open Packaging Conventions.

The Info-ZIP implementations of the ZIP format adds support for Unix filesystem features, such as user and group IDs, file permissions, and support for symbolic links. The Apache Ant implementation is aware of these to the extent that it can create files with predefined Unix permissions. The Info-ZIP implementations also know how to use the error correction capabilities built into the ZIP compression format. Some programs (such as IZArc) do not and will choke on a file that has errors.

The Info-ZIP Windows tools also support NTFS filesystem permissions, and will make an attempt to translate from NTFS permissions to Unix permissions or vice-versa when extracting files. This can result in potentially unintended combinations, e.g. .exe files being created on NTFS volumes with executable permission denied.

Strong encryption controversy

When WinZip 9.0 public beta was released in 2003, WinZip introduced its own AES-256 encryption, using a different file format, along with the documentation for the new specification.^[12] The encryption standards themselves were not proprietary, but PKWARE had not updated APPNOTE.TXT to include Strong Encryption Specification (SES) since 2001, which had been used by PKZIP versions 5.0 and 6.0. WinZip technical consultant Kevin Kearney and StuffIt product manager Mathew Covington accused PKWARE of withholding SES, but PKZIP chief technology officer Jim Peterson claimed that Certificate-based encryption was still incomplete. However, the latest publicly available APPNOTE.TXT at the time was version 4.5 (available on PKWARE's FTP site), which not only omitted SES, but also omitted Deflate64, DCL Implode, BZip2 compression methods used by .ZIP files created by contemporary PKZIP products.

To overcome this shortcoming, contemporary products such as PentaZip 'implemented' strong ZIP encryption by encrypting ZIP archives into a different file format.^[13]

In another controversial move, PKWare applied for a patent in 2003-07-16 describing a method for combining .ZIP and strong encryption to create a secure .ZIP file.^[14]

In the end, PKWARE and WinZip agreed to support each other's products. On 2004-01-21, PKWARE announced the support of WinZip-based AES compression format.^[15] In later version of WinZip beta, it is able to support SES-based ZIP files.^[16] PKWARE eventually released version 5.2 of .ZIP File Format Specification to public, which documented SES.

See also

• PKZIP
• List of archive formats
• LZW compression method
• Comparison of file archivers

References

External links

• Judgment in favor of SEA in SEA v. PKWARE and Phil Katz
• Technical specifications of the PKZIP file formats from info-ZIP
• Current file format specification from PKWARE (including many recent features that are not widely supported)
• 18 Years of ZIP format: Happy Birthday at The Data Compression News Blog
• Comparison of the performances of various methods of data compression (french)
• .NET Zip library - reads and writes zip archives.
• ZIP2 file format specification
• ZIP File Quine
• At4J - Java implementation of Zip.
• ZipStorer: A pure C# class to compress and store files in zip archives