Sparse file

A sparse file: The empty bytes don't need to be saved, thus they can be represented by metadata.

In computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently when blocks allocated to the file are mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual "empty" space which makes up the block, using less disk space. The full block size is written to disk as the actual size only when the block contains "real" (non-empty) data.

When reading sparse files, the file system transparently converts metadata representing empty blocks into "real" blocks filled with zero bytes at runtime. The application is unaware of this conversion.

Most modern file systems support sparse files, including most Unix variants and NTFS, but notably not Apple's HFS+. Sparse files are commonly used for disk images, database snapshots, log files and in scientific applications.

Creating sparse files in Unix

If executed manually, the Unix command:

dd if=/dev/zero of=sparse-file bs=1 count=0 seek=5M

Will create a file of five megabytes in size, but with no data stored on disk (only metadata).

Detecting sparse files in Unix

Sparse files have different apparent and actual file sizes. This can be detected by comparing the output of:

du -s -B1 --apparent-size sparse-file

and:

du -s -B1 sparse-file

Copying sparse files in Unix

Normally, the GNU version of cp is good at detecting whether a file is sparse, so it suffices to run:

cp sparse-file new-file

and new-file will be sparse. However, GNU cp does have a --sparse=WHEN option. This is especially useful if a sparse-file has somehow become non-sparse (i.e. the empty blocks have been written out to disk in full). Disk space can be recovered by doing:

cp --sparse=always formerly-sparse-file recovered-sparse-file

It should be noted that not all cp implementations support the --sparse option and will in all cases expand sparse files, like e.g. FreeBSD's cp. A viable alternative on those systems is to use rsync with its own --sparse option instead of cp.

Advantages

The advantage of sparse files is that storage is only allocated when actually needed: disk space is saved, and large files can be created even if there is insufficient free space on the file system.

Disadvantages

Disadvantages are that sparse files may become fragmented; file system free space reports may be misleading; filling up file systems containing sparse files can have unexpected effects; and copying a sparse file with a program that does not explicitly support them may copy the entire, uncompressed size of the file, including the sparse, mostly zero sections which are not on disk—losing the benefits of the sparse property in the file.

See also

• Comparison of file systems
• Sparse matrix

References

• Giampaolo, Dominic (1999). Practical File System Design with the Be File System. Morgan Kaufmann. ISBN 1-55860-497-9

External links

• NTFS Sparse Files For Programmers
• Creating sparse files in Windows XP Professional using fsutil
• Creating sparse files in Solaris using mkfile
• Understanding Sparse File Sizes in Database Snapshots
• Detecting holes in sparse files