Sparse matrix

A sparse matrix obtained when solving a finite element problem in two dimensions. The non-zero elements are shown in black.

In the subfield of numerical analysis a sparse matrix is a matrix populated primarily with zeros (Stoer & Bulirsch 2002, p. 619). The term itself was coined by Harry M. Markowitz.

Conceptually, sparsity corresponds to systems which are loosely coupled. Consider a line of balls connected by springs from one to the next; this is a sparse system. By contrast, if the same line of balls had springs connecting every ball to every other ball, the system would be represented by a dense matrix. The concept of sparsity is useful in combinatorics and application areas such as network theory, of a low density of significant data or connections.

Huge sparse matrices often appear in science or engineering when solving partial differential equations.

When storing and manipulating sparse matrices on a computer, it is beneficial and often necessary to use specialized algorithms and data structures that take advantage of the sparse structure of the matrix. Operations using standard matrix structures and algorithms are slow and consume large amounts of memory when applied to large sparse matrices. Sparse data is by nature easily compressed, and this compression almost always results in significantly less memory usage. Indeed, some very large sparse matrices are impossible to manipulate with the standard algorithms.

Storing a sparse matrix

The naïve data structure for a matrix is a two-dimensional array. Each entry in the array represents an element a_i,j of the matrix and can be accessed by the two indices i and j. For an m×n matrix we need at least enough memory to store (m×n) entries to represent the matrix.

Substantial memory can be saved by storing only the non-zero entries. Depending on the number and distribution of the non-zero entries, different data structures can be used and yield huge savings in memory when compared to a naïve approach.

One example of such a sparse matrix format is the Yale Sparse Matrix Format. It stores an initial sparse m×n matrix, M, in row form using three one-dimensional arrays. Let NNZ denote the number of nonzero entries of M. The first array is A, which is of length NNZ, and holds all nonzero entries of M in left-to-right top-to-bottom (row-major) order. The second array is IA, which is of length m + 1 (i.e., one entry per row, plus one). IA(i) contains the index in A of the first nonzero element of row i. Row i of the original matrix extends from A(IA(i)) to A(IA(i+1)-1), i.e. from the start of one row to the last index before the start of the next. The third array, JA, contains the column index of each element of A, so it also is of length NNZ.

For example, the matrix

[ 1 2 0 0 ]
[ 0 3 9 0 ]
[ 0 1 4 0 ]

is a three-by-four matrix with six nonzero elements, so

A  = [ 1 2 3 9 1 4 ] // List of non-zero matrix element in order
IA = [ 1 3 5 7 ]     // IA(i) = Index of the first nonzero element of row i in A
JA = [ 1 2 2 3 2 3 ] // JA(i) = Column position of the non zero element A(i)

Note: In this example, the row and line index begin by 1 and not 0.

Example

A bitmap image having only 2 colors, with one of them dominant (say a file that stores a handwritten signature) can be encoded as a sparse matrix that contains only row and column numbers for pixels with the non-dominant color.

Diagonal matrices

A very efficient structure for a diagonal matrix is to store just the entries in the main diagonal as a one-dimensional array, so a diagonal n×n matrix requires only n entries.

Bandwidth

The lower bandwidth of a matrix A is the smallest number p such that the entry a_ij vanishes whenever i > j + p. Similarly, the upper bandwidth is the smallest p such that a_ij = 0 whenever i < j − p (Golub & Van Loan 1996, §1.2.1). For example, a tridiagonal matrix has lower bandwidth 1 and upper bandwidth 1.

Matrices with small upper and lower bandwidth are known as band matrices and often lend themselves to simpler algorithms than general sparse matrices; or one can sometimes apply dense matrix algorithms and gain efficiency simply by looping over a reduced number of indices.

By rearranging the rows and columns of a matrix A it may be possible to obtain a matrix A’ with a lower bandwidth. The Cuthill-McKee algorithm can be used to reduce the bandwidth of a sparse symmetric matrix. There are, however, matrices for which the Reverse Cuthill-McKee algorithm performs better.

The U.S. National Geodetic Survey (NGS) uses Dr. Richard Snay's "Banker's" algorithm because on realistic sparse matrices used in Geodesy work it has better performance.

There are many other methods in use.

Reducing fill-in

"Fill-in" redirects here. For the puzzle, see Fill-In (puzzle).

The fill-in of a matrix are those entries which change from an initial zero to a non-zero value during the execution of an algorithm. To reduce the memory requirements and the number of arithmetic operations used during an algorithm it is useful to minimize the fill-in by switching rows and columns in the matrix. The symbolic Cholesky decomposition can be used to calculate the worst possible fill-in before doing the actual Cholesky decomposition.

There are other methods than the Cholesky decomposition in use. Orthogonalization methods (such as QR factorization) are common, for example, when solving problems by least squares methods. While the theoretical fill-in is still the same, in practical terms the "false non-zeros" can be different for different methods. And symbolic versions of those algorithms can be used in the same manner as the symbolic Cholesky to compute worst case fill-in.

Solving sparse matrix equations

Both iterative and direct methods exist for sparse matrix solving. One popular iterative method is the conjugate gradient method.

See also

• Matrix representation
• Pareto principle
• Ragged matrix
• Skyline matrix
• Sparse array
• Sparse graph code
• Sparse file
• Harwell-Boeing file format

References

• Golub, Gene H.; Van Loan, Charles F. (1996). Matrix Computations (3rd ed.). Baltimore: Johns Hopkins. ISBN 978-0-8018-5414-9. .
• Stoer, Josef; Bulirsch, Roland (2002). Introduction to Numerical Analysis (3rd ed.). Berlin, New York: Springer-Verlag. ISBN 978-0-387-95452-3. .
• Tewarson, Reginald P, Sparse Matrices (Part of the Mathematics in Science & Engineering series), Academic Press Inc., May 1973. (This book, by a professor at the State University of New York at Stony Book, was the first book exclusively dedicated to Sparse Matrices. Graduate courses using this as a textbook were offered at that University in the early 1980s).
• Sparse Matrix Multiplication Package, Randolph E. Bank, Craig C. Douglas [1]
• Pissanetzky, Sergio 1984, "Sparse Matrix Technology", Academic Press
• R. A. Snay. Reducing the profile of sparse symmetric matrices. Bulletin Géodésique, 50:341–352, 1976. Also NOAA Technical Memorandum NOS NGS-4, National Geodetic Survey, Rockville, MD. [2]

External Links

• Oral history interview with Harry M. Markowitz, Charles Babbage Institute, University of Minnesota. Markowitz discusses his development of portfolio theory (for which he received a Nobel Prize in Economics), sparse matrix methods, and his work at the RAND Corporation and elsewhere on simulation software development (including computer language SIMSCRIPT), modeling, and operations research.

Further reading

• Norman E. Gibbs, William G. Poole, Jr. and Paul K. Stockmeyer (1976). "A comparison of several bandwidth and profile reduction algorithms". ACM Transactions on Mathematical Software 2 (4): 322–330. doi:10.1145/355705.355707. http://portal.acm.org/citation.cfm?id=355707. 
• John R. Gilbert, Cleve Moler and Robert Schreiber (1992). "Sparse matrices in MATLAB: Design and Implementation". SIAM Journal on Matrix Analysis and Applications 13 (1): 333–356. doi:10.1137/0613024. http://citeseer.ist.psu.edu/gilbert91sparse.html. 
• Sparse Matrix Algorithms Research at the University of Florida, containing the UF sparse matrix collection.