In numerical analysis, nested dissection is a divide and conquer heuristic for the solution of sparse symmetric systems of linear equations based on graph partitioning. Nested dissection was introduced by George (1973); the name was suggested by Garrett Birkhoff.^[1]

Nested dissection consists of the following steps:

• Form an undirected graph in which the vertices represent rows and columns of the system of linear equations, and an edge represents a nonzero entry in the sparse matrix representing the system.
• Recursively partition the graph into subgraphs using separators, small subsets of vertices the removal of which allows the graph to be partitioned into subgraphs with at most a constant fraction of the number of vertices.
• Perform Cholesky decomposition (a variant of Gaussian elimination for symmetric matrices), ordering the elimination of the variables by the recursive structure of the partition: each of the two subgraphs formed by removing the separator is eliminated first, and then the separator vertices are eliminated.

As a consequence of this algorithm, the fill-in (the set of nonzero matrix entries created in the Cholesky decomposition that are not part of the input matrix structure) is limited to at most the square of the separator size at each level of the recursive partition. In particular, for planar graphs (frequently arising in the solution of sparse linear systems derived from two-dimensional finite element method meshes) the resulting matrix has O(n log n) nonzeros, due to the planar separator theorem guaranteeing separators of size O(√n).^[2] For arbitrary graphs there is a nested dissection that guarantees fill-in within a ${\displaystyle O(\min\{{\sqrt {d}}\log ^{4}n,m^{1/4}\log ^{3.5}n\})}$ factor of optimal, where d is the maximum degree and m is the number of non-zeros. ^[3]

• Cycle rank of a graph, or a symmetric Boolean matrix, measures the minimum parallel time needed to perform Cholesky decomposition
• Vertex separator

• George, J. Alan (1973), "Nested dissection of a regular finite element mesh", SIAM Journal on Numerical Analysis, 10 (2): 345–363, Bibcode:1973SJNA...10..345G, doi:10.1137/0710032, JSTOR 2156361.
• Gilbert, John R. (1988), "Some nested dissection order is nearly optimal", Information Processing Letters, 26 (6): 325–328, doi:10.1016/0020-0190(88)90191-3, hdl:1813/6607.
• Gilbert, John R.; Tarjan, Robert E. (1986), "The analysis of a nested dissection algorithm", Numerische Mathematik, 50 (4): 377–404, doi:10.1007/BF01396660.
• Lipton, Richard J.; Rose, Donald J.; Tarjan, Robert E. (1979), "Generalized nested dissection", SIAM Journal on Numerical Analysis, 16 (2): 346–358, Bibcode:1979SJNA...16..346L, doi:10.1137/0716027, JSTOR 2156840.
• Agrawal, Ajit; Klein, Philip; Ravi, R. (1993), "Cutting down on Fill Using Nested Dissection: Provably Good Elimination Orderings", Graph Theory and Sparse Matrix Computation, The IMA Volumes in Mathematics and its Applications, vol. 56, Springer New York, pp. 31–55, doi:10.1007/978-1-4613-8369-7_2, ISBN 978-1-4613-8371-0.