Background

Definition (LU-Factorization).  The nonsingular matrix A has an LU-factorization if it can be expressed as the product of a lower-triangular matrix L and an upper triangular matrix U:  

          .  

When this is possible we say that A has an LU-decomposition.  It turns out that this factorization (when it exists) is not unique.  If L has 1's on it's diagonal, then it is called a Doolittle factorization.  If U has 1's on its diagonal, then it is called a Crout factorization.  When    (or ),  it is called a Cholesky decomposition.  

Theorem (A = LU;  Factorization with NO Pivoting).  Assume that A has a Doolittle, Crout or Cholesky factorization.  The solution X to the linear system  , is found in three steps:  

    1.  Construct the matrices  , if possible.  
    2.  Solve    for    using forward substitution.
    3.  Solve    for    using back substitution.

Proof  LU Factorization  LU Factorization  

Theorem (A = LU;  Doolittle Factorization).  Assume that A has a Doolittle factorization A = LU.  
=   

The solution X to the linear system  , is found in three steps:  
    1.  Construct the matrices  , if possible.  
    2.  Solve    for    using forward substitution.
    3.  Solve    for    using back substitution.

Proof  Cholesky, Doolittle and Crout Factorization  Cholesky, Doolittle and Crout Factorization  

For curiosity, the reader might be interested in other methods of computing  L  and  U.

Theorem (A = LU;  Crout Factorization).  Assume that A has a Crout factorization A = LU.  
=   

The solution X to the linear system  , is found in three steps:  
    1.  Construct the matrices  , if possible.  
    2.  Solve    for    using forward substitution.
    3.  Solve    for    using back substitution.

Proof  Cholesky, Doolittle and Crout Factorization  Cholesky, Doolittle and Crout Factorization  

Mathematica Subroutine (Doolittle).

Mathematica Subroutine (Crout).

Mathematica Subroutine (Forward Elimination).

Mathematica Subroutine (Back Substitution).

Theorem (Cholesky Factorization).  If A is real, symmetric and positive definite matrix, then it has a Cholesky factorization  

                  

where U an upper triangular matrix.  

Remark.  Observe that is a lower triangular matrix, so that  A = LU.  Hence we could also write Cholesky factorization  

              

where L a lower triangular matrix.  

Theorem (A = LU;  Cholesky Factorization).  Assume that A has a Cholesky factorization  ,  where  .
=   

Or if you prefer to write the Cholesky factorization as  ,  where .
=   

The solution X to the linear system  , is found in three steps:  
    1.  Construct the matrices  , if possible.  
    2.  Solve    for    using forward substitution.
    3.  Solve    for    using back substitution.

Proof  Cholesky, Doolittle and Crout Factorization  Cholesky, Doolittle and Crout Factorization  

    The following Cholesky subroutine can be used when the matrix  A  is real, symmetric and positive definite.
Observe that the loop starting with  For[j=k,j<=n,j++,  is not necessary and that  U  is computed by forming the transpose of  L.

Computer Programs  Cholesky, Doolittle and Crout Factorization  Cholesky, Doolittle and Crout Factorization  

Mathematica Subroutine (Cholesky factorization).

Example 1 (a).  Find the  A = LU factorization for the matrix  .  Use the Doolittle method.  

Example 1 (b).  Find the A = LU  factorization for the matrix  .  Use the Crout method.  

Example 1 (c).  Find the A = LU  factorization for the matrix  .  Use the Cholesky method.  
Solution 1 (a).
Solution 1 (b).
Solution 1 (c).

Example 2.  Consider the linear system  .  

2 (a).  Solve the linear system  AX = B  by using the Doolittle method.  
2 (b).  Solve the linear system  AX = B  by using the Crout method.  
2 (c).  Solve the linear system  AX = B  by using the Cholesky method.  
Solution 2 (a).
Solution 2 (b).
Solution 2 (c).

Application to Polynomial Curve Fitting

Theorem (Least-Squares Polynomial Curve Fitting). Given the    data points  ,  the least squares polynomial of degree  m  of the form  

        

that fits the n data points is obtained by solving the following linear system


          

for the m+1 coefficients .  These equations are referred to as the "normal equations".

Proof  Least Squares Polynomials  Least Squares Polynomials  

Example 3.  Find the "least squares cubic" that for the four data points.   
Solution 3.

Application to Continuous Least Squares Approximation

    The continuous least squares approximation to a function   on the interval [0,1] for the set of functions    can solved by using the normal equations

(1)           for   .

Where the inner product is .  Solve the linear system (1) for the coefficients and construct the approximation function
        
        .

Definition (Gram Matrix).  The Gram matrix G is a matrix of inner products where the elements are  .  

    The case when the set of functions is    will produce the Hilbert matrix.  Since we require the computation to be as exact as possible and an exact formula is known for the inverse of the Hilbert matrix, this is an example where an inverse matrix comes in handy.

Example 4.  Find the continuous least squares polynomial of degree n=4 that approximates the function    over the interval  .  
Solution 4.

Old Lab Project (Doolittle's Method  Doolittle's Method).  Internet hyperlinks to an old lab project.  

Research Experience for Undergraduates

Cholesky, Doolittle and Crout Factorizations  Cholesky, Doolittle and Crout Factorizations  Internet hyperlinks to web sites and a bibliography of articles.  

Download this Mathematica Notebook Cholesky, Doolittle and Crout Factorization

Return to Numerical Methods - Numerical Analysis

(c) John H. Mathews 2004