SINGULAR VALUE DECOMPOSITION - edsmathscholar.com

In the article The Spectral Decomposition of Symmetric Matrices, it has been shown that every symmetric matrix A can be expressed as A = EΛE’ where E is a matrix whose columns are the eigenvectors of A with the norm 1 and Λ is a diagonal matrix whose elements are the eigenvalues of A. Now, what if the matrix A is not a symmetric matrix, but any matrix? The singular value decomposition does the same “work” as the spectral decomposition, so it can be viewed as a more general decomposition or an extension of the spectral decomposition.

The Singular Value Decomposition

If A is any m x n matrix all elements of which are real numbers then there is an orthogonal m x m matrix U and an orthogonal n x n matrix V such that A = UΣV’ where Σ = [σ_ij] is an m x n matrix with σ_ii ≥ 0 for i = 1, 2, …, k, σ₁₁ ≥ σ₂₂ ≥ … ≥ σ_kk, and σ_ij = 0 if i ≠ j; k = min(m,n). The positive constants σ_ii are called the singular values of A.

How to determine U and V?
Let A = UΣV’. As a result, AA’ = (UΣV’)(VΣ’U’) = U(ΣΣ’)U’, which is a symmetric matrix. Noting that AA’ = U(ΣΣ’)U’, it can be concluded that: 1) U in A = UΣV’ can be obtained by applying the spectral decomposition to AA’ and 2) the diagonal elements of ΣΣ’ are the squares of the diagonal elements of Σ. The matrix V is obtained similarly, namely by applying the spectral decomposition to A’A. (Note that A’A = (VΣ’U’)(UΣV’) = VΣ’ΣV’.)

Example 1

Let $A = \begin{pmatrix}1 & -1 \\ 0 & 1 \\ 1 & 0 \end{pmatrix}$ . Determine the singular value decomposition of A.

Answer
U can be obtained from the spectral decomposition of symmetric matrix AA’. Note that:

By solving |λI – AA’| = 0, it turns out that the the eigenvalues of AA’ are as follows: λ₁ = 3, λ₂ = 1, and λ₃ = 0. The eigenvalues λ₁, λ₂, and λ₃ (respectively) result in the eigenspaces E₁, E₂, and E₃ as shown below.

Consequently, we have the orthogonal matrix U as follows.

Note that if A = UΣV’ then A’ = VΣ’U’ and A’U = VΣ’ = VΣ. Since A’U = VΣ, it follows that $A' \vec{u}_i = \sigma_i \vec{v}_i$ where $\sigma_i = \sqrt{\lambda_i}$ , $\vec{u}_i$ = i-th column vector of U and $\vec{v}_i$ = i-th column vector of V. Therefore, $\vec{v}_i = \frac{1}{\sqrt{\lambda_i}} A' \vec{u}_i$ applies. As a consequence,

So, $V = \begin{pmatrix}1/\sqrt{2} & 1/\sqrt{2} \\ -1/\sqrt{2} & 1/\sqrt{2} \end{pmatrix}$ and the singular value decomposition of A is:

Example 2

Find the singular value decomposition of $A = \begin{pmatrix}3 & 2 & 2 \\ 2 & 3 & -2 \end{pmatrix}$ .

Answer

V can be determined from the spectral decomposition of the symmetric matrix A’A. Note that:

By finding the roots of |λI – A’A| = 0, we have the eigenvalues of AA’, i.e. λ₁ = 25, λ₂ = 9, and λ₃ = 0. The eigenvalues λ₁, λ₂, and λ₃ , respectively, result in the eigenspaces E₁, E₂, and E₃ as follows.

As a consequence, we have the orthogonal matrix V as follows:

Note that if A = UΣV’ then A’ = VΣ’U’ and AV = UΣ. From the relationship AV = UΣ we have $A \vec{v}_i = \sigma_i \vec{u}_i$ where $\sigma_i = \sqrt{\lambda_i}$ , $\vec{v}_i$ = i-th column vector of V and $\vec{u}_i$ = i-th column vector of U. So, $\vec{u}_i = \frac{1}{\sqrt{\lambda_i}} A \vec{v}_i$ applies. As a consequence, it follows that: