Decompositions

Define $\Sigma$ as the $m$ by $n$ matrix whose top $n$ rows contain ${\rm diag}(\sigma_1 ,\ldots, \sigma_n )$ and whose bottom $m-n$ rows are zero. Define the $m$ by $m$ matrix $U = [u_1,\ldots,u_m]$ and the $n$ by $n$ matrix $V = [v_1,\ldots,v_n]$. $U$ is called the left singular vector matrix of $A$, and $V$ is called the right singular vector matrix of $A$. Since the $u_i$ are orthogonal unit vectors, we see that $U^*U=I$; i.e., $U$ is a unitary matrix. If $A$ is real then the $u_i$ are real vectors, so $U^TU = I$, and we also say that $U$ is an orthogonal matrix. The same discussion applies to $V$. The $n+m$ equalities $Av_i = \sigma_i u_i$ and $A^* u_i = \sigma_i v_i$ for $i=1,\ldots,n$ and $A^*u_i = 0$ for $i=n+1,\ldots,m$ may also be written $AV = U \Sigma$ and $A^* U= V \Sigma^*$, or $A = U \Sigma V^*$. The factorization

$$A = U \Sigma V^*$$

is called the SVD of $A$. In other words, $A$ is unitarily (orthogonally) equivalent to the diagonal matrix $\Sigma$.

There are several ``smaller'' versions of the SVD that are often computed. Let $U_t = [u_1,\ldots,u_t]$ be an $m$ by $t$ matrix of the first $t$ left singular vectors, $V_t = [v_1,\ldots,v_t]$ be an $n$ by $t$ matrix of the first $t$ right singular vectors, and $\Sigma_t = {\rm diag}(\sigma_1 ,\ldots, \sigma_t)$ be a $t$ by $t$ matrix of the first $t$ singular values. Then we can make the following definitions.

Thin SVD.

$A = U_n \Sigma_n V_n^*$ is the thin (or economy-sized) SVD of $A$. The thin SVD is much smaller to store and faster to compute than the full SVD when $n \ll m$.

Compact SVD.

$A = U_{r} \Sigma_{r} V_{r}^*$ is the compact SVD of $A$. The compact SVD is much smaller to store and faster to compute than the thin SVD when $r \ll n$.

Truncated SVD.

$A_t = U_{t} \Sigma_{t} V_{t}^*$ is the rank-$t$ truncated (or partial) SVD of $A$, where $t < r$. Among all rank-$t$ matrices $B$, $B=A_t$ is the unique minimizer of $\Vert A - B \Vert _F$ and also minimizes (perhaps not uniquely) $\Vert A - B \Vert _2$. The truncated SVD is much smaller to store and cheaper to compute than the compact SVD when $t \ll r$, and is the most common form of the SVD computed in applications.

The thin SVD may also be written $A = \sum_{i=1}^n \sigma_i u_i v_i^*$. Each $(\sigma_i, u_i, v_i)$ is called a singular triplet. The compact and truncated SVDs may be written similarly (the sum going from $i=1$ to $r$, or $i=1$ to $t$, respectively).

If $A$ is $m$ by $n$ with $m<n$, then its SVD is $A = U \Sigma V^*$, where $U$ is $m$ by $m$, $\Sigma$ is $m$ by $n$ with ${\rm diag}(\sigma_1 ,\ldots, \sigma_m)$ in its first $m$ columns and zeros in columns $m+1$ through $n$, and $V$ is $n$ by $n$. Its thin SVD is $A= U_m \Sigma_m V_m^*$, and the compact SVD and truncated SVD are as above.

More generally, if we take a subset of $k$ columns of $U$ and $V$
(say $\hat{U}= U(:,[2,3,5])$ = columns 2, 3, and 5, and $\hat{V}= V(:,[2,3,5])$), then these columns span a pair of singular subspaces of $A$. If we take the corresponding submatrix $\hat{\Sigma}= {\rm diag}(\sigma_2 , \sigma_3 , \sigma_5 )$ of $\Sigma$, then we can write the corresponding partial SVD $\hat{U}^* A \hat{V}= \hat{\Sigma}$. If the columns in $\hat{U}$ and $\hat{V}$ are replaced by $k$ different orthonormal vectors spanning the same invariant subspace, then we get a different partial SVD $\check{U}^* A \check{V}= \check{A}$, where $\check{A}$ is a $k$ by $k$ matrix whose singular values are those of $\hat{\Sigma}$, though $\check{A}$ may not be diagonal.