Gram-Schmidt Orthogonalization

Definition: Vectors

To transform linearly independent set $\{{\mathbf{u}}_1,{\mathbf{u}}_2,\dots ,{\mathbf{u}}_r\}$ into an orthogonal set $\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_r\}$:

$${\mathbf{v}}_i={\mathbf{u}}_i-\sum _{j=1}^{i-1}\frac{⟨{\mathbf{u}}_i,{\mathbf{v}}_j⟩}{||{\mathbf{v}}_j|{|}^2}{\mathbf{v}}_j,i=1,2,\dots ,r$$

where

• $⟨{\mathbf{u}}_i,{\mathbf{v}}_j⟩=u_1{v}_1+u_2{v}_2+\cdots +u_r{v}_r$ (Euclidean Inner Product)
• $||{\mathbf{v}}_j||=\sqrt{v_1^2+v_2^2+\cdots +v_r^2}$ (Norm)

Continuing, the vector

$${\mathbf{q}}_i=\frac{1}{||{\mathbf{v}}_i||}{\mathbf{v}}_i$$

form an orthonormal set.

Definition: Matrices

To transform linearly independent set $\{A_1,A_2,\dots ,A_k\}$ of matrices into an orthogonal set $\{B_1,B_2,\dots ,B_k\}$

$$B_i=A_i-\sum _{j=1}^{i-1}\frac{A_i\cdot B_j}{B_j\cdot B_j}{B}_j,i=1,2,\dots ,k:$$

where $A\cdot B=\mathrm{tr}(A^{T}B)$ (Frobenius Inner Product)

Corollary

Let

• $\mathcal{V}$ : Linear space
• $\{A_1,\dots ,A_k\}$ : Nonempty linearly independent set of matrices in $\mathcal{V}$

If $C_i=||B_i|{|}^{-1}{B}_i,\forall i=1,\dots ,k$

Then the matrices $C_1,\dots ,C_k$ are orthonormal

Example

Assume $R^3$ has the Euclidean Inner Product. Apply the Gram-Schmidt process to transform the basis vectors ${\mathbf{u}}_1=(1,1,1),{\mathbf{u}}_2=(0,1,1),{\mathbf{u}}_3=(0,0,1)$ into an orthogonal basis $\{{\mathbf{v}}_1,{\mathbf{v}}_2,{\mathbf{v}}_3\}$, and then normalize the orthogonal basis vectors to obtain an orthonormal basis $\{{\mathbf{q}}_1,{\mathbf{q}}_2,{\mathbf{q}}_3\}$

1. ${\mathbf{v}}_1={\mathbf{u}}_1=(1,1,1)$

$$\begin{array}{rl}{\mathbf{v}}_2& ={\mathbf{u}}_2-\frac{⟨{\mathbf{u}}_2,{\mathbf{v}}_1⟩}{||{\mathbf{v}}_1|{|}^2}{\mathbf{v}}_1\\ & =(0,1,1)-\frac{0\cdot 1+1\cdot 1+1\cdot 1}{1^2+1^2+1^2}(1,1,1)\\ & =(0,1,1)-\frac{2}{3}(1,1,1)\\ & =\left(-\frac{2}{3},\frac{1}{3},\frac{1}{3}\right)\end{array}$$

$$\begin{array}{rl}{\mathbf{v}}_3& =u_3-\frac{⟨{\mathbf{u}}_3,{\mathbf{v}}_1⟩}{||{\mathbf{v}}_1|{|}^2}{\mathbf{v}}_1-\frac{⟨{\mathbf{u}}_3,{\mathbf{v}}_2⟩}{||{\mathbf{v}}_2|{|}^2}{\mathbf{v}}_2\\ & =(0,0,1)-\frac{1}{3}(1,1,1)-\frac{1/3}{2/3}\left(-\frac{2}{3},\frac{1}{3},\frac{1}{3}\right)\\ & =\left(0,-\frac{1}{2},\frac{1}{2}\right)\end{array}$$

Thus, the vectors

$${\mathbf{v}}_1=(1,1,1),{\mathbf{v}}_2=\left(-\frac{2}{3},\frac{1}{3},\frac{1}{3}\right),{\mathbf{v}}_3=\left(0,-\frac{1}{2},\frac{1}{2}\right)$$

form an orthogonal basis for $R^3$. The norm of these vectors are

$$||{\mathbf{v}}_1||=\sqrt{3},||{\mathbf{v}}_2||=\frac{\sqrt{6}}{3},||{\mathbf{v}}_3||=\frac{1}{\sqrt{2}}$$

so an orthonormal basis for $R^3$ is

$${\mathbf{q}}_1=\frac{{\mathbf{v}}_1}{||{\mathbf{v}}_1||}=\left(\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}}\right),{\mathbf{q}}_2=\left(-\frac{2}{\sqrt{6}},\frac{1}{\sqrt{6}},\frac{1}{\sqrt{6}}\right),{\mathbf{q}}_3=\left(0,-\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}}\right)$$