Let $V$ be a real inner product space. A nonempty set of vectors in $V$ is orthogonal if each vector in the set is orthogonal to all the other vectors in the set. That is, the set $\{\bfv_1,\ldots,\bfv_n\}\subseteq V$ is orthogonal if $$ \langle\bfv_i,\bfv_j\rangle=0,\qquad i\neq j. $$ Let $S$ be a finite set of vectors in $V$ such that ${\bf 0}\notin S$. Then,

An orthogonal set of vectors in $V$ is called orthonormal if all the vectors in the set are unit vectors. That is, the set $\{\bfe_1,\ldots,\bfe_n\}\subset V$ is orthonormal if $$ \langle\bfe_i,\bfe_j\rangle=\delta_{i,j}, $$ where the Kronecker delta is defined by $$ \delta_{i,j}=\begin{cases} \,0,\ & i\neq j,\\[.15cm] \,1,\ &i=j. \end{cases} $$

Examples in $\R^n$ endowed with the usual dot product are given as follows:

Examples in $\R^n$ endowed with the usual dot product are given as follows:

Example 1: $\left\{ (1,0,0), (0,1,0), (0,0,1)\right\}$ orthonormal set.

Example 2: $\mathcal S = \left\{ \left( \frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}} \right), \left(-\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}},0\right) \right\}$ orthonormal set in $\R^3$.

Example 3: $\mathcal S \cup \left\{ \left( \frac{1}{\sqrt{6}},\frac{1}{\sqrt{6}},-\frac{2}{\sqrt{6}} \right) \right\}$ orthonormal set in $\R^3$.

Examples in $\R^n$ endowed with the usual dot product are given as follows:

Example 1: $\left\{ (1,0,0), (0,1,0), (0,0,1)\right\}$ orthonormal set.

Example 2: $\mathcal S = \left\{ \left( \frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}} \right), \left(-\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}},0\right) \right\}$ orthonormal set in $\R^3$.

Example 3: $\mathcal S \cup \left\{ \left( \frac{1}{\sqrt{6}},\frac{1}{\sqrt{6}},-\frac{2}{\sqrt{6}} \right) \right\}$ orthonormal set in $\R^3$.

An orthonormal basis for $V$ is a basis for $V$ that is also an orthonormal set.

The sets in Examples 1 and 3 are also orthonormal basis for $\R^3$, but the set in Example 2 is NOT a basis for $\R^3$.

Let $S=\{\bfe_1,\ldots,\bfe_n\}$ be an orthonormal basis for $V$, and let $\bfu\in V$. Then, $$ \bfu=\langle\bfu,\bfe_1\rangle\bfe_1+\ldots+\langle\bfu,\bfe_n\rangle\bfe_n $$ and $$ ||\bfu||^2=\langle\bfu,\bfe_1\rangle^2+\ldots+\langle\bfu,\bfe_n\rangle^2. $$

Proof: Since $S=\{\bfe_1,\ldots,\bfe_n\}$ is a basis, we can write for any $\u\in V$ \[ \u = a_1\bfe_1 + a_2\bfe_2 + \cdots + a_n\bfe_n. \]

Then, for $i = 1, \ldots n$, we have

Extra: $\norm{\u}^2 = \langle \u ,\u \rangle $ $= \langle \u ,\bfe_1 \rangle ^2+ \cdots +\langle \u ,\bfe_n \rangle ^2.$

Let $U$ be a finite-dimensional subspace of the real inner product space $V$. Then, each $\bfv\in V$ can be written in a unique way as $$ \bfv=\bfu+\bfw,\qquad \bfu\in U,\quad \bfw\in U^\bot. $$ In the proof, we will assume that $U$ has an orthonormal basis $S=\{\bfe_1,\ldots,\bfe_k\}$. As we will see in Section 11.6, this assumption is redundant since every finite-dimensional inner product space has such a basis.

Proof: Let $\v\in S$, then $\v = \u + (\v - \u)$ for any $\u \in U.$ Set \[ \u = \langle \v, \bfe_1 \rangle \bfe_1 + \cdots + \langle \v, \bfe_n \rangle \bfe_n, \quad \text{then }\u \in U. \quad (\star) \] and set $\w = \v-\u$. Then for $i = 1 , \ldots , k$ we have

Then $\w$ is orthogonal to every vector in $U= \text{span} (S)$, which implies $\w\in U^{\perp}.$

Proof: For uniqueness let $\v = \u + \w$ and $\v = \u'+ \w'$, where $\u, \u'\in U$ and $\w, \w'\in U^{\perp}$. Then \[ \u - \u' = \w' - \w \] but $U \cap U^{\perp} = \{\mathbf 0\}.$ Then \[ \u - \u' = \mathbf 0 = \w'- \w. \] That is $\u = \u'$ and $\w=\w'.$ This implies uniqueness. $\; \blacksquare$

The vector $\bfu\in U$ is called the orthogonal projection of $\bfv$ onto $U$ and is given by $$ \mathrm{Proj}_U(\bfv)=\langle\bfv,\bfe_1\rangle\bfe_1+\ldots+\langle\bfv,\bfe_k\rangle\bfe_k. $$ Likewise, the vector $\bfw\in U^\bot$ is called the orthogonal projection of $\bfv$ onto $U^\bot$ and is given by $$ \mathrm{Proj}_{U^\bot}(\bfv)=\bfv-\mathrm{Proj}_U(\bfv). $$

One can show that $$ \dim V=\dim U+\dim U^\bot. $$ This can be very helpful when determining the orthogonal complement of a subspace $U$. Indeed, suppose you have managed to find $\dim V-\dim U$ linearly independent vectors that are all orthogonal to $U$. Then, these vectors will, in fact, form a basis for $U^\bot$. This property will be used in the next example.

Let $\R^3$ be endowed with the usual dot product, and let $$ U= {\rm span}\left(\left\{(0,1,0),\left(-\dfrac{4}{5},0,\dfrac{3}{5}\right)\right\}\right),\quad \bfv=(1,1,1). $$ Find the orthogonal projections of $\bfv$ onto $U$ and $U^\bot$.

First, consider $U = \text{span}(S)$, where $$S= \left\{ \bfe_1 = (0,1,0), \bfe_2= (-4/5, 0, 3/5)\right\}$$ is orthonormal (Check it!📝).

Orthogonal projection onto $U$ means

Orthogonal projection onto $U^{\perp}$ means

It is often convenient to have an orthonormal basis for a given finite-dimensional inner product space. The following algorithm turns a linearly independent set of vectors into an orthonormal set of vectors with the same span as the original set. Applying the algorithm to a basis thus turns the basis into an orthonormal basis. Hence:

It is often convenient to have an orthonormal basis for a given finite-dimensional inner product space. The following algorithm turns a linearly independent set of vectors into an orthonormal set of vectors with the same span as the original set. Applying the algorithm to a basis thus turns the basis into an orthonormal basis. Hence:

Let $\{\bfv_1,\ldots,\bfv_n\}$ be a linearly independent set of vectors in the real inner product space $V$.

The corresponding Gram-Schmidt process is the following algorithm:

The corresponding Gram-Schmidt process is the following algorithm:

Step 1. Let $\w_1 = \v_1$. (Easy! 😃)

Step 2. We can obtain a vector $\w_2$ that is orthogonal to $\w_1$ by computing the component of $\v_2$ that is orthogonal to the space $W_1$ spanned by $\w_1$. To do this we use the formula: \[ \w_2 = \v_2 - \mathrm{Proj}_{W_1} \v_2 = \v_2 - \frac{\langle \v_2, \w_1 \rangle}{\norm{\w_1}^2}\w_1. \]

The corresponding Gram-Schmidt process is the following algorithm:

Step 1. Let $\w_1 = \v_1$. (Easy! 😃)

Step 2. We can obtain a vector $\w_2$ that is orthogonal to $\w_1$ by computing the component of $\v_2$ that is orthogonal to the space $W_1$ spanned by $\w_1$. To do this we use the formula: \[ \w_2 = \v_2 - \mathrm{Proj}_{W_1} \v_2 = \v_2 - \frac{\langle \v_2, \w_1 \rangle}{\norm{\w_1}^2}\w_1. \]

Step 3. To construct a vector $\w_3$ that is orthogonal to both $\w_1$ and $\w_2$, we compute the component of $\v_3$ orthogonal to the space $W_2$ spanned by $\w_1$ and $\w_2$. Here we use the formula: \[ \w_3 = \v_3 - \mathrm{Proj}_{W_2} \v_3 = \v_3 - \frac{\langle \v_3, \w_1 \rangle}{\norm{\w_1}^2}\w_1 - \frac{\langle \v_3, \w_2 \rangle}{\norm{\w_2}^2}\w_2. \]

Step 1. Let $\w_1 = \v_1$. (Easy! 😃)

Step 2. We can obtain a vector $\w_2$ that is orthogonal to $\w_1$ by computing the component of $\v_2$ that is orthogonal to the space $W_1$ spanned by $\w_1$. To do this we use the formula: \[ \w_2 = \v_2 - \mathrm{Proj}_{W_1} \v_2 = \v_2 - \frac{\langle \v_2, \w_1 \rangle}{\norm{\w_1}^2}\w_1. \]

Step 3. To construct a vector $\w_3$ that is orthogonal to both $\w_1$ and $\w_2$, we compute the component of $\v_3$ orthogonal to the space $W_2$ spanned by $\w_1$ and $\w_2$. Here we use the formula: \[ \w_3 = \v_3 - \mathrm{Proj}_{W_2} \v_3 = \v_3 - \frac{\langle \v_3, \w_1 \rangle}{\norm{\w_1}^2}\w_1 - \frac{\langle \v_3, \w_2 \rangle}{\norm{\w_2}^2}\w_2. \]

Step n. Continuing in this way we will produce after $n$ steps an orthogonal set of nonzero vectors $\left\{\w_1, \w_2, \ldots, \w_n \right\}.$

Final step. To convert the orthogonal basis into an orthonormal basis $\left\{\mathbf q_1, \mathbf q_2, \ldots, \mathbf q_n \right\},$ normalize the orthogonal basis vectors.