Math 511: Linear Algebra

5.6 Orthogonal Subspaces

---

5.6.1 Geometry

$$ \require{color} \definecolor{brightblue}{rgb}{.267, .298, .812} \definecolor{darkblue}{rgb}{0.0, 0.0, 1.0} \definecolor{palepink}{rgb}{1, .73, .8} \definecolor{softmagenta}{rgb}{.99,.34,.86} \definecolor{blueviolet}{rgb}{.537,.192,.937} \definecolor{jonquil}{rgb}{.949,.792,.098} \definecolor{shockingpink}{rgb}{1, 0, .741} \definecolor{royalblue}{rgb}{0, .341, .914} \definecolor{alien}{rgb}{.529,.914,.067} \definecolor{crimson}{rgb}{1, .094, .271} \def\ihat{\mathbf{\hat{\unicode{x0131}}}} \def\jhat{\mathbf{\hat{\unicode{x0237}}}} \def\khat{\mathrm{\hat{k}}} \def\tombstone{\unicode{x220E}} \def\contradiction{\unicode{x2A33}} $$

Example 1

Pictured here is the $xy$-plane and the $z$-axis in three dimensional space. The $xy$-plane is the span$\{\ihat,\ \jhat\}$ and the $z$-axis is the span$\{\khat\}$. There are two vector in the illustration

$$ \mathbf{u} = \begin{bmatrix} 3 \\ 4 \\ 0 \end{bmatrix},\ \text{and}\ \mathbf{v} = \begin{bmatrix} 0 \\ 0 \\ 4 \end{bmatrix} $$

These vectors are orthogonal because the angle between them is $\frac{\pi}{2}$, or because their dot product is zero. In fact every vector $\mathbf{u}$ in the $xy$-plane is orthogonal to every vector $\mathbf{v}$ on the $z$-axis.

Definition

Two subspaces $U$ and $W$ of inner product space $V$ are said to be orthogonal subspaces if $\mathbf{x}\cdot\mathbf{y}=0$ for every $\mathbf{x}\in U$ and $\mathbf{y}\in\ W$.

We borrow the symbol for perpendicular $\perp$ from trigonometry and denote two orthogonal subspaces by writing

$$ V\perp W $$

In our example the $xy$-plane is orthogonal to the $z$-axis.

$$ \text{span}\left\{\,\ihat,\ \jhat\,\right\}\perp\text{span}\{\khat\} $$

Example 2

Pictured here is the plane $-x + y + z = 0$ and its normal line $\mathbf{r}(t) = \begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1\ \end{bmatrix}t$

The normal line is the graph of the subspace $V =\ \text{span}\left\{\,\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1\ \end{bmatrix}\,\right\}$.

To determine basis vectors for the plane $-x + y + z = 0$ we need the orthogonal subspace of the normal line. Thinking of linear algebra, this means we need a two dimensional subspace that shares only the zero vector with the normal line.

If we know that the row space $C(A^T)$ of a matrix $A$ is our subspace $V$, then the null space of our matrix would share only the zero vector with it. Let

$$ A = \begin{bmatrix} -1\ & \ \ 1 & \ \ 1\ \end{bmatrix}. $$

Verify that the row space of matrix $A$ is $C(A^T) =\ \text{span}\left\{\,\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1\ \end{bmatrix}\,\right\}$.

This matrix is already in reduced row echelon form! Backward substitution reveals

$$ \begin{align*} x_2 &= \alpha\in\mathbb{R} \\ \\ x_3 &= \beta\in\mathbb{R} \\ \\ -x_1 + \alpha + \beta &= 0 \\ \\ x_1 &= \alpha + \beta \end{align*} $$

Thus vectors in the null space of matrix $A$, $N(A)$, have the form

$$ \begin{bmatrix} \alpha + \beta \\ \alpha \\ \beta \end{bmatrix} = \begin{bmatrix} \alpha \\ \alpha \\ 0 \end{bmatrix} + \begin{bmatrix} \beta \\ 0 \\ \beta \end{bmatrix} = \alpha\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} + \beta\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} $$

The null space of matrix $A$ is given by

$$ \text{null}(A) =\ \text{span}\left\{\,\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix},\ \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}\,\right\} $$

The dot product of any vector on the normal line $\ t\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1 \end{bmatrix}$ and any vector on the plane $\alpha\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} + \beta\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}$ yields

$$ \begin{align*} \left(t\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1 \end{bmatrix}\right)\cdot\left(\alpha\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} + \beta\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}\right) &= \left(t\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1 \end{bmatrix}\right)\cdot\left(\alpha\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}\right) + \left(t\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1 \end{bmatrix}\right)\cdot\left(\beta\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}\right) \\ \\ &= t\alpha\left(\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1 \end{bmatrix}\right)\cdot\left(\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}\right) + t\beta\left(\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1 \end{bmatrix}\right)\cdot\left(\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}\right) \\ \\ &= t\alpha\left( (-1)(1) + (1)(1) + (1)(0) \right) + t\beta\left( (-1)(1) + (1)(0) + (1)(1) \right) \\ \\ &= t\alpha(0) + t\beta(0) = 0 \end{align*} $$

Hence a plane in three dimensional space passing through the origin and its normal line form orthogonal subspaces.

Example 3

The $y$-axis is the span$\{\mathbf{e}_2\}$ and the $z$-axis is the span$\{\mathbf{e}_3\}$. If $\mathbf{u}\in\text{span}\{\mathbf{e}_2\}$ and $\mathbf{v}\in\text{span}\{\mathbf{e}_3\}$, then the dot product

$$ \mathbf{u}\cdot\mathbf{v} = (\alpha\,\mathbf{u})\cdot(\beta\,\mathbf{v}) = \alpha\beta\,(\mathbf{e}_2\cdot\mathbf{e}_3) = \alpha\beta\,0 = 0 $$

The $y$-axis and the $z$-axis form orthogonal subspaces.

---

5.6.2 The Four Fundamental Subspaces

Theorem 5.2.1

The row space of a matrix $C(A^T)$ and the null space $N(A)$ are orthogonal subspaces.

Proof:

For any $m\times n$ matrix $A$, the row space $C(A^T)$ and the null space $N(A)$ are subspaces of $\mathbb{R}^n$. To determine if they are orthogonal subspaces we need to show that the inner product of any vector $\mathbf{x}\in C(A^T)$ and any vector $\mathbf{y}\in N(A^T)$ is zero. That is we need to show these vectors are orthogonal. If $\mathbf{x}\in C(A^T)$ then $\mathbf{x} = A^T\mathbf{u}$ for some vector in $\mathbb{R}^m$; and if $\mathbf{y}\in N(A)$ then $A\mathbf{y}=\mathbf{0}$.

$$ \mathbf{x}\cdot\mathbf{y} = (A^T\mathbf{u})\cdot\mathbf{y} = (A^T\mathbf{u})^T\mathbf{y} = (\mathbf{u}^TA)\mathbf{y} = \mathbf{u}^T(A\mathbf{y}) = \mathbf{u}^T\mathbf{0} = 0 $$

Since the dot product of every vector in the row space of matrix $A$, is orthogonal to every vector in the null space of matrix $A$, we have that they are orthogonal subspaces. $\tombstone$

Corollary 5.6.2

$$ C(A) \perp N(A^T) $$

Proof:

Using the same reasoning if $\mathbf{x}\in C(A)$ and $\mathbf{y}\in N(A^T)$, then $\mathbf{x} = A\mathbf{u}$ for some vector $\mathbf{u}\in\mathbb{R}^n$ and $A^T\mathbf{y}=\mathbf{0}$. Hence the dot product

$$ \mathbf{x}\cdot\mathbf{y} = (A\mathbf{u})\cdot\mathbf{y} = (A\mathbf{u})^T\mathbf{y} = (\mathbf{u}^TA^T)\mathbf{y} = \mathbf{u}^T(A^T\mathbf{y}) = \mathbf{u}^T\mathbf{0} = 0 $$

We have shown that the column space and the null space of the transpose of a matrix are orthogonal subspaces.

$$ C(A)\perp N(A^T) $$ $\tombstone$

---

5.6.3 Orthogonal Complements

Notice that in our four examples

• the $xy$-plane and the $z$-axis
• a plane passing through the origin and its normal line
• the $y$-axis and the $z$-axis

In the first two examples the span of all of the vectors in both subspaces is the entire inner product space $\mathbb{R}^3$. However the span of all of the vectors in both subspaces of the third example is a two dimensional subspace of $\mathbb{R}^3$, the $yz$-plane.

Definition

Let $S$ be a subspace of inner product space $V$. The set of all vectors in $V$ that are orthogonal to every vector in our subspace $S$ is given by

$$ S^{\perp} := \left\{\,x\in V\,:\,\langle\,\mathbf{x},\mathbf{y}\,\rangle = 0\ \text{for every vector}\ \mathbf{y}\in S\,\right\} $$

$S^{\perp}$ is called the orthogonal complement of subspace $S$ in $V$.

• the $xy$-plane and the $z$-axis are orthogonal complements
• a plane passing through the origin and its normal line are orthogonal complements
• the $y$-axis and the $z$-axis are orthogonal subspaces, but not orthogonal complements

Theorem 5.6.3

The intersection of a subspace $S$ of an inner product space $V$ and its orthogonal complement is the trivial subspace of $V$.

$$ S\cap S^{\perp} = \left\{ \mathbf{0} \right\} $$

Proof:

If vector $\mathbf{x}\in S\cap S^{\perp}$, then $\mathbf{x}\in S$ and $\mathbf{x}\in S^{\perp}$. Thus $\langle \mathbf{x},\mathbf{x} \rangle = 0$. This implies that $\mathbf{x}=\mathbf{0}$. \tombstone

This theorem has an immediate consequence for vector sums of a subspace and its orthogonal complement.

Corollary 5.6.4

The vector sum of a subspace $S$ of an inner product space $V$ and its orthogonal complement is a direct sum.

Is the direct sum of a subspace $S$ of an inner product space $V$ and its orthogonal complement the entire inner product space $V$? While this is true for finite dimensional inner product spaces, it is not generally true for infinite dimensional inner product spaces. Fortunately, most of our computer applications require us to use finite dimensional inner product spaces.

Theorem 5.6.5

The direct sum of a subspace $S$ of a finite dimensional inner product space $V$ and its orthogonal complement is $V$.

Proof:

Let $S$ be a subspace of a finite dimensional inner product space $V$ with dimension $n > 0$.

Case 1: $S = \left\{\mathbf{0}\right\}$

Since for every vector $\mathbf{x}\in V$, $\langle \mathbf{x},\mathbf{0} \rangle = 0$, $\left\{\mathbf{0}\right\}^{\perp} = S^{\perp} = V$. Clearly,

$$ S\oplus S^{\perp} = \left\{\mathbf{0}\right\}^{\perp} \oplus V = V $$

Case 2: $S = V$

Since the only vector orthogonal to every vector is the zero vector $V^{\perp} = \left\{\mathbf{0}\right\}$.

Case 3: $\dim(S) = k$ and $0 < k < n$.

Let $\left\{\mathbf{u}_1,\dots,\mathbf{u}_k\right\}$ be a basis for $S$. One may extend this basis to a basis $\left\{\mathbf{u}_1,\dots,\mathbf{u}_k,\mathbf{v}_{k+1},\dots,\mathbf{v}_n\right\}$ for $V$. Using the Gram-Schmidt orthogonalization process, one creates an orthonormal basis $\left\{\mathbf{q}_1,\dots,\mathbf{q}_k,\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\}$ so that

$$ \text{Span}\left\{\mathbf{q}_1,\dots,\mathbf{q}_k\right\} = \text{Span}\left\{\mathbf{u}_1,\dots,\mathbf{u}_k\right\} = S $$

Clearly every basis vector $\mathbf{v}\in\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\}$ is orthogonal to every vector in $S$ since $\mathbf{v}$ is orthogonal to every vector in the orthonormal basis of $S$. Because inner product is multi-linear, every vector in $\text{Span}\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\}$ is also orthogonal to every vector in $S$. Thus

$$ \text{Span}\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\}\subset S^{\perp} $$

Likewise every element of the orthogonal complement of $S$ is orthogonal to every orthonormal basis vector $\left\{\mathbf{q}_1,\dots,\mathbf{q}_k\right\}$ of $S$. Thus a nonzero vector $\mathbf{v}\in S^{\perp}$ must be a linear combination of the remaining orthonormal basis vectors of $V$.

$$ \mathbf{v} = \displaystyle\sum_{i=k+1}^n v_i\mathbf{q}_i \in \text{Span}\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\} $$

Hence we have also that $S^{\perp}$ is a subset of $\text{Span}\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\}$. Consequently

$$ S^{\perp}=\text{Span}\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\} $$

This establishes that

$$ S\oplus S^{\perp} = \text{Span}\left\{\mathbf{q}_1,\dots,\mathbf{q}_k\right\}\oplus\text{Span}\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\} = V $$ $\tombstone$

Recalling that for $m\times n$ matrices, $\left(A^T\right)^T = A$, and for $n\times n$ nonsingular matrices, $\left(A^{-1}\right)^{-1} = A$, will it turn out that $\left(S^{\perp}\right)^{\perp} = S$? As in Theorem 5.6.5, this is only true for finite dimensional inner product spaces.

Corollary 5.6.6

If $S$ is a subspace of a finite dimensional inner product space $V$, then $\left(S^{\perp}\right)^{\perp} = S$.

Proof:

Using the construction in the proof of theorem 5.6.5, one obtains that

$$ \begin{align*} \left(S^{\perp}\right)^{\perp} &= \left( \text{Span}\left\{\mathbf{q}_1,\dots,\mathbf{q}_k\right\}^{\perp} \right)^{\perp} \\ \\ &= \left( \text{Span}\left\{\mathbf{q}_{k+1},\dots,\mathbf{q}_n\right\} \right)^{\perp} \\ \\ &= \text{Span}\left\{\mathbf{q}_1,\dots,\mathbf{q}_k\right\} = S \end{align*} $$ $\tombstone$

Theorem 5.6.7

The Fundamental Subspaces of a Matrix

If $A$ is an $m\times n$ matrix, then the null space and the row space of $A$ are orthogonal complements. Similarly the column space and the null space of the transpose of $A$ are orthogonal complements.

$$ N(A) = C(A^T)^{\perp}\ \text{and}\ N(A^T) = C(A)^{\perp} $$

Furthermore,

$$ C(A^T)\oplus N(A) = \mathbb{R}^n\ \text{and}\ C(A)\oplus N(A^T) = \mathbb{R}^m $$

Proof:

We have already shown that $N(A)\perp C(A^T)$, that is the null space and the row space of a matrix are orthogonal subspaces. Thus every vector in $N(A)$ is also in the orthogonal complement $C(A^T)^{\perp}$.

$$ N(A)\subset C(A^T)^{\perp} $$

If vector $\mathbf{x}\in C(A^T)^{\perp}$, then using our definition we have $\mathbf{x}\cdot\mathbf{y} = 0$ for every vector $\mathbf{y}\in C(A^T)$. That means that if $\mathbf{u}\in\mathbb{R}^m$ is any vector in $\mathbb{R}^m$ and $\mathbf{y}=A^T\mathbf{u}$, then

$$ 0 = \mathbf{y}\cdot\mathbf{x} = (A^T\mathbf{u})^T\mathbf{x} = (\mathbf{u}^TA)\mathbf{x} = \mathbf{u}^T(A\mathbf{x}) = \mathbf{u}\cdot A\mathbf{x} $$

This linear algebra says that the vector $A\mathbf{x}$ is orthogonal to every vector in $\mathbf{R}^m$. That can only be the zero vector

$$ A\mathbf{x} = \mathbf{0} $$

This linear algebra shows that every vector in the orthogonal complement of $C(A^T)$ is also in the null space $N(A)$.

$$ C(A^T)^{\perp}\subset N(A) $$

Together, $N(A)\subset C(A^T)^{\perp}$ and $C(A^T)^{\perp}\subset N(A)$ tell us that

$$ C(A^T)^{\perp} = N(A) $$

Using the same argument with $C(A)$ and the $N(A^T)$ yields

$$ C(A)^{\perp} = N(A^T) $$ $\tombstone$

That is a LOT of theorems that we can use to understand the relationship between the fundamental subspaces of a matrix and orthogonal complements of a subspace of an inner product space.

---

5.6.4 Exercises

Exercise 1

Let $S$ be the subspace of $\mathbf{R}^4$ spanned by $\mathbf{x}_1 = \begin{bmatrix}\ \ 1 \\ \ \ 2 \\ -2 \\ -1 \end{bmatrix}$ and $\mathbf{x}_2 = \begin{bmatrix} -2 \\ -2 \\ \ \ 3 \\ \ \ 2 \\ \end{bmatrix}$. Find a basis for $S^{\perp}$.

Check Your Work

We know that the row space $C\left(A^T\right)$ and the null space $N(A)$ of a matrix are orthogonal complements. So if we could find a matrix $A$ whose row space is
$$ C\left(A^T\right) = \text{Span}\left\{\mathbf{x}_1,\ \mathbf{x}_2\right\} $$
then we could reduce the matrix and use backward substitution to find the null space $N(A)$, and this null space would be the orthogonal complement of the row space $C\left(A^T\right)$. That is $N(A) = S^{\perp}$.

$$ \begin{align*} A &= \begin{bmatrix}\ \ 1 &\ \ 2 & -2 & -1 \\ -2 & -2 &\ \ 3 &\ \ 2 \end{bmatrix}\begin{array}{c} \\ R_2 + 2R_1 \\ \end{array} \\ \\ &\rightarrow \begin{bmatrix}\ \ 1 &\ \ 2 & -2 & -1 \\ \ \ 0 &\ \ 2 & -1 &\ \ 0 \end{bmatrix}\begin{array}{c} R_1 - R_2 \\ \\ \end{array} \\ \\ &\rightarrow \begin{bmatrix}\ \ 1 &\ \ 0 & -1 & -1 \\ \ \ 0 &\ \ 2 & -1 &\ \ 0 \end{bmatrix} \\ \\ x_3 &= \alpha\in\mathbb{R} \\ x_4 &= \beta\in\mathbb{R} \\ \\ 2x_2 - \alpha &= 0 \\ x_2 &= \frac{\alpha}{2} \\ \\ x_1 - \alpha - \beta &= 0 \\ x_1 &= \alpha + \beta \\ \\ N(A) &= \left\{\,\begin{bmatrix} \alpha + \beta \\ \frac{\alpha}{2} \\ \alpha \\ \beta \end{bmatrix}\,:\,\alpha,\beta\in\mathbb{R}\,\right\} \\ \\ &= \left\{\,\alpha\begin{bmatrix} 1 \\ \frac{1}{2} \\ 1 \\ 0 \end{bmatrix} + \beta\begin{bmatrix} 1 \\ 0 \\ 0 \\ 1 \end{bmatrix}\,:\,\alpha,\beta\in\mathbb{R}\,\right\} \\ \\ S^{\perp} &= \text{Span}\left\{\,\begin{bmatrix} 1 \\ \frac{1}{2} \\ 1 \\ 0 \end{bmatrix},\ \begin{bmatrix} 1 \\ 0 \\ 0 \\ 1 \end{bmatrix}\,\right\} \end{align*} $$

Exercise 2

Describe the subspaces $S$ and $S^{\perp}$ geometrically in $\mathbb{R}^4$.

Check Your Work

We see from the upper triangular form of matrix $A$ in Example 1 that the vectors $\mathbf{x}_1$ and $\mathbf{x}_2$ are linearly independent. Thus the subspace $S$ is a plane since it is the span of two linearly independent vectors. The orthogonal complement must have dimension $4 - 2 = 2$ so $S^{\perp}$ is also a plane in $\mathbb{R}^4$ that is orthogonal to the plane $S$. The two orthogonal planes intersect only at the origin $(0,0,0,0)$.

---

Your use of this self-initiated mediated course material is subject to our Creative Commons License 4.0

---