- 5.2.1 Geometry
- 5.2.2 Weighted Inner Product
- 5.2.3 Statistics and Sampling
- 5.2.4 Integral Inner Products
- 5.2.5 Norm of a Vector in an Inner Product Space
- 5.2.6 Angle Between Two Vectors
- 5.2.7 Orthogonal Projection
- 5.2.8 The Cauchy-Schwarz-Bunyakovsky Theorem
- 5.2.9 Norms
- 5.2.10 p-Norms
- 5.2.11 Exercises
- copyleft
Inner Products and Orthogonality
( This video is long. You should take a break or two as you watch this video.)
We have discussed the geometry of vector space $\mathbb{R}^n$ for any positive integer $n$ using the Euclidean scalar product, or dot product. While a great many mathematical models using linear algebra rely on these Euclidean Spaces, we can extend our using of scalar products to include non-Euclidean geometries on the vector space $\mathbb{R}^n$. We can also extend our use of scalar product to describe geometries on more abstract vector spaces.
A scalar product defined on a vector space is called an inner product. The Euclidean inner product on vector space $\mathbb{R}^n$ is called a dot product. An inner product on a vector space endows the vector space with the notions of magnitude or norm and direction. This means that adding an inner product to a vector space gives it a geometry. Together, the vector space and its inner product is called an inner product space.
Exercise 1¶
Consider the function $\langle\cdot,\cdot\rangle\,:\,\mathbb{R}^2\times\mathbb{R}^2\rightarrow\mathbb{R}$ defined for every pair of vectors $\mathbf{x}$, $\mathbf{y}\in\mathbb{R}^2$ by
$$ \langle\mathbf{x},\mathbf{y}\rangle\,:=\,x_1y_1 - x_2y_2 $$
Show that this function is not an inner product on $\mathbb{R}^2$.
Follow Along
This function is symmetric and bilinear, but it is not positive definite. Choose the nonzero vector
$$ \mathbf{x} = \begin{bmatrix} 1\ \\ 1\ \end{bmatrix} \ne \mathbf{0} $$
Then
$$ \langle\mathbf{x},\mathbf{x}\rangle = (1)(1) - (1)(1) = 0 $$
An inner product requires $\langle\mathbf{x},\mathbf{x}\rangle = 0$ only when $\mathbf{x} = \mathbf{0}$. Since this vector is nonzero, the first axiom fails and the function is not an inner product.
Exercise 2¶
Let $\mathbf{x} = \begin{bmatrix}\ 1\ \\ \ 1\ \end{bmatrix}$ and $\mathbf{y} = \begin{bmatrix}\ \ 1\ \\ -1\ \end{bmatrix}$ in $\mathbb{R}^2$. Compute the Euclidean scalar product $\mathbf{x}\cdot\mathbf{y}$, and compute $\langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}} = x_1y_1 + 3x_2y_2$. Are these two vectors orthogonal in both geometries?
Follow Along
Using the Euclidean scalar product,
$$ \mathbf{x}\cdot\mathbf{y} = (1)(1) + (1)(-1) = 0 $$
so the vectors are orthogonal in the Euclidean geometry. Using the weighted scalar product,
$$ \langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}} = (1)(1) + 3(1)(-1) = 1 - 3 = -2 \ne 0 $$
so the vectors are not orthogonal in the weighted geometry. Orthogonality is a property of the inner product, not of the vectors alone.
Example 1 - Weighted Inner Product¶
One can define a non-Euclidean inner product on the vector space $\mathbb{R}^n$. Given a vector $\mathbf{w}\in\mathbb{R}^n$ with positive entries one defines a weighted inner product on $\mathbb{R}^n$ by
$$ \langle \mathbf{x},\mathbf{y} \rangle_{\mathbf{w}}\,= \displaystyle\sum_{i=1}^n x_iy_iw_i = x_1y_1w_1 + x_2y_2w_2 + \cdots + x_ny_nw_n $$
This will work just as well as the Euclidean inner product on $\mathbb{R}^n$. Then for all vectors $x$, $y$, $z\in\mathbb{R}^n$ and scalars $s$, $t$,
The inner product is positive definite. $$ \langle \mathbf{x},\mathbf{x}\rangle_{\mathbf{w}}\,= \sum_{i=1}^n x_ix_iw_i = \sum_{i=1}^n x_i^2w_i \ge 0$$ since $x_i^2\ge 0$ and $w_i\gt 0$ for all $1\le i\le n$. Furthermore, if $\langle \mathbf{x},\mathbf{x}\rangle_{\mathbf{w}}\,=\,0$, then $$ \begin{align*} \displaystyle\sum_{i=1}^n x_i^2w_i &= 0 \\ x_i^2w_i &= 0 \qquad\qquad 1\le i\le n \\ x_i^2 &= 0 \qquad\qquad 1\le i\le n \\ x_i &= 0 \qquad\qquad 1\le i\le n \end{align*} $$ Thus $\mathbf{x}$ is the zero vector $\mathbf{0}$.
The inner product is symmetric. $$ \langle \mathbf{x},\mathbf{y}\rangle_{\mathbf{w}}\,= \displaystyle\sum_{i=1}^n x_iy_iw_i = \displaystyle\sum_{i=1}^n y_ix_iw_i =\,\langle \mathbf{y},\mathbf{x}\rangle_{\mathbf{w}} $$
The inner product is multi-linear. $$ \begin{align*} \langle s\mathbf{x} + t\mathbf{y}, \mathbf{z}\rangle_{\mathbf{w}} &= \displaystyle\sum_{i=1}^n (s x_i + t y_i)z_iw_i \\ &= s\displaystyle\sum_{i=1}^n x_iz_iw_i + t\displaystyle\sum_{i=1}^n y_iz_iw_i \\ &= s\langle\mathbf{x}, \mathbf{z}\rangle_{\mathbf{w}} \,+\, t\langle\mathbf{y}, \mathbf{z}\rangle_{\mathbf{w}} \end{align*} $$
The notational convention $\langle\cdot,\,\cdot\rangle_{\mathbf{w}}$ uses the subscript to distinguish the inner product using weight vector $\mathbf{w}$ from other inner products in our calculations.
Definition¶
Inner Product
An inner product on a vector space $V$ is a function $\langle\cdot,\cdot\rangle:V\times V\rightarrow\mathbb{R}$ that satisfies the following three axioms. For any vectors $x$, $y$, $z\in V$, and scalars $s$ and $t$,
Positive Definite$$\langle \mathbf{x}, \mathbf{x}\rangle\,\ge\,0 \text{ with equality if and only if }\mathbf{x}=\mathbf{0}$$
Conjugate Symmetric$$\langle \mathbf{x}, \mathbf{y}\rangle\,=\,\langle \mathbf{y}, \mathbf{x}\rangle^*$$
Bilinear (multilinear for two inputs) $$\langle s\mathbf{x} + t\mathbf{y}, \mathbf{z} \rangle\,=\,s\langle\mathbf{x}, \mathbf{z}\rangle + t\langle\mathbf{y}, \mathbf{z}\rangle$$
Notice that property 2 is called conjugate symmetric. This will only effect us when we are using complex vector spaces. Since the complex conjugate of a real number is itself, real inner products on real vectors spaces are simply symmetric. If $x$ and $y$ are real numbers, and $z = x + iy$, then $$z^* = (x + iy)^* = x - iy$$ However for real numbers $$x^* = x$$ Since conjugation has no effect on real inner products on real vector spaces, you may see me fail to mention conjugation in these inner product spaces.
Notice that using properties 2. and 3. together give us $$\langle \mathbf{x}, s\mathbf{y} + t\mathbf{z}\rangle = \langle s\mathbf{y} + t\mathbf{z}, \mathbf{x}\rangle^* = s\langle\mathbf{y}, \mathbf{x}\rangle^* + t\langle\mathbf{z}, \mathbf{x}\rangle^* = s\langle\mathbf{x}, \mathbf{y}\rangle + t\langle\mathbf{x}, \mathbf{z}\rangle$$
Example 2 - Low Pass Filter¶
In the A440 Pitch Standard, middle C is 261.626 hz. The octaves of the $C$ pitch is given by
| Scientific designation | Frequency (hz) | Name | | --- | --- | -- | | $C_{-1}$ | 8.176 | | $C_0$ | 16.352 | | $C_1$ | 32.703 | | $C_2$ | 65.406 | | $C_3$ | 130.813 | | $C_4$ | 261.626 | Middle C | | $C_5$ | 523.251 | | $C_6$ | 1046.502 | | $C_7$ | 2093.005 | | $C_8$ | 4186.009 | | $C_9$ | 8372.018 | | $C_{10}$ | 16744.036 |
A frequency vector $\mathbf{v}\in\mathbb{R}^{12}$ is a list of 12 magnitudes or intensities for each of the frequencies listed in the table above. A low pass filter will scale the high and middle frequency elements of the vector by a small number. If one chooses
$$ \mathbf{w} = [\, 1,\ 1,\ 1,\ 1,\ .1,\ .1,\ .1,\ .1,\ .01,\ .01,\ .01,\ .01 \,]^\mathsf{T} $$
the resulting weighted inner product will produce a geometry that leaves low frequency elements of the vector unchanged but diminish the middle and high frequency elements of the vector. The low frequency elements will have significantly more influence on the magnitude of a frequency vector.
The magnitude of the frequency vector
$$\mathbf{v} = [\, 1,\ 0,\ 0,\ 0,\ 0,\ 0,\ 0,\ 0,\ 1,\ 1,\ 1,\ 1 \,]^\mathsf{T}$$
will be
$$\lVert\mathbf{v}\rVert_{\mathbf{w}} = \sqrt{ 1^2 + 1^2(.01) + 1^2(.01) + 1^2(.01) + 1^2(.01)} = \sqrt{1.04} \approx 1.0198$$
Definition¶
Inner Product Space
A vector space $V$ endowed with an inner product $\langle\cdot,\cdot\rangle\,:\,V\times V\rightarrow\mathbb{R}$ is called the Inner Product Space $\left\{\,V,\ \langle\cdot,\cdot\rangle\,\right\}$.
Example 3 - Frobenius Inner Product Space¶
In vector space $\mathbb{R}^{m\times n}$ define $\langle\cdot,\cdot\rangle_F\,:\,\mathbb{R}^{m\times n}\times\mathbb{R}^{m\times n} \rightarrow \mathbb{R}$ by
$$\langle A,B\rangle_F\,:=\,\displaystyle\sum_{i=1}^m\,\displaystyle\sum_{j=1}^n a_{ij}b_{ij}$$
This is the Frobenius inner product on the vector space $\mathbb{R}^{m\times n}$.
Exercise 3¶
Show that the Frobenius function $\langle\cdot,\cdot\rangle_F\,:\,\mathbb{R}^{m\times n}\times\mathbb{R}^{m\times n} \rightarrow \mathbb{R}$ defined by
$$\langle A,B\rangle_F\,:=\,\displaystyle\sum_{i=1}^m\,\displaystyle\sum_{j=1}^n a_{ij}b_{ij}$$
satisfies the axioms of an inner product.
Follow Along
For any matrices $A$, $B$ and $C\in\mathbb{R}^{m\times n}$, and scalars $s$ and $t$,
The Frobenius inner product is positive definite because $$ \begin{align*} \langle A, A\rangle\,&=\,\sum_{i=1}^m\displaystyle\sum_{j=1}^n a_{ij}a_{ij} = \sum_{i=1}^m \sum_{j=1}^n a_{ij}^2 \ge 0. \end{align*} $$ Moreover $\langle A, A\rangle = 0$ if and only if $a_{ij}^2 = 0$ for all $1\le i\le m$ and $1\le j\le n$. Since it is a sum of nonnegative numbers, $\langle A, A\rangle = 0$ if and only if $a_{ij}=0$ for all $1\le i\le m$ and $1\le j\le n$. That is if and only if $A$ is the $m\times n$ zero matrix $O$.
The Frobenius inner product is conjugate symmetric. $$ \begin{align*} \langle A, B\rangle &= \sum_{i=1}^m\displaystyle\sum_{j=1}^n a_{ij}b_{ij} = \sum_{i=1}^m \sum_{j=1}^n b_{ij}a_{ij} = \langle B, A\rangle \end{align*} $$
The Frobenius inner product is multi-linear. Since our vector space is a real vector space, $$ \begin{align*} \langle s A + t B, C\rangle\, &= \langle s [a_{ij}] + t [b_{ij}], [c_{ij}]\rangle \\ &= \langle \left[ s a_{ij} + tb_{ij}\right], [c_{ij}]\rangle \\ &= \sum_{i=1}^m \sum_{j=1}^n (s a_{ij} + t b_{ij})c_{ij} \\ &= s \sum_{i=1}^m \sum_{j=1}^n a_{ij}c_{ij} + t \sum_{i=1}^m \sum_{j=1}^n b_{ij}c_{ij} \\ &= s\langle A, C\rangle + t\langle B, C\rangle \end{align*} $$
Exercise 4¶
On the vector space $\mathbb{R}^2$ use the weight vector $\mathbf{w} = \begin{bmatrix}\ 2\ \\ \ 3\ \end{bmatrix}$ to define the weighted inner product $\langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}} = 2x_1y_1 + 3x_2y_2$. For the vectors $\mathbf{x} = \begin{bmatrix}\ 1\ \\ \ 2\ \end{bmatrix}$ and $\mathbf{y} = \begin{bmatrix}\ \ 4\ \\ -1\ \end{bmatrix}$, compute $\langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}}$ and $\lVert\mathbf{x}\rVert_{\mathbf{w}}$.
Follow Along
The weighted inner product is
$$ \langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}} = 2(1)(4) + 3(2)(-1) = 8 - 6 = 2 $$
The weighted norm follows from the inner product of the vector with itself,
$$ \lVert\mathbf{x}\rVert_{\mathbf{w}} = \sqrt{\langle\mathbf{x},\mathbf{x}\rangle_{\mathbf{w}}} = \sqrt{2(1)^2 + 3(2)^2} = \sqrt{2 + 12} = \sqrt{14} $$
This is a file from the Wikimedia Commons. Information about this image can be obtained from its
Description Page.
The mathematical models of complex systems such as weather, financial markets, disease transmission, and quality assurance are generally determined by measuring the state of the system at various systematically appointed locations in the population. Weather measurements are taken at many geographical locations. Financial data is collected for key products, businesses or markets. The key property these types of applications share is that they are very large and/or embody complicated interactions of the elements of the population. One must obtain a sample of the state of a complex system at a finite number of locations, and use the values of this sample to estimate parameters using an inner product within a mathematical model.
Example 4 - Discrete Inner Product¶
On the vector space $P_n$, let $S = \{\,x_0,\ x_1,\ \dots,\ x_n \,\} \subset [a,b]$ be $n+1$ distinct (different) real numbers in the interval. For any two polynomials $p$, $q\in P_n$, one may define an inner product by
$$\langle p,q\rangle_S\,:=\,\displaystyle\sum_{i=0}^n p(x_i)q(x_i) = \sum_{x\in S} p(x)q(x)$$
Example 5 - Discrete and Weighted Inner Product¶
We may again choose a positive function $w(t)$ defined on $[a,b]$ and define the weighted inner product on $P_n$ for any two polynomials $p$, $q\in P_n$ by
$$ \langle p,q\rangle_{\mathbf{w}}\,:=\,\sum_{i=0}^n p(x_i)q(x_i)w(x_i) = \sum_{x\in S} p(x)q(x)w(x) $$
We can choose these points $\{x_0,\ x_1,\ \dots,\ x_n\}\subset [a,b]$ to be evenly distributed in the interval. These values can be considered a sample of the values in the interval. How these values are chosen comprises an academic field of its own.
Example 6 - Partition¶
We may use a constant weighting factor for a partition, $\pi = \left\{\, x_0,\ x_1,\ \dots,\ x_n \,\right\}$ of $n+1$ points in an interval $[a,b]$, $a = x_0 < x_1 < \cdots < x_n=b$. The inner product on the vector space of all real polynomials $P_n$ is defined by
$$ \langle p,q\rangle_\pi := \dfrac{b-a}{n}\sum_{i=0}^n p(x_i)q(x_i) = \sum_{i=0}^n p(x_i)q(x_i)\Delta x, $$
where $\Delta x = \dfrac{b-a}{n}$ and for any polynomials $p$, $q\in P_n$.
Exercise 5¶
On the vector space $P_2$ use the set of distinct points $S = \left\{-1,\ 0,\ 1\right\}$ to define the discrete inner product $\langle p,q\rangle_S = \displaystyle\sum_{x\in S} p(x)q(x)$. Show that the polynomials $p(x) = x$ and $q(x) = x^2$ are orthogonal, and compute $\lVert p\rVert_S$.
Follow Along
Evaluating each polynomial at the three sample points,
$$ \langle p,q\rangle_S = (-1)(1) + (0)(0) + (1)(1) = -1 + 0 + 1 = 0 $$
The inner product is zero, so the two polynomials are orthogonal with respect to this discrete inner product. For the norm,
$$ \lVert p\rVert_S = \sqrt{\langle p,p\rangle_S} = \sqrt{(-1)^2 + 0^2 + 1^2} = \sqrt{2} $$
Exercise 6¶
Using the same points $S = \left\{-1,\ 0,\ 1\right\}$ and the weight function values $w(-1) = 1$, $w(0) = 2$, and $w(1) = 1$, define the weighted discrete inner product $\langle p,q\rangle_{\mathbf{w}} = \displaystyle\sum_{x\in S} p(x)q(x)w(x)$. Compute $\langle p,q\rangle_{\mathbf{w}}$ and $\lVert p\rVert_{\mathbf{w}}$ for $p(x) = x$ and $q(x) = x^2$.
Follow Along
Each term is now scaled by the weight at that point,
$$ \langle p,q\rangle_{\mathbf{w}} = (-1)(1)(1) + (0)(0)(2) + (1)(1)(1) = -1 + 0 + 1 = 0 $$
so the two polynomials remain orthogonal. For the weighted norm,
$$ \lVert p\rVert_{\mathbf{w}} = \sqrt{(-1)^2(1) + 0^2(2) + (1)^2(1)} = \sqrt{2} $$
The weight at $x = 0$ has no effect here because $p(0) = 0$.
If one increases the number of sampling points in a partition without bound in the interval $[a,b]$, $a=x_0 \lt x_1 \lt \dots \lt x_n=b$, then $\Delta x = \frac{b-a}{n}$ and
$$ \langle p,q\rangle = \lim_{n\rightarrow\infty} \dfrac{b-a}{n} \sum_{i=0}^n p(x_i)q(x_i) = \lim_{n\rightarrow\infty} \sum_{i=0}^n p(x_i)q(x_i)\dfrac{b-a}{n} = \lim_{n\rightarrow\infty} \displaystyle\sum_{i=0}^n p(x_i)q(x_i)\Delta x $$
The last expression is a Riemann Sum so
$$ \langle p,q\rangle = \displaystyle\lim_{n\rightarrow\infty} \displaystyle\sum_{i=0}^n p(x_i)q(x_i)\Delta x = \displaystyle\int_a^b p(x)q(x)\,dx $$
Example 7 - Integral Inner Product¶
On the vector space of continuous functions $C[a,b]$ one may define an inner product for any functions $f$, $g\in C[a,b]$ by
$$ \langle f,g\rangle\,:=\,\displaystyle\int_a^b f(t)g(t)\,dt $$
Example 8 - Weighted Integral Inner Product¶
In the vector space $C[a,b]$, we may choose a positive continuous function $w\in C[a,b]$. This function is called a weight function and one may define a weighted inner product for any continuous functions $f$, $g\in C[a,b]$ by
$$ \langle f,g\rangle_{\mathbf{w}}\,:=\,\displaystyle\int_a^b f(t)g(t)w(t)\,dt $$
Example 9 - Riemann-Stieltjes Inner Product¶
One may also choose a function defined on the interval $[a,b]$ so that its derivative is positive and continuous on the interval $[a,b]$. In this case we have $w\in C^1[a,b]$ and one may define a weighted inner product for any continuous functions $f$, $g\in C[a,b]$ by
$$ \langle f,g\rangle_{\mathbf{w}}\,:=\,\displaystyle\int_a^b f(t)g(t)w'(t)dt = \displaystyle\int_a^b f(t)g(t) dw $$
This is called the Riemann-Stieltjes integral and we used the differential $dw = w'(t)dt$ to define substitution in integral calculus.
Example 10 - Cumulative Distribution Function¶
If we choose a positive increasing function $w\in C^1[a,b]$ so that $w(a) = 0$ and $w(b) = 1$, then we call this function a cumulative probability distribution function or distribution function. Using this distribution function we may define the probability density function $p(t) = w'(t)$. Now we may define the inner product on $C[a,b]$ for any two functions $f$, $g\in C[a,b]$ by
$$ \langle f,g\rangle_{\mathbf{w}}\,:=\,\int_a^b f(t)g(t)p(t)\,dt = \int_a^b f(t)g(t)w'(t)\,dt = \int_a^b f(t)g(t)\,dw $$
Example 7 (continued)¶
Suppose $f\in C[a,b]$ is a continuous function and $f(x_0)\neq 0$ for some value $a\le x_0\le b$. Then as $f$ is continuous at every point in the interval $[a,b]$, $f$ has no jump discontinuities or vertical asymptotes in this interval. This means that the graph of $f$ must rise above zero continuously to $f(x_0)\gt 0$ or descend below zero continuously to the $f(x_0)\lt 0$. As the product of continuous functions is continuous, we have that in either case, the graph of the function $f^2$ rises continuously above zero to $f(x_0)^2\gt 0$. As continuous function $f^2(x)$ is nonzero at $x_0$, there must be a subinterval containing $x_0$, $I=[c,d]\subseteq[a,b]$ for which the continuous function $f^2(x)\gt \frac{f(x_0)^2}{2}$ for every point in $I$, and $a\lt c\lt d\lt b$. Thus
$$ \langle f, f\rangle = \displaystyle\int_a^b f^2(t)\,dt \ge \displaystyle\int_c^d f^2(t)\,dt \ge \displaystyle\int_c^d \frac{f(x_0)^2}{2}\,dt = \dfrac{f(x_0)^2}{2}(d-c) \gt 0 $$
Exercise 7¶
Show that the inner product in Example 7 satisfies the axioms of the definition of an inner product.
Follow Along
For any continuous functions $f$, $g$, $h\in C[a,b]$, and any scalars $s$ and $t$,
We have already shown that this inner product is positive semi-definite in the previous paragraph. We need to show that $\langle f,f \rangle = 0$ if and only if $f$ is the zero function on $C[a,b]$. One way is obvious if $f(x)=0$ for all $x\in[a,b]$, then $\langle f,f \rangle = 0$. The contrapositive of the argument above shows that if $f\in C[a,b]$ is such that $\langle f,f \rangle = 0$, then $f(x) = 0$ for all $x\in[a,b]$.
This inner product is symmetric. $$ \langle f, g\rangle = \int_a^b f(t)g(t)\,dt = \int_a^b g(t)f(t)\,dt = \langle g, f\rangle $$
This inner product is multi-linear $$ \begin{align*} \langle s f + t g, h\rangle &= \int_a^b \left(s f(t) + t g(t)\right)h(t)\,dt \\ &= \int_a^b \left(s f(t)h(t) + t g(t)h(t)\right)\,dt \\ &= s \int_a^b f(t)h(t)\,dt + t \int_a^b g(t)h(t)\,dt \\ &= s\langle f, h\rangle + t\langle g, h\rangle \end{align*} $$
Exercise 8¶
In the vector space $C[0,1]$ with the integral inner product $\langle f,g\rangle = \displaystyle\int_0^1 f(x)g(x)\,dx$, let $f(x) = 1$ and $g(x) = x$. Compute $\langle f,g\rangle$, $\lVert f\rVert$, and $\lVert g\rVert$, and find the angle between $f$ and $g$.
Follow Along
The inner product and the two norms are
$$ \langle f,g\rangle = \displaystyle\int_0^1 x\,dx = \dfrac{1}{2}, \qquad \lVert f\rVert = \sqrt{\displaystyle\int_0^1 1\,dx} = 1, \qquad \lVert g\rVert = \sqrt{\displaystyle\int_0^1 x^2\,dx} = \dfrac{1}{\sqrt{3}} $$
The cosine of the angle between them is
$$ \cos(\theta) = \dfrac{\langle f,g\rangle}{\lVert f\rVert\,\lVert g\rVert} = \dfrac{1/2}{(1)\left(1/\sqrt{3}\right)} = \dfrac{\sqrt{3}}{2} $$
Hence $\theta = \dfrac{\pi}{6}$.
Definition¶
Norm of a Vector in an Inner Product Space
If $\ \mathbf{u}$ is a vector in an inner product space $V$, then the norm or magnitude of the vector in this inner product space is defined by
$$ \lVert\mathbf{u}\rVert := \langle\mathbf{u}, \mathbf{u}\rangle^{1/2} $$
This means that we can define norm or magnitude in any abstract vector space if we can define an inner product on the vector space.
Example 2 (continued)¶
In Example 2, the norm or magnitude of a frequency vector is defined for any frequency vector $\mathbf{f}\in\mathbb{R}^{12}$ by
$$ \lVert\mathbf{f}\rVert := \sqrt{\langle\mathbf{f}, \mathbf{f}\rangle} = \left(\displaystyle\sum_{i=1}^{12} f_i^2w_i\right)^{1/2} $$
Example 3 (continued)¶
Consider the matrices
$$ A = \begin{bmatrix}\ \ 4\ &\ \ 2\ & -3\ \\ \ \ 2\ & -1\ &\ \ 2\ \end{bmatrix}\ \text{and}\ B = \begin{bmatrix} -3\ &\ 0\ &\ 2\ \\ \ \ 0\ &\ 1\ &\ 2\ \end{bmatrix} $$
in vector space $\mathbb{R}^{2\times 3}$ with the Frobenius inner product from Example 3. The inner product of matrices $A$ and $B$ is given by
$$ \begin{align*} \langle A, B\rangle_F &= \left[ (4)(-3) + (2)(0) + (-3)(2)\right] + \left[(2)(0) + (-1)(1) + (2)(2) \right] \\ &= -12 + 0 - 6 + 0 - 1 + 4 = -15 \end{align*} $$
The Frobenius norms of these matrices are
$$ \begin{align*} \lVert A\rVert_F &= \left( 4^2 + 2^2 + (-3)^2 + 2^2 + (-1)^2 + 2^2 \right)^{1/2} = \sqrt{38} \\ \\ \lVert B\rVert_F &= \left( (-3)^2 + 0^2 + 2^2 + 0^2 + 1^2 + 2^2 \right)^{1/2} = \sqrt{18} = 3\sqrt{2} \end{align*} $$
Exercise 9¶
In the vector space $\mathbb{R}^{2\times 2}$ with the Frobenius inner product, compute the norm of the matrix
$$ A = \begin{bmatrix}\ 1\ &\ 2\ \\ \ 2\ &\ 4\ \end{bmatrix} $$
Follow Along
The Frobenius norm is the square root of the inner product of the matrix with itself, which sums the squares of every entry,
$$ \lVert A\rVert_F = \sqrt{\langle A,A\rangle_F} = \sqrt{1^2 + 2^2 + 2^2 + 4^2} = \sqrt{1 + 4 + 4 + 16} = \sqrt{25} = 5 $$
Exercise 10¶
On the vector space $\mathbb{R}^3$ use the weight vector $\mathbf{w} = \begin{bmatrix}\ 1\ \\ \ 4\ \\ \ 4\ \end{bmatrix}$ to define a weighted inner product. Compute the norm of $\mathbf{x} = \begin{bmatrix}\ 1\ \\ \ 1\ \\ \ 1\ \end{bmatrix}$ in this inner product space, and compare it with the Euclidean norm of $\mathbf{x}$.
Follow Along
The weighted norm is
$$ \lVert\mathbf{x}\rVert_{\mathbf{w}} = \sqrt{1(1)^2 + 4(1)^2 + 4(1)^2} = \sqrt{1 + 4 + 4} = \sqrt{9} = 3 $$
The Euclidean norm of the same vector is $\lVert\mathbf{x}\rVert = \sqrt{1 + 1 + 1} = \sqrt{3}$. The vector has a different length in each inner product space, so the norm depends on the choice of inner product.
We define the angle between two vectors in an inner product space by using the definition of angle derived from Euclidean space.
Definition¶
The Angle Between Two Vectors
If $\mathbf{u}$ and $\mathbf{v}$ are vectors in an inner product space $V$, then the cosine of the angle $\theta$ between the vectors is defined by
$$ \cos(\theta) := \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert} $$
To determine the angle between two vectors $\mathbf{u}$ and $\mathbf{v}$ in an inner product space $V$, one uses the principal arc cosine of the signed quotient to obtain an angle $\theta$ in the interval $[0,\pi]$. This does not mean that there is a physical interpretation of the angle between two vectors in an abstract inner product space. One of the most useful concepts of angle in an abstract inner product space is orthogonality. If this angle is $\frac{\pi}{2}$, then $\cos\left(\frac{\pi}{2}\right)=0$, so the cosine
$$ \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert} = \cos\left(\frac{\pi}{2}\right) = 0 $$
Definition¶
Orthogonal
If $\ \mathbf{u}$ and $\mathbf{v}$ are vectors in an inner product space $V$ such that their inner product $\langle\mathbf{u}, \mathbf{v}\rangle = 0$, then the vectors are orthogonal.
Since the inner product of the zero vector with all vectors in an inner product space will be the scalar zero, the zero vector is orthogonal to all vectors in an inner product space.
Moreover many key concepts of length and angle turn out to be true for abstract inner product spaces as well. If two vectors $\mathbf{u}$ and $\mathbf{v}$ are orthogonal in a vector space, then they form the two legs of a right triangle in the inner product space with hypotenuse
$$ \mathbf{p} = \mathbf{u} + \mathbf{v} $$
We use the terms right triangle, leg and hypotenuse loosely here because there is no way to draw an adequate picture of such a triangle in an abstract inner product space. However the picture in $\mathbb{R}^2$ suggests that the intuition about the relationships of the vectors is the same.
Theorem 1¶
The Pythagorean Rule
If $\ \mathbf{u}$ and $\mathbf{v}$ are vectors in an inner product space $V$, then $\ \mathbf{u}$ and $\mathbf{v}$ are orthogonal if and only if
$$ \lVert\mathbf{u} + \mathbf{v}\rVert^2 = \lVert\mathbf{u}\rVert^2 + \lVert\mathbf{v}\rVert^2 $$
Proof of Theorem 1¶
$$ \begin{align*} \lVert\mathbf{u} + \mathbf{v}\rVert^2 &= \langle\mathbf{u}+\mathbf{v}, \mathbf{u}+\mathbf{v}\rangle \\ &= \langle\mathbf{u}, \mathbf{u}\rangle + \langle\mathbf{u},\mathbf{v}\rangle + \langle\mathbf{v},\mathbf{u}\rangle + \langle\mathbf{v}, \mathbf{v}\rangle \\ &= \lVert\mathbf{u}\rVert^2 + \langle\mathbf{u},\mathbf{v}\rangle + \langle\mathbf{u},\mathbf{v}\rangle^* + \lVert\mathbf{v}\rVert^2 \end{align*} $$
$\langle\mathbf{u}, \mathbf{v}\rangle = 0 = \langle\mathbf{u}, \mathbf{v}\rangle^*$ if and only if $\mathbf{u}$ and $\mathbf{v}$ are orthogonal. $\blacksquare$
Example 11 - Orthogonal Functions¶
Consider the vector space $C[-1,1]$ with the inner product defined in Example 7. The vectors $f(x)=1$ and $g(x)=x$ are orthogonal because
$$ \langle 1, x\rangle = \displaystyle\int_{-1}^1 1\cdot x\,dx = \left[ \frac{x^2}{2} \right]_{-1}^1 = \frac{1}{2} - \frac{1}{2} = 0 $$
Computing the norms of these vectors one obtains
$$ \begin{align*} \langle 1, 1\rangle &= \displaystyle\int_{-1}^1 1\cdot 1\,dx = \left[ x \right]_{-1}^1 = 1 - (-1) = 2 \\ \\ \langle x, x\rangle &= \displaystyle\int_{-1}^1 x\cdot x\,dx = \left[ \frac{x^3}{3} \right]_{-1}^1 = \frac{1}{3} - \left(-\frac{1}{3}\right) = \frac{2}{3} \\ \\ \langle x+1, x+1\rangle &= \displaystyle\int_{-1}^1 (x+1)(x+1)\,dx = \left[ \frac{(x+1)^3}{3} \right]_{-1}^1 = \left[ \frac{2^3}{3} - \frac{0}{3} \right] = \frac{8}{3} \\ \\ \lVert 1\rVert^2 + \lVert x\rVert^2 &= 2 + \frac{2}{3} = \frac{8}{3} = \lVert x+1 \rVert^2\qquad\LARGE{\color{green}\checkmark} \end{align*} $$
Example 12 - Normalized Orthogonal Functions¶
In the vector space $C[-\pi,\pi]$ with the inner product defined in Example 8, and using the constant weight function $w(t) = \dfrac{1}{\pi}$
$$ \langle\mathbf{f}, \mathbf{g}\rangle = \dfrac{1}{\pi}\displaystyle\int_{-\pi}^{\pi} f(t)g(t)\,dt $$
Consider the vectors $f(t) = \cos(t)$ and $g(t) = \sin(t)$.
$$ \begin{align*} \langle\cos(t), \sin(t)\rangle &= \dfrac{1}{\pi}\displaystyle\int_{-\pi}^{\pi} \cos(t)\sin(t)\,dt = \dfrac{1}{2\pi}\displaystyle\int_{-\pi}^{\pi} \sin(2t)\,dt = \dfrac{1}{2\pi}\left[ -\dfrac{1}{2}\cos(2t) \right]_{-\pi}^{\pi} \\ \\ &= \dfrac{1}{4\pi}\left[ 1 - 1 \right] = 0 \\ \\ \langle\cos(t), \cos(t)\rangle &= \dfrac{1}{\pi}\displaystyle\int_{-\pi}^{\pi} \cos(t)\cos(t)\,dt = \dfrac{1}{\pi}\displaystyle\int_{-\pi}^{\pi} \cos(t)^2\,dt \\ \\ &= \dfrac{1}{2\pi}\displaystyle\int_{-\pi}^{\pi} \left(1 + \cos(2t)\right)\,dt = \dfrac{1}{2\pi}\left[ t + \dfrac{1}{2}\sin(2t)\right]_{-\pi}^{\pi} \\ \\ &= \dfrac{1}{2\pi}\left[ \pi + \frac{0}{2} - \left( -\pi + \frac{0}{2} \right) \right] = 1 \\ \\ \langle\sin(t), \sin(t)\rangle &= \dfrac{1}{\pi}\displaystyle\int_{-\pi}^{\pi} \sin(t)\sin(t)\,dt = \dfrac{1}{\pi}\displaystyle\int_{-\pi}^{\pi} \sin(t)^2\,dt \\ \\ &= \dfrac{1}{2\pi}\displaystyle\int_{-\pi}^{\pi} \left(1 - \cos(2t)\right)\,dt = \dfrac{1}{2\pi}\left[ t + \dfrac{1}{2}\sin(2t)\right]_{-\pi}^{\pi} \\ \\ &= \dfrac{1}{2\pi}\left[ \pi - \frac{0}{2} - \left( -\pi - \frac{0}{2} \right) \right] = 1 \end{align*} $$
Thus $f(t)=\cos(t)$ and $g(t)=\sin(t)$ are orthogonal unit vectors in this inner product space.
They are orthogonal because their inner product is zero
They are unit vectors because their norms are one.
Exercise 11¶
On the vector space $\mathbb{R}^2$ use the weight vector $\mathbf{w} = \begin{bmatrix}\ 1\ \\ \ 4\ \end{bmatrix}$ to define a weighted inner product. Determine whether $\mathbf{x} = \begin{bmatrix}\ 2\ \\ \ 1\ \end{bmatrix}$ and $\mathbf{y} = \begin{bmatrix}\ \ 2\ \\ -1\ \end{bmatrix}$ are orthogonal in this inner product space, and whether they are orthogonal in the Euclidean inner product space.
Follow Along
In the weighted inner product space,
$$ \langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}} = 1(2)(2) + 4(1)(-1) = 4 - 4 = 0 $$
so the vectors are orthogonal. In the Euclidean inner product space,
$$ \mathbf{x}\cdot\mathbf{y} = (2)(2) + (1)(-1) = 3 \ne 0 $$
so the vectors are not orthogonal there.
Exercise 12¶
In the vector space $C[-1,1]$ with the integral inner product $\langle f,g\rangle = \displaystyle\int_{-1}^1 f(x)g(x)\,dx$, show that $f(x) = 1$ and $g(x) = x$ are orthogonal. Then compute the cosine of the angle between $f(x) = 1$ and $h(x) = x^2$.
Follow Along
The first inner product is
$$ \langle f,g\rangle = \displaystyle\int_{-1}^1 x\,dx = 0 $$
so $f$ and $g$ are orthogonal. For the second pair,
$$ \langle f,h\rangle = \displaystyle\int_{-1}^1 x^2\,dx = \dfrac{2}{3}, \qquad \lVert f\rVert = \sqrt{2}, \qquad \lVert h\rVert = \sqrt{\displaystyle\int_{-1}^1 x^4\,dx} = \sqrt{\dfrac{2}{5}} $$
Therefore
$$ \cos(\theta) = \dfrac{2/3}{\sqrt{2}\,\sqrt{2/5}} = \dfrac{2/3}{2/\sqrt{5}} = \dfrac{\sqrt{5}}{3} $$
Like the definition for the angle between two vectors in an abstract inner product space, the projection is a formal definition that may or may not have a physical interpretation. However like orthogonality, projection has many useful applications in linear algebra.
Definition¶
Component
If $\ \mathbf{u}$ and $\mathbf{v}$ are two nonzero vectors in inner product space $V$, then the component or scalar projection of vector $\mathbf{u}$ onto $\mathbf{v}$ is defined by
$$\alpha = \text{Comp}_{\mathbf{v}}\mathbf{u} := \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{v}\rVert}$$
Definition¶
Orthogonal Projection
The projection or vector projection of nonzero vector $\mathbf{u}$ onto nonzero $\mathbf{v}$ is defined by
$$ \mathbf{p} = \text{Proj}_{\mathbf{v}}\mathbf{u} := \alpha\left(\dfrac{\mathbf{v}}{\lVert\mathbf{v}\rVert}\right) = \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{v}\rVert}\left(\dfrac{\mathbf{v}}{\lVert\mathbf{v}\rVert}\right) = \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{v}\rVert^2}\mathbf{v} = \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\langle\mathbf{v}, \mathbf{v}\rangle}\mathbf{v}, $$
where $\alpha = \text{Comp}_{\mathbf{v}}\mathbf{u}$.
Notice that for nonzero $\mathbf{u}$, $\mathbf{v}$, and $\mathbf{p} = \text{Proj}_{\mathbf{v}}\mathbf{u}$, we have $\alpha = \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{v}\rVert}$. Thus,
$$ \begin{align*} \langle\mathbf{p}, \mathbf{p}\rangle &= \left\langle\dfrac{\alpha}{\lVert\mathbf{v}\rVert}\mathbf{v}, \dfrac{\alpha}{\lVert\mathbf{v}\rVert}\mathbf{v}\right\rangle = \left(\dfrac{\alpha}{\lVert\mathbf{v}\rVert}\right)^2 \langle\mathbf{v}, \mathbf{v}\rangle = \alpha^2 \\ \\ \langle\mathbf{u}, \mathbf{p}\rangle &= \left\langle \mathbf{u}, \dfrac{\alpha}{\lVert\mathbf{v}\rVert}\mathbf{v}\right\rangle = \alpha\dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{v}\rVert} = \alpha^2 \\ \\ \langle\mathbf{u}-\mathbf{p}, \mathbf{p}\rangle &= \langle\mathbf{u}, \mathbf{p}\rangle - \langle\mathbf{p},\mathbf{p}\rangle = \alpha^2 - \alpha^2 = 0 \end{align*} $$
Thus the projection of nonzero vector $\mathbf{u}$ onto nonzero vector $\mathbf{v}$ in an inner product space decomposes $\mathbf{u}$ into two vectors with respect to $\mathbf{v}$.
$\mathbf{p} = \mathrm{Proj}_{\mathbf{v}} \mathbf{u}$ parallel to $\mathbf{v}$
$\mathbf{r} = \mathbf{u}-\mathbf{p}$ orthogonal to $\mathbf{v}$
Definition¶
Orthogonal Component
The orthogonal component of nonzero vector $\mathbf{u}$ with respect to a nonzero vector $\mathbf{v}$ in inner product space $V$ is the vector
$$\mathbf{r} = \mathrm{perp}_{\mathbf{v}} \mathbf{u} = \mathbf{u} - \mathrm{Proj}_{\mathbf{v}} \mathbf{u}.$$
Together $\mathbf{u} = \mathrm{Proj}_{\mathbf{v}} \mathbf{u} + \mathrm{perp}_{\mathbf{v}} \mathbf{u}$ is called an orthogonal decomposition of $\mathbf{u}$ with respect to $\mathbf{v}$.
Theorem 2¶
Projection and Dependence
If nonzero vector $\mathbf{u}$, nonzero vector $\mathbf{v}$, $\mathbf{p} = \mathrm{Proj}_{\mathbf{v}}\mathbf{u}$, and $\mathbf{r} = \mathrm{perp}_{\mathbf{v}}\mathbf{u}$ are vectors in inner product space $V$, then $\mathbf{p}$ and $\mathbf{r}$ are orthogonal to each other. Moreover, $\mathbf{u}$ and $\mathbf{v}$ are linearly dependent if and only if $\mathbf{u}=\mathbf{p}$.
Proof of Theorem 2¶
Suppose that nonzero vector $\mathbf{u}$, nonzero vector $\mathbf{v}$, $\mathbf{p} = \mathrm{Proj}_{\mathbf{v}}\mathbf{u}$, and $\mathbf{r} = \mathrm{perp}_{\mathbf{v}}\mathbf{u}$ are vectors in inner product space $V$. Using the linear algebra above
$$\langle \mathbf{r},\ \mathbf{p} \rangle = \langle\mathbf{u}-\mathbf{p}, \mathbf{p}\rangle = \langle\mathbf{u}, \mathbf{p}\rangle - \langle\mathbf{p},\mathbf{p}\rangle = 0.$$
Furthermore, $\mathbf{u}$ and $\mathbf{v}$ are linearly dependent if and only if $\mathbf{u}$ is a scalar multiple of $\mathbf{v}$, or $\mathbf{u}=\beta\mathbf{v}$. Hence,
$$\mathbf{p} = \dfrac{\langle\beta\mathbf{v}, \mathbf{v}\rangle}{\langle\mathbf{v}, \mathbf{v}\rangle}\mathbf{v} = \beta\mathbf{v} = \mathbf{u}.$$
If $\mathbf{p}=\mathbf{u}$, then $\mathbf{r} = \mathbf{u} - \mathbf{p} = \mathbf{0}$ and $\mathbf{u}$ is parallel to $\mathbf{v}$. $\blacksquare$
Exercise 13¶
In the Euclidean inner product space $\mathbb{R}^3$, let $\mathbf{u} = \begin{bmatrix}\ 3\ \\ \ 4\ \\ \ 0\ \end{bmatrix}$ and $\mathbf{v} = \begin{bmatrix}\ 1\ \\ \ 0\ \\ \ 0\ \end{bmatrix}$. Compute the orthogonal projection $\mathbf{p}$ of $\mathbf{u}$ onto $\mathbf{v}$ and the orthogonal component $\mathbf{r} = \mathbf{u} - \mathbf{p}$. Verify that $\mathbf{p}$ and $\mathbf{r}$ are orthogonal.
Follow Along
Since $\langle\mathbf{u},\mathbf{v}\rangle = 3$ and $\langle\mathbf{v},\mathbf{v}\rangle = 1$, the projection is
$$ \mathbf{p} = \dfrac{\langle\mathbf{u},\mathbf{v}\rangle}{\langle\mathbf{v},\mathbf{v}\rangle}\mathbf{v} = 3\begin{bmatrix} 1\ \\ 0\ \\ 0\ \end{bmatrix} = \begin{bmatrix} 3\ \\ 0\ \\ 0\ \end{bmatrix} $$
The orthogonal component is
$$ \mathbf{r} = \mathbf{u} - \mathbf{p} = \begin{bmatrix} 0\ \\ 4\ \\ 0\ \end{bmatrix} $$
and $\langle\mathbf{p},\mathbf{r}\rangle = (3)(0) + (0)(4) + (0)(0) = 0$, so the two vectors are orthogonal.
Exercise 14¶
In the vector space $C[0,1]$ with the integral inner product, compute the orthogonal projection of $f(x) = x$ onto $g(x) = 1$, and verify that the orthogonal component is orthogonal to $g$.
Follow Along
The projection uses the same formula as in any inner product space,
$$ \mathbf{p} = \dfrac{\langle f,g\rangle}{\langle g,g\rangle}\,g = \dfrac{\displaystyle\int_0^1 x\,dx}{\displaystyle\int_0^1 1\,dx}\,g = \dfrac{1/2}{1}\,g = \dfrac{1}{2} $$
The orthogonal component is $r(x) = x - \dfrac{1}{2}$, and
$$ \langle r,g\rangle = \displaystyle\int_0^1\left(x - \dfrac{1}{2}\right)dx = \dfrac{1}{2} - \dfrac{1}{2} = 0 $$
so $r$ is orthogonal to $g$. The projection of $f$ onto the constant function is the average value of $f$ on the interval.
Theorem 3¶
The Cauchy-Schwarz-Bunyakovsky Theorem
If $\ \mathbf{u}$ and $\mathbf{v}$ are any two vectors in an inner product space $V$, then
$$ \left|\langle\mathbf{u}, \mathbf{v}\rangle\right| \le \lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert $$
Proof of Theorem 3¶
If $\mathbf{v}=\mathbf{0}$, then
$$ \left|\langle\mathbf{u}, \mathbf{v}\rangle\right| = 0 = \lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert $$
If $\mathbf{v}\neq\mathbf{0}$, then define $\mathbf{p}:=\text{Proj}_{\mathbf{v}}\mathbf{u}$. Since vectors $\mathbf{p}$ and $\mathbf{u}-\mathbf{p}$ are orthogonal, the Pythagorean Law implies
$$ \lVert\mathbf{p}\rVert^2 + \lVert\mathbf{u}-\mathbf{p}\rVert^2 = \lVert\mathbf{u}\rVert^2 $$
Replacing vector $\mathbf{p}$ with its definition
$$ \begin{align*} \lVert\mathbf{p}\rVert^2 &= \langle\mathbf{p}, \mathbf{p}\rangle \\ \\ &= \left\langle\dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\langle\mathbf{v}, \mathbf{v}\rangle}\mathbf{v}, \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\langle\mathbf{v}, \mathbf{v}\rangle}\mathbf{v}\right\rangle \\ \\ &= \left(\dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\langle\mathbf{v}, \mathbf{v}\rangle}\right)^2\langle\mathbf{v}, \mathbf{v}\rangle = \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle^2}{\langle\mathbf{v}, \mathbf{v}\rangle} \\ \\ &= \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle^2}{\lVert\mathbf{v}\rVert^2} = \lVert\mathbf{u}\rVert^2 - \lVert\mathbf{u}-\mathbf{p}\rVert^2 \end{align*} $$
Multiplying both sides of the last equation by the nonzero scalar $\lVert\mathbf{v}\rVert^2$ yields
$$ \langle\mathbf{u}, \mathbf{v}\rangle^2 = \lVert\mathbf{u}\rVert^2\,\lVert\mathbf{v}\rVert^2 - \lVert\mathbf{u}-\mathbf{p}\rVert^2\,\lVert\mathbf{v}\rVert^2 \le \lVert\mathbf{u}\rVert^2\,\lVert\mathbf{v}\rVert^2 $$
Computing the square root of both sides of the last equation gives us
$$ \left|\langle\mathbf{u}, \mathbf{v}\rangle\right| \le \lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert \qquad \blacksquare$$
Corollary 4¶
Angle Cosine
For nonzero vectors $\mathbf{u}$ and $\mathbf{v}$ in an inner product space $V$, the Cauchy-Schwarz-Bunyakovsky Theorem implies
$$ -1\le \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert}\le 1 $$
Thus there is a unique angle $\theta\in[0,\ \pi]$ such that
$$ \cos(\theta) = \dfrac{\langle\mathbf{u}, \mathbf{v}\rangle}{\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert} $$
This tells us that the angle between two nonzero vectors in an inner product space is well-defined.
Exercise 15¶
On the vector space $\mathbb{R}^2$ use the weight vector $\mathbf{w} = \begin{bmatrix}\ 1\ \\ \ 2\ \end{bmatrix}$ to define a weighted inner product. Verify the Cauchy-Schwarz-Bunyakovsky Theorem for $\mathbf{x} = \begin{bmatrix}\ 1\ \\ \ 1\ \end{bmatrix}$ and $\mathbf{y} = \begin{bmatrix}\ \ \ 1\ \\ -1\ \end{bmatrix}$.
Follow Along
The weighted inner product and the two norms are
$$ \langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}} = 1(1)(1) + 2(1)(-1) = -1, \qquad \lVert\mathbf{x}\rVert_{\mathbf{w}} = \sqrt{1 + 2} = \sqrt{3}, \qquad \lVert\mathbf{y}\rVert_{\mathbf{w}} = \sqrt{1 + 2} = \sqrt{3} $$
Therefore
$$ \left|\langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}}\right| = 1 \le 3 = \lVert\mathbf{x}\rVert_{\mathbf{w}}\,\lVert\mathbf{y}\rVert_{\mathbf{w}} $$
The inequality is strict because the two vectors are not scalar multiples of one another.
Exercise 16¶
In the Euclidean inner product space $\mathbb{R}^2$, let $\mathbf{x} = \begin{bmatrix}\ 1\ \\ \ 2\ \end{bmatrix}$ and $\mathbf{y} = \begin{bmatrix}\ 2\ \\ \ 4\ \end{bmatrix}$. Show that the Cauchy-Schwarz-Bunyakovsky inequality holds with equality, and explain why.
Follow Along
The inner product and the norms are
$$ \langle\mathbf{x},\mathbf{y}\rangle = (1)(2) + (2)(4) = 10, \qquad \lVert\mathbf{x}\rVert = \sqrt{5}, \qquad \lVert\mathbf{y}\rVert = \sqrt{20} = 2\sqrt{5} $$
Therefore
$$ \left|\langle\mathbf{x},\mathbf{y}\rangle\right| = 10 = \left(\sqrt{5}\right)\left(2\sqrt{5}\right) = \lVert\mathbf{x}\rVert\,\lVert\mathbf{y}\rVert $$
Equality holds because $\mathbf{y} = 2\mathbf{x}$, so the vectors are linearly dependent.
The magnitude or norm of a vector in a vector space can be defined independent of an inner product. In many applications, the magnitude of a vector is required without the notion of angle or direction.
Definition¶
Norm of a vector
A function $\lVert\cdot\rVert\,:\,V\rightarrow[0,\infty)$ that maps every vector $\mathbf{v}\in V$ to a nonnegative real number $\lVert\mathbf{v}\rVert$ and satisfies the following axioms is called a norm on a vector space $V$.
For all vectors $\mathbf{v}$ and $\mathbf{w}$ and scalar $t$,
$\lVert\mathbf{v}\rVert\ge 0$ with equality if and only if $\mathbf{v}=\mathbf{0}$.
$\lVert t\mathbf{v}\rVert = |t|\lVert\mathbf{v}\rVert$.
$\lVert\mathbf{v} + \mathbf{w}\rVert\le\lVert\mathbf{v}\rVert + \lVert\mathbf{w}\rVert$ for all $\mathbf{v},\ \mathbf{w}\in V$
Normed Linear Space
A vector space $V$ endowed with a norm is called a normed linear space, $\left\{\, V,\,\lVert\cdot\rVert \,\right\}$.
The third condition in the definition of norm is called the triangle inequality.
One should verify that the norm induced by an inner product satisfies the definition of a norm here. That is,
Theorem 5¶
Every Inner Product Space is a Normed Linear Space
Every inner product space $V$ is a normed linear space with its norm defined for every vector $\mathbf{v}\in V$ by
$$ \lVert\mathbf{v}\rVert = \sqrt{\langle\mathbf{v}, \mathbf{v}\rangle} $$
Proof of Theorem 5¶
Given an inner product space $V$, then for every pair of vectors $\mathbf{v}$, $\mathbf{w}\in V$, and real scalar $\alpha\in\mathbb{R}$,
$\lVert\mathbf{v}\rVert^2 = \langle\mathbf{v}, \mathbf{v}\rangle \ge 0$, with $\langle\mathbf{v}, \mathbf{v}\rangle = 0$ if and only if $\mathbf{v}=\mathbf{0}$ from property (1) of an inner product.
$\lVert\alpha\mathbf{v}\rVert^2 = \langle\alpha\mathbf{v}, \alpha\mathbf{v}\rangle = \alpha^2\langle\mathbf{v}, \mathbf{v}\rangle$.
Using the Cauchy-Schwarz-Bunyakovsky Inequality we have
$$ \begin{align*} \lVert\mathbf{v} + \mathbf{w}\rVert^2 &= \langle\mathbf{v} + \mathbf{w}, \mathbf{v} + \mathbf{w}\rangle \\ \\ &= \langle\mathbf{v},\mathbf{v}\rangle + \langle\mathbf{v},\mathbf{w}\rangle + \langle\mathbf{w},\mathbf{v}\rangle + \langle\mathbf{w},\mathbf{w}\rangle \\ \\ &= \langle\mathbf{v},\mathbf{v}\rangle + \langle\mathbf{v},\mathbf{w}\rangle + \langle\mathbf{v},\mathbf{w}\rangle^* + \langle\mathbf{w},\mathbf{w}\rangle \\ \\ &= \langle\mathbf{v},\mathbf{v}\rangle + 2\Re\left( \langle\mathbf{v},\mathbf{w}\rangle \right) + \langle\mathbf{w},\mathbf{w}\rangle \\ \\ &\le \langle\mathbf{v},\mathbf{v}\rangle + 2\Re\left| \langle\mathbf{v},\mathbf{w}\rangle \right| + \langle\mathbf{w},\mathbf{w}\rangle \\ \\ &= \langle\mathbf{v},\mathbf{v}\rangle + 2\left| \langle\mathbf{v},\mathbf{w}\rangle \right| + \langle\mathbf{w},\mathbf{w}\rangle \\ \\ &\le \lVert\mathbf{v}\rVert^2 + 2\lVert\mathbf{v}\rVert\,\lVert\mathbf{w}\rVert + \lVert\mathbf{w}\rVert^2 \\ \\ &= \left(\lVert\mathbf{v}\rVert + \lVert\mathbf{w}\rVert\right)^2 \end{align*} $$
Computing the square root of both sides yields the triangle inequality. $\blacksquare$
Definition¶
The Distance Between Two Vectors
For vectors $\mathbf{u}$ and $\mathbf{v}$ in normed linear space $V$, the distance from vector $\mathbf{u}$ to vector $\mathbf{v}$ is defined by
$$ \text{dist}(\mathbf{u},\mathbf{v}) := \lVert\mathbf{v} - \mathbf{u}\rVert $$
What about the converse proposition? Is every normed linear space also an inner product space? The answer is no. While every inner product induces a norm on a vector space, most norms are not induced by an inner product.
Exercise 17¶
In the normed linear space $\mathbb{R}^{2\times 2}$ with the norm induced by the Frobenius inner product, compute the distance between
$$ A = \begin{bmatrix}\ 4\ &\ 6\ \\ \ 3\ &\ 4\ \end{bmatrix} \quad\text{and}\quad B = \begin{bmatrix}\ 1\ &\ 2\ \\ \ 3\ &\ 4\ \end{bmatrix} $$
Follow Along
The distance is the norm of the difference,
$$ A - B = \begin{bmatrix} 3\ &\ 4\ \\ 0\ &\ 0\ \end{bmatrix} $$
so
$$ \text{dist}(A,B) = \lVert A - B\rVert_F = \sqrt{3^2 + 4^2 + 0^2 + 0^2} = \sqrt{25} = 5 $$
Exercise 18¶
Every norm induced by an inner product satisfies the parallelogram law
$$ \lVert\mathbf{u} + \mathbf{v}\rVert^2 + \lVert\mathbf{u} - \mathbf{v}\rVert^2 = 2\lVert\mathbf{u}\rVert^2 + 2\lVert\mathbf{v}\rVert^2 $$
Verify this identity in the Euclidean inner product space $\mathbb{R}^2$ for $\mathbf{u} = \begin{bmatrix}\ 3\ \\ \ 4\ \end{bmatrix}$ and $\mathbf{v} = \begin{bmatrix}\ 1\ \\ \ 2\ \end{bmatrix}$.
Follow Along
The sum and difference of the two vectors are
$$ \mathbf{u} + \mathbf{v} = \begin{bmatrix} 4\ \\ 6\ \end{bmatrix}, \qquad \mathbf{u} - \mathbf{v} = \begin{bmatrix} 2\ \\ 2\ \end{bmatrix} $$
The left side of the identity is
$$ \lVert\mathbf{u} + \mathbf{v}\rVert^2 + \lVert\mathbf{u} - \mathbf{v}\rVert^2 = (16 + 36) + (4 + 4) = 52 + 8 = 60 $$
The right side is
$$ 2\lVert\mathbf{u}\rVert^2 + 2\lVert\mathbf{v}\rVert^2 = 2(25) + 2(5) = 50 + 10 = 60 $$
The two sides agree.
Example 13 - 1-Norm¶
Show that the function $\lVert\cdot\rVert_1\,:\mathbb{R}^n\rightarrow[0,\infty)$ defined for every vector $\mathbf{x}\in\mathbb{R}^n$ by
$$ \lVert\mathbf{x}\rVert_1 := |x_1| + |x_2| + |x_3| + ... + |x_n| = \displaystyle\sum_{i=1}^n |x_i| $$
defines a norm on $\mathbb{R}^n$.
We need to show that $\lVert\cdot\rVert_1$ satisfies all three axioms of the definition of a norm on vector space $\mathbb{R}^n$. For every pair of vectors $\mathbf{x}$, $\mathbf{y}$ and scalar $t$,
$\lVert\mathbf{x}\rVert_1 = \displaystyle\sum_{i=1}^n |x_i| \ge 0$ since it is a sum of nonnegative real numbers. Moreover if $\lVert\mathbf{x}\rVert_1=0$, then $|x_i|=0$ for all $1\le i\le n$, so $x_i=0$ for all $1\le i\le n$. Hence $\mathbf{x}=\mathbf{0}$.
$\lVert\alpha\mathbf{x}\rVert_1 = \displaystyle\sum_{i=1}^n \left|\alpha x_i\right| = \displaystyle\sum_{i=1}^n |\alpha||x_i| = |\alpha|\displaystyle\sum_{i=1}^n |x_i| = |\alpha|\lVert\mathbf{x}\rVert_1$
Using the triangle inequality of real numbers $$ \lVert\mathbf{x} + \mathbf{y}\rVert_1 = \displaystyle\sum_{i=1}^n |x_i + y_i| \le \displaystyle\sum_{i=1}^n \left(|x_i| + |y_i|\right) = \displaystyle\sum_{i=1}^n |x_i| + \displaystyle\sum_{i=1}^n |y_i| = \lVert\mathbf{x}\rVert_1 + \lVert\mathbf{y}\rVert_1.$$
Definition¶
p-Norm
For any real number $p\ge 1$, one can define a norm on the vector space $\mathbb{R}^n$ called a $p$-norm, such that for every vector $\mathbf{x}\in\mathbb{R}^n$, $$ \lVert\mathbf{x}\rVert_p := \left( \sum_{i=1}^n |x_i|^p \right)^{1/p}.$$
Example 14 - 2-Norm¶
In particular, for $p=2$, the $2$-norm is given by
$$ \lVert\mathbf{x}\rVert_2 := \left(\displaystyle\sum_{i=1}^n x_i^2 \right)^{1/2} = \sqrt{\langle\mathbf{x}, \mathbf{x}\rangle} $$
The 2-norm is the only p-norm induced by an inner product, the Euclidean inner product.
Definition¶
Infinity Norm
The uniform norm or infinity norm is defined on the vector space $\mathbb{R}^n$ for every vector $\mathbf{x}\in\mathbb{R}^n$ by
$$ \lVert\mathbf{x}\rVert_{\infty} := \text{max}\left\{ |x_i|\,:\,1\le i\le n \right\} $$
Example 15 - Computing Uniform Norm¶
Let $\mathbf{x}_1$, $\mathbf{x}_2\in\mathbb{R}^2$ be defined by
$$ \mathbf{x}_1 = \begin{bmatrix} -2\ \\ \ \ 2\ \end{bmatrix}\ \text{and}\ \ \mathbf{x}_2 = \begin{bmatrix}\ \ 1\ \\ -3\ \end{bmatrix} $$
The uniform norm
$$ \lVert\mathbf{x}_1\rVert_{\infty} = 2\ \text{and}\ \ \lVert\mathbf{x}_2\rVert_{\infty} = 3 $$
Exercise 19¶
Let $\mathbf{x}=\begin{bmatrix} -2\ \\ \ \ 4\ \\ -1\ \end{bmatrix}$. Compute $\lVert\mathbf{x}\rVert_1$, $\lVert\mathbf{x}\rVert_2$, $\lVert\mathbf{x}\rVert_3$, and $\lVert\mathbf{x}\rVert_{\infty}$.
Follow Along
$$ \begin{align*} 1.&\ \lVert\mathbf{x}\rVert_1 = |-2| + |4| + |-1| = 7 \\ \\ 2.&\ \lVert\mathbf{x}\rVert_2 = \left( 4 + 16 + 1 \right)^{1/2} = \sqrt{21} \\ \\ 3.&\ \lVert\mathbf{x}\rVert_3 = \left( 8 + 64 + 1 \right)^{1/3} = \sqrt[3]{73} \\ \\ 4.&\ \lVert\mathbf{x}\rVert_{\infty} = \text{max}\left\{ |-2|, |4|, |-1| \right\} = 4 \\ \end{align*} $$
Exercise 20¶
Let $\mathbf{x} = \begin{bmatrix}\ \ 1\ \\ -2\ \\ \ \ 2\ \end{bmatrix}$. Compute $\lVert\mathbf{x}\rVert_1$, $\lVert\mathbf{x}\rVert_2$, and $\lVert\mathbf{x}\rVert_{\infty}$, and verify that
$$ \lVert\mathbf{x}\rVert_{\infty} \le \lVert\mathbf{x}\rVert_2 \le \lVert\mathbf{x}\rVert_1 $$
Follow Along
The three norms are
$$ \lVert\mathbf{x}\rVert_1 = |1| + |-2| + |2| = 5 $$ $$ \lVert\mathbf{x}\rVert_2 = \left( 1 + 4 + 4 \right)^{1/2} = \sqrt{9} = 3 $$ $$ \lVert\mathbf{x}\rVert_{\infty} = \text{max}\left\{ |1|,\ |-2|,\ |2| \right\} = 2 $$
Therefore $2 \le 3 \le 5$, which confirms the chain of inequalities for this vector.
These questions test the definitions and theorems of this section, together with material on linear systems, determinants, and vector spaces from earlier chapters. In a true/false question, true means always true — a statement that is only sometimes true is false.
True/False Questions¶
For each statement, decide whether it is always true. If true, give a general argument; if false, give a counterexample with specific numbers.
- Every inner product on the vector space $\mathbb{R}^n$ is the Euclidean scalar product.
Follow Along
FALSE. The Euclidean scalar product is only one of infinitely many inner products on $\mathbb{R}^n$. Counterexample: on $\mathbb{R}^2$ the weighted function
$$ \langle\mathbf{x},\mathbf{y}\rangle_{\mathbf{w}} = 2x_1y_1 + 3x_2y_2 $$
satisfies all three axioms, yet $\langle\mathbf{e}_1,\mathbf{e}_1\rangle_{\mathbf{w}} = 2$ while $\mathbf{e}_1\cdot\mathbf{e}_1 = 1$. The two inner products give different geometries on the same vector space.
- If $\mathbf{x}$ is a vector in an inner product space $V$ and $\langle\mathbf{x},\mathbf{y}\rangle = 0$ for every vector $\mathbf{y}\in V$, then $\mathbf{x} = \mathbf{0}$.
Follow Along
TRUE. The hypothesis holds for every vector in $V$, so it holds for the particular choice $\mathbf{y} = \mathbf{x}$. Then
$$ \langle\mathbf{x},\mathbf{x}\rangle = 0 $$
By the positive definite axiom, $\langle\mathbf{x},\mathbf{x}\rangle = 0$ only when $\mathbf{x} = \mathbf{0}$. Hence $\mathbf{x}$ is the zero vector.
- If $A$ is a $2\times 2$ matrix with $\det(A)\ne 0$, then the function $\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\mathsf{T}A\,\mathbf{y}$ is an inner product on $\mathbb{R}^2$.
Follow Along
FALSE. A nonzero determinant guarantees that $A$ is nonsingular, but it does not make the function positive definite. Counterexample:
$$ A = \begin{bmatrix} 1\ &\ \ \ 0\ \\ 0\ &\ -1\ \end{bmatrix}, \qquad \det(A) = -1 \ne 0 $$
Choosing the nonzero vector $\mathbf{x} = \begin{bmatrix} 0\ \\ 1\ \end{bmatrix}$ gives
$$ \langle\mathbf{x},\mathbf{x}\rangle = \mathbf{x}^\mathsf{T}A\,\mathbf{x} = -1 < 0 $$
which violates the first axiom.
- If $\mathbf{u}$ and $\mathbf{v}$ are vectors in an inner product space $V$ and $\left|\langle\mathbf{u},\mathbf{v}\rangle\right| = \lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert$, then the set $\left\{\mathbf{u},\mathbf{v}\right\}$ is linearly dependent.
Follow Along
TRUE. The Cauchy-Schwarz-Bunyakovsky Theorem gives the inequality $\left|\langle\mathbf{u},\mathbf{v}\rangle\right| \le \lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert$, with equality exactly when one vector is a scalar multiple of the other. A set of two vectors in which one is a scalar multiple of the other is linearly dependent. For instance, in the Euclidean space $\mathbb{R}^2$ the vectors $\mathbf{u} = \langle 1,2\rangle$ and $\mathbf{v} = \langle 2,4\rangle = 2\mathbf{u}$ satisfy $\left|\langle\mathbf{u},\mathbf{v}\rangle\right| = 10 = \lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert$.
- Every norm on a vector space $V$ is induced by some inner product on $V$.
Follow Along
FALSE. Every norm induced by an inner product satisfies the parallelogram law
$$ \lVert\mathbf{x} + \mathbf{y}\rVert^2 + \lVert\mathbf{x} - \mathbf{y}\rVert^2 = 2\lVert\mathbf{x}\rVert^2 + 2\lVert\mathbf{y}\rVert^2 $$
The $1$-norm on $\mathbb{R}^2$ does not. Counterexample: for $\mathbf{x} = \begin{bmatrix} 1\ \\ 0\ \end{bmatrix}$ and $\mathbf{y} = \begin{bmatrix} 0\ \\ 1\ \end{bmatrix}$, we have $\lVert\mathbf{x} + \mathbf{y}\rVert_1 = 2$ and $\lVert\mathbf{x} - \mathbf{y}\rVert_1 = 2$, so the left side is $4 + 4 = 8$, while the right side is $2(1) + 2(1) = 4$. Since $8 \ne 4$, the $1$-norm comes from no inner product.
Multiple Choice Questions¶
Check the box of each statement that must be true, and leave false statements unchecked. No justification is required.
- Let $V$ be an inner product space, and let $\mathbf{v}\in V$ be a fixed vector. Check each statement that must be true.
$\square\ $ $\langle\mathbf{x},\mathbf{x}\rangle \ge 0$ for every vector $\mathbf{x}\in V$.
$\square\ $ $\langle\mathbf{0},\mathbf{y}\rangle = 0$ for every vector $\mathbf{y}\in V$.
$\square\ $ The set $\left\{\mathbf{x}\in V\,:\,\langle\mathbf{x},\mathbf{v}\rangle = 0\right\}$ is a subspace of $V$.
$\square\ $ If $\mathbf{x}\ne\mathbf{0}$ and $\langle\mathbf{x},\mathbf{y}\rangle = \langle\mathbf{x},\mathbf{z}\rangle$, then $\mathbf{y} = \mathbf{z}$.
$\square\ $ Every inner product space is finite dimensional.
Follow Along
Check the first, second, and third boxes; leave the fourth and fifth unchecked. The fourth fails because $\mathbf{y}$ and $\mathbf{z}$ may differ by any vector orthogonal to $\mathbf{x}$, and the fifth fails because $C[a,b]$ with the integral inner product is an infinite dimensional inner product space.
- On the vector space $\mathbb{R}^2$ define $\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\mathsf{T}A\,\mathbf{y}$ using the matrix
$$ A = \begin{bmatrix}\ 2\ &\ 1\ \\ \ 1\ &\ 2\ \end{bmatrix} $$
Check each statement that must be true.
$\square\ $ This function is an inner product on $\mathbb{R}^2$.
$\square\ $ $\langle\mathbf{e}_1,\mathbf{e}_2\rangle = 1$.
$\square\ $ The standard basis vectors $\mathbf{e}_1$ and $\mathbf{e}_2$ are orthogonal in this inner product space.
$\square\ $ The homogeneous linear system $A\mathbf{x} = \mathbf{0}$ has only the trivial solution.
$\square\ $ $\lVert\mathbf{e}_1\rVert = 1$ in this inner product space.
Follow Along
Check the first, second, and fourth boxes; leave the third and fifth unchecked. The matrix $A$ is symmetric with $\det(A) = 3 > 0$ and positive diagonal entries, so the function is positive definite, and $\det(A)\ne 0$ means $A\mathbf{x} = \mathbf{0}$ has only the trivial solution. However $\langle\mathbf{e}_1,\mathbf{e}_2\rangle = 1 \ne 0$, so the standard basis vectors are not orthogonal here, and $\lVert\mathbf{e}_1\rVert = \sqrt{\langle\mathbf{e}_1,\mathbf{e}_1\rangle} = \sqrt{2}$.
- Let $\mathbf{u}$ and $\mathbf{v}$ be nonzero vectors in an inner product space $V$, with $\mathbf{p}$ the orthogonal projection of $\mathbf{u}$ onto $\mathbf{v}$ and $\mathbf{r} = \mathbf{u} - \mathbf{p}$. Check each statement that must be true.
$\square\ $ $\mathbf{p} = \dfrac{\langle\mathbf{u},\mathbf{v}\rangle}{\langle\mathbf{v},\mathbf{v}\rangle}\,\mathbf{v}$.
$\square\ $ $\langle\mathbf{r},\mathbf{v}\rangle = 0$.
$\square\ $ $\lVert\mathbf{x}\rVert_{\infty} \le \lVert\mathbf{x}\rVert_2$ for every vector $\mathbf{x}\in\mathbb{R}^n$.
$\square\ $ $\lVert\mathbf{x}\rVert_1 \le \lVert\mathbf{x}\rVert_2$ for every vector $\mathbf{x}\in\mathbb{R}^n$.
$\square\ $ The distance function $\text{dist}(\mathbf{u},\mathbf{v}) = \lVert\mathbf{u} - \mathbf{v}\rVert$ is a norm on $V$.
Follow Along
Check the first, second, and third boxes; leave the fourth and fifth unchecked. The fourth inequality points the wrong way, since $\lVert\mathbf{x}\rVert_2 \le \lVert\mathbf{x}\rVert_1$; for example $\mathbf{x} = \langle 1,-2,2\rangle$ has $\lVert\mathbf{x}\rVert_2 = 3$ and $\lVert\mathbf{x}\rVert_1 = 5$. The fifth fails because a norm is a function of a single vector, while the distance function takes two vectors as inputs.