Each Section and Topic title are links. At the end of each section, there is a link back to this table.
- 2.1.1 Matrix Notation
- 2.1.2 Vector Algebra
- 2.1.3 Linear Transformation
- 2.1.4 Matrix Arithmetic
- 2.1.5 Matrix-Vector Multiplication
- 2.1.6 Understanding Matrix Multiplication
- 2.1.7 - Dual Matrix Multiplication
- 2.1.8 - Inner Product Form of Matrix Multiplication
- 2.1.9 - Outer Product
- 2.1.10 - Block Multiplication
- 2.1.11 - Transformations in Space
- 2.1.12 - Exercises
- copyleft
2.1.1 Matrix Notation 
Notation is Important!¶
This section covers the details of matrix arithmetic (how to perform algebraic operations like adding, subtracting, and multiplying matrices). Before studying algebraic operations with matrices one must understand the basic notation required in this course for writing and communicating them. Video lectures, online notes, homework submissions, and test answers need to unambiguously denote different types of mathematical objects and the variable names that will represent them. In this course variable names (letters) will denote, scalars, vectors, and matrices differently.
Video Lecture 1: Linear Algebra Notation.¶
Video Lecture 1: Linear Algebra Notation Watch the Notation Video Lecture 1!¶
This is a short video to motivate and demonstrate the notation one uses in Linear Algebra in this class. I will only read what you write or type in your homework or exam. You must use this notation!
2.1.2 Vector Algebra
Algebra with Vectors and Scalars¶
We begin our understanding of vector algebra with scalar multiplication. When we see a column vector or $n\times 1$ matrix $\mathbf{v}$, we usually think of it as a representation of an arrow in an $n$-dimensional space $\mathbb{R}^n$. For example a vector $\mathbf{v}\in\mathbb{R}^3$ such as
$$ \mathbf{v} = \begin{bmatrix}\ \ 2\ \\ \ \ 2\ \\ -1\ \end{bmatrix} $$
Multiplication of a vector $\mathbf{v}$ by a scalar $t$:
- stretches (dilates) the vector if $|t|>1$,
- squishes (contracts) the vector if $0 < |t| < 1$,
- reverses the direction (reflects) the vector if $t < 0$,
- results in the zero vector $\mathbf{0}$ if $t = 0$,
- and leaves the vector unchanged if $t=1$.
This was explained in 
Video Lecture 1.4: Vectors, what even are they?. Algebraically, multiplying a vector $\mathbf{v}$ with coordinates $v_i$ results in a vector with coordinates $tv_i$.
Definition¶
Scalar multiplication of a Vector
If $\mathbf{v}\in\mathbb{R}^n$, and $t\in\mathbb{R}$ is a scalar, then
$$ t\mathbf{v} = t\begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} = \begin{bmatrix} tv_1 \\ tv_2 \\ \vdots \\ tv_n \end{bmatrix}. $$
This is called element-wise multiplication.
Example 1 - Span of a Vector¶
The set of all possible outputs or scalar multiples of vector $\mathbf{x}$ is the span of the set $\left\{\,\mathbf{x}\,\right\}$ or a line.
| Vector Equation | Graph |
|---|---|
| $$ \mathbf{x}(t) = t\mathbf{x} = \begin{bmatrix}\ \ 2t\ \\ \ \ 2t\ \\ -t\ \end{bmatrix}\qquad\qquad $$ |
|
Likewise, our video about vectors shows that vector addition is also performed element-wise.
Definition¶
Vector Addition
If $\mathbf{v}$ and $\mathbf{w}\in\mathbb{R}^n$, then the result $\mathbf{v}+\mathbf{w}$ is algebraically implemented element-wise.
$$\mathbf{v}+\mathbf{w} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} + \begin{bmatrix} w_1 \\ w_2 \\ \vdots \\ w_n \end{bmatrix} = \begin{bmatrix} v_1 + w_1 \\ v_2 + w_2 \\ \vdots \\ v_n + w_n \end{bmatrix}$$
Exercise 1 - A Linear Combination¶
(a) Use subscripts to denote the linear combination of twelve vectors $\mathbf{v}_1$, $\dots$, $\mathbf{v}_{12}$ in $n$-dimensional space, and twelve scalars $c_1$, $\dots$, $c_{12}$, in increasing index order.
View Solution
$$c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_{12}\mathbf{v}_{12}$$ 2.1.3 Linear Transformations
Linear Maps¶
It is easy for students to think that this course and these notes are about matrices. This misconception will seem valid for the first three chapters of this course. The first three chapters introduce students to a very significant amount of vocabulary. Introducing this vocabulary using familiar objects like, lines, planes, 3-dimensional space, and matrices make the topics seem grounded with students previous education. However this linear algebra course concerns linear spaces, vector spaces, and linear mappings from one vector space to another.
Linear transformations, linear functions, or linear maps are one of the basic topics of this course. In our study of calculus, functions with derivatives, anti-derivatives, and absolutely convergent power series were our main concern. In our study of derivatives, two important properties emerged:
- The derivative of a sum of two functions is the sum of the individual derivatives.
- The derivative of a scalar multiple of a function is the same scalar multiple of the function's derivative.
These were called the linearity properties. Mathematicians noticed the same algebraic structure occurring throughout many separate applications in STEM. Linear algebra is the study of the algebraic structures that occur in many disparate sciences. Machine Learning, robotics, computer graphics, physics, and engineering all use linear maps.
Video Lecture 2: Linear Transformations and Matrices.¶
Linear Transformations
Linear TransformationsWhat makes a function a Linear function or Linear Transformation? In our pre-calculus mathematics education, a linear function is one whose graph is a line. In slope-intercept form we have
$$\ell(x) = mx + b$$
Your instructor has been assuming that everyone knows about the equations of a lines and planes, and how to graph them. At this time in your education we must create a distinction between a function that is linear and a function that is affine. Linear functions behave like multiplication, and affine functions behave like translations of linear functions.
The function $\ell(x) = 3x$ is a linear function. One of the many properties linear transformations (functions) share is that $\ell(0) = 0$. The graph of a linear transformation passes through the origin of our coordinate axes. Affine functions translate a linear function away from the origin. Hence the function $f(x) = 3x - 5$ is affine.
Definition¶
Linear Transformation
A linear transformation is a function $L\,:\,V\rightarrow W$, whose domain and codomain are vector spaces (like $\mathbb{R}^n$) such that for any vectors $\mathbf{x}$, $\mathbf{y}$ in the domain of $L$, and scalars $t\in\mathbb{R}$,
$$ \begin{align*} L(\mathbf{x} + \mathbf{y}) &= L(\mathbf{x}) + L(\mathbf{y}) \\ \\ L(t\mathbf{x}) &= tL(\mathbf{x}) \end{align*} $$
$\mathbb{R} = \mathbb{R}^1$ is a vector space too even though it is also a field of scalars.
Example 2 - A Linear Transformation¶
Consider $L\,:\,\mathbb{R}^3\rightarrow\mathbb{R}^2$ defined by $L(\mathbf{x}) = \langle x_1,\ x_2 \rangle$.
$$ \begin{align*} L(\langle 1, 2, -3 \rangle) &= \langle 1, 2 \rangle \\ L(\,3\langle 1, 2, -3 \rangle\,) &= L(\langle 3, 6, -9 \rangle) = \langle 3, 6 \rangle = 3\langle 1, 2 \rangle = 3L(\langle 1, 2, -3 \rangle) \\ L(\,\langle 1, 2, -3 \rangle + \langle -2, 1, 0 \rangle\,) &= L(\langle -1, 3, -3 \rangle) = \langle -1, 3 \rangle = \langle 1, 2 \rangle + \langle -2, 1 \rangle = L(\langle 1, 2, -3 \rangle) + L(\langle -2, 1, 0 \rangle) \\ \end{align*} $$
For any $\mathbf{v}$, $\mathbf{w}\in\mathbb{R}^3$, and any scalars $s$, $t\in\mathbb{R}$,
$$ \begin{align*} L(\,s\mathbf{v} + t\mathbf{w}\,) &= L(\,s\langle v_1,\,v_2,\,v_3 \rangle + t\langle w_1,\,w_2,\,w_3 \rangle\,) \\ &= L(\langle sv_1 + tw_1,\,sv_2+tw_2,\,sv_3+tw_3 \rangle) \\ &= \langle sv_1 + tw_1,\,sv_2+tw_2 \rangle \\ &= s\langle v_1,\,v_2\rangle + t\langle w_1,\,w_2\rangle \\ &= sL(\langle v_1,\,v_2,\,v_3 \rangle) + tL(\langle w_1,\,w_2,\,w_3 \rangle) \\ &= sL(\mathbf{v}) + tL(\mathbf{w}) \end{align*} $$
Notice that we show both linearity properties in one set of computations.
Angle bracket notation allows book publishers and students to save space by writing a column vector in a row. Angle brackets are preferred by physicists for quantum mechanics. A tedious element of our course is exact notation. Mathematicians usually prefer round brackets for everything. We will deal with a great number of vectors in $\mathbb{R}^n$, and these are usually denoted as columns of numbers
$$ \mathbf{v} = \begin{bmatrix} v_1 \\ \vdots\\ v_n \end{bmatrix} = \left( \begin{array}{c} v_1 \\ \vdots \\ v_n \end{array} \right) = \langle v_1,\ \dots,\ v_n \rangle $$
The advantage of angle bracket notation is that it fits on one line. Students can use any of these three notations for vectors. Everyone must pick a notation and stick with it for the duration of the exercise. Mixing notation tells me that you don't understand what you're writing.
Linear Transformations can dilate, constrict, rotate, reflect, shear, and orthogonally project a vector space. We will study all of these types of linear transformations in this course. They are all geometrical transformations that leave the origin fixed; that is $L(\mathbf{0})=\mathbf{0}$. A notable geometrical transformation missing from the list is a translation. It is missing because shifting the vector space to the right, left, or in the direction of translation vector $\mathbf{b}$ does not fix the origin.
Definition¶
Affine Transformation
An affine transformation is a composition of a linear transformation and a translation $T\,:\,\mathbb{R}^m\rightarrow\mathbb{R}^n$ defined by
$$ T(\mathbf{x}) = A\mathbf{x} + \mathbf{b} $$
The easiest way to solve problems involving translations is to move your coordinate axes so that the transformation is linear. Then solve the linear problem, and translate the answer to the original axes.
Example 3 - Online Avatar¶
A game player's avatar, Mika, resides in a virtual world maintained by the game server in world coordinates. This coordinate system is 2-dimensional because this is a simple game and avatars can move only parallel to the floor, not up or down. This coordinate system is absolute in the sense that movements of avatars and NPCs within the world have no effect on the coordinate system. Mika's current location is $M = (235,178)$. When the player rotates Mika $90^{\circ}$ counter-clockwise, the server performs updates to the virtual environment in world coordinates. The affine rotation computed by the game server about $M$ is
$$ T(\mathbf{x}) = R(\mathbf{x}-M) + M = R\mathbf{x} + (M - RM), \qquad R = \begin{bmatrix}\ \ 0\ & -1\ \\ \ \ 1\ &\ \ 0\ \end{bmatrix}. $$
The player's device, however, never works in world coordinates. It renders everything in local coordinates, where Mika sits at the origin. A point with world coordinate $\mathbf{x}$ has local coordinate $\mathbf{x} - M$, and conversely world $=$ local $+\, M$.
Suppose a companion drone hovers beside Mika. On the device it sits at local coordinate $\mathbf{p}_{\text{local}} = (40, 20)$, which the server stores as world coordinate $\mathbf{p} = M + (40,20) = (275, 198)$. The player presses rotate-left. Where does the drone end up?
On the server (world coordinates) the update is affine. First build the translation $\mathbf{b} = M - RM$:
$$ RM = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}\begin{bmatrix} 235 \\ 178 \end{bmatrix} = \begin{bmatrix} -178 \\ 235 \end{bmatrix}, \qquad \mathbf{b} = M - RM = \begin{bmatrix} 235 - (-178) \\ 178 - 235 \end{bmatrix} = \begin{bmatrix} 413 \\ -57 \end{bmatrix}. $$
Then apply $T$ to the drone:
$$ T(\mathbf{p}) = R\mathbf{p} + \mathbf{b} = \begin{bmatrix} -198 \\ 275 \end{bmatrix} + \begin{bmatrix} 413 \\ -57 \end{bmatrix} = \begin{bmatrix} 215 \\ 218 \end{bmatrix}. $$
On the device (local coordinates) the same update is linear. Mika is the origin, so the pivot is the origin, and the rotation is just $R$:
$$ R\,\mathbf{p}_{\text{local}} = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}\begin{bmatrix} 40 \\ 20 \end{bmatrix} = \begin{bmatrix} -20 \\ 40 \end{bmatrix}. $$
Converting that new local coordinate back to world coordinates, $M + (-20, 40) = (215, 218)$ — the same place the server computed.
The difference is the whole point. Test whether the server's world-coordinate map is linear by checking the origin:
$$ T(\mathbf{0}) = \mathbf{b} = (413, -57) \neq \mathbf{0}. $$
A linear map must fix the origin; $T$ does not, so $T$ is affine, not linear. The device's map is the pure rotation $R$, and $R(\mathbf{0}) = \mathbf{0}$, so it is linear. Nothing changed about the rotation itself — only the coordinate system. The world-to-local conversion $\mathbf{x} \mapsto \mathbf{x} - M$ is exactly the change of variables that moves the origin onto the rotation center, stripping the translation $\mathbf{b}$ away and leaving the linear core $R$. That is why the device can rotate Mika with a single matrix while the server has to carry the offset.
2.1.4 Matrix Arithmetic
The Algebra of Linear Maps¶
Definition¶
Matrices
An $m\times n$ matrix $A$ is an algebraic representation of a linear transformation, map, or function $A:\mathbb{R}^n\rightarrow\mathbb{R}^m$.
In pre-calculus mathematics one learns to algebraically combine functions by combining the outputs of the functions.
Definition¶
Algebra of Functions
If $f\,:\,D\rightarrow R$, and $g\,:\,D\rightarrow R$ are functions with domain $D$ and codomain $R$, then for all $x\in D$,
- $f + g\,:D\rightarrow R$ such that $(f+g)(x) = f(x) + g(x)$
- $f - g\,:D\rightarrow R$ such that $(f-g)(x) = f(x) - g(x)$
If $h\,:\,R\rightarrow S$ is a function with domain $R$ and codomain $S$ then, for all $x\in D$,
- $h\circ f\,:\,D\rightarrow S$ such that $(h\circ f)(x) = h\left(f(x)\right)$
Finally, if $t\in\mathbb{R}$ is a real scalar, for all $x\in R$,
- $th\,:\,R\rightarrow S$ such that $(th)(x) = t\cdot h(x)$
We want matrix addition and scalar multiplication to be motivated by this algebra of functions. In other words, if $L(\mathbf{v}) = A\mathbf{v}$ and $M(\mathbf{v}) = B\mathbf{v}$, and $t\in\mathbb{R}$ is a scalar, then we want
$$(L + M)(\mathbf{v}) = L(\mathbf{v}) + M(\mathbf{v}) = A\mathbf{v} + B\mathbf{v} = (A + B)\mathbf{v}$$
and
$$(tL)(\mathbf{v}) = t\cdot L(\mathbf{v}) = t\left(A\mathbf{v}\right) = (tA)\mathbf{v}$$
We use this motivation to derive the definitions of matrix addition and scalar multiplication. We want:
, and $t\in\mathbb{R}$ is a real scalar,
$$ \begin{align*} (A + B)\mathbf{v} &= (L + M)(\mathbf{v}) = L(\mathbf{v}) + M(\mathbf{v}) = A\mathbf{v} + B\mathbf{v} \\ \\ &= \left( v_1\mathbf{a}_1 + \cdots + v_n\mathbf{a}_n \right) + \left( v_1\mathbf{b}_1 + \cdots + v_n\mathbf{b}_n \right) \\ \\ &= v_1\left( \mathbf{a}_1 + \mathbf{b}_1 \right) + \cdots + v_n\left( \mathbf{a}_n + \mathbf{b}_n \right) \\ \\ &= v_1\begin{bmatrix} a_{11} + b_{11} \\ \vdots \\ a_{m1}+b_{m1} \end{bmatrix} + \cdots + v_n\begin{bmatrix} a_{1n} + b_{1n} \\ \vdots \\ a_{mn}+b_{mn} \end{bmatrix} \\ \\ &= \begin{bmatrix} a_{11}+b_{11} & \cdots & a_{1n}+b_{1n} \\ \vdots & \ddots & \vdots \\ a_{m1} + b_{m1} & \cdots & a_{mn} + b_{mn} \end{bmatrix} \mathbf{v} \end{align*} $$
Typing or writing the tedious pattern of elements $a_{ij}+b_{ij}$ for all $m\cdot n$ elements of $A+B$ can be illustrated more succinctly by simply denoting the pattern. One denotes an $m\times n$ matrix in tedious computations by
$$A = \begin{bmatrix} a_{ij} \end{bmatrix}$$
If $A$ is an $m\times n$ matrix, or $A\in\mathbb{R}^{m\times n}$, then one also implies that $1\le i\le m$, and $1\le j\le n$.
Definition¶
Matrix Addition
The sum of two $m\times n$ matrices $A$ and $B$ is defined by
$$A + B = \begin{bmatrix} a_{ij} \end{bmatrix} + \begin{bmatrix} b_{ij} \end{bmatrix} = \begin{bmatrix} a_{ij} + b_{ij} \end{bmatrix}$$
In other words, the sum of two $m\times n$ matrices is performed element-wise; one adds each of the corresponding elements of the matrices.
The effort to learn the notation of vector and matrices will prove to be an excellent investment in this course.
Definition¶
Matrix-Scalar Multiplication
Scalar multiplication of an $m\times n$ matrix $A$ by a scalar $t\in\mathbb{R}$ is defined by
$$tA = t\begin{bmatrix} a_{ij} \end{bmatrix} = \begin{bmatrix} ta_{ij} \end{bmatrix}$$
Exercise 2 - A Linear Combination of Matrices¶
Given matrices $A = \begin{bmatrix}\ \ 2\ &\ \ 2\ & -4\ \\ \ \ 3\ &\ \ 2\ &\ \ 0\ \\ -2\ & -4\ &\ \ 5\ \end{bmatrix}$, and $B = \begin{bmatrix} -2\ &\ \ 3\ &\ \ 2\ \\ \ \ 1\ & -3\ &\ \ 4\ \\ -3\ &\ \ 0\ &\ \ 5\ \end{bmatrix}$,
Compute the linear combination $D = 2A - 3B$
View Solution
$$ D = 2A - 3B = \begin{bmatrix}\ \ 4\ &\ \ 4\ & -8\ \\ \ \ 6\ &\ \ 4\ &\ \ 0\ \\ -4\ & -8\ &\ \ 10\ \end{bmatrix} - \begin{bmatrix} -6\ &\ \ 9\ &\ \ 6\ \\ \ \ 3\ & -9\ &\ \ 12\ \\ -9\ &\ \ 0\ &\ \ 15\ \end{bmatrix} = \begin{bmatrix}\ \ 10\ & -5\ & -14\ \\ \ \ 3\ &\ \ 13\ & -12\ \\ \ \ 5\ & -8\ & -5\ \end{bmatrix} $$ 2.1.5 Matrix-Vector Multiplication
Pre-calculus instruction in matrix-vector multiplication tends to focus on an algorithm for computing the result. These algorithms for computing products of row and column elements work, however they do not provide insight into the meaning of matrix-vector multiplication. Matrix-vector multiplication is a special case of matrix multiplication. In the case of matrix-vector multiplication, a linear map is applied to one vector. The result is the output of the linear map. So
$$\mathbf{y} = L(\mathbf{x})\longleftrightarrow\mathbf{y} = A\cdot\mathbf{x}.$$
Definition¶
Matrix-Vector multiplication
Multiplication of an $m\times n$ matrix $A$ on the right by an $n\times 1$ vector $\mathbf{x}$
yields a linear combination of the columns of matrix $A$ given by
$$ A\mathbf{x} = \begin{bmatrix} \mathbf{a}_1 & \mathbf{a}_2 & \cdots & \mathbf{a}_n \end{bmatrix}\,\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = x_1\mathbf{a}_1 + x_2\mathbf{a}_2 + \cdots + x_n\mathbf{a}_n. $$
Hence the product of an $m\times n$ matrix $A$ on the right by an $n\times1$ vector $\mathbf{x}$ is an $m\times1$ vector $y$
$$ \mathbf{y}=A\mathbf{x} $$
Multiplying a matrix on the right by a vector results in a linear combination of the columns of the matrix.
Notice that the definition uses the notation from Lecture Video 1. In the definition, a matrix $A$ is represented by its columns. The $j^{\text{th}}$ column of matrix $A$ is denoted in vector notation $\mathbf{a}_j$. Subscripts are used to denote a column vector because the index of the column is in the cellar.
$$A = \begin{bmatrix} \mathbf{a}_1 & \mathbf{a}_2 & \cdots & \mathbf{a}_n \end{bmatrix},$$
for $m\times n$ matrix $A$. In addition, vector $\mathbf{x}$ must have the same number of elements as matrix $A$ has columns to obtain the correct linear combination.
Example 4 - A Small Example¶
$$ \begin{align*} A\mathbf{x} &= \begin{bmatrix}\ \ 7\ &\ \ 4\ & -6\ \\ \ \ 6\ &\ \ 0\ &\ \ 3\ \\ \ \ 7\ &\ \ 1\ &\ \ 1\ \end{bmatrix}\,\begin{bmatrix}\ \ 1\ \\ -3\ \\ \ \ 2\ \end{bmatrix} = 1\begin{bmatrix}\ \ 7\ \\ \ \ 6\ \\ \ \ 7\ \end{bmatrix} + (-3)\begin{bmatrix}\ \ 4\ \\ \ \ 0\ \\ \ \ 1\ \end{bmatrix} + 2\begin{bmatrix} -6\ \\ \ \ 3\ \\ \ \ 1\ \end{bmatrix} \\ \\ &= \begin{bmatrix}\ \ 7\ \\ \ \ 6\ \\ \ \ 7\ \end{bmatrix} + \begin{bmatrix} -12\ \\ \ \ \ 0\ \\ \,-3\ \end{bmatrix} + \begin{bmatrix} -12\ \\ \ \ \ 6\ \\ \ \ \ 2\ \end{bmatrix} = \begin{bmatrix} -17\ \\ \ \ 12\ \\ \ \ 6\ \end{bmatrix} \end{align*} $$
Geometric understanding comes from the following matrix-vector multiplications
| Matrix Equation | Graph |
|---|---|
| $$ \begin{align*} A\ihat &= \begin{bmatrix}\ \ 7\ &\ \ 4\ & -6\ \\ \ \ 6\ &\ \ 0\ &\ \ 3\ \\ \ \ 7\ &\ \ 1\ &\ \ 1\ \end{bmatrix}\,\begin{bmatrix}\ \ 1\ \\ \ \ 0\ \\ \ \ 0\ \end{bmatrix} = \begin{bmatrix}\ \ 7\ \\ \ \ 6\ \\ \ \ 7\ \end{bmatrix} = \mathbf{a}_1 \\ \\ A\jhat &= \begin{bmatrix}\ \ 7\ &\ \ 4\ & -6\ \\ \ \ 6\ &\ \ 0\ &\ \ 3\ \\ \ \ 7\ &\ \ 1\ &\ \ 1\ \end{bmatrix}\,\begin{bmatrix}\ \ 0\ \\ \ \ 1\ \\ \ \ 0\ \end{bmatrix} = \begin{bmatrix}\ \ 4\ \\ \ \ 0\ \\ \ \ 1\ \end{bmatrix} = \mathbf{a}_2 \\ \\ A\khat &= \begin{bmatrix}\ \ 7\ &\ \ 4\ & -6\ \\ \ \ 6\ &\ \ 0\ &\ \ 3\ \\ \ \ 7\ &\ \ 1\ &\ \ 1\ \end{bmatrix}\,\begin{bmatrix}\ \ 0\ \\ \ \ 0\ \\ \ \ 1\ \end{bmatrix} = \begin{bmatrix} -6\ \\ \ \ 3\ \\ \ \ 1\ \end{bmatrix} = \mathbf{a}_3 \end{align*} $$ |
|
|
|
|
The linear function $L:\mathbb{R}^3\rightarrow\mathbb{R}^3$ defined by $L(\mathbf{x})=A\mathbf{x}$ maps all of the vectors within the unit cube (the cube with sides $\ihat$, $\jhat$, and $\khat$) onto the parallelepiped with sides $A\ihat$, $A\jhat$, and $A\khat$
$$ \begin{align*} A\mathbf{x} &= A\left( x_1\ihat + x_2\jhat + x_3\khat \right) \\ \\ &= x_1A\ihat + x_2A\jhat + x_3A\khat = x_1\mathbf{a}_1 + x_2\mathbf{a}_2 + x_3\mathbf{a}_3 \end{align*} $$ |
|
In the figures, we have a linear transformation $L:\mathbb{R}^3\rightarrow\mathbb{R}^3$. The input of linear transformation $L$ is a $3$-dimensional vector, and the output is a $3$-dimensional vector. For every vector $\mathbf{v}\in\mathbb{R}^3$,
$$\mathbf{w} = L(\mathbf{v})$$
is the output vector $\mathbf{w}\in\mathbb{R}^3$. In order to connect this familiar function notation of a linear transformation with the matrix $A$ one considers the outputs of the unit vectors $\ihat$, $\jhat$, and $\khat$.
$$ \begin{align*} L(\ihat) &= L\left( \begin{bmatrix}\ \ 1\ \\ \ \ 0\ \\ \ \ 0\ \end{bmatrix} \right) = \begin{bmatrix}\ \ 7\ \\ \ \ 6\ \\ \ \ 7\ \end{bmatrix} \\ L(\jhat) &= L\left( \begin{bmatrix}\ \ 0\ \\ \ \ 1\ \\ \ \ 0\ \end{bmatrix} \right) = \begin{bmatrix}\ \ 4\ \\ \ \ 0\ \\ \ \ 1\ \end{bmatrix} \\ L(\khat) &= L\left( \begin{bmatrix}\ \ 0\ \\ \ \ 0\ \\ \ \ 1\ \end{bmatrix} \right) = \begin{bmatrix} -6\ \\ \ \ 3\ \\ \ \ 1\ \end{bmatrix} \\ \end{align*} $$
Any input vector $\mathbf{v}\in\mathbb{R}^3$ is a linear combination of $\ihat$, $\jhat$, and $\khat$;
$$\mathbf{v} = v_1\ihat + v_2\jhat + v_3\khat$$
Since $L$ is a linear transformation,
$$ \begin{align*} L(\mathbf{v}) &= L(v_1\ihat + v_2\jhat + v_3\khat) \\ \\ &= v_1L(\ihat) + v_2L(\jhat) + v_3L(\khat) \\ \\ &= v_1\begin{bmatrix}\ \ 7\ \\ \ \ 6\ \\ \ \ 7\ \end{bmatrix} + v_2\begin{bmatrix}\ \ 4\ \\ \ \ 0\ \\ \ \ 1\ \end{bmatrix} + v_3\begin{bmatrix} -6\ \\ \ \ 3\ \\ \ \ 1\ \end{bmatrix} \\ \\ &= A\mathbf{v} \end{align*} $$
Example 5 - Matrix Representation¶
Construct the matrix $A$ that represents the linear transformation $L\,:\,\mathbb{R}^3\rightarrow\mathbb{R}^3$ defined by
$$L(\mathbf{x}) = 3\mathbf{x}$$
Recall that every vector $\mathbf{x}\in\mathbb{R}^3$ can be written in terms of the unit vectors,
$$\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = x_1\ihat + x_2\jhat + x_3\khat$$
Since $L$ is a linear map, that means,
$$L(\mathbf{x}) = L(x_1\ihat + x_2\jhat + x_3\khat) = x_1L(\ihat) + x_2L(\jhat) + x_3L(\khat)$$
By packing the results of $\mathbf{a}_1 = L(\ihat)$, $\mathbf{a}_2 = L(\jhat)$, and $\mathbf{a}_3 = L(\khat)$ into matrix $A$, we have
$$L(\mathbf{x}) = x_1L(\ihat) + x_2L(\jhat) + x_3L(\khat) = x_1\mathbf{a}_1 + x_2\mathbf{a}_2 + x_3\mathbf{a}_3 = A\mathbf{x}$$
$$L(\mathbf{v}) = A\mathbf{v}$$
Hence
$$\mathbf{a}_1 = L(\ihat) = \langle 3,\ 0,\ 0 \rangle,\quad \mathbf{a}_2 = L(\jhat) = \langle 0,\ 3,\ 0 \rangle,\quad \mathbf{a}_3 = L(\khat) = \langle 0,\ 0,\ 3 \rangle $$
and
$$A = \begin{bmatrix}\ 3\ &\ 0\ &\ 0\ \\ \ 0\ &\ 3\ &\ 0\ \\ \ 0\ &\ 0\ &\ 3\ \end{bmatrix}$$
In this way, the product of matrix $A$ and vector $\mathbf{x}$ is the application of linear map $L$ to input $\mathbf{x}$. The arithmetic formulation results from the fact that $L$ is a linear map.
Exercise 3 - Matrix Representation¶
Determine the matrix $B$ that represents the linear transformation $M\,:\,\mathbb{R}^3\rightarrow\mathbb{R}^4$ defined by
$$M\left(\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} \right) = \begin{bmatrix} -x_2 \\ x_1 \\ x_1+x_2 \\ x_1-x_2 \end{bmatrix}$$
View Solution
$$M(\ihat) = \begin{bmatrix}\ \ 0\ \\ \ \ 1\ \\ \ \ 1\ \\ \ \ 1\ \end{bmatrix}, \quad M(\jhat) = \begin{bmatrix} -1\ \\ \ \ 0\ \\ \ \ 1\ \\ -1\ \end{bmatrix}, \quad M(\khat) = \begin{bmatrix}\ \ 0\ \\ \ \ 0\ \\ \ \ 0\ \\ \ \ 0\ \end{bmatrix}$$$$B = \begin{bmatrix}\ \ 0\ & -1\ &\ \ 0\ \\ \ \ 1\ &\ \ 0\ &\ \ 0\ \\ \ \ 1\ &\ \ 1\ &\ \ 0\ \\ \ \ 1\ & -1\ &\ \ 0\ \end{bmatrix}$$
Exercise 4 - Rotation Matrix¶
Construct the matrix representation $R$ of the linear map on $\mathbb{R}^2$ that rotates every vector in standard position $90^{\circ}$ counter-clockwise.
View Solution
A map on $\mathbb{R}^2$ means that the domain and codomain are both $\mathbb{R}^2$, and the matrix is a square matrix. We need to remember our trigonometry of the unit circle. If we rotate $\ihat$ by $90^{\circ}$ counter-clockwise we get $\jhat$. If we rotate $\jhat$ $90^{\circ}$ counter-clockwise we get $-\ihat$, so$$L(\ihat) = \jhat,\qquad L(\jhat) = -\ihat$$
$$R = \begin{bmatrix}\ \ 0\ & -1\ \\ \ \ 1\ &\ \ 0\ \end{bmatrix}$$
2.1.6 Understanding Matrix Multiplication
Multiplication accomplishes a change in scale. One may stretch (dilate) a vector by $3$ or shrink (contract) it by $2$ (stretch by $\frac{1}{2}$). If we perform two scalings in a row like "stretch by $3$ and then shrink the result by $2$", we obtain a scaling of $\dfrac{3}{2}$. The definition of scalar multiplication affords our understanding of change in scale.
Matrix multiplication also needs to accommodate a change in geometry and its scale. Like scalar multiplication we need to apply one linear map and then another. What is the resulting change in geometry? Repeated application of one function after another is called function composition.
Our goal is to use the matrix representations $A$ and $B$ of linear maps $L\,:\,\mathbb{R}^n\rightarrow\mathbb{R}^m$ and $M\,:\,\mathbb{R}^p\rightarrow\mathbb{R}^n$ to compute the matrix $C$ that represents the composition of these maps $L\circ M$. Happily our definition of matrix-vector multiplication gives us a succinct formula for matrix product that achieves this goal.
We will study 5 (Five!) different formulations or perspectives about matrix multiplication. This is the first one because it is the most important.
Video Lecture 3: Matrix Multiplication and Composition.¶
Matrix MultiplicationThe product of two matrices is a linear transformation. It is the linear transformation that composes the two factor matrices. If $m\times n$ matrix $A$ represents linear transformation $L\,:\,\mathbb{R}^n\rightarrow\mathbb{R}^m$, and $n\times p$ matrix $B$ represents linear transformation $M\,:\,\mathbb{R}^p\rightarrow\mathbb{R}^n$, then $L\circ M\,:\,\mathbb{R}^p\rightarrow\mathbb{R}^m$ so that for any vector $\mathbf{x}\in\mathbb{R}^p$, we have
$$ (L\circ M)(\mathbf{x}) = L\left(M(\mathbf{x})\right) = L\left( B\mathbf{x} \right) = A(B\mathbf{x}) = (AB)\mathbf{x} $$
Like function composition, which is defined only when the codomain of the function on the right of the operator is precisely the domain of the function on the left, matrix product is defined only when the number of rows of the matrix on the right matches the number of columns of the matrix on the left. This directly results from the relationship between a matrix and the linear map it represents. An $m\times n$ matrix represents a linear map from domain $\mathbb{R}^n$ to codomain $\mathbb{R}^m$.
Video Lecture 4: Visualizing Matrix Multiplication.¶
Visualizing Matrix Multiplication Only study the first 21 minutes of Dr. Strang's lecture. He shows us four different mathematical visualizations of matrix multiplication. We will study the rest of the video in the section 2.3.
How do we define matrix multiplication so that matrix product represents function composition?
Definition¶
Matrix Multiplication
The product of two matrices is a new matrix that represents the function composition of the factor matrices.
$$ C = AB $$
means that matrix $C$ and matrix $AB$ represent the same linear transformation.
For $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{n\times p}$ the matrix product of $A$ and $B$ is the matrix $C\in\mathbb{R}^{m\times p}$ so that
$$ C = AB = A\begin{bmatrix} \mathbf{b}_1 & \mathbf{b}_2 & \cdots & \mathbf{b}_p \end{bmatrix} = \begin{bmatrix} A\mathbf{b}_1 & A\mathbf{b}_2 & \cdots & A\mathbf{b}_p \end{bmatrix} $$
We simply repeat a matrix-vector multiplication $p$ times.
Example 6 - Matrix Multiplication¶
Let us compute the matrix product
$$ C = AB = \begin{bmatrix}\ \ 8 &\ \ 2\ &\ \ 1\ \\ \ \ 9\ &\ \ 1\ & -4\ \\ -8 &\ \ 0 & -2\ \\ \ 10 &\ \ 0 &\ \ 1\ \end{bmatrix}\,\begin{bmatrix}\ \ 1 & -1 &\ \ 0\ \\ \ \ 0 &\ \ 0 &\ \ 1\ \\ \ \ 1 &\ \ 1&\ \ 0\ \end{bmatrix} $$
Reading the matrix $B$, the matrix on the right, the first column of our product matrix is $A\mathbf{b}_1$, which specifies, "the first column plus the third column",
$$\mathbf{c}_1 = A\mathbf{b}_1 = \begin{bmatrix}\ \ 9\ \\ \ \ 5\ \\ -10\ \\ \ 11\ \end{bmatrix}.$$
Similarly, the second column of the product prescribes, "the third column minus the first column",
$$\mathbf{c}_2 = A\mathbf{b}_2 = \begin{bmatrix} -7\ \\ -13\ \\ \ \ 6\ \\ -9\ \end{bmatrix}.$$
Likewise, the last column of the product requires that the third column of the product is "the second column",
$$\mathbf{c}_3 = A\mathbf{b}_3 = \begin{bmatrix}\ 2 \\ \ 1 \\ \ 0 \\ \ 0\ \end{bmatrix}.$$
Hence
$$AB = \begin{bmatrix}\ \ 9 & -7 &\ \ 2\ \\ \ \ 5 & -13\ &\ \ 1\ \\ -10 &\ \ 6 &\ \ 0\ \\ \ \ 11 & -9\ &\ \ 0\ \end{bmatrix}.$$
Exercise 5 - Matrix Multiplication¶
Let $A = \begin{bmatrix}\ \ 3\ &\ \ 1\ & -2\ & -2\ \\ \ \ 5\ &\ \ 0\ & -4\ &\ \ 0\ \\ -4\ & -5\ &\ \ 3\ & -4\ \end{bmatrix}$ and $B = \begin{bmatrix}\ \ 5\ &\ \ 5\ \\ -4\ & -5\ \\ \ \ 4\ & -1\ \\ \ \ 0\ & -4\ \end{bmatrix}$. Compute $C = AB$.
View Solution
$$ \begin{align*} \mathbf{c}_1 = A\mathbf{b}_1 &= \begin{bmatrix}\ \ 3\ &\ \ 1\ & -2\ & -2\ \\ \ \ 5\ &\ \ 0\ & -4\ &\ \ 0\ \\ -4\ & -5\ &\ \ 3\ & -4\ \end{bmatrix}\begin{bmatrix}\ \ 5\ \ &\ 5\ \\ -4\ & -5\ \\ \ \ 4\ & -1\ \\ \ \ 0\ & -4\ \end{bmatrix} = \begin{bmatrix}\ \ 3\ \\ \ \ 9\ \\ \ \ 12\ \end{bmatrix} \\ \mathbf{c}_2 = A\mathbf{b}_2 &= \begin{bmatrix}\ \ 3\ &\ \ 1\ & -2\ & -2\ \\ \ \ 5\ &\ \ 0\ & -4\ &\ \ 0\ \\ -4\ & -5\ &\ \ 3\ & -4\ \end{bmatrix}\begin{bmatrix}\ \ 5\ &\ \ 5\ \\ -4\ & -5\ \\ \ \ 4\ & -1\ \\ \ \ 0\ & -4\ \end{bmatrix} = \begin{bmatrix}\ \ 20\ \\ \ \ 29\ \\ \ \ 18\ \end{bmatrix} \\ C = AB &= \begin{bmatrix}\ \ 3\ &\ \ 20\ \\ \ \ 9\ &\ \ 29\ \\ \ \ 12\ &\ \ 18\ \end{bmatrix} \end{align*} $$In mathematics, a dual relationship occurs when shifting your perspective reveals a symmetric, reciprocal relationship between two mathematical objects. We have been representing vectors like $\mathbf{v}\in\mathbb{R}^n$ as column vectors,
$$\mathbf{v} = \begin{bmatrix} v_1 \\ \vdots \\ v_n \end{bmatrix}.$$
This was an arbitrary choice. Notice that a column vector is also an $n\times 1$ matrix. This matrix represents a linear map $V\,:\,\mathbb{R}^1\rightarrow\mathbb{R}^n$ defined for every scalar $t\in\mathbb{R}$ by
$$V(t) = t\mathbf{v}.$$
(Notice that the graph of $V$ is the span of $\mathbf{v}$ in $\mathbb{R}^n$). Could we lay the entries of $\mathbf{v}$ out horizontally and treat the result as a $1 \times n$ matrix?
$$\left[\, v_1 \ \ v_2 \ \cdots \ v_n \,\right]$$
This is also a matrix, and it also represents a linear map — but in the opposite direction. A $1 \times n$ matrix takes an input from $\mathbb{R}^n$ and produces an output in $\mathbb{R}^1$. Using matrix-vector multiplication as we defined it in Section 2.1.5,
$$\left[\, v_1 \ \ v_2 \ \cdots \ v_n \,\right] \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = v_1 x_1 + v_2 x_2 + \cdots + v_n x_n.$$
The column-vector form of $\mathbf{v}$ represents the map $t \mapsto t\mathbf{v}$ that builds scalar multiples of $\mathbf{v}$. The row-vector form represents a dual map that takes an arbitrary vector $\mathbf{x}$ and combines its entries against the entries of $\mathbf{v}$. Each form is a different perspective on the same underlying data $v_1$, $\dots$, $v_n$. Does this mean we can find a formula for matrix multiplication as a linear combination of rows?
2.1.7 Dual Matrix Multiplication
When considering matrix-vector multiplication $A\mathbf{b}$, what if matrix $A$ is a $1\times n$ matrix and $\mathbf{b}$ is $n\times1$? What does matrix-vector multiplication look like? If matrix $A$ has only one row then each column of matrix $A$ is a $1\times 1$ column vector.
$$ A\mathbf{x} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n}\end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = \begin{bmatrix} a_1 & a_2 & \cdots & a_n \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = a_1x_1 + a_2x_2 + \cdots + a_nx_n. $$
If we multiply a $1\times n$ matrix $A$ by an $n\times p$ matrix $B$ we have
$$ AB = \begin{bmatrix} A\mathbf{b}_1 & A\mathbf{b}_2 & \cdots & A\mathbf{b}_p \end{bmatrix} $$
For each $1\le j\le p$,
$$ A\mathbf{b}_j = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \end{bmatrix} \begin{bmatrix} b_{1j} \\ b_{2j} \\ \vdots \\ b_{nj} \end{bmatrix} = a_{11}b_{1j} + a_{12}b_{2j} + \cdots + a_{1n}b_{nj} $$
Showing all columns of the product,
$$ \begin{align*} AB &= \begin{bmatrix} a_{11}b_{11} + a_{12}b_{21} + \cdots + a_{1n}b_{n1} & \cdots & a_{11}b_{1j} + a_{12}b_{2j} + \cdots + a_{1n}b_{nj} & \cdots & a_{11}b_{1p} + a_{12}b_{2p} + \cdots + a_{1n}b_{np} \end{bmatrix} \\ \\ &= \begin{bmatrix} {\color{crimson}a_{11}b_{11}} + a_{12}b_{21} + \cdots + a_{1n}b_{n1} & \cdots & {\color{crimson}a_{11}b_{1j}} + a_{12}b_{2j} + \cdots + a_{1n}b_{nj} & \cdots & {\color{crimson}a_{11}b_{1p}} + a_{12}b_{2p} + \cdots + a_{1n}b_{np} \end{bmatrix} \\ \\ &= {\color{crimson}\begin{bmatrix} a_{11}b_{11} & \dots & a_{11}b_{1p} \end{bmatrix}} + \begin{bmatrix} a_{12}b_{21} & \cdots & a_{12}b_{2p} \end{bmatrix} + \cdots + \begin{bmatrix} a_{1n}b_{n1} & \cdots & a_{1n}b_{np} \end{bmatrix} \\ \\ &= {\color{crimson}a_{11}\begin{bmatrix} b_{11} & \dots & b_{1p} \end{bmatrix}} + a_{12}\begin{bmatrix} b_{21} & \cdots & b_{2p} \end{bmatrix} + \cdots + a_{1n}\begin{bmatrix} b_{n1} & \cdots & b_{np} \end{bmatrix} \\ \\ &= {\color{crimson}a_{11}b^{1}} + a_{12}b^2 + \cdots + a_{1n}b^n \end{align*} $$
The last equality stems from the fact that for $1\le k\le n$, $\begin{bmatrix} b_{k1} & b_{k2} & \dots & b_{kp} \end{bmatrix}$ is the $k^{\text{th}}$ row of matrix $B$, and is denoted with a roof index, $\mathbf{b}^k$.
Definition¶
Vector-Matrix Multiplication
For any $1\times n$ row vector $\mathbf{v}$ and $n\times p$ matrix $B$, the vector-matrix product is defined by
$$ \mathbf{v}B := v_{11}\mathbf{b}^1 + v_{12}\mathbf{b}^2 + \cdots + v_{1n}\mathbf{b}^n $$
Multiplying a matrix on the left by a row vector results in a linear combination of the rows of the matrix!
Example 7 - Vector-Matrix Multiplication¶
$$ \begin{align*} \begin{bmatrix} -2 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & 3 & 0 \\ 2 & -1 & 7 \\ 2 & 1 & 0 \end{bmatrix} &= \begin{bmatrix} -2 & 0 & 1 \end{bmatrix}\begin{bmatrix} \mathbf{b}^1 \\ \mathbf{b}^2 \\ \mathbf{b}^3 \end{bmatrix} = -2\mathbf{b}^1 + 0\mathbf{b}^2 + \mathbf{b}^3 \\ \\ &= -2\begin{bmatrix} 1 & 3 & 0 \end{bmatrix} + 0\begin{bmatrix} 2 & -1 & 7 \end{bmatrix} + 1\begin{bmatrix} 2 & 1 & 0 \end{bmatrix} \\ \\ &= \begin{bmatrix} -2 & -6 & 0 \end{bmatrix} + \begin{bmatrix} 2 & 1 & 0 \end{bmatrix} = \begin{bmatrix} 0 & -5 & 0 \end{bmatrix} \\ \\ &= \begin{bmatrix} 0 & -5 & 0 \end{bmatrix}. \end{align*} $$
We can describe this as an elementary row operation
We can think of a matrix as a column of row vectors.
$$ A = \begin{bmatrix} \mathbf{a}^1 \\ \mathbf{a}^2 \\ \vdots \\ \mathbf{a}^m \end{bmatrix} $$
Definition¶
Dual Form of Matrix Multiplication
If $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{n\times p}$, then the product matrix $C\in\mathbb{R}^{m\times p}$ can also be given by
$$ C = AB = \begin{bmatrix} \mathbf{a}^1 \\ \mathbf{a}^2 \\ \vdots \\ \mathbf{a}^m \end{bmatrix}B = \begin{bmatrix} \mathbf{a}^1B \\ \mathbf{a}^2B \\ \vdots \\ \mathbf{a}^mB \end{bmatrix}, $$
where the $k^{\text{th}}$ row of the product is the product of a row vector $\mathbf{a}^k$ times matrix $B$. When something like this happens in mathematics, we call it duality.
Example 8 - Dual Matrix Multiplication¶
Let $A = \begin{bmatrix}\ \ 1\ &\ \ 0\ &\ \ 0\ \\ \ \ 4\ &\ \ 1\ &\ \ 0\ \\ \ \ 3\ & -2\ &\ \ 1\ \end{bmatrix}$ and $B = \begin{bmatrix} -4\ &\ \ 4\ & -1\ \\ \ \ \ 3\ &\ \ 0\ &\ \ 4\ \\ \ \ \ 0\ &\ \ 0\ & -3\ \end{bmatrix}$.
$$ \begin{align*} C &= AB = \begin{bmatrix} \mathbf{a}^1 \\ \mathbf{a}^2 \\ \mathbf{a}^3 \end{bmatrix}B = \begin{bmatrix} \mathbf{a}^1B \\ \mathbf{a}^2B \\ \mathbf{a}^3B \end{bmatrix} \\ \\ \mathbf{a}^1 B &= \begin{bmatrix}\ \ 1\ &\ \ 0\ &\ \ 0\ \end{bmatrix}B = \begin{bmatrix} -4\ &\ \ 4\ & -1\ \end{bmatrix} \\ \\ \mathbf{a}^2 B &= \begin{bmatrix}\ \ 4\ &\ \ 1\ &\ \ 0\ \end{bmatrix}B = \begin{bmatrix} -13\ &\ 16\ &\ \ 0\ \end{bmatrix} \\ \\ \mathbf{a}^3B &= \begin{bmatrix}\ \ 3\ & -2\ &\ \ 1\ \end{bmatrix}B = \begin{bmatrix} -18\ &\ 12\ & -14\ \end{bmatrix} \\ \\ C &= \begin{bmatrix} -4\ &\ \ 4\ & -1\ \\ -13\ &\ 16\ &\ \ \ 0\ \\ -18\ &\ 12\ & -14\ \end{bmatrix} \end{align*} $$
Exercise 6 - Dual Matrix Multiplication¶
Let $A = \begin{bmatrix}\ \ 1\ &\ \ 1\ & -5\ & -3\ \\ \ \ 1\ & -2\ & -3\ & -1\ \\ -4\ &\ \ 0\ & -4\ & -5\ \\ \ \ 4\ & -1\ & -3\ &\ \ 4\ \end{bmatrix}$ and $E_{21} = \begin{bmatrix}\ \ 1\ &\ 0\ &\ 0\ &\ 0\ \\ -1\ &\ 1\ &\ 0\ &\ 0\ \\ \ \ 0\ &\ 0\ &\ 1\ &\ 0\ \\ \ \ 0\ &\ 0\ &\ 0\ &\ 1\ \end{bmatrix}$. Compute $E_{21}A$.
View Solution
$$ E_{21}A = \begin{bmatrix}\ \ 1\ &\ 0\ &\ 0\ &\ 0\ \\ -1\ &\ 1\ &\ 0\ &\ 0\ \\ \ \ 0\ &\ 0\ &\ 1\ &\ 0\ \\ \ \ 0\ &\ 0\ &\ 0\ &\ 1\ \end{bmatrix}\begin{bmatrix}\ \ 1\ &\ \ 1\ & -5\ & -3\ \\ \ \ 1\ & -2\ & -3\ & -1\ \\ -4\ &\ \ 0\ & -4\ & -5\ \\ \ \ 4\ & -1\ & -3\ &\ \ 4\ \end{bmatrix} = \begin{bmatrix}\ \ 1\ &\ \ 1\ & -5\ & -3\ \\ \ \ 0\ & -3\ &\ \ 2\ &\ \ 2\ \\ -4\ &\ \ 0\ & -4\ & -5\ \\ \ \ 4\ & -1\ & -3\ &\ \ 4\ \end{bmatrix} $$Exercise 7 - Elementary Matrix Multiplication¶
Let $B = \begin{bmatrix}\ \ 1\ &\ \ 1\ & -5\ & -3\ \\ \ \ 0\ & -3\ &\ \ 2\ &\ \ 2\ \\ -4\ &\ \ 0\ & -4\ & -5\ \\ \ \ 4\ & -1\ & -3\ &\ \ 4\ \end{bmatrix}$. Construct a matrix $E_{31}$ that performs a type III row operation on matrix $B$. $E_{31}$ must add $4$ times row one to row three when the product $E_{31}B$ is computed. Compute $E_{31}B$ to show that the row operation is performed properly in the product.
View Solution
$$ \begin{align*} E_{31} &= \begin{bmatrix}\ 1\ &\ 0\ &\ 0\ &\ 0\ \\ \ 0\ &\ 1\ &\ 0\ &\ 0\ \\ \ 4\ &\ 0\ &\ 1\ &\ 0\ \\ \ 0\ &\ 0\ &\ 0\ &\ 1\ \end{bmatrix} \\ \\ E_{31}B &= \begin{bmatrix}\ 1\ &\ 0\ &\ 0\ &\ 0\ \\ \ 0\ &\ 1\ &\ 0\ &\ 0\ \\ \ 4\ &\ 0\ &\ 1\ &\ 0\ \\ \ 0\ &\ 0\ &\ 0\ &\ 1\ \end{bmatrix}\begin{bmatrix}\ \ 1\ &\ \ 1\ & -5\ & -3\ \\ \ \ 0\ & -3\ &\ \ 2\ &\ \ 2\ \\ -4\ &\ \ 0\ & -4\ & -5\ \\ \ \ 4\ & -1\ & -3\ &\ \ 4\ \end{bmatrix} \\ \\ &= \begin{bmatrix}\ \ 1\ &\ \ 1\ & -5\ & -3\ \\ \ \ 0\ & -3\ &\ \ 2\ &\ \ 2\ \\ \ \ 0\ &\ \ 4\ & -24\ & -17\ \\ \ \ 4\ & -1\ & -3\ &\ \ 4\ \end{bmatrix} \end{align*} $$ 2.1.8 Inner Product Form of Matrix Multiplication
In addition to the views of matrix multiplication as a row of matrix-vector multiplications or a column of vector-matrix multiplications, we can look at each element of the product matrix. This is generally the method taught in a pre-calculus course.
Definition¶
Inner Product Form of Matrix Multiplication
$$ \begin{align*} AB = C &= \left[ c_{ij} \right] \\ \\ &= \begin{bmatrix} \mathbf{a}^1 \\ \mathbf{a}^2 \\ \vdots \\ \mathbf{a}^m \end{bmatrix}\begin{bmatrix} \mathbf{b}_1 & \mathbf{b}_2 & \cdots & \mathbf{b}_p \end{bmatrix} \\ \\ &= \begin{bmatrix} \mathbf{a}^i\mathbf{b}_j \end{bmatrix} \\ \end{align*} $$
Each element $c_{ij}$ of the product matrix $C$ is the matrix product of a $1\times n$ row vector $\mathbf{a}^i$ and an $n\times 1$ column vector $\mathbf{b}_j$.
$$ \left[ c_{ij} \right] = \left[ \mathbf{a}^i\mathbf{b}_j \right] = \sum_{k=1}^n a_{ik}b_{kj} $$
Example 9 - Inner Product Matrix Multiplication¶
Let $A = \begin{bmatrix}\ \ 2\ & -1\ \\ -1\ &\ \ 4\ \end{bmatrix}$ and $B = \begin{bmatrix} -1\ &\ \ 0\ \\ \ 11\ & -14\ \end{bmatrix}$
Then
$$ \begin{align*} c_{11} &= \mathbf{a}^1\mathbf{b}_1= \begin{bmatrix}\ \ 2\ & -1 \end{bmatrix}\begin{bmatrix} -1\ \\ \ \ 11\ \end{bmatrix} = -13 \\ \\ c_{21} &= \mathbf{a}^2\mathbf{b}_1 = \begin{bmatrix} -1\ &\ \ 4\ \end{bmatrix}\begin{bmatrix} -1\ \\ \ \ 11\ \end{bmatrix} = 45 \\ \\ c_{12} &= \mathbf{a}^1\mathbf{b}_2 = \begin{bmatrix}\ \ 2\ & -1 \end{bmatrix}\begin{bmatrix}\ \ 0\ \\ -14\ \end{bmatrix} = 14 \\ \\ c_{22} &= \mathbf{a}^2\mathbf{b}_2 = \begin{bmatrix} -1\ &\ \ 4\ \end{bmatrix}\begin{bmatrix}\ \ 0\ \\ -14\ \end{bmatrix} = -56 \\ \\ C &= \begin{bmatrix} -13\ &\ \ 14\ \\ \ \ 45\ & -56\ \end{bmatrix} \end{align*} $$
The inner product form expresses $c_{ij}$ as the inner product of row $i$ of $A$ and column $j$ of $B$. The product of a row vector and a column vector as in inner product matrix multiplication that produces each element of the product matrix is called an inner product.
$$ \mathbf{a}^i\mathbf{b}_j $$
This product is called an inner product because the result is a scalar that allows formal definitions of the geometric relationships between the vectors, such as lengths and angles. We will study this geometry in chapter 5.
Exercise 8 - Diagonal Products¶
Given matrices $A = \begin{bmatrix} -3\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 2\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ & -3\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ & -4\ \end{bmatrix}$, $B = \begin{bmatrix}\ \ 1\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 3\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ & -2\ \end{bmatrix}$, and $C = \begin{bmatrix}\ \ 2\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ & -1\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 1\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ & -1\ \end{bmatrix}$
Compute:
(a) $AB = $
(b) $AC = $
(c) $B^5 = $
(d) $C^{10} = $
(e) $C^n = $ for any positive integer $n$?
View Solution
(a) $AB = \begin{bmatrix} -3\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ & -9\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 8\ \end{bmatrix}$(b) $AC = \begin{bmatrix} -6\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ & -2\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ & -3\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 4\ \end{bmatrix}$
(c) $B^5 = \begin{bmatrix}\ \ 1\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 243\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ & -32\ \end{bmatrix}$
(d) $C^{10} = \begin{bmatrix}\ 1024\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 1\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 1\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 1\ \end{bmatrix}$
(e) $C^n = \begin{bmatrix}\ \ 2^n\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ & (-1)^n\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 1\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ & (-1)^n\ \end{bmatrix}$
This is the formula students typically meet first. It is correct, but it's only one of five equivalent views, and not always the most useful. This applies especially when people perform multiplications without a computing device. Computing a matrix product takes $O(n^3)$ computations (roughly $n^3$ multiplications and additions for $n\times n$ matrices). Matrix-vector and vector-matrix views often expose structure that lets you compute a particular row, column, or block of the product without doing all $n^3$ operations.
2.1.9 Outer Products
The outer product is the matrix product of a row vector and a column vector. Suppose that $A$ is an $m\times n$ matrix and $B$ is an $n\times p$ matrix. In matrix multiplication these outer products are given by,
$$ \mathbf{a}_i\mathbf{b}^j $$
Since each column $\mathbf{a}_i$ of an $m\times n$ matrix is an $m\times 1$ column vector, and every row $\mathbf{b}^j$ of an $n\times p$ matrix is a $1\times p$ row vector, the matrix product
$$ \mathbf{a}_i\mathbf{b}^j $$
is an $m\times p$ matrix. This is the shape of our matrix product. $C = AB$ is an $m\times p$ matrix when $A$ is $m\times n$ and $B$ is $n\times p$. Recall that the inner product formula for matrix product tells us that every element of the $m\times p$ result $C$ has entries
$$[c_{ij}] = \left[ \sum_{k=1}^n a_{ik}b_{kj} \right] = \left[ \mathbf{a}^i\mathbf{b}_j \right].$$
Using a derivation much like the one in Section 2.1.7, each of the terms in the elements of the product are also terms in one of the $n$ outer products $\mathbf{a}_k\mathbf{b}^k$. This yields yet another formula for the product of two matrices
Definition¶
Outer Form of Matrix Multiplication
Given an $m\times n$ matrix $A$ and an $n\times p$ matrix $B$ the product $C = AB$ is given by
$$C = \sum_{k=1}^n \mathbf{a}_k\mathbf{b}^k$$
Instead of each element $c_{ij}$ written as a sum of $n$ inner products, we now have the product matrix $C$ written as a sum of $n$ matrices. Each of these $n\times p$ matrices is an outer product. This is the dual of the inner product formula.
This outer product formula will be useful in chapter 8 when we study the singular value decomposition and spectral decomposition. Outer products are used in Machine Learning, Neural Networks, Data Compression, and Signal Processing.
Exercise 9 - Outer Product Multiplication¶
Given $E = \begin{bmatrix}\ \ 1\ &\ \ 0\ &\ \ 0\ \\ -1\ &\ \ 1\ &\ \ 0\ \\ \ \ 1\ &\ \ 0\ &\ \ 1\ \end{bmatrix}$ and $A = \begin{bmatrix}\ \ 2\ &\ \ 3\ & -2\ \\ \ \ 2\ &\ \ 4\ &\ \ 0\ \\ -2\ & -3\ &\ \ 3\ \end{bmatrix}$, compute the matrix product $EA$ using the outer product formula.
View Solution
$$ \begin{align*} \mathbf{e}_1\mathbf{a}^1 &= \begin{bmatrix}\ \ 2\ &\ \ 3\ & -2\ \\ -2\ & -3\ &\ \ 2\ \\ \ \ 2\ &\ \ 3\ & -2\ \end{bmatrix} \\ \mathbf{e}_2\mathbf{a}^2 &= \begin{bmatrix}\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 2\ &\ \ 4\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ \end{bmatrix} \\ \mathbf{e}_3\mathbf{a}^3 &= \begin{bmatrix}\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ \\ -2\ & -3\ &\ \ 3\ \end{bmatrix} \\ \\ EA = \mathbf{e}_1\mathbf{a}^1 + \mathbf{e}_2\mathbf{a}^2 + \mathbf{e}_3\mathbf{a}^3 &= \begin{bmatrix}\ \ 2\ &\ \ 3\ & -2\ \\ \ \ 0\ &\ \ 1\ &\ \ 2\ \\ \ \ 0\ &\ \ 0\ &\ \ 1\ \end{bmatrix} \end{align*} $$ 2.1.10 Block Multiplication
We have already been partitioning our matrices into rows and columns in order to multiply matrices, and reduce matrices into simpler forms. Another useful technique for matrix multiplication is block partitioning. Basically one partitions a matrix into smaller blocks to simplify computations.
Example 10 - Block Partitions¶
Consider $A = \left[ \begin{array}{cccc}\ 2\ &\ 1\ &\ 1\ &\ 1\ \\ \ 1\ &\ 2\ &\ 0\ &\ 1\ \\ \ 2\ &\ 2\ &\ 1\ &\ 0\ \end{array} \right]$ and $B = \left[ \begin{array}{cccc}\ 2\ &\ 1\ &\ 1\ &\ 0\ \\ \ 2\ &\ 2\ &\ 0\ &\ 3\ \\ \ 3\ &\ 2\ &\ 2\ &\ 0\ \\ \ 1\ &\ 2\ &\ 1\ &\ 1\ \end{array} \right]$. How do we compute $C = AB$?
Multiplying a $3\times 4$ matrix by a $4\times 4$ matrix seems a lot to accomplish. Can we simplify our task by breaking it up into smaller parts? Consider the partitions:
$$A = \left[ \begin{array}{cc|cc}\ 2\ &\ 1\ &\ 1\ &\ 1\ \\ \ 1\ &\ 2\ &\ 0\ &\ 1\ \\ \hline \ 2\ &\ 2\ &\ 1\ &\ 0\ \end{array} \right] \text{ and }B = \left[ \begin{array}{cc|cc}\ 2\ &\ 1\ &\ 1\ &\ 0\ \\ \ 2\ &\ 2\ &\ 0\ &\ 3\ \\ \hline \ 3\ &\ 2\ &\ 2\ &\ 0\ \\ \ 1\ &\ 2\ &\ 1\ &\ 1\ \end{array} \right]$$
We created four submatrix partitions for each matrix:
$$A = \left[ \begin{array}{c|c} A_{11} & A_{12} \\ \hline A_{21} & A_{22} \end{array} \right] \text{ and }B = \left[ \begin{array}{c|c} B_{11} & B_{12} \\ \hline B_{21} & B_{22} \end{array} \right]$$
Recall that the inner product formula for matrix multiplication defines each element of $C = AB$ as a sum of products of scalars. Furthermore, addition and multiplication are commutative and associative. Can you see that
$$C = AB = \left[ \begin{array}{c|c} A_{11}B_{11} + A_{12}B_{21} & A_{11}B_{12} + A_{12}B_{22} \\ \hline A_{21}B_{11} + A_{22}B_{21} & A_{21}B_{12} + A_{22}B_{22} \end{array} \right] = \left[ \begin{array}{c|c} C_{11} & C_{12} \\ \hline C_{21} & C_{22} \end{array} \right]$$
In other words,
$$C_{ij} = A_{i1}B_{1j} + A_{i2}B_{2j}$$
The block products are easier to resolve. For example
$$ \begin{align*} C_{11} = A_{11}B_{11} + A_{12}B_{21} &= \begin{bmatrix}\ 2\ &\ 1\ \\ \ 1\ &\ 2\ \end{bmatrix}\begin{bmatrix}\ 2\ &\ 1\ \\ \ 2\ &\ 2\ \end{bmatrix} + \begin{bmatrix}\ 1\ &\ 1\ \\ \ 0\ &\ 1\ \end{bmatrix}\begin{bmatrix}\ 3\ &\ 2\ \\ \ 1\ &\ 2\ \end{bmatrix} \\ \\ &= \begin{bmatrix}\ 6\ &\ 4\ \\ \ 6\ &\ 5\ \end{bmatrix} + \begin{bmatrix}\ 4\ &\ 4\ \\ \ 1\ &\ 2\ \end{bmatrix} = \begin{bmatrix}\ 10\ &\ 8\ \\ \ 7\ &\ 7\ \end{bmatrix} \end{align*} $$
How does one compute the $2\times 2$ products quickly? $\begin{bmatrix}\ 2\ &\ 1 \end{bmatrix}$ times a $2\times p$ matrix is "two times the first row plus the second row". $\begin{bmatrix}\ 1\ &\ 2\ \end{bmatrix}$ times a $2\times p$ matrix is "two times the second row plus the first". Likewise,
$$ \begin{align*} C_{12} = A_{11}B_{12} + A_{12}B_{22} &= \begin{bmatrix}\ 2\ &\ 1\ \\ \ 1\ &\ 2\ \end{bmatrix}\begin{bmatrix}\ 1\ &\ 0\ \\ \ 0\ &\ 3\ \end{bmatrix} + \begin{bmatrix}\ 1\ &\ 1\ \\ \ 0\ &\ 1\ \end{bmatrix}\begin{bmatrix}\ 2\ &\ 0\ \\ \ 1\ &\ 1\ \end{bmatrix} \\ \\ &= \begin{bmatrix}\ 2\ &\ 3\ \\ \ 1\ &\ 6\ \end{bmatrix} + \begin{bmatrix}\ 3\ &\ 1\ \\ \ 1\ &\ 1\ \end{bmatrix} = \begin{bmatrix}\ 5\ &\ 4\ \\ \ 2\ &\ 7\ \end{bmatrix} \end{align*} $$
An $m\times 2$ matrix times $\begin{bmatrix}\ 1\ \\ \ 0\ \end{bmatrix}$ is the first column plus none of the second column. An $m\times 2$ matrix times $\begin{bmatrix}\ 0\ \\ \ 3\ \end{bmatrix}$ is none of the first column plus three times the second column.
Indeed, $C = AB = \begin{bmatrix}\ 10\ &\ 8\ &\ 5\ &\ 4\ \\ \ 7\ &\ 7\ &\ 2\ &\ 7\ \\ \ 11\ &\ 8\ &\ 4\ &\ 6\ \end{bmatrix}$
Exercise 10 - Solve the Matrix Equation¶
Let $A\in\mathbb{R}^{3\times3}$ and solve the matrix equation
$$ \begin{bmatrix} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ &-3\ & -2\ \end{bmatrix}A = \begin{bmatrix} -5\ &\ \ 6\ &\ \ 10\ \\ \ \ 0\ &\ \ 5\ &\ \ 9\ \\ -2\ & -6\ & -8\ \end{bmatrix} $$
View Solution
$$ \begin{align*} \begin{bmatrix} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ &-3\ & -2\ \end{bmatrix}\begin{bmatrix} a_1 & a_4 & a_7 \\ a_2 & a_5 & a_8 \\ a_3 & a_6 & a_9 \end{bmatrix} &= \begin{bmatrix} -5\ &\ \ 6\ &\ \ 10\ \\ \ \ 0\ &\ \ 5\ &\ \ 9\ \\ -2\ & -6\ & -8\ \end{bmatrix} \\ \\ -3a_1 - 2a_2 + 5a_3 \qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad &= -5 \\ -a_1 +\ \ a_2 + 3a_3 \qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad &= 0 \\ a_1 - 3a_2 - 2a_3 \qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad &= -2 \\ -3a_4 - 2a_5 + 5a_6 \qquad\qquad\qquad\qquad &= 6 \\ -a_4 +\ \ a_5 + 3a_6 \qquad\qquad\qquad\qquad &= 5 \\ a_4 - 3a_5 - 2a_6 \qquad\qquad\qquad\qquad &= -6 \\ -3a_7 - 2a_8 + 5a_9 &= 10 \\ -a_7 +\ \ a_8 + 3a_9 &= 9 \\ a_7 - 3a_8 - 2a_9 &= -8 \end{align*} $$The full $9\times 9$ linear system can be partitioned into a $3\times 3$ matrix of $3\times 3$ submatrices.
$$ \left[ \begin{array}{ccc|ccc|ccc} -3\ & -2\ &\ \ 5\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ -1\ &\ \ 1\ &\ \ 3\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 1\ & -3\ & -2\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \hline \ \ 0\ &\ \ 0\ &\ \ 0\ & -3\ & -2\ &\ \ 5\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ & -1\ &\ \ 1\ &\ \ 3\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 1\ & -3\ & -2\ &\ \ 0\ &\ \ 0\ &\ \ 0\ \\ \hline \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ & -3\ & -2\ &\ \ 5\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ & -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 0\ &\ \ 1\ & -3\ & -2\ \\ \end{array} \right] \begin{bmatrix} a_1 \\ a_2 \\ a_3 \\ \hline a_4 \\ a_5 \\ a_6 \\ \hline a_7 \\ a_8 \\ a_9 \\ \end{bmatrix} = \begin{bmatrix} -5\ \\ \ \ 0\ \\ -2\ \\ \hline \ \ 6\ \\ \ \ 5\ \\ -6\ \\ \hline \ \ 10\ \\ \ \ 9\ \\ -8\ \end{bmatrix} $$
$$ \left[\begin{array}{c|c|c} \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} & O & O \\ \hline O & \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} & O \\ \hline O & O & \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} \end{array} \right] \begin{bmatrix} a_1 \\ a_2 \\ a_3 \\ \hline a_4 \\ a_5 \\ a_6 \\ \hline a_7 \\ a_8 \\ a_9 \\ \end{bmatrix} = \begin{bmatrix} -5\ \\ \ \ 0\ \\ -2\ \\ \hline \ \ 6\ \\ \ \ 5\ \\ -6\ \\ \hline \ \ 10\ \\ \ \ 9\ \\ -8\ \end{bmatrix} $$
$$ \begin{align*} \left[ \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} \right]\begin{bmatrix} a_1 \\ a_2 \\ a_3 \end{bmatrix} + O\begin{bmatrix} a_4 \\ a_5 \\ a_6 \end{bmatrix} + O\begin{bmatrix} a_7 \\ a_8 \\ a_9 \end{bmatrix} &= \begin{bmatrix} -5\ \\ \ \ 0\ \\ -2\ \end{bmatrix} \\ \\ O\begin{bmatrix} a_1 \\ a_2 \\ a_3 \end{bmatrix} + \left[ \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} \right]\begin{bmatrix} a_4 \\ a_5 \\ a_6 \end{bmatrix} + O\begin{bmatrix} a_7 \\ a_8 \\ a_9 \end{bmatrix} &= \begin{bmatrix}\ \ 6\ \\ \ \ 5\ \\ -6\ \end{bmatrix} \\ \\ O\begin{bmatrix} a_1 \\ a_2 \\ a_3 \end{bmatrix} + O\begin{bmatrix} a_4 \\ a_5 \\ a_6 \end{bmatrix} + \left[ \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} \right]\begin{bmatrix} a_7 \\ a_8 \\ a_9 \end{bmatrix} &= \begin{bmatrix}\ \ 10\ \\ \ \ 9\ \\ -8\ \end{bmatrix} \end{align*} $$
It should be clear that the zero submatrices provide nothing to the solution. This yields three $3\times 3$ systems. Notice also that the coefficient matrix for all three nontrivial linear systems are the same; only the right-hand side of each nontrivial linear system is different.
$$ \begin{align*} \left[ \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} \right]\begin{bmatrix} a_1 \\ a_2 \\ a_3 \end{bmatrix} &= \begin{bmatrix} -5\ \\ \ \ 0\ \\ -2\ \end{bmatrix} \\ \\ \left[ \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} \right]\begin{bmatrix} a_4 \\ a_5 \\ a_6 \end{bmatrix} &= \begin{bmatrix}\ \ 6\ \\ \ \ 5\ \\ -6\ \end{bmatrix} \\ \\ \left[ \begin{array}{ccc} -3\ & -2\ &\ \ 5\ \\ -1\ &\ \ 1\ &\ \ 3\ \\ \ \ 1\ & -3\ & -2\ \end{array} \right]\begin{bmatrix} a_7 \\ a_8 \\ a_9 \end{bmatrix} &= \begin{bmatrix}\ \ 10\ \\ \ \ 9\ \\ -8\ \end{bmatrix} \end{align*} $$
One can perform one more optimization before reducing the partitioned matrix:
$$ \begin{align*} \left[ \begin{array}{ccc|ccc} -3\ & -2\ &\ \ 5\ & -5\ &\ \ 6\ &\ \ 10\ \\ -1\ &\ \ 1\ &\ \ 3\ &\ \ 0\ &\ \ 5\ &\ \ 9\ \\ \ \ 1\ & -3\ & -2\ & -2\ & -6\ & -8\ \end{array} \right] \begin{array}{c} -R_2 \qquad\ \, \\ R_1 \qquad\ \, \\ \\ \end{array} &\longrightarrow \left[ \begin{array}{ccc|ccc}\ \ 1\ & -1\ & -3\ &\ \ 0\ & -5\ & -9\ \\ -3\ & -2\ &\ \ 5\ & -5\ &\ \ 6\ &\ \ 10\ \\ \ \ 1\ & -3\ & -2\ & -2\ & -6\ & -8\ \end{array} \right] \begin{array}{c} \\ R_2+3R_1 \\ R_3-R_1 \\ \end{array} \longrightarrow \\ \\ \left[ \begin{array}{ccc|ccc}\ \ 1\ & -1\ & -3\ &\ \ 0\ & -5\ & -9\ \\ \ \ 0\ & -5\ & -4\ & -5\ & -9\ & -17\ \\ \ \ 0\ & -2\ &\ \ 1\ & -2\ & -1\ &\ \ 1\ \end{array} \right] \begin{array}{c} \\ R_2-3R_3 \\ \\ \end{array} &\longrightarrow \left[ \begin{array}{ccc|ccc}\ \ 1\ & -1\ & -3\ &\ \ 0\ & -5\ & -9\ \\ \ \ 0\ &\ \ 1\ & -7\ &\ \ 1\ & -6\ & -20\ \\ \ \ 0\ & -2\ &\ \ 1\ & -2\ & -1\ &\ \ 1\ \end{array} \right] \begin{array}{c} \\ \\ R_3+2R_2 \\ \end{array} \longrightarrow \\ \\ \left[ \begin{array}{ccc|ccc}\ \ 1\ & -1\ & -3\ &\ \ 0\ & -5\ & -9\ \\ \ \ 0\ &\ \ 1\ & -7\ &\ \ 1\ & -6\ & -20\ \\ \ \ 0\ &\ \ 0\ & -13\ &\ \ 0\ & -13\ & -39\ \end{array} \right] \begin{array}{c} \\ \\ -R_3/13 \\ \end{array} &\longrightarrow \left[ \begin{array}{ccc|ccc}\ \ 1\ & -1\ & -3\ &\ \ 0\ & -5\ & -9\ \\ \ \ 0\ &\ \ 1\ & -7\ &\ \ 1\ & -6\ & -20\ \\ \ \ 0\ &\ \ 0\ &\ \ 1\ &\ \ 0\ &\ \ 1\ &\ \ 3\ \end{array} \right] \begin{array}{c} R_1+3R_3 \\ R_2+7R_3 \\ \\ \end{array} \longrightarrow \\ \\ \left[ \begin{array}{ccc|ccc}\ \ 1\ & -1\ &\ \ 0\ &\ \ 0\ & -2\ &\ \ 0\ \\ \ \ 0\ &\ \ 1\ &\ \ 0\ &\ \ 1\ &\ \ 1\ &\ \ 1\ \\ \ \ 0\ &\ \ 0\ &\ \ 1\ &\ \ 0\ &\ \ 1\ &\ \ 3\ \end{array} \right] \begin{array}{c} R_1+R_2 \\ \\ \\ \end{array} &\longrightarrow \left[ \begin{array}{ccc|ccc}\ \ 1\ &\ \ 0\ &\ \ 0\ &\ \ 1\ & -1\ &\ \ 1\ \\ \ \ 0\ &\ \ 1\ &\ \ 0\ &\ \ 1\ &\ \ 1\ &\ \ 1\ \\ \ \ 0\ &\ \ 0\ &\ \ 1\ &\ \ 0\ &\ \ 1\ &\ \ 3\ \end{array} \right] \end{align*} $$
In reduced-row echelon form the solution is in the right partition.
$$ A = \left[ \begin{array}{ccc}\ \ 1\ & -1\ &\ \ 1\ \\\ \ 1\ &\ \ 1\ &\ \ 1\ \\ \ 0\ &\ \ 1\ &\ \ 3\ \end{array} \right] $$
2.1.11 Transformations in Space
Video Lecture 5: Linear Transformations of Space.¶
3D Linear Transformations of SpaceIf matrix $A$ is an $m\times n$ matrix, then $A$ is an element of the vector space of all $m\times n$ matrices of real numbers
$$ A\in\mathbb{R}^{m\times n} $$
We see that an $m\times n$ matrix $A$ represents a linear transformation from $n$-dimensional space to $m$-dimensional space.
$$ A:\mathbb{R}^n\rightarrow\mathbb{R}^m $$
where for every input $\mathbf{x}\in\mathbb{R}^n$ we get an output $\mathbf{y}\in\mathbb{R}^m$ such that
$$ \mathbf{y} = A\mathbf{x} $$
This allows FIVE different views of matrix multiplication:
Function Composition
The definition of matrix multiplication, a row of matrix-vector products $AB = \begin{bmatrix} A\mathbf{b}_1 & A\mathbf{b}_2 & \cdots & A\mathbf{b}_p \end{bmatrix}$.
A column of vector-matrix products $AB = \begin{bmatrix} \mathbf{a}^1B \\ \mathbf{a}^2B \\ \vdots \\ \mathbf{a}^mB \end{bmatrix}$, the dual to the definition.
A matrix of inner products $c_{ij} = \mathbf{a}^i\mathbf{b}_j = \displaystyle\sum_{k=1}^n a_{ik}b_{kj}$
A sum of outer products $C = \displaystyle\sum_{k=1}^n \mathbf{a}_k\mathbf{b}^k$, the dual to the inner product formulation.
Video Lecture 6: Matrix Multiplication in Practice.¶
Matrix Multiplication In PracticeChoosing a View¶
Each of the five views computes the same product $C = AB$. They are equivalent in result but not in effort or insight.
Use:
- Composition when you care about what the product does rather than what its entries are — for example, when reasoning about whether a sequence of transformations preserves a property.
- Matrix-vector (column view of $AB$) when $B$ has a simple column structure — many zeros, a single nonzero column, or repeated columns. Each column of $C$ is $A$ times one column of $B$, so simple columns of $B$ produce easy columns of $C$.
- Vector-matrix (row view of $AB$) when $A$ has a simple row structure. Each row of $C$ is one row of $A$ times $B$, so simple rows of $A$ produce easy rows of $C$. Exercise 11 illustrates this directly.
- Inner-product when you need exactly one entry $c_{ij}$ and don't care about the rest. It's the wrong choice for computing the whole product by hand.
- Outer-product when you want to write the product as a sum of rank-one pieces — for example, to recognize structure or to drop small contributions. This view is central to the singular value decomposition and to low-rank approximation in data science.
2.1.12 Exercises
Exercise 11¶
Compute the matrix $G = EF$ where $E = \begin{bmatrix}\ \ 0\ & -1\ \\ \ \ 1\ &\ \ 1\ \\ \ \ 1\ &\ \ 1\ \end{bmatrix}$ and $F = \begin{bmatrix}\ \ 1\ & -4\ & -1 \\ -3\ &\ \ 2\ & -2\ \end{bmatrix}$
View Solution
As matrix $E$ is the simplest matrix, so let us multiply matrix $F$ on the left by the three rows of matrix $E$ to obtain the three rows of the product matrix $G$.$$ \begin{align*} G &= EF = \begin{bmatrix}\ \ 0\ & -1\ \\ \ \ 1\ &\ \ 1\ \\ \ \ 1\ &\ \ 1\ \end{bmatrix}\begin{bmatrix}\ \ 1\ & -4\ & -1 \\ -3\ &\ \ 2\ & -2\ \end{bmatrix} \\ \\ \mathbf{g}^1 &= 0\mathbf{f}^1 - \mathbf{f}^2 = \begin{bmatrix}\ \ 3\ & -2\ &\ \ 2\ \end{bmatrix} \qquad &\begin{array}{c} \text{none of row one} \\ \text{minus row two} \end{array} \\ \\ \mathbf{g}^2 &= \mathbf{f}^1 + \mathbf{f}^2 = \begin{bmatrix} -2\ & -2\ & -3\ \end{bmatrix} \qquad &\begin{array}{c} \text{row one plus} \\ \text{row two} \end{array} \\ \\ \mathbf{g}^3 &= \mathbf{g}^2 = \begin{bmatrix} -2\ & -2\ & -3\ \end{bmatrix} \\ \\ G &= \begin{bmatrix}\ \ 3\ & -2\ &\ \ 2\ \\ -2\ & -2\ & -3\ \\ -2\ & -2\ & -3\ \end{bmatrix} \end{align*} $$
Exercise 12¶
Now compute the matrix product $H=FE$ using the same matrices from Exercise 11. Notice the changes.
View Solution
$$ \begin{align*} \mathbf{h}_1 = F\mathbf{e}_1 &= \begin{bmatrix}\ \ 1\ & -4\ & -1 \\ -3\ &\ \ 2\ & -2\ \end{bmatrix}\begin{bmatrix}\ 0\ \\ \ 1\ \\ \ 1\ \end{bmatrix} = \begin{bmatrix} -5\ \\ \ \ 0\ \end{bmatrix} \\ \mathbf{h}_2 = F\mathbf{e}_2 &= \begin{bmatrix}\ \ 1\ & -4\ & -1 \\ -3\ &\ \ 2\ & -2\ \end{bmatrix}\begin{bmatrix} -1\ \\ \ \ 1\ \\ \ \ 1\ \end{bmatrix} = \begin{bmatrix} -6\ \\ \ \ 3\ \end{bmatrix} \\ \\ H &= \begin{bmatrix} -5\ & -6\ \\ \ \ 0\ &\ \ 3\ \end{bmatrix} \end{align*} $$Exercise 13¶
Compute the matrix product $C = AB$ where
$$
A = \begin{bmatrix}
\ \ 1\ &\ \ 2\ &\ \ 0\ & -1\ \\
\ \ 5\ &\ \ 4\ & -6\ &\ \ 1\ \\
\ \ 0\ &\ \ 4\ &\ \ 4\ & -4\ \\
\ \ 2\ &\ \ 1\ & -3\ &\ \ 1\ \end{bmatrix},\qquad\qquad
B = \begin{bmatrix}
\ \ 0\ &\ \ 1\ &\ \ 0\ \\
-1\ &\ \ 1\ & -1\ \\
-1\ & -1\ &\ \ 0\ \\
\ \ 0\ &\ \ 0\ & -1\ \end{bmatrix}
$$
View Solution
As matrix $B$ is the simplest one let us multiply matrix $A$ on the right by the three columns of matrix $B$ to obtain the three columns of the product matrix $C$.$$ \begin{align*} A\mathbf{b}_1 &= \begin{bmatrix} \ \ 1\ &\ \ 2\ &\ \ 0\ & -1\ \\ \ \ 5\ &\ \ 4\ & -6\ &\ \ 1\ \\ \ \ 0\ &\ \ 4\ &\ \ 4\ & -4\ \\ \ \ 2\ &\ \ 1\ & -3\ &\ \ 1\ \end{bmatrix} \begin{bmatrix}\ \ 0\ \\ -1\ \\ -1\ \\ \ \ 0\ \end{bmatrix} = \begin{bmatrix} -2\ \\ \ \ 2 \\ -8\ \\ \ \ 2\ \end{bmatrix} \qquad &\begin{array}{c} \text{the negative of the} \\ \text{sum of column 2 and column 3} \end{array}\\ \\ A\mathbf{b}_2 &= \begin{bmatrix} \ \ 1\ &\ \ 2\ &\ \ 0\ & -1\ \\ \ \ 5\ &\ \ 4\ & -6\ &\ \ 1\ \\ \ \ 0\ &\ \ 4\ &\ \ 4\ & -4\ \\ \ \ 2\ &\ \ 1\ & -3\ &\ \ 1\ \end{bmatrix} \begin{bmatrix}\ \ 1\ \\ \ \ 1\ \\ -1\ \\ \ \ 0\ \end{bmatrix} = \begin{bmatrix}\ \ 3\ \\ \ 15\ \\ \ \ 0\ \\ \ \ 6\ \end{bmatrix} \qquad &\begin{array}{c} \text{the sum of column 1 and} \\ \text{column 2 minus column 3} \end{array} \\ \\ A\mathbf{b}_3 &= \begin{bmatrix} \ \ 1\ &\ \ 2\ &\ \ 0\ & -1\ \\ \ \ 5\ &\ \ 4\ & -6\ &\ \ 1\ \\ \ \ 0\ &\ \ 4\ &\ \ 4\ & -4\ \\ \ \ 2\ &\ \ 1\ & -3\ &\ \ 1\ \end{bmatrix} \begin{bmatrix}\ \ 0\ \\ -1\ \\ \ \ 0\ \\ -1\ \end{bmatrix} = \begin{bmatrix} -1\ \\ -5\ \\ \ \ 0\ \\ -2\ \end{bmatrix} \qquad &\begin{array}{c} \text{the negative of the} \\ \text{sum of column 2 and column 4} \end{array} \\ \\ C &= AB = \begin{bmatrix} -2\ &\ \ 3\ & -1\ \\ \ \ 2\ &\ 15\ & -5\ \\ -8\ &\ \ 0\ &\ \ 0\ \\ \ \ 2\ &\ \ 6\ & -2\ \end{bmatrix}\\ \\ \end{align*} $$
Exercise 14¶
Now compute the matrix product $D = BA$ using the same matrices from Exercise 13.
View Solution
Matrix $B\in\mathbb{R}^{4\times3}$ and matrix $A\in\mathbb{R}^{4\times4}$. The inner dimensions do not match so the matrix product does not exist.Exercise 15 - Block Multiplication¶
Compute the matrix product $C = AB$ using block multiplication, where
$$ A = \left[ \begin{array}{cc|cc} \ \ 1\ &\ \ 2\ &\ \ 0\ &\ \ 0\ \\ \ \ 3\ &\ \ 4\ &\ \ 0\ &\ \ 0\ \\ \hline \ \ 0\ &\ \ 0\ &\ \ 5\ &\ \ 6\ \\ \ \ 0\ &\ \ 0\ &\ \ 7\ &\ \ 8 \end{array} \right] \qquad\text{and}\qquad B = \left[ \begin{array}{cc|cc} \ \ 2\ &\ \ 0\ &\ \ 1\ &\ \ 0\ \\ \ \ 0\ &\ \ 2\ &\ \ 0\ &\ \ 1\ \\ \hline \ \ 1\ &\ \ 0\ &\ \ 2\ &\ \ 0\ \\ \ \ 0\ &\ \ 1\ &\ \ 0\ &\ \ 2 \end{array} \right] $$
Partition each matrix into four $2\times 2$ blocks as shown by the lines, treat the blocks as if they were scalars, and apply the block formula
$$ C = AB = \left[ \begin{array}{c|c} A_{11}B_{11} + A_{12}B_{21} & A_{11}B_{12} + A_{12}B_{22} \\ \hline A_{21}B_{11} + A_{22}B_{21} & A_{21}B_{12} + A_{22}B_{22} \end{array} \right] $$
How much work is saved compared with computing $C$ entry-by-entry from the inner-product formula?
View Solution
Name the four blocks of each matrix:$$A_{11} = \begin{bmatrix}\ 1\ &\ 2\ \\ \ 3\ &\ 4\ \end{bmatrix},\ A_{12}=O,\ A_{21}=O,\ A_{22}=\begin{bmatrix}\ 5\ &\ 6\ \\ \ 7\ &\ 8\ \end{bmatrix}$$
$$B_{11} = 2I,\ B_{12}=I,\ B_{21}=I,\ B_{22}=2I$$
where $I$ is the $2\times 2$ identity matrix and $O$ is the $2\times 2$ zero matrix.
Because $A_{12}$ and $A_{21}$ are zero, six of the eight block products in the formula vanish. The remaining four reduce to
$$\begin{align*} A_{11}B_{11} = A_{11}(2I) = 2A_{11} = \begin{bmatrix}\ 2\ &\ 4\ \\ \ 6\ &\ 8\ \end{bmatrix} \\ A_{11}B_{12} = A_{11}I = A_{11} = \begin{bmatrix}\ 1\ &\ 2\ \\ \ 3\ &\ 4\ \end{bmatrix} \\ A_{22}B_{21} = A_{22}I = A_{22} = \begin{bmatrix}\ 5\ &\ 6\ \\ \ 7\ &\ 8\ \end{bmatrix} \\ A_{22}B_{22} = A_{22}(2I) = 2A_{22} = \begin{bmatrix}\ 10\ &\ 12\ \\ \ 14\ &\ 16\ \end{bmatrix} \end{align*}$$
Assembling the blocks back together yields $$ C = AB = \left[ \begin{array}{cc|cc}\ 2\ &\ 4\ &\ 1\ &\ 2\ \\ \ 6\ &\ 8\ &\ 3\ &\ 4\ \\ \hline \ 5\ &\ 6\ &\ 10\ &\ 12\ \\ \ 7\ &\ 8\ &\ 14\ &\ 16\ \end{array} \right]$$
Work savings. The inner-product formula requires $4^3 = 64$ scalar multiplications to compute a $4\times 4$ product. The block view recognizes that matrix $A$ is block-diagonal (so two block products are zero), and that $B_{11}$, $B_{12}$, $B_{21}$, $B_{22}$ are scalar multiples of the identity (so the non-trivial block products reduce to copying a $2\times 2$ block or doubling its entries). The non-trivial work is two scalar doublings of $2\times 2$ blocks — eight multiplications by two — and the rest is reading. Block structure converts a $64$-multiplication problem into something that can be done by inspection.
