15 Orthogonal Vectors and Orthogonal Complements of Subspaces

Lecture from 06.11.2024 | Video: Videos ETHZ

Orthogonality of Vectors and Subspaces

Definition

Two vectors $\mathbf{v},\mathbf{w}\in {\mathbb{R}}^n$ are orthogonal or perpendicular if their dot product is zero:

${\mathbf{v}}^T\mathbf{w}={\sum }_{i=1}^n{v}_i{w}_i=0$

Two subspaces $V$ and $W$ of ${\mathbb{R}}^n$ are orthogonal if every vector $\mathbf{v}\in V$ is orthogonal to every vector $\mathbf{w}\in W$.

Example in ${\mathbb{R}}^2$

Fact 1

If $\mathbf{v}$ and $\mathbf{w}$ are orthogonal, any scalar multiple of $\mathbf{v}$ is also orthogonal to any scalar multiple of $\mathbf{w}$. This means that the subspaces spanned by $\mathbf{v}$ and $\mathbf{w}$ are orthogonal.

Let’s take $\mathbf{v}=\left[\begin{array}{c}1\\ 2\end{array}\right]$ and $\mathbf{w}=\left[\begin{array}{c}-2\\ 1\end{array}\right]$.

${\mathbf{v}}^T\mathbf{w}=(1)(-2)+(2)(1)=-2+2=0$. So, $\mathbf{v}$ and $\mathbf{w}$ are orthogonal.

Now consider $3\mathbf{v}=\left[\begin{array}{c}3\\ 6\end{array}\right]$ and $-2\mathbf{w}=\left[\begin{array}{c}4\\ -2\end{array}\right]$.

$(3\mathbf{v}{)}^T(-2\mathbf{w})=(3)(4)+(6)(-2)=12-12=0$. As expected, the scalar multiples are also orthogonal.

The subspace spanned by $\mathbf{v}$ is $V=\{\lambda \mathbf{v}|\lambda \in \mathbb{R}\}$ and the subspace spanned by $\mathbf{w}$ is $W=\{\mu \mathbf{w}|\mu \in \mathbb{R}\}$. These subspaces are orthogonal.

Fact 2

If $\mathbf{v}$ and $\mathbf{w}$ are linearly independent and orthogonal vectors in ${\mathbb{R}}^2$, they form a basis for ${\mathbb{R}}^2$. This implies that if a linear combination of $\mathbf{v}$ and $\mathbf{w}$ is zero, then the coefficients must be zero.

Continuing with the same $\mathbf{v}$ and $\mathbf{w}$ as above, we can see they are linearly independent because neither is a scalar multiple of the other. Since they are orthogonal and linearly independent in ${\mathbb{R}}^2$, they form a basis for ${\mathbb{R}}^2$.

Consider the equation $\lambda \mathbf{v}+\mu \mathbf{w}=0$. We want to show that $\lambda =\mu =0$.

$\lambda \left[\begin{array}{c}1\\ 2\end{array}\right]+\mu \left[\begin{array}{c}-2\\ 1\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$

This gives us the system of equations:

$\lambda -2\mu =0\text{ and }2\lambda +\mu =0$

Multiplying the first equation by 2, we get $2\lambda -4\mu =0$. Subtracting this from the second equation gives $5\mu =0$, so $\mu =0$. Substituting $\mu =0$ into the first equation gives $\lambda =0$.

Thus, the only solution is $\lambda =\mu =0$, demonstrating that if a linear combination of orthogonal, linearly independent vectors is zero, the coefficients must be zero. This is a consequence of them forming a basis.

Decomposition of ${\mathbb{R}}^2$ and the Minkowski Sum

We can express the decomposition of ${\mathbb{R}}^2$ in terms of the Minkowski sum. The Minkowski sum of two sets $V$ and $W$ in ${\mathbb{R}}^n$, denoted $V+W$, is defined as:

$V+W=\{\mathbf{v}+\mathbf{w}|\mathbf{v}\in V,\mathbf{w}\in W\}$

Orthogonality of Subspaces and Bases

Lemma: Let $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ be a basis for subspace $V$ and $\{{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ be a basis for subspace $W$. Then $V$ and $W$ are orthogonal if and only if ${\mathbf{v}}_i^T{\mathbf{w}}_j=0$ for all $i\in \{1,\dots ,k\}$ and $j\in \{1,\dots ,l\}$.

Proof

• Forward Direction: If $V$ and $W$ are orthogonal, then any $\mathbf{v}\in V$ and $\mathbf{w}\in W$ are orthogonal. Since the ${\mathbf{v}}_i$ and ${\mathbf{w}}_j$ are basis vectors, they belong to their respective subspaces, so ${\mathbf{v}}_i^T{\mathbf{w}}_j=0$.

• Reverse Direction: Assume ${\mathbf{v}}_i^T{\mathbf{w}}_j=0$ for all $i$ and $j$. Any $\mathbf{v}\in V$ can be written as $\mathbf{v}={\sum }_{i=1}^k{\lambda }_i{\mathbf{v}}_i$, and any $\mathbf{w}\in W$ as $\mathbf{w}={\sum }_{j=1}^l{\mu }_j{\mathbf{w}}_j$. Then:

${\mathbf{v}}^T\mathbf{w}={\left({\sum }_{i=1}^k{\lambda }_i{\mathbf{v}}_i\right)}^T\left({\sum }_{j=1}^l{\mu }_j{\mathbf{w}}_j\right)={\sum }_{i=1}^k{\sum }_{j=1}^l{\lambda }_i{\mu }_j{\mathbf{v}}_i^T{\mathbf{w}}_j=0$

since each ${\mathbf{v}}_i^T{\mathbf{w}}_j=0$. Therefore, $V$ and $W$ are orthogonal.

Linear Independence of Basis Vectors from Orthogonal Subspaces

Lemma: Let $V$ and $W$ be two orthogonal subspaces of ${\mathbb{R}}^n$. Let $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ be a basis of $V$ and $\{{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ be a basis of $W$. The set of vectors $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k,{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ is linearly independent.

Proof

Consider a linear combination of the basis vectors that equals the zero vector:

${\sum }_{i=1}^k{\lambda }_i{\mathbf{v}}_i+{\sum }_{j=1}^l{\mu }_j{\mathbf{w}}_j=0(\ast )$

We want to show that ${\lambda }_i=0$ for all $i$ and ${\mu }_j=0$ for all $j$.

Let $\mathbf{v}={\sum }_{i=1}^k{\lambda }_i{\mathbf{v}}_i$. Then equation $(\ast )$ is equivalent to: $\mathbf{v}=-{\sum }_{j=1}^l{\mu }_j{\mathbf{w}}_j$ Now, take the dot product of both sides of this equation with $\mathbf{v}$: ${\mathbf{v}}^T\mathbf{v}={\mathbf{v}}^T\left(-{\sum }_{j=1}^l{\mu }_j{\mathbf{w}}_j\right)$ $\Vert \mathbf{v}{\Vert }^2=-{\sum }_{j=1}^l{\mu }_j{\mathbf{v}}^T{\mathbf{w}}_j$ Since $\mathbf{v}\in V$ and each ${\mathbf{w}}_j\in W$, and $V$ and $W$ are orthogonal, we have ${\mathbf{v}}^T{\mathbf{w}}_j=0$ for all $j$. Therefore: $\Vert \mathbf{v}{\Vert }^2=0$ This implies that $\mathbf{v}=0$. Since $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ is a basis for $V$ (and thus linearly independent), the only way for ${\sum }_{i=1}^k{\lambda }_i{\mathbf{v}}_i=0$ is if ${\lambda }_i=0$ for all $i$. Analogously ${\mu }_j=0$ for all $j$.

Alternatively, substituting $\mathbf{v}=0$ back into equation $(\ast )$ gives: ${\sum }_{j=1}^l{\mu }_j{\mathbf{w}}_j=0$

Since $\{{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ is a basis for $W$ (and thus linearly independent), we must have ${\mu }_j=0$ for all $j$.

Thus, we have shown that all the coefficients ${\lambda }_i$ and ${\mu }_j$ are zero, which proves that the set $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k,{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ is linearly independent.

Intersection and Sum of Orthogonal Subspaces

Corollary: Let $V$ and $W$ be orthogonal subspaces of ${\mathbb{R}}^n$. Then:

1. $V\cap W=\{0\}$
2. $\text{dim}(V+W)=\text{dim}(\{\mathbf{v}+\mathbf{w}|\mathbf{v}\in V,\mathbf{w}\in W\})=\text{dim}(V)+\text{dim}(W)\le n$

Proof

1. Intersection: Suppose $\mathbf{x}\in V\cap W$. Since $V$ and $W$ are orthogonal, ${\mathbf{x}}^T\mathbf{y}=0$ for all $\mathbf{y}\in W$. But since $\mathbf{x}\in W$ as well, we can choose $\mathbf{y}=\mathbf{x}$, which means ${\mathbf{x}}^T\mathbf{x}=||\mathbf{x}|{|}^2=0$. This implies $\mathbf{x}=0$. Therefore, $V\cap W=\{0\}$.
2. Sum and Dimension: The sum of two subspaces $V$ and $W$, denoted $V+W$, is defined as $\{\mathbf{v}+\mathbf{w}|\mathbf{v}\in V,\mathbf{w}\in W\}$. Let $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ be a basis for $V$ and $\{{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ be a basis for $W$. Since $V$ and $W$ are orthogonal subspaces, their bases can be combined to form a basis for $V+W$, given by $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k,{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$. In simpler terms, any vector in $V+W$ can be uniquely represented as a linear combination of vectors from the combined basis. Hence, the dimension adds up to form the dimension of the combined subspace. As both V and W are subspaces of ${\mathbb{R}}^n$, their sum is also a subspace in ${\mathbb{R}}^n$. Hence, its dimension will always be less than or equal to n. This leads to $\text{dim}(V+W)=\text{dim}(V)+\text{dim}(W)$. Since $V+W$ is a subspace of ${\mathbb{R}}^n$, its dimension must be less than or equal to $n$. Therefore, $\text{dim}(V)+\text{dim}(W)\le n$.

The Orthogonal Complement of a Subspace

Definitions

Let $V$ be a subspace of ${\mathbb{R}}^n$. The orthogonal complement of $V$, denoted $V^{\perp }$, is the set of all vectors in ${\mathbb{R}}^n$ that are orthogonal to every vector in $V$:

$V^{\perp }=\{\mathbf{w}\in {\mathbb{R}}^n|{\mathbf{w}}^T\mathbf{v}=0\text{ for all }\mathbf{v}\in V\}$

$V^{\perp }$ is itself a subspace of ${\mathbb{R}}^n$. Intuitively, imagine a plane in 3D space. Its orthogonal complement would be the line through the origin that’s perpendicular to the plane. Every vector in the line is orthogonal to every vector in the plane.

Theorem: Nullspace and Row Space

Let $A\in {\mathbb{R}}^{m\times n}$ be a matrix. Then $N(A)=C(A^T{)}^{\perp }=R(A{)}^{\perp }$. This theorem states that the nullspace of a matrix is the orthogonal complement of its row space (or equivalently, the column space of its transpose).

Proof

• $\mathbf{N}(\mathbf{A})\subseteq \mathbf{C}({\mathbf{A}}^{\mathbf{T}}{)}^{\perp }$: Let $\mathbf{x}$ be a vector in the nullspace of $A$. This means $A\mathbf{x}=0$. We want to show that $\mathbf{x}$ is orthogonal to every vector in the column space of $A^T$. Any vector $\mathbf{b}$ in $C(A^T)$ can be written as $\mathbf{b}=A^T\mathbf{y}$ for some $\mathbf{y}\in {\mathbb{R}}^m$. The dot product of $\mathbf{b}$ and $\mathbf{x}$ is ${\mathbf{b}}^T\mathbf{x}=(A^T\mathbf{y}{)}^T\mathbf{x}={\mathbf{y}}^{T}A\mathbf{x}={\mathbf{y}}^{T}0=0$. Since their dot product is zero, $\mathbf{x}$ is orthogonal to $\mathbf{b}$. Because $\mathbf{b}$ was an arbitrary vector in $C(A^T)$, this means $\mathbf{x}$ is orthogonal to every vector in $C(A^T)$, and therefore $\mathbf{x}\in C(A^T{)}^{\perp }$.

• $\mathbf{C}({\mathbf{A}}^{\mathbf{T}}{)}^{\perp }\subseteq \mathbf{N}(\mathbf{A})$: Let $\mathbf{x}\in C(A^T{)}^{\perp }$. This means ${\mathbf{b}}^T\mathbf{x}=0$ for all $\mathbf{b}\in C(A^T)$. Let $\mathbf{y}=A\mathbf{x}$. Since $\mathbf{y}\in {\mathbb{R}}^m$, $A^T\mathbf{y}=A^T(A\mathbf{x})$ is in $C(A^T)$. Because $\mathbf{x}$ is orthogonal to every vector in $C(A^T)$, it must be orthogonal to $A^T\mathbf{y}$: $0={\mathbf{x}}^T(A^T\mathbf{y})={\mathbf{x}}^T(A^{T}A\mathbf{x})=(A\mathbf{x}{)}^T(A\mathbf{x})=||A\mathbf{x}|{|}^2=||\mathbf{y}|{|}^2$ For the squared norm of a vector to be zero, the vector itself must be zero. Thus, $A\mathbf{x}=\mathbf{y}=0$, which means $\mathbf{x}\in N(A)$.

Since the row space of $A$ is the same as the column space of $A^T$ ($R(A)=C(A^T)$), we also have $N(A)=R(A{)}^{\perp }$.

Example

Consider the matrix:

$A=\left[\begin{array}{cc}1& 2\\ 2& 4\end{array}\right]$

The row space $R(A)$ is spanned by the vector $[1,2]$. The nullspace $N(A)$ contains vectors of the form $[-2,1]$ (and their scalar multiples). Notice that the dot product of $[1,2]$ and $[-2,1]$ is $(1)(-2)+(2)(1)=0$. This confirms that the nullspace is orthogonal to the row space.

This relationship is fundamental in linear algebra, particularly in understanding the solutions to systems of linear equations and in concepts like least squares approximations. The orthogonal complement provides a powerful tool for decomposing vector spaces and analyzing the behavior of linear transformations.

Multiplying by Invertible Matrix Doesn’t Change Row Space

If $A$ is an $m\times n$ matrix and $M$ is an invertible $m\times m$ matrix, then $R(A)=R(MA)$. See: Invariance of Row Space under Multiplication by an Invertible Matrix

The Orthogonal Complement of a Subspace II

Theorem: Properties of Orthogonal Complements

Let $V$ and $W$ be orthogonal subspaces of ${\mathbb{R}}^n$. The following statements are equivalent:

(i) $W=V^{\perp }$ (ii) $\text{dim}(V)+\text{dim}(W)=n$ (iii) Every $\mathbf{u}\in {\mathbb{R}}^n$ can be written uniquely as $\mathbf{u}=\mathbf{v}+\mathbf{w}$, where $\mathbf{v}\in V$ and $\mathbf{w}\in W$.

Proof

• (i) implies (ii): Let $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ be a basis for $V$. Form a matrix $A$ whose rows are the basis vectors of $V$. Then $V=R(A)=C(A^T)$. By the previous theorem, $V^{\perp }=N(A)$. From the rank-nullity theorem, $\text{dim}(V)+\text{dim}(N(A))=n$. Since $W=V^{\perp }=N(A)$, we have $\text{dim}(V)+\text{dim}(W)=n$.

• (ii) implies (iii): Let $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ be a basis for $V$ and $\{{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ be a basis for $W$. Since $V$ and $W$ are orthogonal, ${\mathbf{v}}_i^T{\mathbf{w}}_j=0$ for all $i$ and $j$. The set $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k,{\mathbf{w}}_1,\dots ,{\mathbf{w}}_l\}$ is linearly independent. Since $k+l=n$, this set forms a basis for ${\mathbb{R}}^n$. Thus, any $\mathbf{u}\in {\mathbb{R}}^n$ can be written uniquely as a linear combination of these basis vectors, which is equivalent to $\mathbf{u}=\mathbf{v}+\mathbf{w}$ for unique $\mathbf{v}\in V$ and $\mathbf{w}\in W$.

• (iii) implies (i): We need to show $W=V^{\perp }$. Since $V$ and $W$ are orthogonal, $W\subseteq V^{\perp }$. For the reverse inclusion, let $\mathbf{u}\in V^{\perp }$. By (iii), we can write $\mathbf{u}=\mathbf{v}+\mathbf{w}$ for unique $\mathbf{v}\in V$ and $\mathbf{w}\in W$. Then $0={\mathbf{u}}^T\mathbf{v}=(\mathbf{v}+\mathbf{w}{)}^T\mathbf{v}={\mathbf{v}}^T\mathbf{v}+{\mathbf{w}}^T\mathbf{v}=\Vert \mathbf{v}{\Vert }^2$. This implies $\mathbf{v}=0$, so $\mathbf{u}=\mathbf{w}\in W$. Thus, $V^{\perp }\subseteq W$, and we conclude $W=V^{\perp }$.

Continue here: 16 Orthogonal Complementary Subspaces, Projections, Normal Form