23 Eigenvalues and Eigenvectors, Complex Numbers

Lecture from 04.12.2024 | Video: Videos ETHZ

Introduction

This week, we delve into a fundamental concept in linear algebra: eigenvalues and eigenvectors.

Intro to Eigenvalues and Eigenvectors... (MIT)

A good and intuitive lecture…

For a shorter and more visual understanding…

Informal Definition

Given a square matrix $A$, an eigenvalue $\lambda$ (a scalar) and its corresponding eigenvector $\mathbf{v}$ (a non-zero vector) satisfy the following equation:

$A\mathbf{v}=\lambda \mathbf{v}$

This equation signifies that when the linear transformation represented by matrix $A$ is applied to the eigenvector $\mathbf{v}$, the resulting vector is simply a scaled version of the original vector $\mathbf{v}$. The scaling factor is the eigenvalue $\lambda$.

Rewriting the equation, we get:

$(A-\lambda I)\mathbf{v}=0$

where $I$ is the identity matrix. For this equation to hold with a non-zero vector $\mathbf{v}$, the matrix $(A-\lambda I)$ must be singular (non-invertible). Therefore:

$\det (A-\lambda I)=0$

This equation is crucial for finding eigenvalues.

Strategy for Finding Eigenvalues

We can find eigenvalues by solving the equation $\det (A-\lambda I)=0$. The expression $\det (A-\lambda I)$ is a polynomial in $\lambda$ of degree $n$ (where $A$ is an $n\times n$ matrix), called the characteristic polynomial. The roots of this polynomial are the eigenvalues of $A$.

Drawback: Complex Numbers

A significant challenge is that not all polynomials have real roots. This means that some matrices might have eigenvalues that are complex numbers. Therefore, we need to take a detour into the realm of complex numbers to fully understand and analyze eigenvalues and eigenvectors.

A Bit More Formal

Definition:

Given a square matrix $A\in {\mathbb{R}}^{n\times n}$, a scalar $\lambda \in \mathbb{C}$ is an eigenvalue of $A$, and a non-zero vector $\mathbf{v}\in {\mathbb{C}}^n$ is the corresponding eigenvector if:

$A\mathbf{v}=\lambda \mathbf{v}$

$(\lambda ,\mathbf{v})$ is called an eigenvalue-eigenvector pair. If $\lambda$ is a real number ($\lambda \in \mathbb{R}$), then it’s called a real eigenvalue, and the pair is a real pair.

Example: Consider the matrix $A=\left[\begin{array}{cc}0& -1\\ 1& 0\end{array}\right]$. To find the eigenvalues, we solve: $\det (A-\lambda I)=\det \left(\left[\begin{array}{cc}-\lambda & -1\\ 1& -\lambda \end{array}\right]\right)={\lambda }^2+1=0$

This equation has no real solutions. The solutions are complex: $\lambda =i$ and $\lambda =-i$, where $i=\sqrt{-1}$ is the imaginary unit.

Complex Numbers

What are complex numbers?

Complex numbers are numbers of the form $z=a+ib$, where $a,b\in \mathbb{R}$ and $i$ is the imaginary unit ($i^2=-1$). $a$ is the real part of $z$ (denoted as $\Re (z)$), and $b$ is the imaginary part (denoted as $\Im (z)$). The set of all complex numbers is denoted by $\mathbb{C}=\{a+ib:a,b\in \mathbb{R}\}$.

Operations with Complex Numbers

Keeping in mind that $i^2=-1$, we can perform arithmetic operations on complex numbers:

• Addition: $(a+ib)+(x+iy)=(a+x)+i(b+y)$
• Multiplication: $(a+ib)(x+iy)=(ax-by)+i(ay+bx)$ (Distribute and use $i^2=-1$).
• Complex Conjugate: The complex conjugate of $z=a+ib$ is $\bar{z}=a-ib$. The product of a complex number and its conjugate is a real number: $z\bar{z}=a^2+b^2$. This is useful for division.
• Division: To divide complex numbers, we multiply the numerator and denominator by the complex conjugate of the denominator: $\frac{a+ib}{x+iy}=\frac{(a+ib)(x-iy)}{(x+iy)(x-iy)}=\frac{(ax+by)+i(bx-ay)}{x^2+y^2}=\frac{ax+by}{x^2+y^2}+i\frac{bx-ay}{x^2+y^2}$ This expresses the division as a complex number in standard form.

More on Complex Numbers

Notation: Given a complex number $z=a+ib\in \mathbb{C}$:

1. $\Re (a+ib)=a$: the real part of $z$.
2. $\Im (a+ib)=b$: the imaginary part of $z$.
3. $|z|=\sqrt{a^2+b^2}$: the modulus (or magnitude or absolute value) of $z$, representing its distance from the origin in the complex plane.
4. $\bar{z}=a-ib$: the complex conjugate of $z$.

Properties: For $z=a+ib\in \mathbb{C}$:

• $|z{|}^2=a^2+b^2=(a+ib)(a-ib)=z\bar{z}$ (This relates the modulus to the complex conjugate)
• $\frac{1}{z}=\frac{\bar{z}}{|z{|}^2}=\frac{a-ib}{a^2+b^2}$ (This is the derivation of the division formula)
• $|z_1{z}_2|=|z_1||z_2|$ and $|z_1/z_2|=|z_1|/|z_2|$ provided $z_2\ne 0$

Complex Vectors and Matrices

Complex Matrices; Fast Fourier Transform... (MIT)

I suggest watching the first part of this lecture as it introduces the topic quite well…

We extend the concepts of vectors and matrices to the complex domain.

• ${\mathbb{C}}^n$ denotes the set of all $n$-dimensional column vectors with complex entries.
• ${\mathbb{C}}^{m\times n}$ denotes the set of all $m\times n$ matrices with complex entries.

Transposing Complex Vectors and Matrices

For a complex vector $\mathbf{v}$ and a complex matrix $A$:

• ${\mathbf{v}}^{\ast }$ denotes the conjugate transpose (or Hermitian transpose) of $\mathbf{v}$, obtained by taking the transpose of $\mathbf{v}$ and then taking the complex conjugate of each entry. If $\mathbf{v}=\left[\begin{array}{c}{v}_1\\ ⋮\\ v_n\end{array}\right]$, then ${\mathbf{v}}^{\ast }=[{\bar{v}}_1\dots {\bar{v}}_n]$.
• $A^{\ast }$ denotes the conjugate transpose of $A$. In other words, $A^{\ast }=(\bar{A}{)}^T=\overline{(A^T)}$.

Inner Product in ${\mathbb{C}}^n$:

For $\mathbf{v},\mathbf{w}\in {\mathbb{C}}^n$, the inner product is defined as:

$⟨\mathbf{v},\mathbf{w}⟩={\mathbf{w}}^{\ast }\mathbf{v}={\sum }_{i=1}^n{\bar{w}}_i{v}_i$

The norm (length) of a complex vector $\mathbf{v}$ is defined as:

$||\mathbf{v}||=\sqrt{⟨\mathbf{v},\mathbf{v}⟩}=\sqrt{{\mathbf{v}}^{\ast }\mathbf{v}}=\sqrt{{\sum }_{i=1}^n|v_i{|}^2}$

where $|v_i|$ is the modulus of the complex number $v_i$. Note that $||\mathbf{v}|{|}^2={\mathbf{v}}^{\ast }\mathbf{v}$

Canonical Notation (Linear Algebra in ${\mathbb{C}}^n$)

The standard linear algebra concepts of linear independence, span, and basis extend naturally to the complex vector space ${\mathbb{C}}^n$:

• Linear Independence: Vectors ${\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\in {\mathbb{C}}^n$ are linearly independent if ${\alpha }_1{\mathbf{v}}_1+\cdots +{\alpha }_k{\mathbf{v}}_k=0$ implies that all scalars ${\alpha }_i$ are zero: ${\alpha }_1=\cdots ={\alpha }_k=0$, where ${\alpha }_i\in \mathbb{C}$.

• Span: The span of vectors ${\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\in {\mathbb{C}}^n$ is the set of all their linear combinations: $\text{Span}\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}=\left\{\mathbf{x}\in {\mathbb{C}}^n|\mathbf{x}={\sum }_{i=1}^k{\alpha }_i{\mathbf{v}}_i,{\alpha }_i\in \mathbb{C}\right\}$

• Basis: A basis for a subspace $S\subseteq {\mathbb{C}}^n$ is a set of linearly independent vectors in $S$ that spans $S$.

Why Complex Numbers, Vectors, and Matrices?

The use of complex numbers, vectors, and matrices in the study of eigenvalues and eigenvectors is motivated by the Fundamental Theorem of Algebra, which guarantees the existence of eigenvalues for any square matrix.

Theorem (Fundamental Theorem of Algebra)

Any non-constant polynomial $P(z)$ of degree $n\ge 1$ with complex coefficients has at least one complex root (or zero). In other words, there exists at least one $\lambda \in \mathbb{C}$ such that $P(\lambda )=0$. A formal proof of this theorem requires complex analysis and is beyond the scope of this lecture.

Repeated Roots and Multiplicity: A polynomial can have repeated roots. For example, the polynomial $(z-2{)}^3(z+1)$ has the root 2 repeated three times and the root $-1$ appearing once.

Corollary (Complete Factorization)

Any non-constant polynomial $P(z)$ of degree $n\ge 1$ with complex coefficients $a_i\in \mathbb{C}$ ($a_n\ne 0$) can be factored completely into linear terms:

$P(z)=a_n(z-{\lambda }_1)(z-{\lambda }_2)\dots (z-{\lambda }_n)$

Where:

• $a_n$ is the leading coefficient (non-zero).
• ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n\in \mathbb{C}$ are the roots (zeros) of the polynomial, not necessarily distinct.
• The number of times a root $\lambda$ appears in the factorization is its algebraic multiplicity.

Connection to Eigenvalues:

The characteristic polynomial of a matrix $A\in {\mathbb{R}}^{n\times n}$ is given by:

$p_A(\lambda )=\det (A-\lambda I)$

This is a polynomial of degree $n$ in the variable $\lambda$. By the Fundamental Theorem of Algebra, the characteristic polynomial always has exactly $n$ complex roots, counting multiplicities.

Key Implication:

• Existence of Eigenvalues: Every square matrix $A\in {\mathbb{R}}^{n\times n}$ has exactly $n$ eigenvalues, counting algebraic multiplicities. These eigenvalues might be real or complex, and some might be repeated.

This fact justifies our use of complex numbers. Even if we start with a real matrix, its eigenvalues (and consequently, its eigenvectors) might be complex. Working with complex numbers ensures we can always find a complete set of eigenvalues for any square matrix, forming a foundational concept in linear algebra. Without complex numbers we wouldn’t be able to fully analyze the behavior of many linear transformations described by matrices.

Revisiting Example 1

Recall the matrix $A=\left[\begin{array}{cc}0& -1\\ 1& 0\end{array}\right]$ from above. We found the eigenvalues to be ${\lambda }_1=i$ and ${\lambda }_2=-i$. Now let’s find the corresponding eigenvectors.

Suggestion

I’d suggest you simply try to generate independent vectors in the null space using Gauss-Jordan instead…

For ${\lambda }_1=i$:

We need to solve $(A-{\lambda }_{1}I)\mathbf{v}=0$:

$\left[\begin{array}{cc}-i& -1\\ 1& -i\end{array}\right]\left[\begin{array}{c}{v}_1\\ v_2\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$

This leads to the system of equations:

Let $v_1=a+ib$ for some real numbers $a$ and $b$. From the first equation, $v_2=-i{v}_1=-i(a+ib)=b-ia$. Substituting in the second equation yields $a+ib-i(b-ia)=a+ib-ib-a=0$ so the equations are consistent and $v_1=a+ib,v_2=b-ia$ is a solution for any $a$ and $b$. Choosing $a=1$ and $b=0$ gives $v_1=1$, $v_2=-i$. Hence the eigenvector corresponding to ${\lambda }_1=i$ is ${\mathbf{v}}_1=\left[\begin{array}{c}1\\ -i\end{array}\right]$.

For ${\lambda }_2=-i$:

Similarly, solving $(A-{\lambda }_{2}I)\mathbf{v}=0$:

$\left[\begin{array}{cc}i& -1\\ 1& i\end{array}\right]\left[\begin{array}{c}{v}_1\\ v_2\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$ Let $v_1=a+ib$ for some real numbers $a$ and $b$. From the first equation, $v_2=i{v}_1=i(a+ib)=-b+ia$. Substitute in the second equation to get $a+ib+i(-b+ia)=a+ib-ib-a=0$ Choosing $a=1$ and $b=0$ gives the eigenvector corresponding to ${\lambda }_2=-i$ as ${\mathbf{v}}_2=\left[\begin{array}{c}1\\ i\end{array}\right]$.

The Theory in General

This section delves into the core concepts and general results concerning eigenvalues and eigenvectors.

Observations and Definitions:

Let $A\in {\mathbb{R}}^{n\times n}$ be a square matrix.

Eigenvalue and Eigenvector

A scalar $\lambda \in \mathbb{C}$ is an eigenvalue of $A$ if there exists a non-zero vector $\mathbf{v}\in {\mathbb{C}}^n$ such that $A\mathbf{v}=\lambda \mathbf{v}$. The vector $\mathbf{v}$ is called an eigenvector corresponding to the eigenvalue $\lambda$. This fundamental equation means that applying the linear transformation represented by $A$ to the eigenvector $\mathbf{v}$ simply scales the vector by a factor of $\lambda$.

Real Eigenvalues and Real Eigenvectors

If $A$ is a real matrix and $\lambda$ is a real eigenvalue, then there exists a corresponding real eigenvector.

Justification: Suppose $\lambda$ is a real eigenvalue. The equation $(A-\lambda I)\mathbf{v}=0$ is a system of linear equations with real coefficients (since $A$ is real and $\lambda$ is real). Gaussian elimination, our method for solving systems of linear equations, involves only arithmetic operations (addition, subtraction, multiplication, division) on the coefficients. Therefore, if the system has a non-trivial solution (which it must, since $\lambda$ is an eigenvalue), it must have a real solution. Thus, there exists a real eigenvector $\mathbf{v}$ corresponding to the real eigenvalue $\lambda$.

Real vs. Complex Eigenvalues

While the characteristic equation always has roots in the complex numbers, we often focus on real eigenvalues and eigenvectors for applications in real vector spaces. However, the theory extends naturally to complex vector spaces. If $\lambda$ is a complex eigenvalue, the corresponding eigenvectors will generally also be complex.

Eigenspace

The set of all eigenvectors corresponding to a particular eigenvalue $\lambda$, together with the zero vector, forms a subspace of ${\mathbb{C}}^n$ called the eigenspace associated with $\lambda$. It’s denoted by $E_{\lambda }=N(A-\lambda I)$, the nullspace of $(A-\lambda I)$. The dimension of the eigenspace is called the geometric multiplicity of the eigenvalue.

Complex Eigenvalues and Conjugate Pairs

If $A$ is a real matrix and $(\lambda ,\mathbf{v})$ is a complex eigenvalue-eigenvector pair, then the complex conjugate $(\bar{\lambda },\bar{\mathbf{v}})$ is also an eigenvalue-eigenvector pair. Where $\bar{\mathbf{v}}$ is the component-wise conjugate of the eigenvector $\mathbf{v}$.

Slightly different than what the prof did…

Proof: Since $(\lambda ,\mathbf{v})$ is an eigenvalue-eigenvector pair, we have $A\mathbf{v}=\lambda \mathbf{v}$. Taking the complex conjugate of both sides:

$\overline{A\mathbf{v}}=\overline{\lambda \mathbf{v}}$

Since $A$ is a real matrix $\bar{A}=A$, and using properties of complex conjugation:

$A\bar{\mathbf{v}}=\bar{\lambda }\bar{\mathbf{v}}$

This equation shows that $\bar{\mathbf{v}}$ is an eigenvector of $A$ corresponding to the eigenvalue $\bar{\lambda }$. Therefore, complex eigenvalues of real matrices always come in conjugate pairs.

Eigenvalues of Orthogonal Matrices

If $Q\in {\mathbb{R}}^{n\times n}$ is an orthogonal matrix ($Q^{T}Q=I$, and since $Q$ is real, $Q^T=Q^{\ast }$) and $\lambda \in \mathbb{C}$ is an eigenvalue of $Q$, then $|\lambda |=1$. This means complex eigenvalues of real orthogonal matrices lie on the unit circle in the complex plane.

Proof: Let $\mathbf{v}\in {\mathbb{C}}^n$ be a (possibly complex) eigenvector corresponding to the eigenvalue $\lambda$. Then $Q\mathbf{v}=\lambda \mathbf{v}$.

1. Inner Product: Consider the inner product of $Q\mathbf{v}$ with itself in ${\mathbb{C}}^n$:

   $(Q\mathbf{v}{)}^{\ast }(Q\mathbf{v})=(\lambda \mathbf{v}{)}^{\ast }(\lambda \mathbf{v})$

2. Properties of Conjugate Transpose: Using properties of the conjugate transpose, we have $(Q\mathbf{v}{)}^{\ast }={\mathbf{v}}^{\ast }{Q}^{\ast }$ and $(\lambda \mathbf{v}{)}^{\ast }=\bar{\lambda }{\mathbf{v}}^{\ast }$. Note that for a real matrix like $Q,Q^{\ast }=Q^T$. Thus:

   ${\mathbf{v}}^{\ast }{Q}^{\ast }Q\mathbf{v}=\bar{\lambda }\lambda {\mathbf{v}}^{\ast }\mathbf{v}$

3. Orthogonality and Modulus: Since $Q$ is orthogonal, $Q^{T}Q=I=Q^{\ast }Q$. Also, ${\mathbf{v}}^{\ast }\mathbf{v}=||\mathbf{v}|{|}^2$ and $\bar{\lambda }\lambda =|\lambda {|}^2$:

   ${\mathbf{v}}^{\ast }I\mathbf{v}=|\lambda {|}^2||\mathbf{v}|{|}^2$

   $||\mathbf{v}|{|}^2=|\lambda {|}^2||\mathbf{v}|{|}^2$

4. Conclusion: Since $\mathbf{v}$ is an eigenvector, it is non-zero, so $||\mathbf{v}|{|}^2\ne 0$. Dividing both sides by $||\mathbf{v}|{|}^2$:

   $|\lambda {|}^2=1⟹|\lambda |=1$

This result has important implications in geometry and transformations. Orthogonal transformations preserve lengths and angles. The fact that their eigenvalues have modulus 1 reflects this preservation of length: eigenvectors are only scaled by a factor of 1 under the transformation, possibly with a rotation in the complex plane if $\lambda$ is complex.

Continue here: 24 Explicit Fibonacci Formula, Eigenvalue and Eigenvector properties, Trace