Lecture from 06.12.2024 | Video: Videos ETHZ
Motivation for Eigenvalues/Eigenvectors: Fibonacci Sequence
The Fibonacci sequence provides a classic example of how eigenvalues and eigenvectors can be used to elegantly solve problems that are otherwise computationally challenging.
The Fibonacci Sequence
The Fibonacci sequence is defined recursively:
- Base Cases: and
- Recursive Step: For ,
This generates the sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, …
The Challenge: An Explicit Formula
The recursive definition, while simple, is inefficient for calculating large Fibonacci numbers. Finding recursively requires computing all preceding terms. Our goal is to derive an explicit formula for , allowing direct calculation without recursion.
Matrix Representation: A Clever Trick
Let’s represent the Fibonacci recurrence using a matrix equation. Notice that:
Let’s define:
- as the vector of consecutive Fibonacci numbers.
- as the Fibonacci matrix.
Now, we can express the recurrence as: . Applying this repeatedly:
where .
This reveals that calculating reduces to efficiently computing . This type of system, where the output is fed back as input, is called a dynamical system.
Finding Eigenvalues and Eigenvectors of M
To compute efficiently, we’ll utilize eigenvalues and eigenvectors.
1. Finding Eigenvalues:
Eigenvalues are the solutions to the characteristic equation :
Using the quadratic formula:
Let (the golden ratio, often denoted by ) and .
2. Finding Eigenvectors:
For each , we solve to find the corresponding eigenvector . This is equivalent to finding the nullspace of the matrix .
For :
We need to solve:
Substituting the value of :
Let’s perform Gaussian elimination (row reduction) on the augmented matrix:
-
Swap rows:
-
Subtract times the first row from the second row:
Let . The first row gives us: Let , then Thus, an eigenvector corresponding to is .
For :
Following a similar procedure (row reduction on the augmented matrix of ), we obtain the eigenvector:
3. Linear Independence and Basis:
Since , the eigenvectors and are linearly independent. Because we have two linearly independent vectors in a 2-dimensional space (), they form a basis for . (Note: Eigenvectors corresponding to distinct eigenvalues are always linearly independent. However, having linearly independent eigenvectors for an matrix is not always guaranteed if eigenvalues are repeated.)
4. Expressing as a Linear Combination of Eigenvectors:
Since forms a basis for , we can express any vector in , including , as a linear combination of and :
This leads to the following system of linear equations:
From the second equation, we have . Substituting this into the first equation:
Therefore, . Since , we have .
Thus, we have expressed as a linear combination of the eigenvectors:
Deriving the Explicit Formula
-
Expressing :
-
Using Eigenvalue Property: Since , it follows (by induction) that . Therefore:
-
Extracting : Recall that . We are interested in the second component:
This is the explicit formula for the -th Fibonacci number. Notice that , so as becomes large, the second term becomes negligible, and grows approximately as , demonstrating exponential growth dominated by the golden ratio.
This general technique is applicable to many other problems involving recurrence relations and dynamical systems.
Key Properties and Results
Eigenvalues of Matrix Powers
If is an eigenvalue-eigenvector pair for , then for any positive integer , is an eigenvalue-eigenvector pair for . This can be proven by induction: . Intuitively, repeated applications of the transformation scale the eigenvector by powers of the eigenvalue.
Eigenvalues of the Inverse Matrix
If is invertible and is an eigenvalue-eigenvector pair for , then is an eigenvalue-eigenvector pair for . This holds because . (Since is invertible, ). Geometrically, if stretches an eigenvector by a factor , compresses it by the reciprocal factor.
Linear Independence of Eigenvectors
Proposition: If are eigenvectors of corresponding to distinct eigenvalues , then these eigenvectors are linearly independent. This result is crucial because it ensures that eigenvectors associated with distinct eigenvalues span different directions in the vector space.
Proof (by contradiction and induction)
This will be a proof by induction on the number of eigenvectors, .
Base Case (): A single eigenvector is non-zero by definition, and a single non-zero vector is always linearly independent.
Inductive Hypothesis: Assume that the statement is true for eigenvectors with distinct eigenvalues. That is, any set of eigenvectors corresponding to distinct eigenvalues is linearly independent.
Inductive Step: Consider a set of eigenvectors corresponding to distinct eigenvalues . Suppose, for the sake of contradiction, that these eigenvectors are linearly dependent. Then there exist scalars , not all zero, such that:
Without loss of generality, assume . Now, multiply both sides of equation () by the matrix :
Using the fact that , we get:
Since , the last term vanishes:
Now, we have a linear combination of eigenvectors equal to the zero vector. By the inductive hypothesis, these eigenvectors are linearly independent. Therefore, all the coefficients in equation () must be zero:
Since all eigenvalues are distinct ( for ), it must be that for . However, we assumed , which is a contradiction!
Therefore, our initial assumption that the eigenvectors are linearly dependent must be false. Hence, are linearly independent. This completes the induction.
This proof demonstrates a fundamental property of eigenvectors: those associated with distinct eigenvalues point in essentially different directions, contributing to a basis for the vector space if there are enough of them. This property is crucial for diagonalization, a powerful technique we’ll explore later, enabling us to simplify matrix operations and analyses.
Characteristic Polynomial and Eigenvalues
The characteristic polynomial of a matrix is a fundamental tool for finding the eigenvalues of the matrix. It provides a direct link between the determinant, a scalar value associated with the matrix, and the eigenvalues, which reveal crucial information about the matrix’s behavior as a linear transformation.
Definition: The characteristic polynomial of , denoted by , is defined as:
where:
- is a scalar variable (which we will solve for to find eigenvalues).
- is the identity matrix.
- denotes the determinant of a matrix.
Why is this called the characteristic polynomial?
The characteristic polynomial encodes intrinsic properties of the linear transformation represented by the matrix . Its roots, which are the eigenvalues, determine how the transformation scales vectors along specific directions (eigenvectors). The polynomial’s coefficients are also related to other important quantities associated with the matrix, such as its trace and determinant, as we’ll see later.
Derivation and Explanation
Recall that an eigenvalue and its corresponding eigenvector (non-zero) satisfy:
This can be rewritten as:
For this equation to have a non-trivial solution (i.e., ), the matrix must be singular (non-invertible). A matrix is singular if and only if its determinant is zero. Therefore:
This equation is called the characteristic equation. The left-hand side of this equation is the characteristic polynomial. The values of that satisfy this equation are the eigenvalues of .
Degree of the Polynomial
When you compute the determinant of , you will find that is a polynomial of degree with leading coefficient of .
The Fundamental Theorem of Algebra and Eigenvalues
The Fundamental Theorem of Algebra states that every non-constant polynomial with complex coefficients has at least one complex root. A corollary of this theorem states that a polynomial of degree has exactly complex roots, counting multiplicities.
Consequences for Eigenvalues
-
Existence: Since is a polynomial of degree , the Fundamental Theorem of Algebra guarantees that it has exactly complex roots. These roots are the eigenvalues of . Therefore, every matrix has exactly eigenvalues, counting multiplicities. Some eigenvalues might be real, some might be complex, and some might be repeated.
-
Counting Multiplicities: If an eigenvalue appears times as a root of the characteristic polynomial (i.e., the polynomial has a factor of ), then we say that has algebraic multiplicity .
Eigenvalues of and
A crucial property related to the characteristic polynomial is that a matrix and its transpose share the same eigenvalues. This stems from the fact that the determinant of a matrix is equal to the determinant of its transpose:
Applying this to the characteristic polynomial:
Since and have the same characteristic polynomial, they must have the same eigenvalues. This doesn’t mean they have the same eigenvectors; they usually don’t. However, the eigenvalues, and their algebraic multiplicities are identical for a matrix and its transpose. This simplifies certain analyses as we can choose to work with either or , whichever is more convenient, when determining eigenvalues.
The Trace of a Matrix
The trace of a square matrix is a fundamental scalar quantity that provides valuable information about the matrix’s properties, particularly its eigenvalues.
Definition
The trace of a square matrix (or ), denoted by or sometimes as , is defined as the sum of its diagonal elements:
Properties of the Trace
The trace operator possesses several important properties that make it a useful tool in linear algebra:
Lemma: For matrices (or ) and scalar :
- Linearity: and
- Invariance under Cyclic Permutations:
- , and so on for products of more matrices.
- Trace of Transpose: since the diagonal elements remain unchanged when transposing.
- Trace of a Scalar Multiple of the Identity: For any scalar , , since each diagonal entry of is equal to .
Proof of Cyclic Property:
-
: The entry of is given by . Therefore:
Changing the order of summation:
-
: Using the property repeatedly:
The Trace and Eigenvalues
One of the most important results concerning the trace is its relationship to the eigenvalues of a matrix.
Theorem
Let (or ), and let be its eigenvalues (counting algebraic multiplicities). Then:
-
Sum of Eigenvalues: The trace of is equal to the sum of its eigenvalues:
-
Product of Eigenvalues: The determinant of is equal to the product of its eigenvalues:
The core idea of this proof is to leverage the two different ways we can express the characteristic polynomial: the determinant form and the factored form (using eigenvalues). By comparing specific coefficients in these two forms, we establish the relationship between the trace and the eigenvalues.
Theorem: Let (or ) be a square matrix, and let be its eigenvalues (counting algebraic multiplicities). Then:
- (The determinant of A is the product of its eigenvalues).
- (The trace of A is the sum of its eigenvalues).
Proof
The characteristic polynomial of is defined as:
where:
- is a complex variable. We use a complex variable because even if A is a real matrix, its eigenvalues (the roots of the characteristic polynomial) might be complex. This ensures that we capture all possible eigenvalues.
- is the identity matrix.
Note to readers: I’m not entirely sure I got everything right, so please take this with a grain of salt. If you spot any mistakes, I’d appreciate it if you let me know.
1. Determinant as Product of Eigenvalues
By the Fundamental Theorem of Algebra, the characteristic polynomial can be factored as:
Where did the come from? The degree of in is , and the coefficient of is . The ‘s are precisely the eigenvalues of — the values of that make the characteristic polynomial zero. This factorization arises because the eigenvalues are the roots of the characteristic equation .
To prove that , we evaluate the characteristic polynomial at :
Using the factored form: Thus, .
2. Trace as Sum of Eigenvalues
We’ll prove by comparing the coefficients of the term in both the determinant form and the factored form of the characteristic polynomial.
(a) Factored Form: Expanding the factored form , the coefficient of the term is .
(b) Determinant Form: Consider . The only terms in the determinant expansion that contribute to the term are those formed by taking diagonal elements and one off-diagonal element. However, to obtain a term, we must choose all diagonal entries in the determinant expansion of . Every other permutation will result in fewer than factors of . The product of the diagonal elements of () gives us:
When you expand this, the coefficient of is . Since the sign of the permutation selecting only the diagonal elements is positive, the term in is .
(c) Equating Coefficients: Equating the coefficients of the term from both forms:
Continue here: 25 Eigenvalues, Eigenvectors, Complete Set of Eigenvectors, Diagonalization, Eigendecomposition