Symmetric Matrices, Quadratic Forms, and the Spectral Theorem

Every diagonalizable matrix has an eigenvector basis, but that basis need not be orthogonal. A real symmetric matrix has stronger structure: it scales independently along mutually perpendicular directions.

Symmetry connects a matrix to the inner product

A real matrix is symmetric when $A^{\mathsf T}=A$. It satisfies

$$\langle A\mathbf x,\mathbf y\rangle =\langle\mathbf x,A\mathbf y\rangle.$$

If $A\mathbf v=\lambda\mathbf v$ and $A\mathbf w=\mu\mathbf w$ with $\lambda\ne\mu$, symmetry gives

$$\lambda\langle\mathbf v,\mathbf w\rangle =\mu\langle\mathbf v,\mathbf w\rangle.$$

Thus eigenvectors for distinct eigenvalues are orthogonal.

The real spectral theorem

TheoremSpectral theorem for real symmetric matrices

Every real symmetric matrix admits an orthogonal matrix $Q$ and a real diagonal matrix $\Lambda$ such that

$$A=Q\Lambda Q^{\mathsf T}.$$

The statement contains three facts: every eigenvalue is real, an orthonormal eigenvector basis exists, and the matrix is diagonal in that basis. Conversely, every matrix $Q\Lambda Q^{\mathsf T}$ of this form is symmetric.

Complete the induction behind the spectral theorem

A real symmetric matrix also satisfies $A^*=A$ over the complex numbers. For $A\mathbf v=\lambda\mathbf v$, the scalar $\mathbf v^*A\mathbf v$ equals its conjugate, so

$$\lambda=\frac{\mathbf v^*A\mathbf v}{\mathbf v^*\mathbf v}\in\mathbb R.$$

The real and imaginary parts of the eigenvector satisfy the same real eigenvalue equation, and at least one is nonzero. Choose a real unit eigenvector $\mathbf q$. For $\mathbf w\perp\mathbf q$,

$$\langle A\mathbf w,\mathbf q\rangle =\langle\mathbf w,A\mathbf q\rangle=0.$$

Thus $\mathbf q^\perp$ is invariant and the restriction is still symmetric. Induction supplies an orthonormal eigenbasis there. The initial existence of a complex eigenvalue uses the fundamental theorem of algebra.

A quadratic form measures direction-dependent energy

A square matrix defines

$$q(\mathbf x)=\mathbf x^{\mathsf T}A\mathbf x.$$

The skew-symmetric part contributes zero, so $q$ depends only on $(A+A^{\mathsf T})/2$. With the symmetric spectral decomposition and $\mathbf y=Q^{\mathsf T}\mathbf x$,

$$q(\mathbf x)=\sum_i\lambda_i y_i^2.$$

Cross terms disappear, leaving independent squares along the eigenvector directions.

How a coordinate change removes cross terms

For $A=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, unit eigenvectors are $(1,1)/\sqrt2$ and $(1,-1)/\sqrt2$. Set

$$u=\frac{x+y}{\sqrt2},\qquad v=\frac{x-y}{\sqrt2}.$$

Then

$$2x^2+2xy+2y^2=3u^2+v^2.$$

The level set $q=1$ has semiaxes $1/\sqrt3$ and $1$ along these directions. Diagonalization identifies both simpler coordinates and the actual axes of the ellipse.

Eigenvalue signs determine definiteness

A matrix is positive definite when $\mathbf x^{\mathsf T}A\mathbf x>0$ for every nonzero $\mathbf x$. Positive semidefinite allows equality. For a real symmetric matrix,

$$A\succ0\Longleftrightarrow \lambda_i>0\text{ for every }i,$$ $$A\succeq0\Longleftrightarrow \lambda_i\ge0\text{ for every }i.$$

Level sets of a positive-definite quadratic form are ellipsoids. Small eigenvalues give flat directions and large eigenvalues give steep directions, explaining why condition number affects optimization speed.

Proof

If all eigenvalues are positive, every nonzero coordinate vector has at least one nonzero square, so $\sum_i\lambda_i y_i^2>0$. Conversely test the quadratic form on each unit eigenvector to obtain $\lambda_i>0$. Replacing strict inequalities by non-strict ones proves the semidefinite equivalence. For a level $c>0$, the equation is $\sum_i y_i^2/(c/\lambda_i)=1$, giving semiaxes $\sqrt{c/\lambda_i}$. The zero level contains only the origin, and negative levels are empty.

Positive definiteness gives a unique minimum

Let $A$ be symmetric positive definite and define $f(\mathbf x)=\tfrac12\mathbf x^{\mathsf T}A\mathbf x-\mathbf b^{\mathsf T}\mathbf x$. For $\mathbf x_*=A^{-1}\mathbf b$, expansion gives

$$f(\mathbf x)-f(\mathbf x_*) =\tfrac12(\mathbf x-\mathbf x_*)^{\mathsf T}A(\mathbf x-\mathbf x_*).$$

This is positive unless $\mathbf x=\mathbf x_*$, proving a unique minimum without multivariable calculus. In Hessian language, the Hessian is precisely $A$.

The Rayleigh quotient finds extreme directions

For nonzero $\mathbf x$, define

$$R_A(\mathbf x) =\frac{\mathbf x^{\mathsf T}A\mathbf x} {\mathbf x^{\mathsf T}\mathbf x}.$$

In an orthonormal eigenbasis, this is a weighted average of eigenvalues. Therefore

$$\lambda_{\min}\le R_A(\mathbf x)\le\lambda_{\max},$$

with equality in the corresponding extreme eigendirections. This variational view connects principal components, stability, and spectral graph theory.

More explicitly, the weights are $y_i^2/\sum_jy_j^2$, are nonnegative, and sum to one. Subtracting $\lambda_{\min}$ gives a sum of nonnegative terms. Equality requires every coordinate attached to a strictly larger eigenvalue to vanish. Thus equality holds on the entire minimum eigenspace; the maximum case is identical. This includes repeated extreme eigenvalues.

The complex counterpart is Hermitian

In a complex vector space, symmetry becomes $A^*=A$, the Hermitian condition. Hermitian matrices also have real eigenvalues and a unitary orthonormal eigenbasis. The algebra of conjugation is reviewed in complex numbers.

The same induction proves the Hermitian statement: the fundamental theorem of algebra supplies an eigenvector, the quotient $\mathbf v^*A\mathbf v/(\mathbf v^*\mathbf v)$ makes its eigenvalue real, and its complex orthogonal complement is invariant. The restriction remains Hermitian. Normalize and induct, starting from dimension one. Unlike the real case, there is no need to extract real or imaginary parts. The required fundamental theorem of algebra is proved in the complex-numbers chapter.

Exercises

ExerciseTest positive definiteness

Determine whether $A=\begin{pmatrix}2&1\\1&2\end{pmatrix}$ is positive definite.

Solution

Its eigenvalues are $3$ and $1$, both positive, so $A$ is positive definite.

ExerciseThe skew part disappears

For a real matrix $K^{\mathsf T}=-K$, prove $\mathbf x^{\mathsf T}K\mathbf x=0$.

Solution

The scalar equals its transpose, $\mathbf x^{\mathsf T}K^{\mathsf T}\mathbf x=-\mathbf x^{\mathsf T}K\mathbf x$, so it must vanish.

ExerciseWeights in a Rayleigh quotient

For $A=\operatorname{diag}(1,4)$, write the Rayleigh quotient of the unit vector $(\cos\theta,\sin\theta)$.

Solution

$R_A=\cos^2\theta+4\sin^2\theta$, which ranges from $1$ to $4$. The trigonometric parameterization enforces unit length.