Math 371 - Alexiades Condition Number of a matrix A
Let A be n×n nonsingular (invertible) matrix, and consider solving the linear system Ax=b.
Any numerical solution is an approximation x̄ ≈ x.
The error e = x−x̄ is related to the residual r = b−Ax̄   by: Ae = A(x-x̄) = b − Ax̄ = r, so e = A⁻¹ r.
It is possible that r is "small" but e is "large", if A⁻¹ is "large".
The system Ax=b is called well-conditioned if small residual implies small error.
It is called ill-conditioned if small residual does NOT imply small error.

The condition number κ(A) quantifies the effect of errors on the solution x of Ax=b.

1. κ(A) = ∥A∥ ∥A⁻¹∥ = max ( ∥Ax∥ / ∥x∥ ) / min ( ∥Ax∥ / ∥x∥ ) = |largest eigenvalue| / |smallest eigenvalue| = (max stretching) / (min stretching)
   
   
2. κ(A) ≥ 1 for any A ,  κ(I) = 1 ,  κ(singular matrix) = ∞ ,  κ(cA) = κ(A) for c≠0 ,  κ( diag(d₁,d₂,...,d_n) ) = max|d_i| / min|d_i|
     κ IS affected by scaling of problem (relative size of entries).
   
   
3. Theorem: absolute error estimate:   ∥A∥⁻¹ ∥r∥ ≤ ∥x−x̄∥ ≤ ∥A⁻¹∥ ∥r∥  
     relative error estimate:   1/κ (∥r∥ / ∥b∥) ≤ ∥x−x̄∥ / ∥x∥ ≤ κ (∥r∥ / ∥b∥)   i.e. 1/κ (rel.residual) ≤ rel.error ≤ κ (rel.residual)
   
   
4. κ(A) is a relative error magnification factor measuring sensitivity of solution to errors in data:
     rel.error in sol ≤ κ • rel.error in A :   ∥Δx∥ / ∥x∥ ≤ κ ∥ΔA∥ / ∥A∥
     rel.error in sol ≤ κ • rel.error in b :   ∥Δx∥ / ∥x∥ ≤ κ ∥Δb∥ / ∥b∥
     rel.error in sol ≤ κ • rel.error in residual:   ∥Δx∥ / ∥x∥ ≤ κ ∥Δr∥ / ∥b∥
   κ near 1 means well-conditioned system ,  κ large means ill-conditioned
   
   
5. κ(A) is a better measure of "nearness to singularity" than det(A) in solving Ax=b !!!
     e.g. A = diag(0.1, ... , 0.1) of size 100×100 has det=10⁻¹⁰⁰ but κ(A) = 1, perfectly well-conditioned!
        in fact this Ax=b is trivially (and exactly) solvable: x_i = b_i/0.1 = 10 b_i
        So, detA=0 iff A singular BUT detA ≈ 0 does NOT imply A ≈ singular!!!
   
   
6. Rule of thumb:   In solving Ax=b, the computation loses about log₁₀κ(A) digits relative to accuracy of input,
   i.e. if κ = 10^p then about p decimals will be lost in solution process
   
   
7. Hilbert matrix: H = [h_ij] = [ 1 / (i+j−1) ]   (Hilbert 1894)   e.g. hilb(3) = [ 1 , 1/2 , 1/3 ; 1/2 , 1/3, 1/4 ; 1/3 , 1/4 , 1/5 ]
     is notoriously ill-conditioned:   Matlab: H=hilb(3) has cond(H)=524.06 ;   H=hilb(30) has cond(H)=8.5e19 !!!
   
   
8. Residual correction method: Accuracy of an ill-conditioned system may be improved:
   solve Ae = r for e , then   x̄_new = x̄ + ē ; repeat. Often a couple of iterations improve accurary...

---