Numerical Differentiation

Numerical Differentiation

Learning Objectives

Understand the derivation of numerical differentiation from Taylor series expansions.
Apply forward, backward, and centered finite difference formulas to approximate derivatives.
Evaluate the impact of truncation and round-off errors in numerical differentiation.
Apply techniques such as Richardson extrapolation and data smoothing to improve accuracy.
Describe advanced methods like complex step differentiation and automatic differentiation.

Numerical differentiation deals with approximating the derivative of a mathematical function or discrete data. It is fundamentally derived from Taylor series expansions.

A collection of algorithms for estimating the derivative of a mathematical function or function data using values of the function and perhaps other knowledge about the function.

Truncation Error

The error created by approximating a mathematical procedure, such as using a finite number of terms in a Taylor series expansion to represent an infinite sum.

Round-off Error

The error introduced by the fact that computers can only represent numbers with a finite number of significant digits, becoming particularly problematic when subtracting two nearly equal numbers.

Derivation from Taylor Series

Taylor Series

An infinite sum of terms that are expressed in terms of the function's derivatives at a single point, used to approximate complex functions.

The mathematical foundation for numerical differentiation is the Taylor series. By expanding $f(x_{i+1})$ around $x_i$ , we get:

Taylor Series Expansion

Expands a function into an infinite sum of terms calculated from the values of its derivatives at a single point.

f(x_{i+1}) = f(x_i) + f'(x_i)h + \frac{f''(x_i)}{2!}h^2 + O(h^3)

Variables

Symbol	Description	Unit
$f(x_{i+1})$	Function value at the next point	-
$f(x_i)$	Function value at the current point	-
$f'(x_i)$	First derivative of the function at the current point	-
$f''(x_i)$	Second derivative of the function at the current point	-
$h$	Step size between points	-
$O(h^3)$	Truncation error of the order of $h^3$	-

Solving for $f'(x_i)$ yields the finite divided difference formulas. The truncated terms represent the theoretical truncation error of the approximation.

Finite Divided Differences for the First Derivative

The simplest approximations of the first derivative are based on truncating the Taylor series after the first derivative term.

Forward Difference Formula

Approximates the first derivative using the current and next points. Truncation Error $E_t = -\frac{f''(x_i)}{2}h = O(h)$

f'(x_i) \approx \frac{f(x_{i+1}) - f(x_i)}{h}

Variables

Symbol	Description	Unit
$f'(x_i)$	Approximate first derivative	-
$f(x_{i+1})$	Function value at the next point	-
$f(x_i)$	Function value at the current point	-
$h$	Step size	-

Backward Difference Formula

Approximates the first derivative using the current and previous points. Truncation Error $E_t = \frac{f''(x_i)}{2}h = O(h)$

f'(x_i) \approx \frac{f(x_i) - f(x_{i-1})}{h}

Variables

Symbol	Description	Unit
$f'(x_i)$	Approximate first derivative	-
$f(x_i)$	Function value at the current point	-
$f(x_{i-1})$	Function value at the previous point	-
$h$	Step size	-

Centered Difference Formula

Approximates the first derivative using the next and previous points, offering higher accuracy. Truncation Error $E_t = -\frac{f'''(x_i)}{6}h^2 = O(h^2)$

f'(x_i) \approx \frac{f(x_{i+1}) - f(x_{i-1})}{2h}

Variables

Symbol	Description	Unit
$f'(x_i)$	Approximate first derivative	-
$f(x_{i+1})$	Function value at the next point	-
$f(x_{i-1})$	Function value at the previous point	-
$h$	Step size	-

Centered Difference Accuracy

The centered difference formula is mathematically more accurate, possessing a truncation error of $O(h^2)$ , compared to $O(h)$ for the standard forward and backward differences. This is because the $O(h^2)$ term in the Taylor series exactly cancels out when subtracting the forward and backward expansions.

Formulas for Higher-Order Derivatives

By manipulating Taylor series expansions for multiple points (e.g., $x_{i+1}, x_{i-1}, x_{i+2}$ ), we can derive formulas for higher-order derivatives. The centered difference formulas for the second and third derivatives are:

Second Derivative Centered Difference

Approximates the second derivative with an error of $O(h^2)$ .

f''(x_i) \approx \frac{f(x_{i+1}) - 2f(x_i) + f(x_{i-1})}{h^2}

Variables

Symbol	Description	Unit
$f''(x_i)$	Approximate second derivative	-
$f(x_{i+1})$	Function value at next point	-
$f(x_i)$	Function value at current point	-
$f(x_{i-1})$	Function value at previous point	-
$h$	Step size	-

Third Derivative Centered Difference

Approximates the third derivative with an error of $O(h^2)$ .

f'''(x_i) \approx \frac{f(x_{i+2}) - 2f(x_{i+1}) + 2f(x_{i-1}) - f(x_{i-2})}{2h^3}

Variables

Symbol	Description	Unit
$f'''(x_i)$	Approximate third derivative	-
$f(x_{i+2})$	Function value at $x_i + 2h$	-
$f(x_{i+1})$	Function value at $x_i + h$	-
$f(x_{i-1})$	Function value at $x_i - h$	-
$f(x_{i-2})$	Function value at $x_i - 2h$	-
$h$	Step size	-

High-Accuracy Differentiation Formulas

Higher-accuracy formulas can be generated by including more terms from the Taylor series expansion. For example, a more accurate forward difference formula requires points $x_i$ , $x_{i+1}$ , and $x_{i+2}$ , achieving $O(h^2)$ without using a centered span.

Richardson Extrapolation

Richardson extrapolation is an elegant method to improve the accuracy of a derivative estimate by combining two less accurate estimates computed with different step sizes.

Generalized Richardson Extrapolation Concept

If $D(h)$ is an approximation of order $O(h^p)$ , we can combine estimates using step sizes $h_1$ and $h_2$ (where $h_2 = h_1/r$ , typically $r=2$ ) to eliminate the leading error term and obtain a higher-order estimate $O(h^{p+1})$ .

Generalized Richardson Extrapolation

Increases the accuracy of a derivative estimate by combining estimates from different step sizes.

D \approx D(h_2) + \frac{D(h_2) - D(h_1)}{r^p - 1}

Variables

Symbol	Description	Unit
$D$	Higher-order derivative estimate	-
$D(h_1)$	Lower-order derivative estimate with step size $h_1$	-
$D(h_2)$	Lower-order derivative estimate with step size $h_2$	-
$r$	Ratio of step sizes ( $h_1 / h_2$ )	-
$p$	Order of the approximation error	-

Classic Richardson Extrapolation for Centered Differences

Increases the accuracy from $O(h^2)$ to $O(h^4)$ when halving the step size ( $r=2$ ) for a centered difference ( $p=2$ ).

D \approx \frac{4}{3}D(h_2) - \frac{1}{3}D(h_1)

Variables

Symbol	Description	Unit
$D$	Higher-order derivative estimate ( $O(h^4)$ )	-
$D(h_1)$	Lower-order centered difference estimate with step size $h_1$	-
$D(h_2)$	Lower-order centered difference estimate with step size $h_2 = h_1 / 2$	-

Formulas for Unequally Spaced Data

In practice, experimental data points are often not evenly spaced. In this case, standard finite difference formulas cannot be applied. Instead, a Lagrange interpolating polynomial is fit through the adjacent points and then differentiated.

Unequally Spaced Differences

Computes the derivative at a point for three unequally spaced data points by differentiating a Lagrange interpolating polynomial.

f'(x_i) = \frac{-h_i}{h_{i-1}(h_{i-1} + h_i)}f(x_{i-1}) + \frac{h_i - h_{i-1}}{h_{i-1}h_i}f(x_i) + \frac{h_{i-1}}{h_i(h_{i-1} + h_i)}f(x_{i+1})

Variables

Symbol	Description	Unit
$f'(x_i)$	First derivative at the point $x_i$	-
$x_i$	Current data point	-
$x_{i-1}$	Previous data point	-
$x_{i+1}$	Next data point	-
$h_{i-1}$	Spacing between previous and current point ( $x_i - x_{i-1}$ )	-
$h_i$	Spacing between current and next point ( $x_{i+1} - x_i$ )	-

When $h_{i-1} = h_i = h$ , this formula correctly collapses to the standard centered difference.

Errors in Numerical Differentiation and Condition Number

Numerical differentiation is inherently unstable and represents an ill-conditioned problem. While reducing the step size $h$ initially decreases the truncation error, it significantly magnifies the round-off error.

Condition Number

A measure of how sensitive a function's output is to small changes or errors in its input. In numerical differentiation, it scales proportionally to $1/h$ .

Loss of Significance

The numerical calculation of a derivative requires subtracting two nearly equal function values, $f(x+h) - f(x)$ , and dividing by a very small number, $h$ . This process is highly susceptible to loss of significance (subtractive cancellation). The condition number for numerical differentiation approaches infinity as $h \to 0$ , meaning the problem becomes infinitely sensitive to the finite precision limits of floating-point arithmetic. Thus, total error forms a "U-shaped" curve: decreasing $h$ beyond an optimal point causes the round-off error to dominate, drastically degrading accuracy.

Differentiating Raw Data

Because numerical differentiation amplifies error by a factor proportional to $1/h$ , differentiating raw, scattered data directly often leads to useless results. The noise completely masks the true derivative.

Data Smoothing Before Differentiation

Because of the ill-conditioned nature of numerical differentiation, raw experimental data must almost always be smoothed before derivatives are taken.

Smoothing and Differentiating Data

STEP-BY-STEP

Visual Inspection: Plot the data to identify the level of noise and potential outliers.
Smoothing or Regression: Apply a low-pass filter (like a moving average) or fit a smooth curve (like a low-order polynomial regression or a smoothing spline) to the data.
Differentiation: Differentiate the fitted curve analytically, or apply numerical differentiation to the smoothed data points.

Partial Derivatives

For functions of multiple variables, partial derivatives are approximated by holding all other variables constant and applying the standard finite difference formulas to the variable of interest.

Advanced Differentiation Techniques

Beyond finite differences, modern engineering and machine learning heavily rely on computational differentiation techniques to avoid round-off errors and the need for analytical derivations.

Complex Step Differentiation

A method to compute exact derivatives of analytic functions using complex arithmetic, avoiding subtractive cancellation errors.

Complex Step Differentiation

Computes the derivative using a very small complex step $ih$ without subtracting function values.

f'(x) \approx \frac{\text{Im}(f(x + ih))}{h}

Variables

Symbol	Description	Unit
$f'(x)$	Derivative of the function	-
$f(x + ih)$	Function evaluated at a complex step	-
$\text{Im}(\dots)$	Imaginary part of the complex number	-
$h$	Very small step size (can be near machine epsilon)	-

Since there is no subtraction in the numerator, $h$ can be chosen as small as machine epsilon (e.g., $10^{-16}$ ) to achieve near-exact precision without catastrophic cancellation.

Automatic Differentiation (AD)

A set of techniques to numerically evaluate the derivative of a function specified by a computer program by repeatedly applying the chain rule to elementary operations.

Automatic Differentiation (AD)

AD computes exact derivatives by systematically applying the chain rule to the elementary operations (addition, multiplication, trigonometric functions) that make up a computer program. It is neither symbolic differentiation (which produces massive equations) nor numerical differentiation (which suffers from truncation and round-off error). AD allows for exact gradient evaluations at the cost of one forward evaluation, which is the foundational technology behind modern deep learning frameworks.

Civil Engineering Applications

In civil engineering, numerical differentiation is often used when analyzing experimental or field data:

Structural Health Monitoring: Calculating velocity and acceleration by taking the first and second numerical derivatives of discrete position data collected from displacement sensors on a bridge.
Geotechnical Engineering: Estimating hydraulic gradients from discrete piezometer head readings across an earth dam to analyze seepage forces.

Truncation vs. Round-off Errors

The total error in numerical differentiation is the sum of truncation error (from ignoring higher-order Taylor series terms) and round-off error (from floating-point precision limits in computers).

Truncation Error: Decreases as step size $h$ decreases.
Round-off Error: Increases rapidly as step size $h$ decreases due to loss of significance when subtracting two nearly equal numbers. Because of this inverse relationship, there is always an optimal step size $h$ that minimizes total error; making $h$ arbitrarily small will ruin the calculation.

Key Takeaways

Finite difference formulas are directly derived from the Taylor series expansion.
Derivatives can be approximated using forward, backward, or centered finite divided differences. Truncation errors are derived from the unused Taylor series terms.
For unequally spaced data, the derivative is derived from differentiating a Lagrange interpolating polynomial over adjacent points.
Centered differences are generally more accurate ( $O(h^2)$ ) than forward or backward differences ( $O(h)$ ) due to the cancellation of the $O(h^2)$ terms in the Taylor series.
Formulas for higher-order derivatives (2nd, 3rd) and high-accuracy formulas involve more neighboring data points.
Richardson extrapolation combines two estimates of lower accuracy to produce one of higher accuracy, with the generalized formula eliminating the leading error term.
Numerical differentiation is an inherently ill-conditioned problem because the condition number scales with $1/h$ . It suffers from severe loss of significance (subtractive cancellation) when $h$ becomes too small.
Because of noise amplification, data smoothing or regression must almost always precede the numerical differentiation of experimental data.
Complex step differentiation avoids subtractive cancellation entirely, allowing extremely small step sizes.
Automatic Differentiation (AD) provides exact analytical derivatives numerically by programmatically applying the chain rule to atomic operations.

Previous TopicCurve Fitting and Interpolation - Examples & Applications

Quiz Me

Next TopicNumerical Differentiation - Examples & Applications

Prev Next

Quiz Me