Tests of Hypotheses

While estimation focuses on finding the value of a population parameter, hypothesis testing focuses on making decisions about a population parameter based on sample data. An engineer might ask: "Does this new steel alloy have a mean tensile strength greater than $400 \text{ MPa}$?" or "Is the variance in asphalt thickness less than $5 \text{ mm}^2$?" Hypothesis testing provides a formal, objective framework to answer these yes-or-no questions.

The formal steps required to set up and evaluate a statistical test involve defining the null and alternative hypotheses, calculating a test statistic, and making a decision based on a P-value or critical value.

• A very small P-value (typically $\le 0.05$) indicates the observed data is highly unlikely under $H_0$, leading us to reject $H_0$.
• A large P-value indicates the data is consistent with $H_0$, so we "fail to reject $H_0$."

If the P-value $\le \alpha$, reject $H_0$. If the P-value $> \alpha$, fail to reject $H_0$.

Because we rely on partial information (a sample) to make decisions about a population, we are at risk of making mistakes. The risks inherent in statistical inference are classified into two types of errors.

• Power increases as the true difference (effect size) increases.
• Power increases as the significance level $\alpha$ increases (but this raises the risk of a Type I error).
• Power increases as the sample size $n$ increases. Engineers often calculate the minimum sample size needed to achieve a specific power (e.g., 80%) before running an expensive test.

Testing claims about the center of a population. Depending on whether the population variance is known or unknown, different test statistics are used.

• Z-Test (Variance Known): Rarely used in practice. Assumes population variance $\sigma^2$ is known.
• t-Test (Variance Unknown): The standard test. Uses the sample standard deviation $s$.

Comparing two different groups (e.g., compressive strength of Mix A vs. Mix B).

• Independent Samples (Pooled t-Test): Assumes the two populations have equal (but unknown) variances. The sample variances are pooled to estimate a single standard error.
• Independent Samples (Welch's t-Test): Does not assume equal variances. More robust and generally preferred.
• Paired t-Test (Dependent Samples): Used when observations are naturally paired or matched (e.g., measuring the stiffness of the exact same beam before and after a retrofitting procedure). The test is performed on the differences between paired values, treating them as a single sample.

Testing categorical outcomes (e.g., percentage of defective items). Uses the normal approximation (Z-test) if $n\pi_0 \ge 5$ and $n(1-\pi_0) \ge 5$.

Testing claims about the variability or consistency of a process. Uses the Chi-square ($\chi^2$) distribution. Highly sensitive to departures from normality in the population.

Checking if data follows a specific theoretical distribution. This is essential for validating the assumptions underlying other statistical methods.

• $H_0$: The data follows the specified distribution.
• $H_1$: The data does not follow the specified distribution.
• A large $\chi^2$ value means the observed data deviates significantly from what was expected, leading to rejection of $H_0$.

There is a direct, mathematical duality between Confidence Intervals and two-sided Hypothesis Tests. If a 95% CI for the mean $\mu$ is $[390, 410]$, then a two-sided hypothesis test (with $\alpha = 0.05$) will:

• Fail to reject $H_0: \mu = 400$ (because $400$ is inside the interval).
• Reject $H_0: \mu = 380$ (because $380$ is outside the interval).

Visualize the relationships between the null distribution, critical value, significance level ($\alpha$), Type I/II errors, and p-value by adjusting the sliders in the simulation below.