Joint Probability Distributions

Joint Probability Distributions

Learning Objectives

Understand joint probability mass and density functions for multiple random variables.
Calculate and interpret marginal and conditional probability distributions.
Determine whether two random variables are statistically independent.
Calculate and interpret covariance and the correlation coefficient.
Understand the properties and applications of the Bivariate Normal Distribution.

Joint probability distributions extend the concepts of single random variables to multiple variables, allowing engineers to analyze complex systems where multiple factors interact. This lesson covers joint, marginal, and conditional distributions, as well as measures of linear relationship like covariance and correlation.

Simultaneous Variables

In many engineering applications, we need to understand the relationship between two or more random variables simultaneously. For example, a structural engineer might study the joint distribution of wind speed ( $X$ ) and atmospheric pressure ( $Y$ ) during a hurricane, or a transportation engineer might model the number of cars ( $X$ ) and trucks ( $Y$ ) arriving at a toll booth.

Joint Probability Mass and Density Functions

Describing the simultaneous behavior of multiple random variables.

Joint Probability Mass Function (Discrete)

For two discrete random variables $X$ and $Y$ , the joint probability mass function $f(x,y)$ gives the probability that $X$ takes the specific value $x$ AND $Y$ takes the specific value $y$ simultaneously.

Joint Probability Mass Function

Formula for calculating joint discrete probability.

f(x, y) = P(X = x, Y = y)

Variables

Symbol	Description	Unit
$f(x, y)$	Joint probability mass function	-
$X, Y$	Discrete random variables	-
$x, y$	Specific values of the random variables	-
$P$	Probability	-

Joint PMF Conditions

$f(x, y) \ge 0$ for all $(x, y)$ .
$\sum_{x} \sum_{y} f(x, y) = 1$ .

Joint Probability Density Function (Continuous)

For two continuous random variables $X$ and $Y$ , the joint probability density function $f(x,y)$ represents the probability that $(X, Y)$ falls within a specific two-dimensional region $R$ in the $xy$ -plane. The probability is the volume under the surface $f(x,y)$ over region $R$ .

Joint Probability Density Function

Formula for calculating joint continuous probability over a region R.

P((X, Y) \in R) = \iint_R f(x, y) \, dx \, dy

Variables

Symbol	Description	Unit
$P((X, Y) \in R)$	Probability that X and Y fall in region R	-
$R$	Two-dimensional region in the xy-plane	-
$f(x, y)$	Joint probability density function	-
$dx, dy$	Differentials for x and y	-

Joint PDF Conditions

$f(x, y) \ge 0$ for all $(x, y)$ .
$\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f(x, y) \, dx \, dy = 1$ .

Marginal Distributions

Isolating the behavior of one variable from the joint distribution.

Isolating Variables

Sometimes we have the joint distribution of $X$ and $Y$ , but we only care about the distribution of $X$ alone, regardless of $Y$ . This is called the marginal distribution.

Marginal Probability Distributions

To find the marginal distribution of one variable, we sum (or integrate) out the other variable over its entire range. Similarly, $h(y)$ is the marginal distribution for $Y$ found by summing or integrating out $x$ .

Marginal Distribution (Discrete)

Formula for calculating marginal discrete probability.

g(x) = \sum_{y} f(x, y)

Variables

Symbol	Description	Unit
$g(x)$	Marginal distribution of X	-
$f(x, y)$	Joint probability mass function	-
$y$	All possible values of Y	-

Marginal Distribution (Continuous)

Formula for calculating marginal continuous probability.

g(x) = \int_{-\infty}^{\infty} f(x, y) \, dy

Variables

Symbol	Description	Unit
$g(x)$	Marginal distribution of X	-
$f(x, y)$	Joint probability density function	-
$dy$	Differential for y	-

Conditional Distributions and Independence

How knowledge of one variable affects the probability distribution of another.

Conditional Probability Distribution

The probability distribution of $X$ , given that $Y$ has taken a specific value $y$ . This is analogous to basic conditional probability ( $P(A|B) = P(A \cap B) / P(B)$ ). Similarly, $f(y|x) = \frac{f(x, y)}{g(x)}$ provided $g(x) > 0$ .

Conditional Probability Distribution

Formula for conditional probability distribution.

f(x|y) = \frac{f(x, y)}{h(y)} \quad \text{provided } h(y) > 0

Variables

Symbol	Description	Unit
$f(x\|y)$	Conditional distribution of X given Y=y	-
$f(x, y)$	Joint probability function	-
$h(y)$	Marginal distribution of Y	-

Independence of Random Variables

Two random variables $X$ and $Y$ are independent if and only if their joint probability distribution is the product of their marginal distributions for all possible values of $(x, y)$ . If this holds true, knowing the value of $X$ gives no information about the value of $Y$ . (e.g., The compressive strength of concrete from Plant A vs. Plant B).

Independence Condition

Mathematical condition for independence of random variables.

f(x, y) = g(x) \cdot h(y)

Variables

Symbol	Description	Unit
$f(x, y)$	Joint probability function	-
$g(x)$	Marginal distribution of X	-
$h(y)$	Marginal distribution of Y	-

Covariance and Correlation

Measuring the linear relationship between two random variables.

Covariance ( $\sigma_{xy}$ )

A measure of how much two random variables change together. A positive covariance indicates that when $X$ is above its mean, $Y$ also tends to be above its mean (e.g., traffic volume and noise levels on a highway). A negative covariance indicates an inverse relationship (e.g., the age of an asphalt pavement and its flexibility).

Covariance Formula

Formula for calculating the covariance between two random variables.

\sigma_{xy} = E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - \mu_X\mu_Y

Variables

Symbol	Description	Unit
$\sigma_{xy}$	Covariance between X and Y	-
$E$	Expected value operator	-
$X, Y$	Random variables	-
$\mu_X, \mu_Y$	Means of X and Y	-

Properties of Covariance

If $X$ and $Y$ are statistically independent, their covariance is zero ( $\sigma_{xy} = 0$ ).
However, a covariance of zero does not necessarily mean they are independent (they could have a non-linear relationship).

Correlation Coefficient ( $\rho_{xy}$ )

A standardized measure of the linear relationship between two variables. Covariance depends on the units of $X$ and $Y$ , making it hard to interpret the strength of the relationship. The correlation coefficient scales covariance by the standard deviations of both variables, producing a dimensionless value between -1 and 1.

Correlation Coefficient Formula

Formula for calculating the correlation coefficient.

\rho_{xy} = \frac{\sigma_{xy}}{\sigma_x \sigma_y}

Variables

Symbol	Description	Unit
$\rho_{xy}$	Correlation coefficient between X and Y	-
$\sigma_{xy}$	Covariance between X and Y	-
$\sigma_x, \sigma_y$	Standard deviations of X and Y	-

Interpretation of Correlation

$\rho_{xy} = 1$ : Perfect positive linear relationship.
$\rho_{xy} = -1$ : Perfect negative linear relationship.
$\rho_{xy} = 0$ : No linear relationship.

The Bivariate Normal Distribution

The foundational model for two correlated continuous variables.

Bivariate Normal Distribution

When two continuous random variables are individually normally distributed and correlated, their joint behavior is described by the bivariate normal distribution. Its PDF forms a 3-dimensional bell surface (a mound) whose orientation depends on the correlation $\rho$ .

Key Properties of Bivariate Normal Distribution

The marginal distributions $g(x)$ and $h(y)$ are both normal.
The conditional distributions $f(x|y)$ and $f(y|x)$ are both normal.
If the correlation $\rho_{xy} = 0$ for a bivariate normal distribution, then $X$ and $Y$ are strictly independent. (This is a special property; for other distributions, $\rho=0$ does not guarantee independence).

Interactive Simulation

Interact with the simulation below to visualize joint probability distributions and marginals.

Engineering Data Analysis

Discrete Joint Probability Explorer

Probability Distribution Table

$X \setminus Y$	$Y = 1$	$Y = 2$	$Y = 3$	$g(x)$
$X = 10$				0.150
$X = 20$				0.600
$X = 30$				0.250
$h(y)$	0.200	0.500	0.300	1.000

Mean

\mu_X

21.00

Mean

\mu_Y

2.10

Covariance

\text{Cov}(X,Y)

2.9000

If $\text{Cov}(X,Y) > 0$ , X and Y tend to increase together. If $\text{Cov}(X,Y) < 0$ , they vary inversely. If $0$ , there is no linear relationship.

Probability Distribution Visualizer

Y=1

Y=2

Y=3

g(x)

X=10

0.10

0.05

X=20

0.10

0.40

0.10

X=30

0.05

0.20

h(y)

Bubble size & opacity indicate the magnitude of the joint probability $f(x,y)$ .

Green bars represent the marginal distributions $g(x)$ and $h(y)$ .

Interactive Simulation

Explore the contours, density surface, and marginal distributions of a Bivariate Normal distribution by adjusting standard deviations and the correlation coefficient.

Engineering Data Analysis • Topic 7

Bivariate Normal Distribution Contours

Std Dev X (

\sigma_X

)2.0

Std Dev Y (

\sigma_Y

)2.0

Correlation (

\rho

)0.50

Covariance Matrix (

\mathbf{\Sigma}

)

\mathbf{\Sigma} = \begin{bmatrix} 4.00 & 2.00 \\ 2.00 & 4.00 \end{bmatrix}

Contours at 1σ, 2σ, and 3σ

Key Takeaways

Joint Distributions ( $f(x,y)$ ): Describe the simultaneous behavior of two random variables.
Marginal Distributions ( $g(x), h(y)$ ): Isolate one variable by summing or integrating out the other.
Conditional Distributions ( $f(x|y)$ ): The behavior of $X$ given a specific value of $Y$ .
Independence: If $X$ and $Y$ are independent, $f(x,y) = g(x) \cdot h(y)$ .
Covariance and Correlation: Measure the linear relationship between variables. Correlation ( $\rho$ ) is standardized, always falling between -1 and 1.
Bivariate Normal: The standard 3D bell-shaped curve for two correlated continuous variables.

Previous TopicContinuous Probability Distributions - Examples & Applications

Quiz Me

Next TopicJoint Probability Distributions - Examples & Applications

Prev Next

Quiz Me

Learning Objectives

Simultaneous Variables

Joint Probability Mass and Density Functions

Joint Probability Mass Function (Discrete)

Joint Probability Mass Function

Joint PMF Conditions

Joint Probability Density Function (Continuous)

Joint Probability Density Function

Joint PDF Conditions

Marginal Distributions

Isolating Variables

Marginal Probability Distributions

Marginal Distribution (Discrete)

Marginal Distribution (Continuous)

Conditional Distributions and Independence

Conditional Probability Distribution

Conditional Probability Distribution

Independence of Random Variables

Independence Condition

Covariance and Correlation

Covariance (σxy\sigma_{xy}σxy​)

Covariance Formula

Properties of Covariance

Correlation Coefficient (ρxy\rho_{xy}ρxy​)

Correlation Coefficient Formula

Interpretation of Correlation

The Bivariate Normal Distribution

Bivariate Normal Distribution

Key Properties of Bivariate Normal Distribution

Interactive Simulation

Engineering Data Analysis

Probability Distribution Table

Probability Distribution Visualizer

Interactive Simulation

Engineering Data Analysis • Topic 7

Covariance ( $\sigma_{xy}$ )

Correlation Coefficient ( $\rho_{xy}$ )