Table of Contents

Assigned Reading

Although our text does not conduct a formal probability review, it would be advised that you consult your prerequisite course's probability textbook. The free, online Khan Academy also has a nice collection of short probability and statistics tutorials covering a wide variety to topics applicable to this course.

Useful Equations

Rules of Integration

Chain Rule for Integration

Rules of Differentiation

Chain Rule for Differentiation

Laws for Probabilities of Events

Venn Diagram

Law of Addition

Definition of Conditional Probability

Bayes' Rule

Random Variables (RVs)

Conditional Probability

If X and Y are discrete RVs, the definition of conditional probability for all values of y such that Pr(Y = y) > 0 is

If X and Y are continuous RVs and have joint PDF f_XY(x,y), then the conditional PDF for all values of y such that f_Y(y) > 0 is

For RV X, we can find Pr(X <= x) by conditioning on RV Y. If Y is a discrete RV, then

If Y is a continuous RV with desity f_Y(y), then

Conditional Expectation

If X and Y are jointly discrete RVs, then for all values of y, such that Pr(Y = y) > 0,

Likewise, if X and Y are jointly continuous RVs, then for all values of y such that f_Y(y) > 0,

Computing Expectations by Conditioning

For RV X, we can find E[X] by conditioning on RV Y. If Y is a discrete RV, then

If Y is a continuous RV with density f_Y(y), then

Integer Series Identities

If 0 < p < 1, then

If a ≠ 1, then

If a ≠ 1, then

where, k factorial = k! = (k)(k - 1)(k - 2)(...)(1).

Combinations of Choosing k from n Elements

Frequently Used Random Variables

Exponential Continuous Random Variable

RV X with parameter α > 0 that supports [0, ∞)

Memoryless property:

Uniform Continuous Random Variable

RV X that supports [a, b]

Bernoulli Discrete Random Variable

RV X with parameter 0 < p < 1 that supports k = {0, 1} 

Binomial Discrete Random Variable

RV X with parameters positive integer n and 0 < p < 1 that supports k = {0, 1, ..., n} 

Geometric Discrete Random Variable

RV X with parameter 0 < p < 1 indicating the probability of success.

The number X of Bernoilli trials needed to get one success, supported on the set k = {1, 2, ...}

Memoryless property:

The number of Y = X - 1 failures before the first success, supported on the set k = {0, 1, ...}

Poisson Discrete Random Variable

RV X with parameter λ > 0 that supports k = {0, 1, ...}

Review of Random Variables (RVs)

Cumulative Distribution Function

A RV X is defined by a function F_X(x), called a cumulative distribution function (CDF) of X, which has the following properties:

Example

CDF of a RV:

Probability

For a RV X and any value a, F_X(a) represents the probability that X is less than or equal to a, i.e.:

Probability Space

F_X(x) defines a probability space, and

Continuous RVs

If the CDF of F_X(x) of the RV X is a continuous function of x, then X is a continuous RV.

Example

CDF of a continuous RV:

CDF of a non-continous RV:

Probability Density Function

If a RV X is continuous, and there is a function f_X(x), called the probability density function (PDF), such that:

then X is an absolutely continuous RV.

If X is differentiable at x, then:

The following are properties of a PDF:

Example

PDF of a continuous RV that is uniformly distributed on the interval [0,C]:

PDF of a continuous RV that is exponentially distributed with parameter α > 0:

Discrete Random Variables

If the CDF F_X(x) of a RV X is constant between any two consecutive values in a discrete set of increasing values of x, X is referred to as a discrete random variable.

Example

CDF of a discrete RV:

Probability Mass Function

If a RV X is discrete, and F_X(x), has step increases at the points {..., x_-1, x₀, x₁, ...}, then:

If there is a smallest value x_j at which a step increase occurs in F_X(x), then:

p_i^X is referred to as the probability mass function (PMF) of X. {..., x_-1, x₀, x₁, ...} is referred to as the support of X. Note that p_i^X at x = x_i is often referred to as p_X(x_i).

The following are properties of a PMF:

Example

A discrete RV X that is binomially distributed with parameters N and 0 <= p <= 1 with support x₀ = 0, x₁ = 1, ..., x_N = N and a PMF:

Expected Value

The expected value (i.e. mean or statistical average) of a RV X is given by:

Autocorrelation

The autocorrelation of a RV X is given by:

nth Moment

The nth moment of a RV X is given by:

Variance

The variance of a RV X is given by:

Standard Deviation

The standard deviation of a RV X is given by:

Expected Value of a Function

The expected value of a function of a RV X is given by:

Examples of Mean, Variance, Standard Deviation, etc.

Continuous RVs

Consider the RV X with uniform distribution over [0,1]:

Suppose h(x) = x^3/2:

Discrete RVs

Consider the RV X with binomial distribution and parameters N = 2 and p = 0.2. Support of X is {x₀ = 0, x₁ = 1, x₂ = 2}.

Suppose h(x) = x^1/2:

Joint Characterization of RVs

Jointly Continuous RVs

Two jointly (absolutely) continuous RVs X and Y are defined by a joint PDF:

and a joint CDF:

Jointly continuous RVs have the following properties:

Example

Consider RVs X and Y with a joint PDF:

Find Pr(X - Y < 1):

Two jointly discrete RVs X and Y are characterized by a support:

a joint PMF {p_i,j} for all i and j, and a joint CDF given by:

with the following properties:

Marginal PDFs

The PDFs of the individual absolutely continuous RVs, or marginal PDFs, are determined from the joint PDF by:

Marginal PMFs

The PMFs of the individual absolutely discrete RVs, or marginal PMFs, are determined from the joint PMF by:

Shorthand Notation

If no ambiguity results, subscripts or superscripts for the PDF, PMF, or CDF are often omitted. For example:

Cross-Correlation

Continuous RVs

The cross-correlation of two jointly absolutely continuous RVs X and Y is given by:

Discrete RVs

The cross-correlation of two jointly discrete RVs X and Y is given by:

Covariance

Conditional Probability

Continuous RVs

Conditional PDF

If X and Y are jointly absolutely continuous RVs, the conditional PDF of X given Y is defined as:

If f_Y(y) ≠ 0:

So, f_X|Y(y) is a valid PDF for the RV X given Y = y. This is written as X | Y = y.

Conditional CDF

If X and Y are jointly absolutely continuous RVs and f_Y(y) ≠ 0, the conditional CDF of X given Y = y is defined as:

If X and Y are jointly absolutely continuous RVs and Pr(Y in the set B) ≠ 0, the conditional CDF of X given Y in the set B is defined as:

Examples

Example 1

B = (-∞, a], then Y in the set B ↔ Y ≤ a.

Example 2

B = [0,1] U [3,4], then Y in the set B ↔ (0 ≤ Y ≤ 1 or 3 ≤ Y ≤ 4).

Example 3

For 0 ≤ y ≤ 1:

For y < 0 or y > 1, f_X|Y(x|y) = 0 and F_X|Y(x|y) is not defined.

Discrete RVs

Conditional PMF

Suppose X and Y are jointly discrete RVs with support {(x_i,y_j)}, i and j integers. The conditional PMF of X given Y is defined as:

If p_j^Y != 0:

So, p_i|j^X|Y, i = ..., -1, 0, 1, ... is a valid PMF for the RV X given Y = y_j, with support {..., x_-1, x₀, x₁, ...}.

Conditional CDF

If X and Y are jointly discrete RVs and p_j^Y != 0, the conditional CDF of X given Y = y_j is defined as:

If X and Y are jointly discrete RVs and Pr(Y in the set B) != 0, the conditional CDF of X given Y in the set B is defined as: