Table of Contents
Assigned Reading
Although our text does not conduct a formal probability review, it would be advised that you consult your prerequisite course's probability textbook. The free, online Khan Academy also has a nice collection of short probability and statistics tutorials covering a wide variety to topics applicable to this course.
Useful Equations
Rules of Integration
Chain Rule for Integration
Rules of Differentiation
Chain Rule for Differentiation
Laws for Probabilities of Events
Venn Diagram
Law of Addition
Definition of Conditional Probability
Bayes' Rule
Random Variables (RVs)
Conditional Probability
If X and Y are discrete RVs, the definition of conditional probability for all values of y such that Pr(Y = y) > 0 is
If X and Y are continuous RVs and have joint PDF fXY(x,y), then the conditional PDF for all values of y such that fY(y) > 0 is
For RV X, we can find Pr(X <= x) by conditioning on RV Y. If Y is a discrete RV, then
If Y is a continuous RV with desity fY(y), then
Conditional Expectation
If X and Y are jointly discrete RVs, then for all values of y, such that Pr(Y = y) > 0,
Likewise, if X and Y are jointly continuous RVs, then for all values of y such that fY(y) > 0,
Computing Expectations by Conditioning
For RV X, we can find E[X] by conditioning on RV Y. If Y is a discrete RV, then
If Y is a continuous RV with density fY(y), then
Integer Series Identities
If 0 < p < 1, then
If a != 1, then
If a != 1, then
where, k factorial = k! = (k)(k - 1)(k - 2)(...)(1).
Combinations of Choosing k from n Elements
Frequently Used Random Variables
Exponential Continuous Random Variable
RV X with parameter α > 0 that supports [0, ∞)
Memoryless property:
Uniform Continuous Random Variable
RV X that supports [a, b]
Bernoulli Discrete Random Variable
RV X with parameter 0 < p < 1 that supports k = {0, 1}
Binomial Discrete Random Variable
RV X with parameters positive integer n and 0 < p < 1 that supports k = {0, 1, ..., n}
Geometric Discrete Random Variable
RV X with parameter 0 < p < 1 indicating the probability of success.
The number X of Bernoilli trials needed to get one success, supported on the set k = {1, 2, ...}
Memoryless property:
The number of Y = X - 1 failures before the first success, supported on the set k = {0, 1, ...}
Poisson Discrete Random Variable
RV X with parameter λ > 0 that supports k = {0, 1, ...}
Review of Random Variables (RVs)
Cumulative Distribution Function
A RV X is defined by a function FX(x), called a cumulative distribution function (CDF) of X, which has the following properties:
Example
CDF of a RV:
Probability
For a RV X and any value a, FX(a) represents the probability that X is less than or equal to a, i.e.:
Probability Space
FX(x) defines a probability space, and
Continuous RVs
If the CDF of FX(x) of the RV X is a continuous function of x, then X is a continuous RV.
Example
CDF of a continuous RV:
CDF of a non-continous RV:
Probability Density Function
If a RV X is continuous, and there is a function fX(x), called the probability density function (PDF), such that:
then X is an absolutely continuous RV.
If X is differentiable at x, then:
The following are properties of a PDF:
Example
PDF of a continuous RV that is uniformly distributed on the interval [0,C]:
PDF of a continuous RV that is exponentially distributed with parameter α > 0:
Discrete Random Variables
If the CDF FX(x) of a RV X is constant between any two consecutive values in a discrete set of increasing values of x, X is referred to as a discrete random variable.
Example
CDF of a discrete RV:
Probability Mass Function
If a RV X is discrete, and FX(x), has step increases at the points {..., x-1, x0, x1, ...}, then:
If there is a smallest value xj at which a step increase occurs in FX(x), then:
piX is referred to as the probability mass function (PMF) of X. {..., x-1, x0, x1, ...} is referred to as the support of X. Note that piX at x = xi is often referred to as pX(xi).
The following are properties of a PMF:
Example
A discrete RV X that is binomially distributed with parameters N and 0 <= p <= 1 with support x0 = 0, x1 = 1, ..., xN = N and a PMF:
Expected Value
The expected value (i.e. mean or statistical average) of a RV X is given by:
Autocorrelation
The autocorrelation of a RV X is given by:
nth Moment
The nth moment of a RV X is given by:
Variance
The variance of a RV X is given by:
Standard Deviation
The standard deviation of a RV X is given by:
Expected Value of a Function
The expected value of a function of a RV X is given by:
Examples of Mean, Variance, Standard Deviation, etc.
Continuous RVs
Consider the RV X with uniform distribution over [0,1]:
Suppose h(x) = x3/2:
Discrete RVs
Consider the RV X with binomial distribution and parameters N = 2 and p = 0.2. Support of X is {x0 = 0, x1 = 1, x2 = 2}.
Suppose h(x) = x1/2:
Joint Characterization of RVs
Jointly Continuous RVs
Two jointly (absolutely) continuous RVs X and Y are defined by a joint PDF:
and a joint CDF:
Jointly continuous RVs have the following properties:
Example
Consider RVs X and Y with a joint PDF:
Find Pr(X - Y < 1):
Jointly Discrete RVs
Two jointly discrete RVs X and Y are characterized by a support:
a joint PMF {pi,j} for all i and j, and a joint CDF given by:
with the following properties:
The PDFs of the individual absolutely continuous RVs are determined from the joint PDF by:
The PMFs of the individual absolutely discrete RVs are determined from the joint PMF by: