Discrete Random Variables

Chapter 2

Discrete Random Variables

Chapter Outline

• Random variables

• Probability mass function (PMF)

• Expectation

• Variance

• Conditional PMF

• Geometric PMF

• Total expectation theorem

• Joint PMF of two random variables

Chapter Outline

• Multiple random variables

– Joint PMF – Conditioning – Independence

• More on expectations

• Binomial distribution revisited

• A hat problem

3

2.1. Random Variables

• An assignment of a value (number) to every possible outcome

• Mathematically: A function from the sample space Ω to the real numbers

– discrete or continuous values

• Can have several random variables defined on the same sample space

• Notation:

– random variable 𝑋 – numerical value 𝑥

Random Variables

5

Main Concepts Related to Random Variables

• A random variable is a real-valued function of the outcome of the experiment.

• A function of a random variable defines another random variable.

• We can associate with each random variable certain

“averages" of interest, such as the mean and the variance.

• A random variable can be conditioned on an event or on another random variable.

• There is a notion of independence of a random variable from an event or from another random variable.

Concepts Related to Discrete Random Variables

• A discrete random variable is a real-valued function of the outcome of the experiment that can take a finite or countably infinite number of values.

• A discrete random variable has an associated probability mass function (PMF) which gives the probability of each numerical value that the random variable can take.

• A function of a discrete random variable defines another

discrete random variable, whose PMF can be obtained from the PMF of the original random variable.

7

2.2. Probability Mass Function (PMF)

• Probability distribution of 𝑋

• Notation:

𝑝_𝑋 𝑥 = 𝑃 𝑋 = 𝑥

= 𝑃 𝜔 ∈ Ω s. t. 𝑋 𝜔 = 𝑥

• 𝑝_𝑋 𝑥 ≥ 0 𝑝_{𝑥 𝑋} 𝑥 = 1

How to Compute a PMF

9

Calculation of the PMF of a Random Variable X

• Collect all the possible outcomes that give rise to the event 𝑋 = 𝑥 .

• Add their probabilities to obtain 𝑝_𝑋(𝑥).

How to Compute a PMF

• collect all possible outcomes for which 𝑋 is equal to 𝑥

• add their probabilities

• repeat for all 𝑥

• Example: Two independent rolls of a fair tetrahedral die

𝐹: outcome of first throw

𝑆: outcome of second throw 𝑋 = min (𝐹, 𝑆)

𝑝_𝑋 2 =

11

Binomial PMF

• 𝑋: number of heads in 𝑛 independent coin tosses

• 𝑃(𝐻) = 𝑝

• Let 𝑛 = 4

𝑝_𝑋 2 = 𝑃 𝐻𝐻𝑇𝑇 + 𝑃 𝐻𝑇𝐻𝑇 + 𝑃 𝐻𝑇𝑇𝐻 +𝑃 𝑇𝐻𝐻𝑇 + 𝑃 𝑇𝐻𝑇𝐻 + 𝑃 𝑇𝑇𝐻𝐻 = 6𝑝² 1 − 𝑝 ²

= 4

2 𝑝² 1 − 𝑝 ² In general:

𝑝_𝑋 𝑘 = 𝑛

𝑘 𝑝^𝑘 1 − 𝑝 ^𝑛−𝑘, 𝑘 = 0,1, … , 𝑛

Binomial PMF

13

Geometric PMF

• 𝑋: number of independent coin tosses until first head

𝑝_𝑋 𝑘 = 1 − 𝑝 ^𝑘−1𝑝, 𝑘 = 1,2, …

𝐸 𝑋 = 𝑘𝑝_𝑋 𝑘

∞

𝑘=1

= 𝑘 1 − 𝑝 ^𝑘−1𝑝

∞

𝑘=1

Geometric PMF

• Memoryless property: Given that 𝑋 > 2, the r.v. 𝑋 − 2 has same geometric PMF

15

Poisson PMF

𝑝_𝑋 𝑘 = 𝑒^−𝜆 𝜆^𝑘

𝑘! , 𝑘 = 0,1,2, …

2.3. Functions of Random Variables

17

2.4. Expectation

• We define the expected value (also called the expectation or the mean) of a random variable 𝑋, with PMF 𝑝_𝑋, by

𝐸 𝑋 = 𝑥𝑝_𝑋 𝑥

𝑥

• Interpretations:

– Center of gravity of PMF

– Average in large number of repetitions of the experiment (to be substantiated later in this course)

Expectation

• Example: Uniform on 0, 1, . . . , 𝑛

𝐸 𝑋 = 0 × _𝑛+1¹ + 1 × _𝑛+1¹ ⋯ 𝑛 × _𝑛+1¹ =

19

Expected Value Rule for Functions of Random Variables

• Let 𝑋 be a random variable with PMF 𝑝_𝑋, and let 𝑔(𝑋) be a

function of 𝑋. Then, the expected value of the random variable 𝑔(𝑋) is given by

𝐸 𝑔 𝑋 = 𝑔 𝑥 𝑝_𝑋(𝑥)

𝑥

.

Average Speed vs. Average Time

• Traverse a 200 mile distance at constant but random speed 𝑉

• time in hours = 𝑇 = 𝑡 𝑉 = 200/𝑉

• 𝐸 𝑇 = 𝐸 𝑡 𝑉 = 𝑡 𝑣 𝑝_{𝑣 𝑉} 𝑣 =

• 𝐸 𝑇𝑉 = 200 ≠ 𝐸 𝑇 ⋅ 𝐸 𝑉

• 𝐸 200/𝑉 = 𝐸 𝑇 ≠ 200/𝐸 𝑉 .

21

Properties of Expectations

• Let

X

– Hard: 𝐸 𝑌 = 𝑦𝑝_{𝑦 𝑌} 𝑦

– Easy: 𝐸 𝑌 = 𝑔 𝑥 𝑝_{𝑥 𝑋} 𝑥

• Caution: In general, E 𝑔 𝑋 ≠ 𝑔 E 𝑋 Properties: If 𝛼, 𝛽 are constants, then:

• 𝐸 𝛼 =

• 𝐸 𝛼𝑋 =

• 𝐸 𝛼𝑋 + 𝛽 =

Variance

• The variance var(𝑋) of a random variable 𝑋 is defined by var 𝑋 = 𝐸 𝑋 − 𝐸 𝑋 ^𝟐 ,

and can be calculated as

var 𝑋 = 𝑥 − 𝐸 𝑋 ^𝟐

𝑥

𝑝_𝑋 𝑥 .

It is always nonnegative. Its square root is denoted by 𝜎_𝑋 and is called the standard deviation.

23

Variance

• Second moment: 𝐸 𝑋² = 𝑥_𝑥 ²𝑝_𝑋 𝑥

• Variance: var(𝑋) = 𝐸 𝑋 − 𝐸 𝑋 ² = 𝑥 − 𝐸 𝑋_𝑥 ²𝑝_𝑋 𝑥 = 𝐸 𝑋² − 𝐸 𝑋 ² Properties:

• var 𝑋 ≥ 0

• var 𝛼𝑋 + 𝛽 = 𝛼²var 𝑋

Example 2.6. (Discrete Uniform Random Variable)

25

Example 2.8. (The Quiz Problem)

27

Review

• Random variable 𝑋: function from sample space to the real numbers

• PMF (for discrete random variables):

𝑝_𝑋 𝑥 = 𝑃 𝑋 = 𝑥

• Expectation:

𝐸 𝑋 = 𝑥𝑝_𝑋 𝑥

𝑥

𝐸 𝑔 𝑋 = 𝑔 𝑥 𝑝_𝑋 𝑥 𝐸 𝛼𝑋 + 𝛽 = 𝛼𝐸 𝑋 + 𝛽 𝑥

Review

• 𝐸 𝑋 − 𝐸 𝑋 =

var 𝑋 = 𝐸 𝑋 − 𝐸 𝑋 ²

= 𝑥 − 𝐸 𝑋_𝑥 ²𝑝_𝑋 𝑥 = 𝐸 𝑋² − 𝐸 𝑋 ²

Standard deviation: 𝜎_𝑋 = var 𝑋

29

Conditional PMF and Expectation

• 𝑝_𝑋|𝐴 𝑥 = 𝑃 𝑋 = 𝑥|𝐴

• 𝐸 𝑋|𝐴 = 𝑥𝑝_{𝑥 𝑋|𝐴} 𝑥

• Let 𝐴 = 𝑋 ≥ 2 𝑝_𝑋|𝐴 𝑥 =

𝐸 𝑋|𝐴 =

Total Expectation Theorem

• Partition of sample space into disjoint events 𝐴₁, 𝐴₂, . . . , 𝐴_𝑛

𝑃 𝐵 = 𝑃 𝐴₁ 𝑃 𝐵|𝐴₁ + ⋯ + 𝑃 𝐴_𝑛 𝑃 𝐵|𝐴_𝑛 𝑝_𝑋 𝑥 = 𝑃 𝐴₁ 𝑝_𝑋|𝐴1 𝑥 + ⋯ + 𝑃 𝐴_𝑛 𝑝_{𝑋|𝐴𝑛} 𝑥

𝐸 𝑋 = 𝑃 𝐴₁ 𝐸 𝑋|𝐴₁ + ⋯ + 𝑃 𝐴_𝑛 𝐸 𝑋|𝐴_𝑛

31

Total Expectation Theorem

• Geometric example:

𝐴₁ ∶ {𝑋 = 1}, 𝐴₂ ∶ {𝑋 > 1}

𝐸 𝑋 = 𝑃 𝑋 = 1 𝐸 𝑋|𝑋 = 1 +𝑃 𝑋 > 1 𝐸 𝑋|𝑋 > 1

• Solve to get 𝐸 𝑋 = 1/𝑝

2.5. Joint PMFs

• 𝑝_𝑋,𝑌 𝑥, 𝑦 = 𝑃 𝑋 = 𝑥 and 𝑌 = 𝑦

• 𝑝_{𝑥 𝑦 𝑋,𝑌 𝑥,𝑦} =

• 𝑝_𝑋 𝑥 = 𝑝_{𝑦 𝑋,𝑌} 𝑥, 𝑦

• 𝑝_𝑋|𝑌 𝑥|𝑦 = 𝑃 𝑋 = 𝑥|𝑌 = 𝑦 = ^{𝑝𝑋,𝑌}_𝑝 ^𝑥,𝑦

𝑌 𝑦

• 𝑝_{𝑥 𝑋|𝑌} 𝑥|𝑦 =

33

Joint and Marginal PMFs

Summary of Facts About Joint PMFs

Let 𝑋 and 𝑌 be random variables associated with the same experiment.

• The joint PMF 𝑝_𝑋,𝑌 of 𝑋 and 𝑌 is defined by 𝑝_𝑋,𝑌 𝑥, 𝑦 = 𝑃 𝑋 = 𝑥, 𝑌 = 𝑦 .

• The marginal PMFs of 𝑋 and 𝑌 can be obtained from the joint PMF, using the formulas

𝑝_𝑋 𝑥 = 𝑝_𝑋,𝑌 𝑥, 𝑦

𝑦

, 𝑝_𝑌 𝑦 = 𝑝_𝑋,𝑌 𝑥, 𝑦

𝑥

.

35

Summary of Facts About Joint PMFs

• A function 𝑔(𝑋, 𝑌) of 𝑋 and 𝑌 defines another random variable, and

𝐸 𝑔 𝑋, 𝑌 = 𝑔(𝑋, 𝑌)

𝑦 𝑥

𝑝_𝑋,𝑌 𝑥, 𝑦 .

If 𝑔 is linear, of the form 𝑎𝑋 + 𝑏𝑌 + 𝑐, we have 𝐸 𝑎𝑋 + 𝑏𝑌 + 𝑐 = 𝑎𝐸 𝑋 + 𝑏𝐸 𝑌 + 𝑐.

• The above have natural extensions to the case where more than two random variables are involved.

Summary of Conditional PMF

37

Summary of Conditional PMF

Summary of Facts About Conditional PMFs

• Conditional PMFs are similar to ordinary PMFs, but pertain to a universe where the conditioning event is known to have

occurred.

• The conditional PMF of 𝑋 given an event 𝐴 with 𝑋(𝐴) > 0, is defined by

𝑝_𝑋|𝐴 𝑥 = 𝑃(𝑋 = 𝑥|𝐴) and satisfies

𝑝_𝑋|𝐴 𝑥 = 1.

𝑥

39

Summary of Facts About Conditional PMFs

• If 𝐴₁, … , 𝐴_𝑛 are disjoint events that form a partition of the sample space, with 𝑃(𝐴_𝑖) > 0 for all 𝑖, then

𝑝_𝑋 𝑥 = 𝑃(

𝑛

𝑖=1

𝐴_𝑖)𝑝_{𝑋|𝐴𝑖} 𝑥 .

(This is a special case of the total probability theorem.)

Summary of Facts About Conditional PMFs

• The conditional PMF of 𝑋 given 𝑌 = 𝑦 is related to the joint PMF by

𝑝_𝑋,𝑌 𝑥, 𝑦 = 𝑝_𝑌 𝑦 𝑝_𝑋|𝑌 𝑥 𝑦 .

• The conditional PMF of 𝑋 given 𝑌 can be used to calculate the marginal PMF of 𝑋 through the formula

𝑝_𝑋 𝑥 = 𝑝_𝑌 𝑦 𝑝_𝑋|𝑌 𝑥 𝑦 .

𝑦

• There are natural extensions of the above involving more than two random variables.

41

Summary of Facts About Conditional Expectations

• The conditional expectation of 𝑋 given an event 𝐴 with 𝑃(𝐴) > 0, is defined by

𝐸 𝑋 𝐴 = 𝑥𝑝_𝑋|𝐴 𝑥 . For a function 𝑔(𝑋), we have 𝑥

𝐸 𝑔(𝑋) 𝐴 = 𝑔(𝑥)𝑝_𝑋|𝐴 𝑥 .

𝑥

• The conditional expectation of 𝑋 given a value 𝑦 of 𝑌 is defined by

Summary of Facts About Conditional Expectations

• If 𝐴₁, … , 𝐴_𝑛 be disjoint events that form a partition of the sample space, with 𝑃(𝐴_𝑖) > 0 for all 𝑖, then

𝐸 𝑋 = 𝑃(

𝑛

𝑖=1

𝐴_𝑖)𝐸[𝑋|𝐴_𝑖].

Furthermore, for any event 𝐵, with 𝑃(𝐴_𝑖 ∩ 𝐵) > 0 for all 𝑖, we have

𝐸 𝑋|𝐵 = 𝑃(

𝑛

𝑖=1

𝐴_𝑖|𝐵)𝐸[𝑋|𝐴_𝑖 ∩ 𝐵].

• We have

𝐸 𝑋 = 𝑝_𝑌 𝑦 𝐸 𝑋 𝑌 = 𝑦 .

𝑦

43

Review

𝑝

𝑥 = 𝑃 𝑋 = 𝑥

𝑝

𝑥, 𝑦 = 𝑃(𝑋 = 𝑥, 𝑌 = 𝑦) 𝑝

𝑥 𝑦 = 𝑃(𝑋 = 𝑥|𝑌 = 𝑦)

𝑝

𝑥 = 𝑝

(𝑥, 𝑦)

𝑦

𝑝

𝑥, 𝑦 = 𝑝

(𝑥)𝑝

(𝑦|𝑥)

45

2.7. Independent Random Variables

𝑝

𝑥, 𝑦, 𝑧 = 𝑝

𝑥 𝑝

𝑦 𝑥 𝑝

𝑧 𝑥, 𝑦

• Random variables 𝑋, 𝑌, 𝑍 are independent if:

𝑝

𝑥, 𝑦, 𝑧 = 𝑝

(𝑥) ∙ 𝑝

(𝑦) ∙ 𝑝

(𝑧)

for all 𝑥, 𝑦, 𝑧

Independent Random Variables: Example

• Independent?

47

Expectations

• In general: 𝐸 𝑔 𝑋, 𝑌 ≠ 𝑔 𝐸 𝑋 , 𝐸 𝑌

• 𝐸 𝛼𝑋 + 𝛽 = α𝐸 𝑋 + 𝛽

• 𝐸 𝑋 + 𝑌 + 𝑍 = 𝐸 𝑋 + 𝐸 𝑌 + 𝐸 𝑍

• If 𝑋, 𝑌 are independent:

– 𝐸 𝑋𝑌 = 𝐸 𝑋 𝐸 𝑌

– 𝐸 𝑔 𝑋 ℎ 𝑌 = 𝐸[𝑔 𝑋 ] ∙ 𝐸[ℎ 𝑌 ] 𝐸 𝑋 = 𝑥𝑝_𝑋 𝑥

𝑥

𝐸 𝑔 𝑋, 𝑌 = 𝑔(𝑥, 𝑦)𝑝_𝑋,𝑌(𝑥, 𝑦)

𝑦 𝑥

Variances

• var 𝛼𝑋 = 𝛼²var 𝑋

• var 𝑋 + 𝛼 = var 𝑋

• Let 𝑍 = 𝑋 + 𝑌 . If 𝑋, 𝑌 are independent:

var 𝑋 + 𝑌 = var 𝑋 + var 𝑌

• Examples:

− If 𝑋 = 𝑌, var 𝑋 + 𝑌 = − If 𝑋 = −𝑌, var 𝑋 + 𝑌 =

− If 𝑋, 𝑌 are indepent and 𝑍 = 𝑋 − 3𝑌, var 𝑍 =

49

Binomial Mean and Variance

• 𝑋 = number of successes in

n

• 𝑋_𝑖 = 1, if success in trial 𝑖, 0, otherwise

• 𝐸 𝑋_𝑖 = 𝐸 𝑋 =

• var 𝑋_𝑖 = var 𝑋 =

𝐸 𝑋 = ^𝑛_𝑘=0 𝑘 ^𝑛_𝑘 𝑝^𝑘(1 − 𝑝)^𝑛−𝑘

The Hat Problem

• 𝑛 people throw their hats in a box and then pick one at random.

– 𝑋: number of people who get their own hat – Find 𝐸 𝑋

𝑋_𝑖= 1, if 𝑖 selects own hat 0, otherwise

• 𝑋 = 𝑋₁ + 𝑋₂ + ⋯ + 𝑋_𝑛

• 𝑃 𝑋_𝑖 = 1 =

• 𝐸 𝑋_𝑖 =

• Are the 𝑋_𝑖 independent?

• 𝐸 𝑋 =

51

Variance in the Hat Problem

• var 𝑋 = 𝐸 𝑋² − 𝐸 𝑋 ² = 𝐸 𝑋² − 1 𝑋² = 𝑋_𝑖²

𝑖

+ 𝑋_𝑖𝑋_𝑗

𝑖,𝑗:𝑖≠𝑗

• 𝐸 𝑋_𝑖² =

𝑃 𝑋₁𝑋₂ = 1 = 𝑃 𝑋₁ = 1 ∙ 𝑃 𝑋₂ = 1 𝑋₁ = 1 =

• 𝐸 𝑋² =

• var 𝑋 =

52

Summary of Facts About Independent Random Variables

• Let 𝐴 be an event, with 𝑃(𝐴) > 0, and let 𝑋 and 𝑌 be random variables associated with the same experiment.

• 𝑋 is independent of the event 𝐴 if

𝑝_𝑋|𝐴 𝑥 = 𝑝_𝑋 𝑥 , for all 𝑥,

that is, if for all 𝑥, the events {𝑋 = 𝑥} and 𝐴 are independent.

• 𝑋 and 𝑌 independent if for all pairs (𝑥, 𝑦), the events {𝑋 = 𝑥}

and {𝑌 = 𝑦} are independent, or equivalently

𝑝_𝑋,𝑌 𝑥, 𝑦 = 𝑝_𝑋 𝑥 𝑝_𝑌 𝑦 , for all 𝑥, 𝑦.

53

Summary of Facts About Independent Random Variables

• If 𝑋 and 𝑌 are independent random variables, then 𝐸 𝑋𝑌 = 𝐸 𝑋 𝐸 𝑌 .

Furthermore, for any functions 𝑔 and ℎ, the random variables 𝑔(𝑋) and ℎ(𝑌) are independent, and we have

𝐸 𝑔(𝑋)ℎ(𝑌) = 𝐸 𝑔(𝑋) 𝐸 ℎ(𝑌) .

• If 𝑋 and 𝑌 are independent, then