Probability and Stochastic Processes
A Friendly Introduction for Electrical and Computer Engineers SECOND EDITION
Roy D. Yates David J. Goodman
08_1 Yates Chapter 3 1
Probability and Stochastic Processes
A Friendly Introduction for Electrical and Computer Engineers SECOND EDITION
Roy D. Yates David J. Goodman
Definitions, Theorems, Proofs, Examples, Quizzes, Problems, Solutions
Chapter 3
08_1 Yates Chapter 3 2
08_1 Yates Chapter 3 3
08_1 Yates Chapter 3 4
greater than or equal to a.
Example 3.1 Solution (continued)
Note that the event which implies that
At the limit,
{
X = x}
Ì{
Y = éênxùú}
,[ ]
1.P X x P Y nx
é ù n
= £ ë = éê ùúû =
[ ]
lim lim1 0.n n
P X x P Y nx
n
®¥ é ù ®¥
= £ ë = éê ùúû = =
08_1 Yates Chapter 3 5
[ ]
lim lim1 0.n n
P X x P Y nx
n
®¥ é ù ®¥
= £ ë = éê ùúû = =
08_1 Yates Chapter 3 6
08_1 Yates Chapter 3 7
08_1 Yates Chapter 3 8
08_1 Yates Chapter 3 9
08_1 Yates Chapter 3 10
08_1 Yates Chapter 3 11
08_1 Yates Chapter 3 12
08_1 Yates Chapter 3 13
08_1 Yates Chapter 3 14
08_1 Yates Chapter 3 15
08_1 Yates Chapter 3 16
08_1 Yates Chapter 3 17
08_1 Yates Chapter 3 18
08_1 Yates Chapter 3 19
08_1 Yates Chapter 3 20
08_1 Yates Chapter 3 21
08_1 Yates Chapter 3 22
08_1 Yates Chapter 3 23
08_1 Yates Chapter 3 24
08_1 Yates Chapter 3 25
08_1 Yates Chapter 3 26
08_1 Yates Chapter 3 27
08_1 Yates Chapter 3 28
08_1 Yates Chapter 3 29
08_1 Yates Chapter 3 30
08_1 Yates Chapter 3 31
08_1 Yates Chapter 3 32
08_1 Yates Chapter 3 33
08_1 Yates Chapter 3 34
08_1 Yates Chapter 3 35
08_1 Yates Chapter 3 36
08_1 Yates Chapter 3 37
08_1 Yates Chapter 3 38
08_1 Yates Chapter 3 39
08_1 Yates Chapter 3 40
08_1 Yates Chapter 3 41
08_1 Yates Chapter 3 42
08_1 Yates Chapter 3 43
08_1 Yates Chapter 3 44
08_1 Yates Chapter 3 45
08_1 Yates Chapter 3 46
08_1 Yates Chapter 3 47
08_1 Yates Chapter 3 48
08_1 Yates Chapter 3 49
08_1 Yates Chapter 3 50
08_1 Yates Chapter 3 51
08_1 Yates Chapter 3 52
08_1 Yates Chapter 3 53
08_1 Yates Chapter 3 54
08_1 Yates Chapter 3 55
08_1 Yates Chapter 3 56
08_1 Yates Chapter 3 57
08_1 Yates Chapter 3 58
08_1 Yates Chapter 3 59
08_1 Yates Chapter 3 60
08_1 Yates Chapter 3 61
Gamma Random Variable
The pdf of the gamma random variable has two parameters, and and is given by
where is the gamma function, which is defined by the integral
The gamma function has the following properties:
a >0 l >0,
1 x
( )
( ) 0 ,
( ) x e x
f x x
a l
l l a
- -
= < < ¥
G ( )z
G
1
( )a 0¥xz- e dx-x z 0.
G =
ò
>08_1 Yates Chapter 3 62
The pdf of the gamma random variable has two parameters, and and is given by
where is the gamma function, which is defined by the integral
The gamma function has the following properties:
1
( )a 0¥xz- e dx-x z 0.
G =
ò
>1 ,
2 p
Gæ öç ÷=
è ø G(z+1) = Gz ( ) for z z > 0, and (m 1) m! for a nonnegative integer.m
G + =
Gamma Random Variable (continued)
a =1:
Special cases:
exponential r. v.
1 and ( a positive integer):
2 2
k k
l = a = chi-square r. v.
( a positive integer):
a = m m m-Erlang r. v.
08_1 Yates Chapter 3 63
( a positive integer):
a = m m m-Erlang r. v.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0
0.5 1 1.5
l=1/2
a = 1/2
a=1
a=2
Gamma Random Variable (continued)
08_1 Yates Chapter 3 64
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
0 0.5 1 1.5
l=1/2
a = 1/2
a=1
a=2
Gamma pdfs for a =1/ 2, a =1, and a = 2 and l =1/ 2
08_1 Yates Chapter 3 65
08_1 Yates Chapter 3 66
08_1 Yates Chapter 3 67
08_1 Yates Chapter 3 68
08_1 Yates Chapter 3 69
08_1 Yates Chapter 3 70
08_1 Yates Chapter 3 71
08_1 Yates Chapter 3 72
Define
2, then
0
2
x terf (x) e dt
p
= ò
-( )
( ) 2 2 1 or
erf z = F z - ( ) 1 1 ( )
2 2
z é erf z ù
F = ê + ú
ë û
( ) 1 1 ;
2 2
X
F x erf x m
s
é æ - ö ù
= ê + ç ÷ ú
è ø
ë û [ ] 1
2 2 2
b a
P a X b erf m erf m
s s
é æ - ö æ - ö ù
< £ = ê ç ÷ - ç ÷ ú
è ø è ø
ë û
08_1 Yates Chapter 3 73
08_1 Yates Chapter 3 74
or
08_1 Yates Chapter 3 75
2 2
0 0
2 2
x x
( )
t t
erf (-x) e dt e dt erf x
p p
- - -
= ò = - ò = -
08_1 Yates Chapter 3 76
08_1 Yates Chapter 3 77
08_1 Yates Chapter 3 78
Using the error function,
[ ]
[ ]
1 1 46 61
46 1 ( ) 1 ( )
2 2 2 2 10
1 1 (1.0607) 0.0668.
2
P X erf x erf
erf
m s
é - ù é - ù
£ = ê + ú = ê + ú
ë û ë × û
= - =
08_1 Yates Chapter 3 79
Using the error function,
08_1 Yates Chapter 3 80
Using the error function,
[ 1 1 ] 1 ( 1 ) ( 1 ) ( 1 ) 0.6827.
2 2 2 2
P Z é erf erf ù erf
- < £ = ê - - ú = =
ë û
08_1 Yates Chapter 3 81
08_1 Yates Chapter 3 82
08_1 Yates Chapter 3 83
08_1 Yates Chapter 3 84
08_1 Yates Chapter 3 85
08_1 Yates Chapter 3 86
08_1 Yates Chapter 3 87
08_1 Yates Chapter 3 88
08_1 Yates Chapter 3 89
08_1 Yates Chapter 3 90
08_1 Yates Chapter 3 91
08_1 Yates Chapter 3 92
08_1 Yates Chapter 3 93
08_1 Yates Chapter 3 94
08_1 Yates Chapter 3 95
08_1 Yates Chapter 3 96
08_1 Yates Chapter 3 97
08_1 Yates Chapter 3 98
08_1 Yates Chapter 3 99
08_1 Yates Chapter 3 100
Probability Models of Derived Random Variables
Consider
y = g(x)
where y = g(x) is a real function of x and x is a r. v. with F xx( ) and ( ).f xx
Y X Y X
Express F ( ) in terms of y F x( ) and ( ) in terms of ( ).f y f x
08_1 Yates Chapter 3 101
By definition
y( ) [y ] [ (x) ] x y
F y = P £ y = P g £ y = Péë ÎR ùû where Ry ={ : ( )x g x < y}.
Distribution Function of g(x)
y x
We shall express the distribution function ( ) of the random variables y (x)
in terms of the distribution function ( ) of the random variable X and the function ( ).
F y g
F x g x
=
1. Continuous and Smooth Function, g(x)
08_1 Yates Chapter 3 102
Consider the function ( ) in the above figure where ( ) is between and for any . g x g x a b x
1 1 1
If , then ( ) for every , hence [y ] 1;
if , then there is no such that ( ) , hence [y ] 0. Thus, 1
0
with and ( ) as shown, we observe tha
Y
y b g x y x P y
y a x g x y P y
y b F (y)
y a
x y g x
³ £ £ =
< £ £ =
ì ³
= íî <
= 1 1
1 1 x 1
2 2 2 2
y 2 2 2 2
x 2
t ( ) for . Hence,
[ x ] ( ).
We finally note that ( ) if or . Hence,
[ x ] [ x
( )
Y
g x y x x
F (y ) P x F x
g x y x x x x x
F (y ) P x P x x } F x
£ £
= £ =
¢ ¢¢ ¢¢¢
£ £ £ £
¢ ¢¢ ¢¢¢
= £ + £ £
= ¢ + F x F xx( )2¢¢¢ - x( ).2¢¢
08_1 Yates Chapter 3 103
1 1 1
If , then ( ) for every , hence [y ] 1;
if , then there is no such that ( ) , hence [y ] 0. Thus, 1
0
with and ( ) as shown, we observe tha
Y
y b g x y x P y
y a x g x y P y
y b F (y)
y a
x y g x
³ £ £ =
< £ £ =
ì ³
= íî <
= 1 1
1 1 x 1
2 2 2 2
y 2 2 2 2
x 2
t ( ) for . Hence,
[ x ] ( ).
We finally note that ( ) if or . Hence,
[ x ] [ x
( )
Y
g x y x x
F (y ) P x F x
g x y x x x x x
F (y ) P x P x x } F x
£ £
= £ =
¢ ¢¢ ¢¢¢
£ £ £ £
¢ ¢¢ ¢¢¢
= £ + £ £
= ¢ + F x F xx( )2¢¢¢ - x( ).2¢¢
Example (Y = aX + b )
To find Y( ), we must find the values of such that .
( )
(a) If 0, for . (Fig. 3.7-2(a)) Hence,
[x ] ( ), a>0.
Y X
F y x ax b y
a ax b y x y b
a
y b y b
F (y) P F
a a
+ £
> + £ £ -
- -
= £ =
08_1 Yates Chapter 3 104
Figure 3.7-2. y =ax +b
(a) (b)
y x
( )
(b) If 0, for . (Fig. 3-2(b)) Hence, [x ] 1 ( ) , a<0.
y b
a ax b y x
a
y b y b
F (y) P F
a a
< + £ > -
- -
= ³ = -
08_1 Yates Chapter 3 105
Example (Y = X
2)
{ }
2
(1) When 0
for
Y
X X
y
x y y x y
F (y) P y x y
F ( y ) F ( y )
³
£ - £ £
= - £ £
= - -
(a) y = x2
y
1 1
2 2
(2) When 0,
( ) [ ] 0 ( ) 1
0 else
1 1
then, ,
2 2
and
X
X
Y X X
y
F y P
x
Let f x
F (x) x x
F (y) F ( y ) F ( y )
f
<
= =
- £ £
= íì î
= + <
= - -
= y + y =2 y for 0£ y £ 14.
08_1 Yates Chapter 3 106
y
1 1
2 2
(2) When 0,
( ) [ ] 0 ( ) 1
0 else
1 1
then, ,
2 2
and
X
X
Y X X
y
F y P
x
Let f x
F (x) x x
F (y) F ( y ) F ( y )
f
<
= =
- £ £
= íì î
= + <
= - -
= y + y =2 y for 0£ y £ 14.
(b) F xx( )
y
In summary,
1 1/ 4
( ) 2 0 1/ 4
0 0.
y
F y y y
y ì >
= ïí £ £
ï <
î
(c) F yy( )
08_1 Yates Chapter 3 107
2. Continuous but not Smooth Function, g(x)
0 1
1 0 1
1 0 1 x 1 x 0
y 1 x
Suppose the function ( ) is constant in an interval ( , ) such that
( ) , .
In this case
[y ] [ x x ] ( ) ( ).
Hence ( ) is discontinuous at y and its discontinuity equals (
g x x x
g x y x x x
P y P x F x F x
F y y F
= < £
= = < £ = -
= x1)- F xx( ).0
Example (Continuous but not Smooth Function)
Y
Consider the function shown in Fig. 3.7-4 (center-level clipper) for
( ) 0 for for
In this case, ( ) is discontinuous for 0 and its discon
x c x c
g x c x c
x c x c
F y y
- >
ìï
= í - £ £
ï + < - î
= x x
x x
tinuity equals ( ) ( ).
Furthermore, if 0 then [y ] [x ] ( );
if 0 then [y ] [x ] ( ).
F c F c
y P y P y c F y c
y P y P y c F y c
- -
³ £ = £ + = +
< £ = £ - = -
08_1 Yates Chapter 3 108
Y
Consider the function shown in Fig. 3.7-4 (center-level clipper) for
( ) 0 for for
In this case, ( ) is discontinuous for 0 and its discon
x c x c
g x c x c
x c x c
F y y
- >
ìï
= í - £ £
ï + < - î
= x x
x x
tinuity equals ( ) ( ).
Furthermore, if 0 then [y ] [x ] ( );
if 0 then [y ] [x ] ( ).
F c F c
y P y P y c F y c
y P y P y c F y c
- -
³ £ = £ + = +
< £ = £ - = -
y
0 c x
)
x(x F
c x
1
0
(a) y= g x( ) (b) F xx( )
)
y(y F
1 -c -c
08_1 Yates Chapter 3 109
0 y
(c) F yY( )
Fig. A Continuous But Not Smooth Function and Its Distribution Function
3. Discontinuous, Staircase Function, g(x)
1
Assume that ( ) is a discontinuous, staircase function ( ) ( ) for .
In this case, the random variable y (x) is of discrete type taking the values with
i i i i
i
g x
g x g x y x x x
g y
= = - < <
=
1 i x x 1
[yP = yi]= P x[ i- < x £ x ]= F x( )i -F x( i- ).
Example (Hard Limiter Function)
x x y
If
1 for 0 ( )
1 for 0, then, takes the values 1 with [y 1] [x 0] (0) [y 1] [x 0] 1 (0).
Hence, ( ) is a staircase function as in Fig. 3.7-5.
g x x
x y
P P F
P P F
F y ì >
= íî- £
±
= - = £ =
= = > = -
08_1 Yates Chapter 3 110
x x y
If
1 for 0 ( )
1 for 0, then, takes the values 1 with [y 1] [x 0] (0) [y 1] [x 0] 1 (0).
Hence, ( ) is a staircase function as in Fig. 3.7-5.
g x x
x y
P P F
P P F
F y ì >
= íî- £
±
= - = £ =
= = > = -
x( ) F x
0
Y( ) F y
g(x) 1
-1
0 x
1
x -1 1
1
y
Figure 3.7-5. The Hard Limiter Function and Its Distribution Function
08_1 Yates Chapter 3 111
Figure 3.7-5. The Hard Limiter Function and Its Distribution Function
Determination of ( ) f
yy
We wish to determine the density of Y ( ) in terms of the density of .
Suppose, first, that the set of the -axis is not in the range of the function ( ).
In other words, ( ) is not a point of
g X X
R y g x
g x R
=
y
for any . In this case, the probability that ( ) is in equals 0. Hence, ( ) 0 for . It suffices, therefore,
to consider the values of such that for some , ( ) .
x g x
R f y y R
y x g x y
= Î
= Fundamental Theorem
08_1 Yates Chapter 3 112
We are given a continuous r.v. with pdf ( ) and a differentiable function ( ) of the real variable . What is the pdf of = ( )?
X fX x g x
x Y g X
1 2
1 1
Theorem: Let , , , , be the real values satisfying ( ). Then, ( )
( ) ( )
( ) ( )
where ( ) ( ) .
i
n
X n
X Y
n
i
x x
x x x y g x
f x f x
f y
g x g x
g x dg x
dx =
=
= + + +
¢ ¢
¢ =
L L
L L
{ }
{ }
{ }
1 2 3
1 1 1
2 2 2
(Proof)
For simplicity, we prove the case that ( ) has three roots, , , and . ( )
Y
y g x x x x
f y dy P y Y y dy
P x X x dx
P x dx X x
=
= < £ +
= < £ +
+ + < £
{
3 3 3}
1 1 2 2 3 3
1 2
1 2
( ) ( ) ( )
for small
( ) ( )
( ) ( )
X X X
i
X X
P x X x dx
f x dx f x dx f x dx
dx
f x f x
dy dy
g x g x
+ < £ +
= + +
= +
¢ ¢
1
3 3
1
( ) ( ) where
( ) .
X
x x
f x g x dy
g x dy
dx =
+ ¢
¢ =
08_1 Yates Chapter 3 113
{ }
{ }
{ }
1 2 3
1 1 1
2 2 2
(Proof)
For simplicity, we prove the case that ( ) has three roots, , , and . ( )
Y
y g x x x x
f y dy P y Y y dy
P x X x dx
P x dx X x
=
= < £ +
= < £ +
+ + < £
{
3 3 3}
1 1 2 2 3 3
1 2
1 2
( ) ( ) ( )
for small
( ) ( )
( ) ( )
X X X
i
X X
P x X x dx
f x dx f x dx f x dx
dx
f x f x
dy dy
g x g x
+ < £ +
= + +
= +
¢ ¢
1
3 3
1
( ) ( ) where
( ) .
X
x x
f x g x dy
g x dy
dx =
+ ¢
¢ =
Example (Y = aX + b and Its Density Function)
( )
2 2
2
( ) 2
Given
( ) , 1
2
find ( ).
X
x m
Y
f x N m
e
Y aX b f y
s
s
ps
- -
=
=
= +
1
1
1 1
1
(Solution)
Apply the fundamental theorem and we obtain ( ) ( ) where .
( )
Considering the following equation ( ) ( )
.
X Y
x
f x y b
f y x
g x a
g x d ax b a
dx
= = -
¢
¢ = + =
08_1 Yates Chapter 3 114
1
1
1 1
1
(Solution)
Apply the fundamental theorem and we obtain ( ) ( ) where .
( )
Considering the following equation ( ) ( )
.
X Y
x
f x y b
f y x
g x a
g x d ax b a
dx
= = -
¢
¢ = + =
( )
2 2
2 2
( ) 2
( ) 2
we get,
( ) 1
2
1 .
2
Note that is the Gaussian with mean ( ) and standard deviatio
X Y
y b m a
y b a m a
Y
f y b f y a
a
e
a
e
a
f b am
s
s
ps p s
- - -
- - -
æ - ö
ç ÷
è ø
=
=
=
+ n ,
which illustrates that a linear transformation of Gaussian r.v. is also Gaussian.
a
s
08_1 Yates Chapter 3 115
( )
2 2
2 2
( ) 2
( ) 2
we get,
( ) 1
2
1 .
2
Note that is the Gaussian with mean ( ) and standard deviatio
X Y
y b m a
y b a m a
Y
f y b f y a
a
e
a
e
a
f b am
s
s
ps p s
- - -
- - -
æ - ö
ç ÷
è ø
=
=
=
+ n ,
which illustrates that a linear transformation of Gaussian r.v. is also Gaussian.
a
s
Example ( y = a x and Its Density Function)
22
2
Given
, 0
( 0 , ) Find ( ).
X Y
Y a X a X N
f y
s
= >
=
2
1 2
1 1
2 2
(Answer)
Solving for gives
, .
And
| ( ) | | 2 | 2 2
| ( ) | | 2 | 2 2 .
y ax x
y y
x x
a a
g x a x ay ay
g x a x ay ay
=
= = -
¢ = = =
¢ = = - =
08_1 Yates Chapter 3 116
2
1 2
1 1
2 2
(Answer)
Solving for gives
, .
And
| ( ) | | 2 | 2 2
| ( ) | | 2 | 2 2 .
y ax x
y y
x x
a a
g x a x ay ay
g x a x ay ay
=
= = -
¢ = = =
¢ = = - =
( )2 2 ( )2 2
2
1 2
1 2
2 2
2 2
Now
( ) ( )
( )
( ) ( )
1 ( ) ( ) , 0
2
1 1
2 2
1 ( ). (1)
2
X X
X
X X
Y
y y
X a X a
y a y a
X
y a X
f x f x
f y
g x g x
f f y
ay
e e
ay
e U y
a y
s s
s
ps
p s
- - -
-
= +
¢ ¢
é ù
= + - >
ë û
é ù
= × ê + ú
ë û
=
× ×
08_1 Yates Chapter 3 117
( )2 2 ( )2 2
2
1 2
1 2
2 2
2 2
Now
( ) ( )
( )
( ) ( )
1 ( ) ( ) , 0
2
1 1
2 2
1 ( ). (1)
2
X X
X
X X
Y
y y
X a X a
y a y a
X
y a X
f x f x
f y
g x g x
f f y
ay
e e
ay
e U y
a y
s s
s
ps
p s
- - -
-
= +
¢ ¢
é ù
= + - >
ë û
é ù
= × ê + ú
ë û
=
× ×
1
2
Gamma density function is defined by
( ) ( ). (2) ( )
Comparing Eqs. (1) and (2), we can tell that ( ) is the Gamma density function
1 1
with and .
2 2
v V
Y
X
v e
f v U v
f y a
a a l
l
a
l a
s
- -
= G
= =
( )
( )
12 2
12
2
12
Note: 1) 1 2
1 1
2) For 1 and 1, with and ,
2 2
( ) . 2
This is the chi - square density with 1 degree of freedom.
y
X
Y
a
y e
f y p
s l a
- -
Gæ öç ÷ = è ø
= = = =
= G
08_1 Yates Chapter 3 118
08_1 Yates Chapter 3 119
08_1 Yates Chapter 3 120
08_1 Yates Chapter 3 121
08_1 Yates Chapter 3 122
08_1 Yates Chapter 3 123
08_1 Yates Chapter 3 124
08_1 Yates Chapter 3 125
08_1 Yates Chapter 3 126
08_1 Yates Chapter 3 127
08_1 Yates Chapter 3 128
08_1 Yates Chapter 3 129
08_1 Yates Chapter 3 130
08_1 Yates Chapter 3 131
08_1 Yates Chapter 3 132
08_1 Yates Chapter 3 133