Chapter 5: MULTIPLE DISCRETE RANDOM VARIABLES

Transcription

1 Charles Bocelet, robablty, Statstcs, ad Radom Sgals," Oford Uversty ress, 06. ISBN: Chapter 5: MULTIL DISCRT RANDOM VARIABLS Sectos 5. Multple Radom Varables ad MFs 5. Idepedece 5. Momets ad pected Values 5.. pected Values for Two Radom Varables 5.. Momets for Two Radom Varables 5.4 ample: Two Dscrete Radom Varables 5.4. Margal MFs ad pected Values 5.4. Idepedece 5.4. Jot CDF Trasformatos Wth Oe Output Trasformatos Wth Several Outputs Dscusso 5.5 Sums of Idepedet Radom Varables 5.6 Sample robabltes, Mea, ad Varace 5.7 Hstograms 5.8 tropy ad Data Compresso 5.8. tropy ad Iformato Theory 5.8. Varable Legth Codg 5.8. codg Bary Sequeces Mamum tropy Summary roblems Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

2 Multple Dscrete Radom Varables The ot probablty mass fucto, ropertes of the ot pmf., 0 (all probabltes are postve). The summato of the pmf for all k s equal to.,.0 The margal pmf of the dvdual radom varables ca be computed,, ample 5. Let, be y= y= =0 = = = The table eplctly shows each of the allows,, 0, 0 0,0,0,0,0, 0,,,, Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

3 , 0,0 0,,,0,,,0,,,0, Notce that the sum or Y s.0! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

4 The ot Cumulatve Dstrbuto Fucto ests, ropertes 0,,,, Calculatg the probablty of a area,,,, Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 4 of C 800

5 Codtoal robablty Let ad From the prevous eample, If we cosder three radom varables:,, ad We ca cosder the followg or,..,. Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 5 of C 800

6 Idepedece If ad Y are depedet, ad, For three or more radom varables, ) ach par s depedet ) The ot pmf of all three factors for all outcomes s depedet Useful termology ad cocept: Idepedet ad Idetcally Dstrbuted (IID) For ths case, the R.V. are depedet ad all have the same pmf! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 6 of C 800

7 Momets ad pected Values,,, roperty Addtve,,,,,,,,,,,,,,, Multplcatve f ad oly f ad Y depedet, ad geeral Not that f ad Y are ot depedet, they may be sad to be correlated ad Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 7 of C 800

8 Covarace ad Correlato Coeffcets Ths gves rse to aother term Covarace, also,,,, or Note that f ad Y are depedet ad 0 As more otato Correlato Coeffcet Lettg The mea value The varace Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 8 of C 800

9 , Thkg terms of a product fucto we defe Wth the correlato coeffcet defed as or AS a result of the ormalzed scalg we epect A specal ote Idepedet radom varables are ucorrelated. 0 However, ucorrelated radom varables are ot ecessarly depedet! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 9 of C 800

10 Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 0 of C 800 ample 4.-5 from Stark ad Woods: Gve Y, 0 0 y 0 0 y 0,, Y y,, Y y,, 0, y y y y Y Y Y Y Note, ot depedet Y Y y y,, Y y y Y ,, Y y y Y Therefore 0, Y COV The covarace ad correlato coeffcet are zero, but the R.V. are ot depedet!

11 ample 5.4 Two dmesoal probablty eample equally lkely pots ad Y Determg margal probabltes 0, 0 5 6, 5, 4, Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

12 ,,,,, Computg a CDF Determe the bouds of terest.5,..., 6 Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

13 5.5 Sums of Idepedet Radom Varables Lettg The mea value The varace, 0 Lettg The mea value The varace Lettg Lettg The pmf from total probablty Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

14 But ths s equvalet to Wth dedece a ot probablty s the product of probabltes; therefore, Resultg Ths s the dscrete covoluto of the two pmf fuctos! ad Revst flppg cos Revst two far de Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 4 of C 800

15 Dscrete Covolutos The sum of two far de pmf k k k k k 4 k 5 k Z Y A dscrete covoluto s the form. pmf 6 m pmf z k pmf k, m k Matlab covoluto. pmf=[ ] pmf = >> cov(pmf,pmf) as = or pmf=[ ]/6 pmf = >> cov(pmf,pmf) as = Colums through Matlab ca multply polyomals whe correctly costructed usg the cov fucto! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 5 of C 800

16 Tetbook ample Commet 5.5 The maual meas of covoluto show (p. 5) s very hady for log-had computatos. Matlab >> =[ -4]; >> Y = [ 4 5]; >> cov(,y) as = Bomal epaso b= [ ]; >> b = cov(b,b) b = >> b4 = cov(b,b) b4 = >> b5 = cov(b4,b) b5 = Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 6 of C 800

17 Tetbook Momet Geeratg Fucto of two d. R.V. From Laplace covoluto the tme doma s multplcato the Laplace doma. For the sum of depedet R.V., the MGF s the product of the MGF! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 7 of C 800

18 5.6 Sample robabltes, Mea, ad Varace (The begg of the relatoshp betwee statstcs ad probablty!) Statstcs Defto: The scece of assemblg, classfyg, tabulatg, ad aalyzg data or facts: Descrptve statstcs the collectg, groupg ad presetg data a way that ca be easly uderstood or assmlated. Iductve statstcs or statstcal ferece use data to draw coclusos about or estmate parameters of the evromet from whch the data came from. Theoretcal Areas: Samplg Theory stmato Theory Hypothess Testg Curve fttg ad regresso Aalyss of Varace selectg samples from a collecto of data that s too large to be eamed completely. cocered wth makg estmates or predctos based o the data that are avalable. attempts to decde whch of two or more hypotheses about the data are true. attempt to fd mathematcal epressos that best represet the data. (Show Chap. 4) attempt to assess the sgfcace of varatos the data ad the relato of these varaces to the physcal stuatos from whch the data arose. (Moder term ANOVA) We wll focus o parameter estmato of the mea ad varace to beg! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 8 of C 800

19 Samplg Theory The Sample Mea How may samples are requred to fd a represetatve sample set that provdes cofdece the results? Defect testg, opo polls, fecto rates, etc. Deftos opulato: Sample: Sample Mea: the collecto of data beg studed N s the sze of the populato a radom sample s the part of the populato selected all members of the populato must be equally lkely to be selected! s the sze of the sample the average of the umercal values that make of the sample opulato: Sample set: N,,,, Sample Mea To geeralze, descrbe the statstcal propertes of arbtrary radom samples rather tha those of ay partcular sample. Sample Mea a pdf., where are radom varables wth Notce that for a pdf, the true mea,, ca be compute whle for a sample data set the above sample mea, ˆ s computed. Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 9 of C 800

20 As may be oted, the sample mea s a combato of radom varables ad, therefore, ca also be cosdered a radom varable. As a result, the hoped for result ca be derved as: If ad whe ths s true, the estmate s sad to be a ubased estmate. Though the sample mea may be ubased, the sample mea may stll ot provde a good estmate. What s the varace of the computato of the sample mea? Varace of the sample mea (the mea tself, ot the value of ) You would epect the sample mea to have some varace about the probablstc or actual mea; therefore, t s also desrable to kow somethg about the fluctuatos aroud the mea. As a result, computato of the varace of the sample mea s desred. For N>> or N fty (or eve a kow pdf), usg the collected samples based o the pror defto of varace, a statstcal estmate of the d momet ad the square of the mea. For Var Var ˆ ˆ ˆ Var Var ˆ ˆ depedet (measuremets should be depedet of each other) ˆ,, for for Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 0 of C 800

21 Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800 As a result we ca defe two summato where = ad <>,, ˆ Var ˆ Var ˆ Var Var ˆ where s the true varace (probablstc) of the radom varable,. Therefore, as approaches fty, ths varace the sample mea estmate goes to zero! It s referred to as a cosstet estmate. Thus a larger sample sze leads to a better estmate of the populato mea. Note: ths varace s developed based o samplg wth replacemet.

22 ample: How may samples of a ftely log tme waveform would be requred to sure the mea s wth % of the true (probablstc) mea value? For ths relatoshp, let Var ˆ Ifte set, therefore assume that you use the wth replacemet equato : Var ˆ Assume that the true meas s 0 ad that the true varace s 9 so that the mea =/- a stadard devato would be 0. The, Var 9 ˆ A very large sample set sze to estmate the mea wth the % desred boud! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

23 Samplg Theory The Sample Varace Whe dealg wth probablty, both the mea ad varace provde valuable formato about the DC ad AC operatg codtos (about what value s epected) ad the varace ( terms of power or squared value) about the operatg pot. Therefore, we are also terested the sample varace as compared to the true data varace. The sample varace of the populato (stdevp) s defed as: S ˆ ad cotug utl (show the comg pages) S where s the true varace of the radom varable. Note: the sample varace s ot equal to the true varace; t s a based estmate! To create a ubased estmator, scale by the basg factor to compute (stdev): ~ S ˆ ˆ S Ths s equato 5. the tetbook! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800

24 Addtoal otes: MATLAB ad MS cel Smulato ad statstcal software packages allow for ether based or ubased computatos. I MS cel there are two dstct fuctos stdev ad stdevp. stdev uses (-) - H asp stdevp uses () - BC F7DC8BB95 I MATLAB, there s a addtoal flag assocate wth the std fucto. std std var, flag mpled as 0 var,,, flag specfed as >> help std std Stadard devato. For vectors, Y = std() returs the stadard devato. For matrces, Y s a row vector cotag the stadard devato of each colum. For N-D arrays, std operates alog the frst o-sgleto dmeso of. std ormalzes Y by (N-), where N s the sample sze. Ths s the sqrt of a ubased estmator of the varace of the populato from whch s draw, as log as cossts of depedet, detcally dstrbuted samples. Y = std(,) ormalzes by N ad produces the square root of the secod momet of the sample about ts mea. std(,0) s the same as std(). Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 4 of C 800

25 Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 5 of C 800 Samplg Theory The Sample Varace - roof The sample varace of the populato s defed as S ˆ S Determg the epected value S S k k S k k S k k S k k k S, S S

26 Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 6 of C 800 S S S Therefore, S To create a ubased estmator, scale by a (u-) basg factor to compute: ~ S S

27 Statstcal Mea ad Varace Summary For takg samples ad estmatg the mea ad varace Mea Varace (based) ˆ The stmate ˆ A ubased estmate ˆ ˆ S ˆ A based estmate S Varace of stmate Var Var ˆ 4 ~ 4 S 4 4 Varace (ubased) ~ S S A ubased estmate ~ S Var S ~ S ˆ Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 7 of C 800

28 5.7 Hstograms Hstogrammg ca be used to determe the values of a pmf! However, a sgfcat umber of trals may have to be ru before the correct pmf ca be observed. Remember the MATLAB smulato of the marble selecto homework #?! Sec_Marble.m Sec_Marble.m Sec_Marble.m See Uform_hst.m See Bomal_hst.m Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 8 of C 800

29 5.8 tropy ad Data Compresso See The bass for formato theory ad of partcular beeft data compresso volves the cocept of etropy. Whe evaluatg formato, a measure of the formato cotet radomess volves the probablty of the occurrece of varous letters the alphabet ad the umber of bts actually eeded to represet the alphabet. For the glsh alphabet, there are m=6 letters. For ormal laguage, each letters has a probablty of occurrece. The measure of the etropy of each potetal symbol s Typcally the log base s used ad the the etropy ca be measured bts. If we assume 6 equally lkely letter a alphabet But realty, the letters have owhere ear equal probablty! O-le t was observed that the average glsh word has a etropy of.6 glsh_9.htm Overall, ths s a specfc applcato ad dscusso related to ecodg that s qute volved ad very mportat. but somewhat uque to a area of terest. Therefore, read t at your lesure. Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 9 of C 800

30 Homework roblem 5.0: rove the Cauchy-Schwarz equalty: where the s ad y s are arbtrary umbers. Ht: Start wth the followg equalty (why s ths true?): 0 Fd the value of a that mmzes the rght had sde above, substtute that value to the same equalty, ad rearrage the terms to the Cauchy-Schwarz equalty at the top. Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 0 of C 800

31 or 0 You may have heard the phrase, The square of the sum of the product s less tha or equal to the product of the sums of the squares! Notes ad fgures are based o or take from materals the course tetbook: Charles Bocelet,, robablty, Statstcs, ad Radom Sgals, Oford Uversty ress, February 06. B.J. Bazu, Sprg 08 of C 800