Statistics for EES Introduction to R and Descriptive Statistics

Transcription

1 Statistics for EES Introduction to R and Descriptive Statistics Dirk Metzler April 23, 2017 Contents Contents 1 Intro: What is Statistics? 2 2 Data Visialization Histograms und Polygons Histograms: Densities or Numbers? Stripcharts and Boxplots Example: Darwin Finches Conclusions Summarizing Data Numerically Median and other Quartiles Mean, Standard Deviation and Variance Computing σ with n or n 1? Mean values are usually nice but sometimes mean example: picky wagtails example: spider men & spider women example: copper-tolerant browntop bent

2 1 Intro: What is Statistics? It is easy to lie with statistics. It is hard to tell the truth without it. What is Statistics? Andrejs Dunkels Nature is full of Variability How to make sense of variable data? Use mathematical theory of randomness:[0.5ex] Probability. Statistics = Data Analysis based on Probabilistic Models Descriptive Statistics Descriptive Statistcs is the first look at the data. Statistics Software R 2

3 2 Data Visialization Data Example Data from a biology diploma thesis, 2001, Forschungsinstitut Senckenberg, Frankfurt am Main Crustacea section Advisor: Prof. Dr. Michael Türkay Charybdis acutidens TÜRKAY 1985 The Squat Lobster Galathea intermedia Squat Lobsters, caught 6. Sept 1988 Helgoländer Tiefe Rinne, North Sea Carpace Lengths (mm): Females, not egg-carrying (n = 215) Female Galathea, not carrrying eggs, caught 6. Sept. '88, n=215 Index Carapax Length [mm] 3

4 2.1 Histograms und Polygons 6. Sept. '88, n=215 Female Galathea, not egg carrying, caught 6. Sept. 88, n=215 Number Number Carapax Length [mm] Female Galathea, not egg carrying, caught 6. Sept. 88, n=215 Female Galathea, not egg carrying, caught 6. Sept. 88, n=215 Number How many have Carapace Length between 2.0 and 2.2? Number How many have Carapace Length between 2.0 and 2.2? Female Galathea, not egg carrying, caught 6. Sept. 88, n=215 Female Galathea, not egg carrying, caught 6. Sept. 88, n=215 Number How many have Carapace Length between 2.0 and 2.2? Number How many have Carapace Length between 2.0 and 2.2? Two Months Later (3. Nov 88) 4

5 Nichteiertragende Weibchen am 3. Nov. '88, n=57 Number Carapax Length [mm] Comparing the two Distributions Nichteiertragende Weibchen Female Crabs, not egg carrying, caught 6. Sept. 88, n=215 Number Carapax Length [mm] : n = 215 Problem: different sample sizes : n = 57 axis such that each distribution has total area Idea: scale y- 5

6 Female Crabs, not egg carrying, caught 6. Sept. 88, n=215 Female Crabs, not egg carrying, caught 6. Sept ? = Proportion of Total per mm Total Area=1 Which Proporion had a length between 2.8 and 3.0 mm? ( ) 0.5 =

7 How to compare the two distributions? Nichteiertragende Weibchen Nichteiertragende Weibchen Carapax Length [mm] Carapax Length [mm] My Advice If you are a commercial artist: Impress everybody with cool 3D graphics! If you are a scientist: Visualize your data in clear and simple 2D plots. (As long as you print on 2D paper and project your slides on 2D screens) Simple and Clear: Polygons 7

8 Female Crabs, not egg carrying, caught 6. Sept. 88, n= Sept. '88, n= Carapax Length [mm] 3. Nov. '88, n= Carapax Length [mm] Convenient to show two or more Polygons in one plot Number Nov. '88 6. Sept. ' Carapax Length [mm] Biological Interpretation: What may be the reason for this shift? 8

9 2.1.1 Histograms: Densities or Numbers? Number vs. Number Number Histograms with unequal intervals should show densities, not numbers! Stripcharts and Boxplots Carapax 1.5 Carapax 9

10 Stripchart + Boxplots, horizontal Carapax 1.5 Carapax Boxplots, horizontal Boxplots, vertikal Simplify to understand Comparison of four groups Histograms and density polygons allow a comprehensive view on the data. Sometimes too comprehensive. Dichte Dichte Dichte Dichte

11 The Boxplot Boxplot, simple type Boxplot, simple type Boxplot, simple type Boxplot, simple type 50 % of the data 50 % of the data 50 % of the data 50 % of the data Median Boxplot, simple type 50 % of the data 50 % of the data Boxplot, simple type 25% 25% 25% 25% Min Median Max Min Median Max 11

12 Boxplot, simple type Boxplot, Standard Type 25% 25% 25% 25% Min 1. Quartile 3. Quartile Median Max Boxplot, Standard Type Interquartile Range Boxplot, Standard Type Interquartile Range 1.5*Interquartile Range 1.5*Interquartile Range Boxplot, Standard Type Boxplot Professional 12

13 Boxplot Professional 95 % Confidence Interval for the Median 2.3 Example: Darwin Finches Charles Robert Darwin ( ) Darwin Finches Darwin s collection of Finches References [1] Sulloway, F.J. (1982) The Beagle collections of Darwin s Finches (Geospizinae). Bulletin of the British Museum (Natural History), Zoology series 43: [2] 13

14 Wing Sizes of Darwin s Finches Wing Lengths by Island Flor_Chrl SCris_Chat Snti_Jams Flor_Chrl SCris_Chat Snti_Jams WingL Wing Lengths by Island Barplot of Wing Lengths (Numbers) Flor_Chrl SCris_Chat Snti_Jams SCris_Chat Flor_Chrl Snti_Jams Histogramm (Densities!) with transparen colors Polygons SCris_Chat Flor_Chrl Snti_Jams SCris_Chat Flor_Chrl Snti_Jams Wing Lengths Wing Lengths Beak Sizes of Darwin s Finches 14

15 Beak Sizes by Species Beak Sizes by Species Cam.par Cer.oli Geo.dif Geo.for Geo.ful Geo.sca Pla.cra Cam.par Cer.oli Geo.dif Geo.for Geo.ful Geo.sca Pla.cra 2.4 Conclusions Conclusions Histograms give detailed information. Polygons allow multiple comparisons. Boxplots can simplify large datasets. Stripcharts more appropriate for small datasets. Sophisticated graphics with 3D or semi-transperent colors do not always improve clarity. 3 Summarizing Data Numerically Idea It is often possible to summarize essential information about a sample numerically. e.g.: How large? Location Parameters How variable? Dispersion Parameters 15

16 Already known from Boxplots Location (How large?) Median Dispersion (How variable?) Inter quartile range (Q 3 Q 1 ) 3.1 Median and other Quartiles The median is the 50% quantile of the data. i.e.: half of the data are smaller or equal to the median, the other half are larger or equal. The Quartiles The first Quartile, Q 1 : A quarter of the observations are smaller than or equal to Q 1. Three quarters are larger or equal. i.e. Q 1 is the 25%-Quantile The third Quartile, Q 3 : Tree quarters of the observations are smaller than or equal to Q 3. One quarter are larger or equal. i.e. Q 3 is the 75%-Quantile 3.2 Mean, Standard Deviation and Variance NOTATION: Most frequently used Location Parameter: The Mean x Dispersion Parameter: The Standard Deviation s Given data named x 1, x 2, x 3,..., x n it is common to write x for the mean. 16

17 DEFINITION: Mean[1ex] =[1ex] Sum of oberved values Number of Observations Sum Number The formula for the mean of x 1, x 2,..., x n : x = (x 1 + x x n )/n = 1 n x i n i=1 Geometric Interpretation of the Mean Center of Gravity Mean = Center of Gravity Where is the center of gravity? x m = 1.5? m = 2? m = 1.8? 17

18 The Standard Deviation How far do typical overservations deviate from the mean? typical Deviation Mean = 2.8 = =? ,8 2.8 = 0,8 1,8 1, Die Standard Deviation σ ( sigma ) ist a slightly weired weighted mean of the deviations: σ = Sum(Deviations 2 )/n The formula for the Standard Deviation of x 1, x 2,..., x n : σ = 1 n (x i x) n 2 σ 2 = 1 n n i=1 (x i x) 2 is the Variance. i=1 18

19 Rule of Thumb for the Standard Deviation In more or less bell-shaped (i.e. single peak, symmetic) distributions: probability density ca. 2/3 are located between x σ und x+σ. x σ x x + σ Standard Deviation of Carapace lengths from females, not carrying eggs, caught 6. Sept. '88 females, not carrying eggs, caught 6. Sept. ' x = σ = 0.28 σ 2 = x = Carapax Length [mm] Carapax Length [mm] In this case 72% are between x σ and x + σ Variance of Carapace lengths from All Carace Lengths in North Sea: X = (X 1, X 2,..., X N ).Carapace Length in our Sample: S = (S 1, S 2,..., S n=215 )Sample Variance: σ 2 S = 1 n 215 i=1 (S i S) 2 0,

20 Can we use as estimation for σx 2, the variance in the whole population?yes, we can! However, σs 2 n 1 is on average by a factor of (= 214/215 n 0, 995) smaller than σx 2. Variances Variance in the Population: σ 2 X = 1 N N i=1 (X i X) 2 Sample Variance: σ 2 S = 1 n n i=1 (S i S) 2 (Corrected) Sample Variance: s 2 = = = n n 1 σ2 S n n 1 1 n 1 n 1 n (S i S) 2 i=1 n (S i S) 2 Usually, Standard Deviation (SD) of S refers to the corrected s. i=1 Example: Computing SD Given Data x =? x = 10/5 = 2 x x x (x x) s 2 = ( (x x) 2 ) /(n 1) = 16/(5 1) = 4 s = Computing σ with n or n 1? 20

21 Simulated population (N=10000 adults) Mean: Standard deviation: Length [cm] Sample from the population (n=10) M: SD with (n 1): 1.15 SD with n: Length [cm] Another sample from the population (n=10) M: SD with (n 1): 1.61 SD with n: Length [cm] 1000 samples, each of size n= SD computed with n SD computed with n Computing σ with n or n 1? 21

22 The standard deviation σ of a random variable with n equally probable outcomes x 1,..., x n (z.b. rolling a dice) is clearly defined by 1 n (x x i ) 2. n i=1 If x 1,..., x n is a sample (the usual case in statistics) you should rather use the formula 1 n (x x i ) 2. n 1 i=1 3.3 Mean values are usually nice but sometimes mean Mean and SD... characterize data well if the distribution is bell-shaped and must be interpreted with caution in other cases We will exemplify this with textbook examples from ecology, see e.g. References [BTH08] M. Begon, C. R. Townsend, and J. L. Harper. Ecology: From Individuals to Ecosystems. Blackell Publishing, 4 edition, When original data were not available, we generated similar data sets by computer simulation. So do not believe all data points example: picky wagtails Conjecture Size of flies varies. efficiency for wagtail = energy gain / time to capture and eat lab experiments show that efficiency is maximal when flies have size 7mm 22

23 References [Dav77] N.B. Davies. Prey selection and social behaviour in wagtails (Aves: Motacillidae). J. Anim. Ecol., 46:37 57, available dung flies captured dung flies dung flies: available, captured number mean= 7.99 sd= 0.96 number mean= 6.79 sd= 0.69 fraction per mm captured available length [mm] length [mm] length [mm] numerical comparison of size distributions dung flies: available, captured captured available mean 6.29 < 7.99 sd 0.69 < 0.96 fraction per mm captured available length [mm] Interpretation The birds prefer dung-flies from a relatively narrow range around the predicted optimum of 7mm. The distributions in this example were bell-shaped, and the 4 numbers (means and standard deviations) were appropriate to summarize the data example: spider men & spider women Nephila madagascariensis (c) by Bernard Gagnon 23

24 Simulated Data: 70 sampled spiders mean size: mm sd of size :12.94 mm????? Nephila madagascariensis (n=70) Nephila madagascariensis (n=70) Frequency Frequency Frequency mean= size [mm] size [mm] size [mm] Nephila madagascariensis (n=70) Nephila madagascariensis (n=70) Frequency males females Frequency males females size [mm] size [mm] Conclusion from spider example If data comes from different groups, it may be reasonable to compute mean an sd separately for each group example: copper-tolerant browntop bent References [Bra60] A.D. Bradshaw. Population Differentiation in agrostis tenius Sibth. III. populations in varied environments. New Phytologist, 59(1):92 103, [MB68] T. McNeilly and A.D Bradshaw. Evolutionary Processes in Populations of Copper Tolerant Agrostis tenuis Sibth. Evolution, 22: , Again, we have no access to original data and use simulated data. 24

25 Adaptation to copper? root length indicates copper tolerance measure root lengths of plants near copper mine take seeds from clean meadow and sow near copper mine measure root length of these meadow plants in copper environment Browntop Bent (n=50) Browntop Bent (n=50) density per cm Copper Mine Grass density per cm Grass seeds from a meadow root length (cm) root length (cm) Browntop Bent (n=50) Browntop Bent (n=50) density per cm Grass seeds from a meadow copper tolerant? density per cm meadow plants copper mine plants root length (cm) root length (cm) Browntop Bent (n=50) Browntop Bent (n=50) density per cm copper mine plants m s m m+s density per cm m s m m+s meadow plants root length (cm) root length (cm) 25

26 Browntop Bent n=50+50 copper mine plants meadow plants root length (cm) 2/3 of the data within [m-sd,m+sd]???? No! quartiles of root length [cm] min Q 1 median Q 3 max copper adapted from meadow Conclusion from browntop bent example Sometimes the two numbers m and sd give not enough information. In this example the four quartiles max, Q 1, median, Q 3, max that are shown in the boxplot are more approriate. Conclusions from this section Always visually inspect the data! Never rely on summarising values alone! 26