Simple Linear Regression

Transcription

1 Simple Linear Regression Technical: lin regression sd(errors) Conf Ints Pred Ints Conceptual Normal dist response y parameters non-linear A what-if prob model for prediction A screening process A scientific tool for testing/thinking non-normal predictors x scales (inc logs) MINITAB A simple model of the data generating system Cert Stats: Intro to Regression Week 1 1

2 Simple Lin Reg Y = +x + Revision Examples Extensions/Variations Scales Additive / Multiplicative Multiple regression Concepts in Statistical Modelling Statistical Inference Why model? The purpose of a statistical model is not: to fit the data is rather: to refine the question Cert Stats: Intro to Regression Week 1 2

3 Statistical Models (Some aspect of) Y simply related to (Some aspect of) X (Some aspect of) in reported units; some other units; more general. Simply related to Small change in X (Same?) Small change in Y Change Additive; Multiplicative; other Apart from some (small) (unpredictable) (error) Which is and Why model? Additive; Multiplicative; other Of (same) (avg size) for all values of X Cert Stats: Intro to Regression Week 1 3

4 Using models Conceptual A summary of the data to communicate to others A what-if prob model for prediction A screening process A scientific tool for testing/thinking A simple model of the data generating system Summary / Descriptive Comparison Insight into process Cert Stats: Intro to Regression Week 1 4

5 Scales and Change Time Salary, price, volume Quality Temperature Earthquake magnitude, ph Approval/Rating Likelihood/proportion Before/after; with/without Treatments/Admin unit: 1,2,3,4 or A,B,C,D Fuel consumption Miles/gallon Miles/litre Litres/1 km!!!!!!!!!!! Keep global warming to 2C, equiv 35.6F!!!!!!!!!!! Cert Stats: Intro to Regression Week 1 5

6 Completions House Completions vs Time Fitted Line Plot Completions = time S R-Sq 87.9% R-Sq(adj) 87.6% House completions data series. Based on the number of new dwellings connected by ESB Networks to the electricity supply 8 6 Source: Department of the Environment, Heritage and Local Government 4 Objective: To summarise time Question? Cert Stats: Intro to Regression Week 1 6

7 Price Sing $ Diamonds Carat wt vs Price Fitted Line Plot Price Sing $ = Carat Data on Carat - Wt of stones in carat units Price (Singapore $) 15 1 Regression 95% CI 95% PI S R-Sq 89.3% R-Sq(adj) 89.2% 5 Objective Nominal: Predict Price from Wt Actual: Is relationship linear? Question? Carat Prediction bands 2s Pricing the C's of Diamond Stones Singfat Chu National University of Singapore Journal of Statistics Education Volume 9, Number 2 (21) See Cert Stats: Intro to Regression Week 1 7

8 Volume Trees: Vol Fitted Line Plot Volume = Height 7 75 Height Regression 95% CI 95% PI S R-Sq 35.8% R-Sq(adj) 33.6% Sample of 31 black cherry trees in the Allegheny National Forest, Pennsylvania, Y = volume (cubic feet), X 1 X 2 = height (feet) = diameter (inches) (at 54 inches above ground Objective: Use height as proxy for volume Later Use diam and ht combined measure as proxy for volume Objective: Prediction via Calibrated Model Question? Prediction bands 2s Cert Stats: Intro to Regression Week 1 8

9 Compensation M.Stuart CEO Compensation (US$) and Company Sales (US$m) (Forbes Magazine, May 1994) Fitted Line Plot Compensation = Sales Total comp Industry Sales Regression Financial 95% 52 PI ComputersComm Insurance ComputersComm Financial Entertainment 415 S R-Sq 13.7% R-Sq(adj) 13.5% Sales Question? Cert Stats: Intro to Regression Week 1 9

10 Gas Gas Gas Comsumption vs Temp Period 1 7 Fitted Line Plot Gas = Temperature S R-Sq 94.4% R-Sq(adj) 94.1% Weekly gas consumption (in 1 cubic feet) and average outside temperature (in deg C) for two "heating seasons : 26 weeks before / 3 weeks after cavity-wall insulation was installed. Thermostat was set at 2 C throughout Objective: Test: Measureable effect of insulation? Question? Temperature 8 1 Period 2 5 Fitted Line Plot Gas = Temperature S R-Sq 81.3% R-Sq(adj) 8.6% Compare Intercepts & Slopes Before/After Temperature Cert Stats: Intro to Regression Week 1 1

11 Stat Stat Math Marks Fitted Line Plot Stat = Alg Regression 95% CI 95% PI S R-Sq 44.2% R-Sq(adj) 43.5% 88 students in each of 5 maths exams Objective: Nominal: Predict one mark from another Actual: Understand inter-relationships Question? Alg Prediction bands 2s Fitted Line Plot Stat = Anal Regression 95% CI 95% PI S R-Sq 36.9% R-Sq(adj) 36.1% 4 2 Mardia, K.V., Kent, J.T. & Bibby, (1979). Multivariate analysis. London: Academic Press Anal Cert Stats: Intro to Regression Week

12 Galton s heights data N=178 pairs; Y=Offspring (inches) X=Mid-parent (inches) Slope =.514 Galton s Data Objective: Nominal: Predict off-spring ht Actual: Quantify heredity Question? slope 1 Reversion to the mediocre Regression to the mean Corr =.51 Cert Stats: Intro to Regression Week 1 12

13 Plotting Scales Logs Revision Statistical Inference Normal distribution Prediction and Confidence Intervals Statistical significance Cert Stats: Intro to Regression Week 1 13

14 C o m p l e t i o n s Plotting Revision Fitted Line Plot Completions = time Over-plot fitted line S R-Sq 87.9% R-Sq(adj) 87.6% Visually estimate residuals % prediction interval line 2s parallel lines time Cert Stats: Intro to Regression Week 1 14

15 Gas Straight Line: changing the scale Change Temp to Fahrenheit (C, F) (, 32) Freezing (C, F) (1, 212) Boiling Fitted Line Plot Gas = Temperature Exercise Change Gas to ft³ 1 ft³ = L S R-Sq 94.4% R-Sq(adj) 94.1% Temperature 8 1 Temp in F Cert Stats: Intro to Regression Week 1 15

16 Straight Line: change the equation Gas = Temp C Give equations for use with Fahrenheit X axis Slope + 1 C reduction of.393 in Gas +1.8 F reduction of.393 in Gas +1 F reduction of.393/1.8 =.218 in Gas Intercept C Gas consumption = F Gas cons = 6.85 Gas F Gas cons = (-32)=13.82 = Temp F See EXCEL Change.Scale y Temp C Cert Stats: Intro to Regression Week Temp F 32 / / / Temp F Temp F

17 Gas CenGas Alt parameterisation: centred y a bx y y bx x y y bx bx Centred Coeffs before centering a=6.85 b=-.393 Means Gas 4.75 Temp 5.35 X =? Fitted Line Plot Gas = Temperature Fitted Line Plot CenGas = CenTemp 7 6 S R-Sq 94.4% R-Sq(adj) 94.1% 3 2 S R-Sq 94.4% R-Sq(adj) 94.1% Temperature 8 1 Temp Gas Cen Temp Cen Gas CenTemp Cert Stats: Intro to Regression Week 1 17

18 Gas StGas Alt parameterisation: standardised y a bx Standardised y y s x x x x x b s y s y sx sx sy sy y y x x s s Now: intercept ; slope x x Coeffs before standardising a=6.85 b=-.393 Means- Gas: 4.75 Temp:5.35 SDs Fitted Line Plot Gas = Temperature Fitted Line Plot StGas = StTemp 7 S R-Sq 94.4% R-Sq(adj) 94.1% 2 S R-Sq 94.4% R-Sq(adj) 94.1% Temperature 8 1 Temp Gas StTemp StGas StTemp 1 2 Scale free Cert Stats: Intro to Regression Week 1 18

19 Revision: scale Descriptive Statistics: Stat, Anal, Alg, Vect, Mech 8 4 Stat Matrix Plot of Stat, Anal, Alg, Vect, Mech Variable Mean SE Mean StDev Stat Anal Alg Vect Mech Anal 2 Alg 45 7 Vect Mech Correlations: Stat, Anal, Alg, Vect, Mech Stat Anal Alg Vect Anal.67 Alg Vect Mech Cell Contents: Pearson correlation Derive Equations: Stats vs Algebra Stat Alg Stats vs Analysis Cert Stats: Intro to Regression Week 1 19

20 Scale: Additive or Multiplicative Y depends on x; some random variation involved Data (y 1, x 1 ), (y 2, x 2 ) Additive or Multiplicative changes? (y 2 - y 1 ) depends on (x 2 - x 1 ) (y 2 / y 1 ) depends on (x 2 / x 1 ) Multiplicative change eg Doubles 1% increase Log Transform renders Mult as Additive (log(y 2 ) - log(y 1 ) ) depends on (log(x 2 ) - log(x 1 ) ) Cert Stats: Intro to Regression Week 1 2

21 Price Sing $ Price Sing $ Fitted Line Plot Price Sing $ = Carat Diamonds Regression 95% CI 95% PI S R-Sq 89.3% R-Sq(adj) 89.2% Minitab? Model details? Compare? Improve? Fitted Line Plot log1(price Sing $) = log1(carat) Carat Prediction bands 2s Regression 95% CI 95% PI S R-Sq 95.7% R-Sq(adj) 95.7% Prediction bands? Carat Cert Stats: Intro to Regression Week 1 21

22 Volume Log1Vol Volume Trees: Vol vs Height Fitted Line Plot Log1Vol = Log1Ht Fitted Line Plot log1(volume) = log1(height) Regression 95% CI 95% PI 1 Regression 95% CI 95% PI S R-Sq 42.1% R-Sq(adj) 4.1% S R-Sq 42.1% R-Sq(adj) 4.1% Log1Ht Height Fitted Line Plot log1(volume) = log1(height) Height Regression 95% CI 95% PI S R-Sq 42.1% R-Sq(adj) 4.1% Minitab? Model details? Compare? Improve? Cert Stats: Intro to Regression Week 1 22

23 Multiplicative Change Salary increases by 2% Salary Salary + 2% of Salary Salary + Salary.2 Salary 1.2 Salary decreases by 2% Salary Salary.8 Salary increases by 2% then deceases by 2% Salary (Salary 1.2).8 = Salary? Cert Stats: Intro to Regression Week 1 23

24 Recall: Logs and Percentages 1 log 1 log1 1 log log 5 log 5 1 log 5 log log 1 log 1 1 log 1 log 1 2 log Antilog Antilog Salary increases by 2% Salary Salary 1.2 log Salary log Salary log1.2= log Salary.79 Salary decreases by 2% Salary Salary.8 log Salary log Salary log.8= log Salary.97 Log Salary increases by.79 then decreases by.79 Salary unchanged by 2% by 16.6% Not symmetric by by 1.2 by ie by Cert Stats: Intro to Regression Week 1 24

25 Changing to/from Log scale Equation in log scale: log( y) a blog( x) ab log( x) a b log( x) a log( x) b a b x a log( x) b a b a/ b Antilog( y) Equation in lin. scale: y y x OR x b y x OR Rescaled x b b Increase log( x) by1 Increase log( y) by b (linearly) Multiply x by1 b Increase y by factor of 1 (multiplicatively) b b y y changes to y 11 (linear change of 1 ) Cert Stats: Intro to Regression Week 1 25

26 Trees: Changing to/from Log scale Equation in log scale: log( Volume) log( Height) log( ) b ( Height) Equation in lin. scale: Volume Volume a x a b Volume.867 ( Height) x (1 ) OR Volume Height (.3 Height) Volume Height doubles (. 3 Height) 4 Refine the question? Cert Stats: Intro to Regression Week 1 26

27 CEO Compensation Which scale to use? What does this tell us about the nature of the relationship? Compensation Rescaled Sales 1/4 Sales Double Question? Cert Stats: Intro to Regression Week 1 27

28 Error Term in Log Scale Linear Scale Model y a bx y predicted y 2 Error Model, 95%of random error lie within 2 N s s Notation sometimes used y predicted y 2 s s small; y close to pred y Error band width constant Log Scale Model log y a blog x log y predictedlog y 2 s y 2 Linear Scale Plot y predicted y 11 Notation sometimes used y predicted 1 1 s s small; y close to pred y Error band y Multiplicative error Cert Stats: Intro to Regression Week 1 28

29 Error Term in Log Scale Not symmetric Cert Stats: Intro to Regression Week 1 29

30 Frequency FITS1 Revision Computing Scatterplot of FITS1 vs time Comps Time RESI1 FITS Etc time Histogram of RESI1 Normal Mean StDev N RESI1 Cert Stats: Intro to Regression Week 1 3

31 Frequency FITS1 Revision Computing Scatterplot of FITS1 vs time Comps Time RESI1 FITS Etc time Histogram of RESI1 Normal Mean StDev N RESI1 Cert Stats: Intro to Regression Week 1 31

32 Frequency Histogram of RESI1 Normal Revision Normal Dist Mean StDev N 44 Long run props 68% within ± 1 SD of mean 95% within ± 2 SD of mean -2-1 RESI % within ± 3 SD of mean 44 resids Mean. SD of outside ( 915.8) of 44 = % Cert Stats: Intro to Regression Week 1 32

33 Revision Normal Dist Sketch Normal Mean 25 SD 4 68% within ± 1 SD of mean 95% within ± 2 SD of mean 99.7% within ± 3 SD of mean Cert Stats: Intro to Regression Week 1 33

34 Stat Revision: SumSq Fitted Line Plot Stat = Anal 3 4 Anal Regression 95% PI S R-Sq 36.9% R-Sq(adj) 36.1% Descriptive Statistics: Sum of Variable Count Mean StDev Variance Squares Stat Anal RES FITS CenStat CenAnal CenFit Regression Analysis: Stat versus Anal The regression equation is Stat = Anal S = R-Sq = 36.9% R-Sq(adj) = 36.1% Analysis of Variance Source DF SS MS F P Regression Error Total Stat Anal RESI1 FITS Cert Stats: Intro to Regression Week 1 34

35 Revision: Covar,Corr and R 2 Regression Analysis: Stat versus Alg Stat = Alg S = R-Sq = 44.2% Analysis of Variance Source DF SS MS Regression Error Total Covariances: Stat Alg FITS1 RESI1 Stat 298 Alg FITS RESI Covariances: Stat Alg FITS1 RESI1 Stat Alg FITS RESI Correlations Stat Alg RESI1 Alg.665 RESI FITS Correlations Stat Alg FITS1 RESI1 Stat 1 Alg.66 1 FITS RESI Cert Stats: Intro to Regression Week 1 35

36 Revision: Corr, R 2, S Descriptive Statistics: Stat, Anal, Alg, Vect, Mech 8 4 Stat Matrix Plot of Stat, Anal, Alg, Vect, Mech Variable Mean SE Mean StDev Stat Anal Alg Vect Mech Anal 2 Alg 45 7 Vect Mech Correlations: Stat, Anal, Alg, Vect, Mech Revision Stats vs Anal Stat Anal Alg Vect Anal.67 Alg Vect Mech Derive R 2, S Cell Contents: Pearson correlation Cert Stats: Intro to Regression Week 1 36

37 Revision: Statistical Inference Point Estimates Best single values for intcpt/slope Conceptually Propose many values Compute SumSq of implied residuals Choose values with min SSQ Margin for Error Confidence Intervals Prediction intervals Statistical Testing Cert Stats: Intro to Regression Week 1 37

38 Are precision details important? Precision about what? Coefficients Gas = Temp (-.29 ±?) Need to test null hypotheses? Is there evidence against insulation does not change Gas/Temp relationship? Cert Stats: Intro to Regression Week 1 38

39 Concept: Statistical Model data like this Replication Data Generating System Comp Generated Random Numbers y i ~ N, i 2 Cert Stats: Intro to Regression Week 1 39

40 Concept: Simple linear regression model y i ~ N, i 2 Data like this Line Data "like completions" High Low Pinch of salt Cert Stats: Intro to Regression Week 1 4

41 Concept: Sampling Distribution What results with data like this? Monte Carlo Experiments Sampling Distribution St Dev (Samp Dist) = St Error Alt Formulae methods Used by MINITAB Cert Stats: Intro to Regression Week 1 41

42 M.Stuart Concept: Statistical Inference Choose the best fitting model Treat it with scepticism How much? Rely on formulae based on the Normal dist Use random computer replication eg modern bootstrap methods Cert Stats: Intro to Regression Week 1 42

43 Science and Statistical Inference Assumptions: Well stated and focussed science Adequate and relevant data Adequate model for data like this Independent sources of info random variation Normal distribution Cert Stats: Intro to Regression Week 1 43

44 More than one predictor? Multiple Regression Statistics Marks: Four other Math marks Tree Volume: Tree Height and Diam at Chest Ht Diamond Price: Carat wt and Aspects of Quality Gas: Temp and Insulation Status Housing Comps: Seasonality Cert Stats: Intro to Regression Week 1 44