COMP6053 lecture: Principal components analysis.

Size: px

Start display at page:

Download "COMP6053 lecture: Principal components analysis."

Edwin Bond
6 years ago
Views:

1 COMP6053 lecture: Principal components analysis

2 Dealing with many variables Last week we looked at multivariate ANOVA, in which multiple continuous outcome variables are related to categorical predictor variables. What about situations in which we have many variables and are not yet sure which ones we want to regard as predictors and which as outcomes?

3 Dealing with many variables Suppose you've given personality tests to a large sample of people, and recorded scores on multiple traits such as extraversion, neuroticism, openness, etc. Or you've asked 500 people to score 100 different foods on a scale between 1 and 10 to say whether they like that food.

4 Dealing with many variables You have a large number of continuous variables for each subject or case. What structure might there be in the relationships between these variables? You don't have explicit predictors and outcomes: you just want to know whether liking wine goes with liking cheese, whether or not one causes or predicts the other.

5 Dealing with many variables Principal components analysis (PCA) is the right tool for the job here. Closely related technique: factor analysis. Essentially we're looking for structure in the covariance or correlation matrix. Can also view it as reducing our large set of variables to a smaller set that can explain much of the variance.

6 An example data set: heptathlon Heptathlon is a seven-event Olympic sport for women. It includes: hurdles high jump shot put 200m run long jump javelin 800m run.

7 An example data set: heptathlon Our sample data set is the results from the heptathlon at the 1988 Olympic games in Seoul. Won by Jackie Joyner-Kersee (right).

8 An example data set: heptathlon We're asking about the relationship between the results for component events. Is there a "running factor", or a "jumping factor", or a "throwing factor", for example? Seven (variables) is not a large number: PCA comes into its own in larger data sets, but seven will keep things manageable for presentation here.

9 Some housekeeping in R To get access to the heptathlon data set, we need to download the HSAUR package ("Handbook of Statistical Analyses Using R"). In the R GUI, set a CRAN mirror from the packages window (e.g., London or Bristol). Then type install.packages("hsaur") library(hsaur)

10 Examining the data There were 25 competitors in the heptathlon; here are the top 3. hurdles highjump shot run200m longjump javelin run800m score Joyner-Kersee (USA) John (GDR) Behmer (GDR) Some results are measured in seconds (lower numbers better), others in metres (higher numbers better).

11 Transforming the data In order to get all of the scores pointing in the same direction, we transform the times to be differences from the slowest time. heptathlon$hurdles = max(heptathlon$hurdles) - heptathlon$hurdles heptathlon$run200m = max(heptathlon$run200m) - heptathlon$run200m heptathlon$run800m = max(heptathlon$run800m) - heptathlon$run800m Now higher numbers are better in all cases.

12 The correlation matrix hurdles highjump shot run200m longjump javelin run800m hurdles highjump shot run200m longjump javelin run800m Can we see any structure here? What goes with what?

13 The correlation matrix hurdles highjump shot run200m longjump javelin run800m hurdles highjump shot run200m longjump javelin run800m With some colour-coding we see a sprinting / jumping factor start to emerge. Javelin results stand out as being unrelated.

15 Pairs plot Scatterplots of all seven variables against each other reinforces the impression we get from the correlation matrix. Clear linear relationships between hurdles, high jump, shot put, 200m, and long jump. Javelin and, to some extent, 800m results are less correlated with the other events.

16 Scaling the data Note that the seven events have very different variances. Standard deviation for the 800m is 8.29 (sec) whereas for the high jump it's only (m). If we work with unscaled scores, the 800m results will have a disproportionate effect. Thus we will tell the PCA function to scale all results to have a variance of 1.0.

17 What are the components in PCA? We're ready to run the PCA analysis now. PCA is going to give us seven components: but what are they? The first component is a line through the centroid of the data that covers as much variance as possible. Your score on C1 is where you sit on that line.

19 What are the components in PCA? The second component also goes through the centroid but is orthogonal to (i.e., uncorrelated with) the first. It is also chosen to explain the maximum amount of the remaining variance. Third and subsequent components continue in the same way, explaining less variance each time.

20 What are the components in PCA? Knowing only a woman's score on component 1 (i.e., where they fit along the first line) we could still recover a pretty good guess about their results, i.e., their location in 7-dimensional data-space. If we knew components 1 and 2, the guess would be more accurate.

21 What are the components in PCA? If we knew all 7 components we could completely reconstruct the person's scores. However, there would be little point to this: we started out knowing their scores. The idea is to choose a small subset of the most useful components and thereby compress the high-dimensional data down to a shorter description.

22 Running the PCA The R commands are easy. heppca = prcomp(heptathlon[,c(1:7)],scale=true) print(heppca) summary(heppca) The orange part is about specifying the first seven columns only. The green part is to force automatic scaling of the event results to a variance of 1.0.

23 What results do we get? print(heppca) gives us the seven principal components, expressed in terms of their loadings on each event result. heppca$rotation[,1] gives us the loadings just for the first component. hurdles highjump shot run200m longjump javelin run800m

24 Component loadings We can see that component 1 contains a little of everything, except for the javelin. It also happens to be negative: the sign is not really important here, as C1 is an artificial variable designed to explain variance. With predict(heppca)[,1] we can ask what each woman's C1 score is.

25 Component 1 scores Joyner-Kersee (USA) John (GDR) Behmer (GDR) Sablovskaite (URS) Choubenkova (URS) Schulz (GDR) Fleming (AUS) Greiner (USA) Lajbnerova (CZE) Bouraga (URS) Wijnsma (HOL) Dimitrova (BUL) Scheider (SWI) Braun (FRG) Ruotsalainen (FIN) Yuping (CHN) Hagger (GB) Brown (USA) Mulliner (GB) Hautenauve (BEL) Kytola (FIN) Geremias (BRA) Hui-Ing (TAI) Jeong-Mi (KOR) Launa (PNG)

26 Component 1 and total heptathlon score In the real heptathlon, there's a scoring system that gives each competitor a points total. The correlation between scores on our component 1 and the official scores is Thus we have clearly explained a lot of the relevant variance with our first factor alone.

27 How much variance does each component explain? summary(heppca) gives us information on how much variance each component covers. Importance of components: PC1 PC2 PC3 PC4 PC5 PC6 PC7 Standard deviation Proportion of Variance Cumulative Proportion C1 covers 63.7%, C2 covers 17.1%, C3 covers 7.4%, etc. plot(heppca) shows this graphically.

28 Variance explained by each component Plot shows the scaled variance explained by each component. Also known as a "scree plot".

29 Variance explained by each component There were 7 event results, each scaled to have a variance of 1.0. This makes a total of 7.0 units of scaled variance to be explained, so the area under the scree plot is 7.0. The first factor always "takes" the largest share of this variance.

30 How many components do we need? If we take all 7 components, we've achieved nothing in terms of compressing the highdimensional data. So we want a small subset of the components: one? Two? Three? Later components explain very little variance anyway.

31 How many components do we need? The usual criterion is to accept components that do at least their share of explanation. The original measurements explain 1.0 units of variance each (given the scaling). Thus only a component that explains more than 1.0 units of variance is really helpful.

32 How many components do we need? By this criterion we only want components 1 and 2 (i.e., we've compressed the data set from 7 dimensions to 2). In doing that we still explain 81% of the variance, so we have lost little. This criterion is sometimes expressed in terms of wanting a factor to have an "eigenvalue" of 1.0 or more.

33 How many components do we need? The criterion is not absolute. Some people argue for choosing all components before the scree plot "falls away". If a component explains slightly less than 1.0 units of variance but has an easily interpreted meaning (based on its loading scores) we would be inclined to keep it.

34 Assessing components 2 and 3 > heppca$rotation[,2] hurdles highjump shot run200m longjump javelin run800m > heppca$rotation[,3] hurdles highjump shot run200m longjump javelin run800m Component 2 has a strong (negative) loading on javelin ability. Component 3 seems to cover shot put ability.

35 Interpreting our PCA Let's go with 2 components that explain 81% of the total variance. Component 1 seems to reflect general athletic ability. Component 2 covers the specific skill of javelin throwing, but with a minor influence from shot put and 200m running ability.

37 Interpreting our PCA We can plot our 25 competitors in our reduced 2-D component space with biplot (heppca). This plot also shows the way each event loads on each component. We now have a better understanding of the structure implicit in the correlation matrix, and could use our two components as predictor or outcome variables in further analyses.

38 Relationship to factor analysis? Factor analysis is a closely related technique. Rather than breaking the data down into principal components, FA hypothesizes hidden variables that explain variation in the observed data. There is only one PCA result for a data set, but there can be many reasonable FA models.

39 Additional material The R script for running the PCA analysis of the heptathlon data set. The lecture borrows heavily from the PCA tutorial in "A Handbook of Statistical Analyses Using R" by Brian S. Everitt and Torsten Hothorn.

Variable Selection Methods for Multivariate Process Monitoring

Proceedings of the World Congress on Engineering 04 Vol II, WCE 04, July - 4, 04, London, U.K. Variable Selection Methods for Multivariate Process Monitoring Luan Jaupi Abstract In the first stage of a