On the Irrifutible Merits of Parceling

Transcription

1 On the Irrifutible Merits of Parceling Todd D. Little Director, Institute for Measurement, Methodology, Analysis and Policy Director & Founder, Stats Camp (Statscamp.org) CARMA Webinar November 10 th, 2017

2 2 Parcels and Parceling Understand reasons for Parcels Understand what Parcels are and how to make them Understand the conditions under which Parcels are beneficial Understand the conditions under which Parcels can be problematic

3 What is Parceling? Parceling: Averaging (or summing) two or more items to create more reliable indicators of a construct Packaging items, tying them together Data pre-processing strategy 3

4 Initial CFA: No Parcels Positive Negative 2 1* 1* (6.1.6items.per.Construct) Great Cheer ful Happy Good Glad Super Sad Down Un happy Blue Bad Terr ible Model Fit: χ 2 (53, n=759) = ; RMSEA =.057 ( ) ; CFI =.987; TLI/NNFI =.984 4

5 Using IFEEL items Example: Positive and Negative Affect In the last two weeks, I have felt 1. Great 7. Sad 2. Cheerful 8. Down 3. Happy 9. Unhappy 4. Good 10. Blue 5. Glad 11. Bad 6. Super 12. Terrible 5

6 Modification Indices for LAMBDA No reason to add a dual factor loading Positive Negative Great Cheerful Happy Good Glad Super Sad Down Unhappy Blue Bad Terrible

7 Modification Indices for THETA Not much evidence for estimating a correlated residual Great Cheerful Happy Good Glad Super Great - - Cheerful Happy Good Glad Super Sad Down Unhappy Blue Bad Terrible Sad Down Unhappy Blue Bad Terrible Sad - - Down Unhappy Blue Bad Terrible

8 crmda.ku.edu 8 CFA: Using Parcels Ψ 21 (6.2.Parcels) Ψ 11 1* 1* Ψ 22 Positive 1 Negative Great &Glad Cheerful & Good Happy & Super Terrible &Sad Down & Blue Unhappy & Bad θ 11 θ 22 θ 33 θ 44 θ 55 θ 66 Average 2 items to create 3 parcels per construct

9 CFA: Using Parcels Similar solution Similar factor correlation Higher loadings, more reliable info Good model fit, improved χ2 Positive Negative 2 1* 1* (6.2.Parcels) Great &Glad Cheerful & Good Happy & Super Terrible &Sad Down & Blue Unhappy & Bad Model Fit: χ 2 (8, n=759) = 26.76; RMSEA =.055 ( ) ; CFI =.994; TLI/NNFI =.989 9

10 Modification Indices for LAMBDA & THETA Modification Indices for LAMBDA-Y Positive Negative PosAFF PosAFF PosAFF NegAFF NegAFF NegAFF Note decrease in modification index values 6 possible MI s for factor loadings (vs. 12 with unparceled items) Modification Indices for THETA-EPS PosAFF1 PosAFF2 PosAFF3 NegAFF1 NegAFF2 NegAFF PosAFF1 - - PosAFF PosAFF NegAFF NegAFF NegAFF Note: 15 possible MI s for correlated residuals (vs. 66 with unparceled items) 10

11 Philosophical Issues To parcel, or not to parcel? 11

12 crmda.ku.edu 12 Empiricist / Conservative View Parceling is akin to cheating because modeled data should be as close to the response of the individual as possible in order to avoid the potential imposition, or arbitrary manufacturing of a false structure Preferred terms: mask, conceal, camouflage, hide, disguise, cover-up, etc. From Little et al., 2002

13 Pragmatic View Given that measurement is a strict, rulebound system that is defined, followed, and reported by the investigator, the level of aggregation used to represent the measurement process is a matter of choice and justification on the part of the investigator Preferred terms: remove unwanted, clean, reduce, minimize, strengthen, etc. From Little et al.,

14 crmda.ku.edu 14 Psuedo-Hobbesian View Parcels should be avoided because researchers are ignorant (perhaps stupid) and prone to mistakes. And, because the unthoughtful or unaware application of parcels by unwitting researchers can lead to bias, they should be avoided. Preferred terms: most (all) researchers are un as in unaware, unable, unwitting, uninformed, unscrupulous, etc.

15 I can be philosophical too: I think, therefore, I eat 15

16 Empirical Pros Psychometric Characteristics of Parcels (vs. Items) Higher reliability, communality, &ratio of common-tounique factor variance Lower likelihood of distributional violations More, Smaller, and more-equal intervals Never Seldom Often Always Happy Glad Mean Sum

17 More Empirical Pros Model Estimation and Fit with Parcels (vs. Items) Fewer parameter estimates Lower indicator-to-subject ratio Reduces sources of parsimony error (population misfit of a model) Lower likelihood of correlated residuals & dual factor loading Reduces sources of sampling error Makes large models tractable/estimable 17

18 Sources of Variance in Items crmda.ku.edu 18

19 Simple Parcel var I I I s2 s3 s4 x e e e var var var var var T1 3 9 var 2 var 3 var 4 T S 2 + e 2 T s 3 e 3 + T S x S 4 e 4 3 = T 1/9 of their original size! 19

20 Correlated Residual T x s3 s4 s5 e e e I I I 4 1 var var var var var 1 var var 3 var 4 var 5 T s 3 e 3 T + + S x S 4 e 4 T S x S 5 e 5 3 = T 4/9 its original size! 1/9 of their original size! 20

21 Cross-loading: Correlated Factors U C U2 s2 s3 s6 e e e I I I 16 1 var var var var var var 1 var var 2 var 3 var 6 U 1 C U 1 C + + U 1 C U 2 s 2 s 3 S 5 e 2 e 3 e 5 T 1 T 2 3 = U 1 C 16/9 its original size! 1/9 of their original size! 21

22 Cross-loading: Uncorrelated Factors U C U3 s1 s2 s3 e e e I I I 1 var var var var var var 1 var 3 9 var 1 var 2 var 3 U 1 C U 1 C + + U 1 C U 3 s 2 s 3 S 1 e 2 e 3 e 1 T 1 T 3 3 = U 1 C 1/9 of their original size! 22

23 Empirical Cons Multidimensionality Constructs and relationships can be hard to interpret if done improperly Model misspecification Can get improved model fit, regardless of whether model is correctly specified Increased Type II error rate if question is about the items Parcel-allocation variability Solutions depend on the parcel allocation combination (Sterba & McCallum, 2010; Sterba, in press) Applicable when the conditions for sampling error are high 23

24 Psychometric Issues Principles of Aggregation (e.g., Rushton et al.) Any one item is less representative than the average of many items (selection rationale) Aggregating items yields greater precision Law of Large Numbers More is better, yielding more precise estimates of parameters (and a person s true score) Normalizing tendency 24

25 Construct Space with Centroid 25

26 Potential Indicators of the Construct 26

27 Selecting Six (Three Pairs) 27

28 take the mean 28

29 and find the centroid 29

30 Construct Space with Centroid 30

31 Selecting Six (Three Pairs) 31

32 take the mean 32

33 find the centroid 33

34 How about 3 sets of 3? 34

35 taking the means 35

36 yields more reliable & accurate indicators crmda.ku.edu 36

37 Variance Components Construct Specific Error 37

38 Aggregation of Variance Components Construct Specific Error + 2 = Taking the mean of 2 indicators 38

39 crmda.ku.edu 39 Building Parcels Theory Know thy S and the nature of your items Random assignment of items to parcels (e.g., fmri) Use Sterba s calculator to find allocation variability when sampling error is high Balancing technique Combine items with higher loadings with items having smaller loadings [Reverse serpentine pattern] Using a priori designs (e.g., CAMI) Develop new tests or measures with parcels as the goal for use in research

40 Techniques: Multidimensional Case Example: Intelligence ~ Spatial, Verbal, Numerical Domain Representative Parcels Has mixed item content from various dimensions Parcel consists of: 1 Spatial item, 1 Verbal item, and 1 Numerical item Facet Representative Parcels Internally consistent, each parcel is a facet or singular dimension of the construct Parcel consists of: 3 Spatial items Recommended method 40

41 Domain Representative Parcels S + V + N = Parcel #1 S + V + N = Parcel #2 S + V + N = Parcel #3 Spatial Verbal Numerical 41

42 Domain Representative crmda.ku.edu 42

43 Domain Representative Intellective Ability, Spatial Ability, Verbal Ability, Numerical Ability But which facet is driving the correlation among constructs? crmda.ku.edu 43

44 crmda.ku.edu 44 Facet Representative Parcels S + S + S = Parcel: Spatial V + V + V = Parcel: Verbal N + N + N = Parcel: Numerical

45 Facet Representative 45

46 crmda.ku.edu 46 Facet Representative Intellective Ability Diagram depicts smaller communalities (amount of shared variance)

47 Facet Representative Parcels + + = + + = + + = A more realistic case with higher communalities crmda.ku.edu 47

48 Facet Representative 48

49 Facet Representative Intellective Ability Parcels have more reliable information 49

50 2 nd Order Representation Capture multiple sources of variance? 50

51 crmda.ku.edu 51 2 nd Order Representation Variance can be partitioned even further

52 crmda.ku.edu 52 2 nd Order Representation Lower-order constructs retain facet-specific variance

53 Functionally Equivalent Models Explicit Higher-Order Structure Implicit Higher-Order Structure 53

54 When Facet Representative Is Best 54

55 When Domain Representative Is Best crmda.ku.edu 55

56 Key Sources and Acknowledgements Special thanks to: Mijke Rhemtulla, Kimberly Gibson, Alex Schoemann, Wil Cunningham, Golan Shahar, John Graham & Keith Widaman Little, T. D., Rhemtulla, M., Gibson, K., & Schoemann, A. M. (2013). Why the items versus parcels controversy needn t be one. Psychological Methods, 18, Little, T. D., Cunningham, W. A., Shahar, G., &Widaman, K. F. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9, Little, T. D., Lindenberger, U., & Nesselroade, J. R. (1999). On selecting indicators for multivariate measurement and modeling with latent variables: When "good" indicators are bad and "bad" indicators are good. Psychological Methods, 4,

57