An Introduction to Psychometrics. Sharon E. Osborn Popp, Ph.D. AADB Mid-Year Meeting April 23, 2017

Transcription

1 An Introduction to Psychometrics Sharon E. Osborn Popp, Ph.D. AADB Mid-Year Meeting April 23, 2017

2 Overview A Little Measurement Theory Assessing Item/Task/Test Quality Selected-response & Performance Tests Psychometric Big Picture

3 Psychometrics Statistics Testing Measurement

4 Tests are Measurement Instruments Precision of measurement Accuracy of use and results An ideal test provides a model to guide development and improvement

5 Measurement Theory Can a test be perfect? An accurate reflection of ability, skill, knowledge?

6 If a Test is Perfect The test would measure an ability or skill precisely, like a perfect ruler Higher score = higher ability Lower score = lower ability

7 If a Test is Perfect Examinee answers would reflect their ability or skill level perfectly No failures of memory No careless mistakes No lucky guesses

8 If a Test is Perfect Test environment & administration conditions would be ideal and the scoring would be impeccable

9 A Perfect Test would have No test-based error No examinee-based error No error at all

10

11 But - there is no perfect test However, we can gauge test effectiveness by using the perfect test as our model So, what would the results of a perfect test look like?

12 Response Pattern of a Perfect Test Score ABCDEFGHIJKLMNO

13 Name Score ABCDEFGHIJKLMNO Pat Ollie Nell Mahesh Lily Kate Joe Inez Helga Gary Felipe Ella Dave Chris Brenda Alex

14 What do we expect? Easy Items Hard Items Low Ability Chris Item N High Ability

15 What do we expect? Easy Items Hard Items Item D Low Ability Lily High Ability

16 What do we expect? Easy Items Hard Items Low Ability Item H Helga High Ability

17 Name Score ABCDEFGHIJKLMNO Pat Ollie Nell Mahesh Lily Realistic Test Response Pattern Kate Joe Inez Helga Gary Felipe Ella Dave Chris Brenda Alex

18 Patterns of differences from what we expect provides valuable information to support test development and improvement

19 Minimizing Error Classical Test Theory Every test result has an error component: X = T + E Latent Trait Theories (Rasch/ IRT) Every test result is a function of ability and difficulty: O = A / D

20 Name Score Item P Pat 15 1 Ollie 14 1 Nell 13 0 Mahesh 12 1 Lily 11 0 High Difficulty Kate 10 1 Joe 9 0 Inez 8 0 Helga 7 0 Gary 6 0 Felipe 5 0 Ella 4 0 Dave 3 0 Chris 2 0 Brenda 1 0 Alex 0 0

21 Name Score Item Q Pat 15 1 Ollie 14 1 Nell 13 1 Mahesh 12 1 Lily 11 1 Low Difficulty Kate 10 1 Joe 9 1 Inez 8 1 Helga 7 1 Gary 6 1 Felipe 5 1 Ella 4 0 Dave 3 1 Chris 2 0 Brenda 1 0 Alex 0 0

22 Name Score Item R Pat 15 0 Ollie 14 0 Nell 13 0 Mahesh 12 1 Lily 11 0 Kate 10 0 Joe 9 0 Inez 8 1 Helga 7 0 Gary 6 0 Felipe 5 0 High Difficulty & Unexpected Pattern Ella 4 0 Dave 3 1 Chris 2 0 Brenda 1 0 Alex 0 0

23 Name Score Item S Pat 15 1 Ollie 14 0 Nell 13 1 Mahesh 12 0 Lily 11 1 Kate 10 1 Joe 9 0 Inez 8 1 Helga 7 0 Gary 6 1 Felipe 5 0 Unexpected pattern Ella 4 1 Dave 3 1 Chris 2 0 Brenda 1 1 Alex 0 0

24 Name Score Item T Pat 15 0 Ollie 14 0 Nell 13 0 Mahesh 12 1 Lily 11 0 Kate 10 0 Joe 9 1 Inez 8 1 Helga 7 0 Gary 6 1 Felipe 5 1 Unexpected pattern Ella 4 1 Dave 3 1 Chris 2 1 Brenda 1 1 Alex 0 1

25 Basic Item Quality Indicators Item Difficulty Proportion Correct Discrimination Index Difference in Difficulty between high and low scorers Point-biserial Correlation Item-Total Correlation Option Review Diff / Disc / Pt-Bis per option Who is choosing which options?

26 Basic Overall Test Indicators Descriptive Statistics N, Mean, Median, Min, Max, Standard Deviation Shape of score distribution Means of indicators (e.g., Disc, Pt-Bis) Error Estimates Standard Error of Measurement (SEM) Conditional SEM at Cut-score Internal Consistency Reliability

27 Questions, so far?

28 What about Performance Tests? Human-judged (usually) Levels of quality or quantity Degree of correct response Presence or Absence of skills/behaviors

29 Quality of Performance Tests Rating Scale Quality Examiner Quality

30 Examiner Quality Tests in which the results depend on human judgments have a different kind of potential testing error to identify and minimize

31 Common Rater Errors Harshness Lenience Erratic

32 Median The middle grade moderates the impact of extreme grades

33 However If extreme grades are commonplace, measurement accuracy is threatened

34 Assessing Examiner Quality Calibration Testing Assess understanding of grading criteria Interrater Agreement Proportion of grades in agreement Interrater Reliability Consistency of agreement within the pool Rater Severity Estimation Controls for other components of the system

35 Best Practices to Ensure High Interrater Agreement Rigorous examiner training re: criteria Well-planned examiner teams Team linkage across sites Extensive post-exam review and analyses Performance feedback after exams

36 Rating Scales Levels of Quality or Quantity 4 - Optimal 2 - Correct 1 - Partially Correct 0 - Incorrect 3 - Appropriate 2 - Acceptable 1 - Inadequate 0 - Unacceptable

37 Rating Scales Clear Definitions Examples at each level Well-functioning Categories Not too many & Not too few Levels proceed with ability

38 Assessing Rating Scale Quality Most psychometric indicators are natural extensions of selected-response measures

39 Category Quality Higher categories = Higher performance levels Lower categories = Lower performance levels

40 Category Probability Curves Categories proceed along ability continuum Threshold where the probability of getting rated a 4 exceeds the probability of getting rated a 3

41 Real Example

42 Real Example

43 Detecting Category Dysfunction From J.M. Linacre in Journal of Applied Measurement (2002)

44 Basic Rating Scale Quality Indicators Basic Descriptive Statistics Count within each category Average Score within each category Item-Total Correlations Category Probability Curves Threshold Estimates

45 Questions at this point?

46 What else is included in Psychometrics?

47 The Big Picture: Evidence of Test Validity Content Administration Scoring Technical Quality Fidelity, Reliability, Fairness

48 Other Important Analyses and Investigations Form Equivalence, Equating, Scaling Differential Item/Test Functioning Dimensionality & Construct Validity Classification Consistency Impact Estimation for Changes/ Revisions

49 Different Psychometric Contexts Criterion- or Norm-referenced Summative or Formative Psychomotor, Cognitive or Affective Large- or Small-scale High- or Low Stakes

50 Psychometrics Identification and minimization of error sources Examining data for departures from results expected under optimal test quality Systematic collection of validity evidence

51