Modeling position effects within an IRT framework

Transcription

1 Modeling position effects within an IRT framework Debeer & Janssen University of Leuven NCME annual conference, Vancouver, Saterday april 14th

2 Content A. Brief introduction. B. Item position effects C. IRT Modeling D. Discussion

3 A. Brief introduction Attribute of interest Administration conditions Test score / Item response

4

5 A. Brief introduction BUT: Context effects within a test.. Test length Testlets Item context (content related) Item order / item position

6 B. Item position effects Differences in item order / item position between administrations Examples: Alternate test forms (same items) Anchor items Adaptive testing

7 Alternate test forms A B Reversed order

8 Alternate test forms A B Random orders

9 Alternate test forms A B C Booklet design, partly overlapping

10 Anchor items A B

11 B. Item position effects Do Item Positions Matter? Literature Mollenkopf (1950) Whitely & Dawis (1976) Yen (1980) Pommerich and Harris (2003) Hohensinn et al. (2008) Schweizer, Schreiner and Gold (2009) Meyers, Miller and Way (2009) Debeer & Janssen (1) and (2)

12 Debeer & Janssen (1) Set 1 Set 2 Booklet 29 items 28 items 29 items N = 805 Booklet Booklet Booklet Booklet 4 186

13 Debeer & Janssen (1) Difference in item difficulty between two reversed orders, according to the order in the first booklet. Difference in item difficulty

14 Difference in item difficulty Debeer & Janssen (1)

15 Debeer & Janssen (2) PISA 2006 (Turkey)

16 PISA 2006 (Turkey): Science Example of increase in item difficulty for one item

17 PISA 2006 (Turkey): Reading Example of increase in item difficulty for one item

18 PISA 2006 (Turkey): Math Example of increase in item difficulty for one item

19 B. Item Position effects Debeer & Janssen: Item position does matter! Item difficulty depends on item position How to interpret Items become more difficult in later positions? Measured attribute changes during test?

20 B. Item position effects How to deal with them? Model Take the position effect into account = include the position effect in the measurement model. IRT framework (De Boeck & Wilson, 2004)

21 C. IRT Modeling Until now: Position effect specific for each item: logit Y pik = 1 = α i θ p β i + δ ik β DIF

22 Debeer & Janssen (1) Difference in item difficulty between two reversed orders, according to the order in the first booklet.

23 C. IRT Modeling Until now: Position effect specific for each item Position effect across items logit Y pik = 1 = α i θ p β i + δ k β

24 Debeer & Janssen (2):

25 C. IRT Modeling Until now: Position effect specific for each item Position effect across items Further: Trend in position effect: logit P Y ipk = 1 = θ p (β i +γ k 1 ) LLTM method (proposed by Kubinger, 2009)

26 C. IRT Modeling Debeer & Janssen (1): Position effect Change in ODDS (Y = 1) P(Y = 1) a + 1 position + 30 positions + 1 position + 30 positions a When the item has a discrimination equal to 1 and the probability of a correct response in the reference position is 0.50.

27 C. IRT Modeling Further: Trend in position effect Individual differences => position dimension logit P Y ipk = 1 = θ p (β i +γ p k 1 ) RW-LLTM method (Rijmen & De Boeck, 2002) Interpretation!

28 C. IRT Modeling Debeer & Janssen (1): Size of the random linear position effect for Item set 1 and 2 combined Position effect Change in ODDS (Y = 1) P(Y = 1) a z(γ) γ + 1 position + 30 positions + 1 position + 30 positions a When the item has a discrimination equal to 1 and the probability of a correct response in the reference position is 0.50.

29 C. IRT Modeling Debeer & Janssen (2): PISA 2006 (Turkey): Random linear cluster effect Literacy weight p-value SD r Math.132 < Reading.241 < Science.106 <

30 Building blocks DIF LLTM RW-LLTM C. IRT Modeling Extentions: Position effect on discrimination Non-uniform DIF 2PL constrained model (Embretson, 1999)

31 C. IRT Modeling Position effect at the level of the item Position effect at the level of the test score

32 C. IRT Modeling Modelled at the level of the item Interpretion: Person level Fatigue / motivation? Extra dimension = individual differences

33 D. Discussion Practical relevance: Detecting and modeling the effect of item position. Important for item kalibration Flexibility of the framework Estimation with R (GLMM) or SAS (NLMM)

34 D. Practical relevance Tetst construction: Items from hard to easy:

35 D. Discussion Further questions: Position effect on specific test, or more general? Is the position effect on different test the same? Interpretation of the position effect / individual differences in the effect of item position?

36 Take home message Item order / position does matter! Items seem to become more difficult when placed later in the test. Not to be ignored The IRT offers a flexible framework for detecting and modeling position effects. Position dimension with individual differences Further research for interpretation.

37 your Thank attention for you! Position does matter!