Introduction to Facets

Transcription

1 Introduction to Facets Aims and Objectives Preparing data for running in Facets Running Facets to familiarise you with the output An examination of examinations dataset Three facets: candidates, examiner, task; complex rating scale Small dataset but real and rather interesting Can be fitted in constraints of MiniFac (2000 data points) Similar to OSCEs etc in data structure General principles Never write a Facets program from scratch, but always edit an existing program Find something similar in the manual and work from there Facets is not the most elegant of programming languages, seeming to have just grown, but it does do good things

2

3

4 Bn = ability of examinee n Am = difficulty of task m Di = difficulty of skill item i Cj = severity of judge j Fk = difficulty of category k relative to category k-1 Pnmijk = probability of rating of k Pnmijk-1 = probability of rating of k-1

5 Bn = ability of examinee n Am = difficulty of task m Di = difficulty of skill item i Cj = severity of judge j Fk = difficulty of category k relative to category k-1 Pnmijk = probability of rating of k Standard Rasch modelling is a subset of the models that Facets runs Pnmijk-1 = probability of rating of k-1

6 Running Facets/MiniFac Easiest for today if all of the files are in the Facets directory where MiniFac was installed Facets does though handle directories with no trouble Files for this workshop are at: OfExaminations/ExOfEx Download the five files into the c:\facets directory A tip: Although the output goes to a Notepad file, it can be easier to use the free text editor Notepad++ Multiple tabs can be open at once Can see various program and output files Reloads file when changed (after asking)

7 Data Data have to be long and thin Each row is a single datapoint with descriptors Facets can read.csv files or SPSS.sav files SPSS data for exams are typically stored with candidates in rows and items in columns Make data long and thin using the VarsToCases command Menu: Data > Restructure > Restructure selected variables into cases Candidate 2, Paper 1, Examiner 4, Mark = 17 1,1,4,15 1,1,11,12 1,1,15,11 1,1,16,7 1,1,17,5 1,2,1,14 1,2,2,9 1,2,3,19 1,2,6,9 1,2,8,16 1,2,10,11 1,2,11,11 1,2,12,6 1,2,14,5 1,2,18,10 1,3,1,16 1,3,2,12 1,3,3,11 1,3,4,18 1,3,5,13 1,3,6,13 1,3,7,12 1,3,8,11 1,3,9,5 1,3,10,13 1,4,1,11 1,4,2,12 1,4,3,9 1,4,4,10 1,4,5,10 1,4,6,12 1,4,7,14 1,4,8,10 1,4,9,14 1,4,10,6 2,1,4,17 2,1,11,10 2,1,15,13 2,1,16,12 2,1,17,11 2,2,1,12 2,2,2,13

8 (1935)

9

10 Not all examiners marked all papers. Facets doesn t worry about this as long as all subsets are connected

11

12

13

14 The Program: ExOfEx-ExampleMark24.txt Programs are just text files. Naming them as.txt means they can be edited easily First line is a title; anything after semi-colon is a comment Lines can be in almost any order but sensible to keep them ordered First block after title specifies input and output files Title= An Examination of Examinations 24 point marking scheme ; anything following a semi-colon is a comment ;input and output files data=c:\facets\examinationofexaminations.csv ; ; data in the.csv file Output=ExOfEx-ExampleMark24.out ;name of output file for text ScoreFile=Tables-OutputScoresMark24; ; generate csv files for output summaries csv=yes Gstats=Yes HeadingLines=No

15 The Program: ExOfEx-ExampleMark24.txt Next commands specify the facets ; data description Facets=3 ; number of facets * Labels= 1,Candidate ;!!!check that max N is high enough for labels * 2, Paper 1=Ancient 2=MedMod 3=Essay 4=Polit * 3,Examiner ;!!!check that max N is high enough for labels 1=A 2=B 3=C [ lines deleted for other examiners] 17=Q 18=R *

16 The Program: ExOfEx-ExampleMark24.txt Now the model ;; the Model line specifies the model ;?,?,? means that all three facets 1,2 and 3 are to be estimated ; R24 means it is a 24-point rating scale, alpha+ to delta Model=?,?,?,R24 ;Rating scale model * ; at least one facet has to be non-centred or the model is not identified noncenter = 2,3

17 The Program: ExOfEx-ExampleMark24.txt Now specify the layout of the results ; specify the layout in the yardstick columns ; N: Numbers, L: Labels; * Counts Vertical=1N,2L,3L,SL ; specify the vertical axis of the yardstick Yardstick (columns lines low high extreme)= 0,25,-1,1,End ; the output listings to be produced Arrange = 1mN,2mN,3mN * inter-rater=3 ; facet 3 is the rater Note that the inter-rater reliability is akin to Cohen s kappa, and Linacre says that one does not want raters to be more similar than the model would suggest, so that their random errors are locally independent ( independent experts rather than rating machines [human optical scanners]). Treat with care!

18 Running Facets/MiniFac Open MiniFac in the usual way Opens in a new window Click on Files > Specification File Name Go to the c:\facets directory if you are not already there Select ExOfEx-ExampleMark24.txt and Open

19 Running Facets/MiniFac Click on OK Lots of output Output to ExOfEx-ExampleMark24.out Should/might open in a Notepad window If not, go to c:\facets and click on the.out file

20 The output Lots and lots, some only of technical interest (e.g. iterations) The first and most important part is the Yardstick (developed by Ben Wright) Put different measures onto a common scale

21 Vertical = (1N,2A,3A,SL) Y Yardstick (columns lines low high extreme)= 0,25,-1,1,End Measr +Candidate -Paper -Examiner Scale (23) 21 Strong Hard Hawk Q P H L 14 N Essay Polit I J Ancient MedMod F G * 0 * 4 8 * * K * 12 * B D E O 11 A C R Weak Easy Dove (1) Measr +Candidate -Paper -Examiner Scale =alpha D,K,O,P,Q marked Paper 1 (Ancient) = alpha/beta borderline (~ I-II.i) ~ II.i-II.ii borderline 6-7 = beta/gamma borderline (~II.ii-III) 1=delta I have edited the output from Facets to make it look nicer: - Courier Font - Line Spacing =.65 - Looks much clearer The Yardstick command sets the range and the number of steps

22 Estimates for individual candidates, sorted by average score An Examination of Examinations 24 point marking scheme 22/11/ :13:52 Table Candidate Measurement Report (arranged by 1mN). Total Total Obsvd Fair(M) + Model Infit Outfit Estim. Corr. Score Count Average Average Measure S.E. MnSq ZStd MnSq ZStd Discrm PtBis Num Candidate Mean (Count: 16) S.D. (Population) S.D. (Sample) Model, Populn: RMSE.05 Adj (True) S.D..23 Separation 4.37 Strata 6.16 Reliability.95 Model, Sample: RMSE.05 Adj (True) S.D..24 Separation 4.52 Strata 6.36 Reliability.95 Model, Fixed (all same) chi-square: d.f.: 15 significance (probability):.00 Model, Random (normal) chi-square: 14.1 d.f.: 14 significance (probability):.44

23 Estimates for individual candidates, sorted by candidate ID An Examination of Examinations 24 point marking scheme 22/11/ :13:52 Table Candidate Measurement Report (arranged by 1N) Total Total Obsvd Fair(M) + Model Infit Outfit Estim. Corr. Score Count Average Average Measure S.E. MnSq ZStd MnSq ZStd Discrm PtBis Num Candidate Mean (Count: 16) S.D. (Population) S.D. (Sample) Model, Populn: RMSE.05 Adj (True) S.D..23 Separation 4.37 Strata 6.16 Reliability.95 Model, Sample: RMSE.05 Adj (True) S.D..24 Separation 4.52 Strata 6.36 Reliability.95 Model, Fixed (all same) chi-square: d.f.: 15 significance (probability):.00 Model, Random (normal) chi-square: 14.1 d.f.: 14 significance (probability):.44

24 Estimates for Exam Type, sorted by average score n Examination of Examinations 24 point marking scheme 22/11/ :13:52 Table Paper Measurement Report (arranged by 2mN) Total Total Obsvd Fair(M) - Model Infit Outfit Estim. Corr. Score Count Average Average Measure S.E. MnSq ZStd MnSq ZStd Discrm PtBis N Paper Essay Polit MedMod Ancient Mean (Count: 4) S.D. (Population) S.D. (Sample) Model, Populn: RMSE.03 Adj (True) S.D..00 Separation.16 Strata.55 Reliability.03 Model, Sample: RMSE.03 Adj (True) S.D..02 Separation.61 Strata 1.14 Reliability.27 Model, Fixed (all same) chi-square: 4.2 d.f.: 3 significance (probability):.24 Model, Random (normal) chi-square: 1.7 d.f.: 2 significance (probability):.42

25 Estimates for individual examiners, sorted by average score An Examination of Examinations 24 point marking scheme 22/11/ :13:52 Table Examiner Measurement Report (arranged by 3mN) Total Total Obsvd Fair(M) - Model Infit Outfit Estim. Corr. Exact Agree. Score Count Average Average Measure S.E. MnSq ZStd MnSq ZStd Discrm PtBis Obs % Exp % Nu Examiner Q P H L N I J G F K O E D B A C R Mean (Count: 17) S.D. (Population S.D. (Sample) Model, Populn: RMSE.06 Adj (True) S.D..16 Separation 2.66 Strata 3.88 Reliability (not inter-rater).88 Model, Sample: RMSE.06 Adj (True) S.D..17 Separation 2.75 Strata 4.00 Reliability (not inter-rater).88 Model, Fixed (all same) chi-square: d.f.: 16 significance (probability):.00 Model, Random (normal) chi-square: 13.6 d.f.: 15 significance (probability):.55 Inter-Rater agreement opportunities: 2320 Exact agreements: 208 = 9.0% Expected: = 9.4%

26 Estimates for individual examiners, sorted by examiner ID An Examination of Examinations 24 point marking scheme 22/11/ :13:52 Table Examiner Measurement Report (arranged by 3N) Total Total Obsvd Fair(M) - Model Infit Outfit Estim. Corr. Exact Agree. Score Count Average Average Measure S.E. MnSq ZStd MnSq ZStd Discrm PtBis Obs % Exp % Nu Examiner A B C D E F G H I J K L N O P Q R Mean (Count: 17) S.D. (Population S.D. (Sample) Model, Populn: RMSE.06 Adj (True) S.D..16 Separation 2.66 Strata 3.88 Reliability (not inter-rater).88 Model, Sample: RMSE.06 Adj (True) S.D..17 Separation 2.75 Strata 4.00 Reliability (not inter-rater).88 Model, Fixed (all same) chi-square: d.f.: 16 significance (probability):.00 Model, Random (normal) chi-square: 13.6 d.f.: 15 significance (probability):.55 Inter-Rater agreement opportunities: 2320 Exact agreements: 208 = 9.0% Expected: = 9.4%

27 Estimates for scale values, 24 points An Examination of Examinations 24 point marking scheme 22/11/ :13:52 Table 8.1 Category Statistics. Model =?,?,?,R DATA QUALITY CONTROL RASCH-ANDRICH EXPECTATION MOST RASCH- Cat Category Counts Cum. Avge Exp. OUTFIT Thresholds Measure at PROBABLE THURSTONE PEAK Score Total Used % % Meas Meas MnSq Measure S.E. Category -0.5 from Thresholds Prob % 1% ( -2.71) low low 100% % 2% -.84* % % 4% % % 11% % % 15% % % 20% % % 24% -.30* % % 25% % % 33% % % 40% % % 51% -.08* % % 62% % % 76% % % 81% % % 87% % % 91% % % 95% % % 96% % % 97% % % 99%.22* % % 99% % % 100%.22* % % 100%.22* ( 3.02) % (Mean) (Modal)--(Median)

28 The Program: ExOfEx-ExampleMark7.txt A 24-point score is a bit unwieldy Replace it with a 7-point score (<4, 5-7, 8-10, etc) Re-run the program but using ExOfEx-ExampleMark7.txt

29 Vertical = (1N,2A,3A,SL) Y Vertical = (1N,2A,3A,SL) Yardstick (columns lines low high extreme)= 0,25,-1,1,End Yardstick (columns lines low high extreme)= 0,10,-2.5,2.5,En Measr +Candidate -Paper -Examiner Scale Measr +Candidate -Paper -Examiner Scale (23) (7) Q Q P P L H 2 6 H L 14 2 N I J N 10 Essay Polit I J Essay Polit G 1 3 Ancient MedMod F G 1 3 Ancient MedMod F K * 0 * 4 8 * * K * 12 * * 0 * 4 * * * 4 * D E 12 B D E O 11 A B C O A C R 10 R (1) (1) Measr +Candidate -Paper -Examiner Scale Measr +Candidate -Paper -Examiner Scale

30 Graphs

31

32 <

33

34 <

35