ASSOCIATION OF TEST PUBLISHERS. Division/Committee/SlG: Health Sector SIG (New) Division/Committee/SlG Goals & Activities

Transcription

1 ATP Division/Committee Report ASSOCIATION OF TEST PUBLISHERS D IVI S I O N/GOM M ITTE E/S IG REPORT TO THE BOARD 23 Mrch 2016 Orlnd, Florid Division/Committee/SlG: Helth Sector SIG (New) Division/Committee/SlG Gols & Activities ATP's mission is to promote nd preserue the generl welfre of testing nd its vlue to society; to orgnize test publishers into permnent body to foster nd mintin collegil reltions mong themselves nd with other professionl nd business groups; to encourge high level of professionlism nd business efhics; to se ve s the principl orgniztion tht monitors nd responds to regultory nd legl rulings; nd to increse the strength nd cohesiveness of the test publisher community. Plese report on your gols/ctivities which support this mission: The objectives of the Helth Sector SIG include bringing together the helth cre sector with the testing industry in order to dvnce ssessment nd testing in helth cre through innovtion, best prctice, networking, nd the exchnge of informtion nd expertise. Specificlly, within the context of the ATP progrm, the SIG will fcilitte dilogue on key issues nd chllenges in testing for eduction, licensure, nd workforce recruitment in helthcre. The Helth Sector SIG hs identified number of strtegies to chieve its purposes, including:. serv ng s voice for helth-relted issues in testing nd ssessment;. encourging the exchnge of informtion nd expertise relevnt to the helth sector by encourging networking t ATP conferences nd events; o mking presenttions nd contributions to ATP progrms; nd o fcilitting the prticiption of individuls nd orgniztions within helth-relted industries who re not currently members of the ATP. Updtes since the lst report: The Helth Sector SIG is new inititive tht ws identified following nd initil meeting held in conjunction with the 2015 lnnovtions Conference. The new Helth Sector SIG will be lunched t designted evening event on22 Mrch 2016 tthe 2016 ATP lnnovtions Conference. At this event, interested persons will be invited to join the SIG in order to identify specific work groups nd projects. This step will lso identify future topics nd sessions for subsequent conferences. lnsert Nme, Dte Pge 1 of 4

2 ATP Division/Committee Report Other news of interest to the Bord, prticulrly threts to your Sector of the lndustry: 1. Ledership Group A ledership group hs been estblished to initite the new SIG nd prepre the ground work for this new inititive. The members of the ledership group re: o ln Frnk AM, Chief Executive Officer, Austrlin Medicl Council. Dvid Wldschmidt, Americn Dentl Assocition, Director of Testing Services nd Secretry, Joint Commission on Ntionl Dentl Exmintions. Ad Woo, Director of Mesurement nd Testing, Ntionl Council of Stte Bords of Nursing 2. Survey of ATP Prticipnts Prticipnts t the 2015 ATP lnnovtions Conference, who hd identified themselves s involved in "helth" ssessments, were surveyed ln reltion to the proposed Helth Sector SlG. The survey estblished tht (s percentge of respondents): o The disciplines represented t the 2015 ATP lnnovtions bsed on the survey responses were: Medicine 7Oo/ol Dentistry 25o/olAllied Helth 20%/ Nursing 10o/ol Phrmcy 5% /Physiotherpy 5% /Helth Service Delivery 5%; Reson for ttending the ATP Conference: lnnovtion 86.36%/ Networking 59%/ Secu rity o/o I S howcsi ng Devel o pments % ; Assessment Activity: Test developer/provider 73o/ol Test Administrtor 18% Acdemic/consultnt 13%/ Testing Client 9%; nd lnvolvement with ATP: Member 31.82o/o / lndividul Subscriber 4.55o/ol Lpsed Member 4.55o/ol Non-member 59.09%. 3. Ghrter The ledership group hs developed drft Chrter for the new Helth Sector SlG. This Chrter will provide the bsis for efforts by the SIG to ttrct further interest nd prticiption t the 2016 ATP lnnovtions Conference. The drft Chrter is ppended to this report. 4. ATP Conference Guide A one-pge summry of the Heth SIG hs been prepred to lert presenters t the 2016 lnnovtions Conference bout this ctivity. By shring this informtion, the SIG ledership hopes to encourge presenters to focus their sessions on helth ssessment issues where pproprite. SIG ledership will be reviewing the entire progrm of ech conference to identify "Helth" trck. The drft Conference Guide is ppended to this report. 5. lnitil Project To lunch the new SlG, n initil project is being developed to produce best prctice guide/mnul on the production of high qulity Multiple-Choice Questions (MCQ) items lnsert Nme, Dte Pge 2 of 4

3 ATP Division/Committee Report for testing in helthcre. This purpose of the MCQ Workshop Mnul is to synthesize the technicl mteril tht is now vilble in number of sources into concise "hndbook" to guide item developers who my be new to constructing MCQ questions for high stkes exmintions in the helth sector. lt will lso provide dvice on ll spects of MGQ development from item writing to psychometric nlysis. The MCQ Workshop Mnul proposl will be presented t the 2016 ATP lnnovtions lunch of the Helth Sector SIG on 22Mrch. The grphics tht will be used to illustrte the project re ppended to this report. 6. Potentil Membership of the Helth Sector SIG Through personl contcts, the ledership group hs reched out to individuls who my be interested in joining the new Helth Sector SlG. Some thirteen individuls, representing cross section of professionl, eductionl nd regultory bodies within the helth sector who will be ttending the 2016 ATP lnnovtions conference hve registered their interest in prticipting in the new SlG. ln ddition further seven individuls who re not ttending the 2016 ATP lnnovtions conference, but could be considered s future prticipnts, hve lso registered their interest. lnformtion bout the SIG lunch nd relted developments will be provided to these individuls. Divis ion/gomm ittee Officers As new SIG the officers of the new SIG will be confirmed fter the lunch t the ATP lnnovtions Conference. Divis ion/gommittee Opertions The Committee opertions will be confirmed fter the lunch of the new SIG Active Members nd Volunteers lnclude tble listing ll ctive Division/Committee members nd volunteers nd their compny ffilition ln Frnk Nme Dvid Wldschmidt Ad Woo Crl Mtheson Gompny Affilition Austrlin Medicl Council Americn Dentl Assocition Ntionl Council of Stte Bords of Nursing Austrlin Medicl Council Emil inf@mc.org.u wldschmidtd@d.org woo@ncsbn.org crl. mtheson@mc. org. u Susn Buick Austrlin Medicl Council susnb@mc.org.u O:\ATP\2016 USA DOCUMENTS\Division CommitteeReporting 2016 Mrch.docx lnsert Nme, Dte Pge 3 of 4

4 Drft Chrter ATP Assocition of Test Publishe15 ','1,'.-,',.-.'.''l-'1, f' Helth Sector Specil lnterest Group Associtíon of Test Publishers Mission The ATP Helth Sector Specil lnterest Group ("SlG") promotes nd dvnces best prctices nd professionl principles tht guide nd inform testing nd ssessment of helth cre workers nd professionls, s well s ptients, by providing n engging community forum tht fcilittes collbortion, communiction, nd innovtion. 1. Bckground Helth cre is mjor concern for ll developed nd developing societies. Blncing society's needs with respect to its helth cre workforce - given limited vilbility of resources nd high expecttions for ptient outcomes nd sfety - represents mjor chllenge. For mny yers testing hs plyed n integrl role in helth sector trining, licensure nd prctice, not to mention the role of ssessment in dignosing nd treting ptients. This includes the following: o Evluting eductionl progress s helth cre workers cquire the necessry skills; o Certifyingnd/or credent l ng completion of forml eduction or specific units of trining; o Determining miniml competency for licensure purposes; o Screening individuls trined outside prticulr eduction or regultory system; o Recruiting nd employing workers within the helth sector; Monitoring the performnce of licensed prctitioners to identify sub-optiml performnce for ptient sfety purposes; nd o Assessin g nd/or screening individuls/ptients for specific disorders/chrcteristics, Significnt dvnces in testing theory nd prctice hve occurred within the helth industries, with copious volumes published in the scientific literture. Concurrently, dvnces in informtion nd communictions tdchnology hve led to rpid growth in the cpcity of the test ng industry to deliver high qulity nd resource efficient ssessments. Despite these significnt developments, much work reminsto dvnce ssessment in helth-relted settings in recognition of the unique needs of this sector, nd the implictions for public helth nd sfety. 2. Purpose The purpose of the SIG is to promote prcticlsolutions nd mechnisms tht help ddress the issuesndchllengessurroundingtheuseoftestingwithinthehelthsector. TheSlGctively explores venues trgeting the prticulr needs of the helth professions, show csing innovtions nd developments in testing through networking opportunities nd the free exchnge of informtion. The SIG opertes within the frmework of the ATP nd its conferences. lt is not intended to replce existing ATP Divisions (i.e., Eduction, Certifiction/Licensure, Workforce Skills Credentiling, lndustril/orgniztionl, nd Clinicl), but rther to complement them by providing n industry specific lens through which to view testing inititives in helth-relted situtions.

5 ln ccordnce with its purpose, the SIG Drws together individuls nd orgniztions from both testing nd the helth sector professions, to promote constructive dilogue concerning key issues nd chllenges; Fcilittes the development of best prctices nd professionl principles tht guide nd inform testing nd ssessment in the helth sector; Advnces the current understnding of helth cre relted chllenges in testing for eduction, trining, licensure, nd workforce/recruitment; Encourges networking, shred development nd collbortion with the broder testing community, including those involved in scientific reserch, test design, test delivery, psychometrics, nd test security; Serves s voice regrding helth cre relted issues in testing, contributing the SIG's unique perspective within ATP through conference contributions nd direct prticiption; Fcilittes the prtic ption nd enggement of individuls nd orgniztions within the helth sector who ttend ATP conferences but who re not currently members of the ATP 3. Structure nd Governnce The SIG opertes underthe uspices of the North Americn Assocition of Test Publishers, in lignment with the gols nd objectives of the ATP. The SIG provides strtegic updtes on its progress to the ATP Bord of Directors t regulr intervls. The SIG chrter my be mended by mjority vote of voting members of the SlG, subject to pprovl of the ATP Bord. Officer positions for the SIG include the following: Chir, Leds nd oversees meetings, opertions, nd generl ctivities of the SlG, in ccordnce with ATP Bylws nd the mission of the SIG; the Chir delegtes ctivities s pproprite to best ccomplish the objectives of the SlG, nd hs uthority to form d hoc committees nd workgroups to meet the needs of the SIG; Vice Chir. Acts under the direction of the Chir, pursuing ctivities delegted by the Chir; nd Secretry. Records nd mintins minutes of meetings nd other records, nd mnges communictions for the SlG. SIG officers re vriously elected for one, two, or three one- yer terms, duríng which they rotte sequentilly through the bove positions s follows: Secretry, Vice Chir, Chir. The Chir my serve further yer s Pst Chir. The Secretry will be elected on n nnul bsis by voting members of the SlG, nd will serve three yers, one in ech officer position. The SIG meets t lest on n nnul bsis t conference of the ATP. Non-ATP members my prticipte in the Helth Sector SlG, but officers must hold minimum of Subscriber membershíp in the ATP. Drft: Ver:sion 3 September O:WTP\ATP Helth SIG\DOCUMENTS\ATP Helth Sector Specil lnierest Group Preliminry Updte Version 3 Drft.docx

6 Possible Helth SIG sessions of interest Mondy, Mrch 21, :30 m - 11:30 m FEATURED SPEAKER SESSION: Which Test is Best? Exploring New Wys to Assess Conti 10:30 m - 11:30 m Nviqtino the Politics of Privcv in Testino 10:30 m - 11:30 m Technology-Enhnced ltems: When to Use, How to Develop, How to Evlute 10:30 m - 11:30 m Automted Scoring: An lntroduction for Certifiction nd Licensure Testing Professionls 11:45 m - 12:45 om Adptinq to Chnqe: lmplementinq nd Mintining n Adptive Testing Progrm 11:45 m - 12:45 pm Do Filing Cndidtes Rely on Subscore lnformtion? Evluting Widely-Held Assumption 11:45 m - 12:45 pm Dumping the Dopes Who Use Brindump Sites: How IBM Turned the Tbles Using Dt Forensics 11'.45 m - 12:45 om FEATURED SPEAKER SESSION: Evidence-Bsed Assessment 11:45 m - 12:45 pm Remote Scoring vs Remote Proctoring: An Alterntive Approch to Dispersed Testing for Hioh Stkes Assessment 11:45 m - 12:45 pm The Drg nd Plce ltem Tvpe nd lts Mny Uses 11:45 m - 12:45 pm Gooqle Glss, Bluetooth nd Spy Glsses...Oh My!!!!! 11:45 m - 12:45 om FISHBOWL SESSION: Blst from the Pst or Meetinq the Future? 2:00 pm - 3:00 pm Universl Design, Accessibility nd Firness: lmplictions for Test nd ltem Develooment 2:00 pm - 3:00 pm FISHBOWL SESSION: Finding the Gp: How to Dignose nd Address Hrd to Write Blueorint Ares 4:00 pm - 5:00 pm FISHBOWL SESSION; Herding Cts - How to Keep Trck Of Your Test Security lncidents nd Report Your Results 4:00 om - 5:00 om The Movinq Troet: Mintininq Exm Vlidity in Rpidlv Chnqinq Disciplines 4:00 pm - 5:00 pm Virtul ltem-writer Trining: The Role of Assessment 5:00 pm - 7:00 pm eposter SESSION: A DIF Anlysis Using Tsk Anlysis Dt to Differentite Similr Job Roles 5:00 pm - 7:00 pm eposter SESSION: lmproving the User Experience: Testing the Usbility of Widescreen lnterfce nd Decision Support Tool in Medicl Certifiction Exms 5:00 pm - 7:00 pm eposter SESSION: Pulse of Securitv in Testinq: ATP Survev of Securitv Prctices 5:00 pm - 7:00 pm eposter SESSION: The lmpct of Using Automted Scoring of Written Responses for Multilinqul High-Stkes Medicl Licensing Exmintion 5:00 pm - 7:00 pm eposter SESSION: Web-bsed Formtive Assessment: An lnnovtive Wy to Assess nd lmprove Ptient-Physicin Communiction

7 Mrch :30 m - 10:00 m WORKSHOP: How to ldentifu, Develop nd lmplement lnnovtive ltems 8:30 m - 10:00 m WORKSHOP: Stndrd Setting Essentils nd n Evidence-Bsed Approch for Determininq if it's Time for Reset 8:30 m - 10:00 m WORKSHOP: Competency Modeling nd Job Anlysis for Certifiction nd Licensure Tests 8:30 m - 10:00 m WORKSHOP: Rel-World High-Stkes Test Security lnvestigtions 8:30 m - 10:00 m WORKSHOP: The lnnovtion Chllense 10:30 m - 12:00 pm WORKSH Msterclss in C Technol Enhnced ltems 10:30 m - 12:00 pm WORKSHOP: Accommodting Test Tkers: Hot Developments, Cse Studies, nd Prcticl T 10:30 m - 12:00 pm WORKSHOP: Essentil Policies nd Prctices for Test Score Vlidtion 10:30 m - 12:00 pm WORKSHOP: The Medi is Knock, Knock, Knockinq t Your Door...Now \Nht??? 1:15 om -2'.15om Pssinq Stndrds for Multilinqul nd lnterntionl Exmintion Progrms 1:15 pm - 2:15 pm PLATINUM SPONSOR SESSION (Alpine Testing Services): Building High- Performing Test Development-to-Delivery Ecosystem: Mximizing Technology nd lntegrtion Benefits 1:15 pm - 2:15 pm Tblet-Bsed Administrtion of the Brief Assessment of Cognition: Adpttion of I Pen nd P Assessment 2:30 pm - 3:15 pm IGNITE SESSION: Structured Blueprint Review: Setting the Foundtion for Test Devel 4:15 pm - 5:15 pm Blncing Act: Mintining Rigorous Security While Providing Positive Cndidte 4:15 pm - 5:15 pm Does Your Exm Accommodte Accommodtions? 6 m Tuesd Mrch 20'16 - reserved for ATP Helth SIG lunch Mrch 23 20'16 8:45 m - 9:45 m Development nd Vlidtion of Computerized Virtul Relity-Bsed Assessment of Functionl Skills 8:45 m - 9:45 m Assessinq Your Risk:Why Security is Not One Size Fits All 8:45 m - 9:45 m Development nd Vlidtion Computerized Virtul Relity-Bsed Assessment of Functionl Skills 10:00 m - 11:00 m The the Bd nd the To-Be-Determined of Credentils 10:00 m - 1 1:00 m Two Tips for Helthy Hert: Or How to ldentify Problem Ares nd Mintin Helthy ltem Bnk. ATP Helth Sector Specil lnterest Group (SlG) The ATP, following n initil positive response t the 2015 lnnovtions Conference, hs decided to estblish Specil lnterest Group for the Helth Sector. The im of the new SIG will be to fcilitte dilogue nd focus on key issues in testing for eduction, licensure nd workforce recruitment in the helth sector within the context oftheatplnnovtionsprogrm. AfunctiontolunchthenewSlGwill beheldtthe2016 lnnovtions Conference in Orlndo, Florid from 6 to 7 pm on Tuesdy 22Mrch20t6. As prt of this inititive, the entire progrm of the 2016 lnnovtions Conference hs been viewed in order to identify presenttions tht my be of interest to those working in the helth sector nd cover rnge of pplictions of interest to those strting out in testing through to the more dvnced. The listing is ttched s guide to the ATP Progrm. Version 2 - s t 3'd Mrch 2016 O:\ATP\2016 USA DOCUMENTS\Finl - Helth SIG suggested sessions USA 2016.docx

8

9 A BEST PRACTICE GUIDE, MCO TESTING FOR THE HEALTH

10 THIS MANUAL IS INTENDED TO PROVIDE A BEST PRACTICE GUIDE FOR THOSE INVOLVED IN THE DEVELOPMENT AND ADMINISTRATION OF HIGH STAKES MCQ EXAMINATIONS IN THE HEALTH SECTOR. THE MANUAL WILL BE A USEFUL TOOL IN ADVANCING KNOWLEDGE AND PRACTICE OF TESTING IN THE HEALTH SECTOR. o

11

12 PLACE OF MCQ TESTS IN HEALTH SETTING PA55 STANDARDS l CHOOSING AN ITEM FORMAT TRAINING ITEM WRITERS BLUEPRINTING COMMON ITEM FLAWS o

13 COMMON ITEM FLAWS lnbuilt clues nd cueing Content i rrelevnt v rince Item not trgeted/missing the point Reverting to multiple tru elllse formt Fctoid Flws in eductionl design o

14

15 STATISTICAL ANALYSIS Descri pt ive sttistics ClssicI test theory Item response theory Mking sense of, nd synthesising, qutity ssurnce outcomes Relibitity nd vlidity lnterprettion nd use of item performnce dt Clssicl test theory nlysis Item response theory nlysis Exmples of interprettion nd use of item dt D I t items lllustrtive exmple - item tht ws ltered ech step of the wy o