TIPS ON BUILDING AND USING COMPETENCY ASSESSMENTS

Transcription

1

2 TIPS ON BUILDING AND USING COMPETENCY ASSESSMENTS 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 3 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

3 1 Written Examinations Types of Assessment Tools / Strategies 5 2 Reflective, Self Analysis Structured Interviews 6 Collection of Evidence Observation, Demonstrations, Simulations Credential Evaluation Portfolio 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 6 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

4 Types of Assessment Tools Written Examinations DESCRIPTION Individuals respond to a series of specific questions, often in the form of essays, short answer or multiple-choice test items. They are most often used to assess knowledge about a specific subject. THE MANUAL EXAMPLES ON HOW IT IS USED Various formats, e.g. Multiple-choice exams, abstract, articles, critical incidents reports, dissertations, essays Summative, high stakes, e.g. placement tests, certification exams Formative, e.g. test of learning offered in a course 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 7 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

5 Types of Assessment Tools Structured Interviews DESCRIPTION Interviews allow individuals to describe in detail what they know and can do, how they have solved problems, and responded to contingencies, etc. THE MANUAL EXAMPLES ON HOW IT IS USED Various formats, e.g. job or placement interviews, component of simulation or demonstration Summative, high stakes, e.g. element of certification or licensure exam Formative, e.g. preparing for job interviews 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 8 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

6 Observation, Demonstrations, Simulations Types of Assessment Tools Demonstrations, Simulations, DESCRIPTION Observations Individuals are assessed against the assessment criteria as they perform the task. They are observed in either a natural setting or in a simulated situation. THE MANUAL EXAMPLES ON HOW IT IS USED Various formats, e.g. Objective Structured Clinical Evaluation (OSCE), performance evaluations Summative, high stakes, e.g. element of certification or licensure exam Formative, e.g. practicing skills in a learning setting 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 9 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

7 Types of Assessment Tools DESCRIPTION Portfolio A portfolio is generally a more formal and organized collection of evidence developed by an individual. It may be paper based or digital. THE MANUAL Evidence of achievement of learning outcomes/occupational standards and experience EXAMPLES ON HOW IT IS USED To prepare for, and use in various opportunities, e.g. entrance requirements, academic credit, job interviews 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 10 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

8 Types of Assessment Tools Reflection, Self-Analysis, Self-Assessment DESCRIPTION Self-assessments allow an individual to match specific knowledge, skills, and abilities against established criteria. THE MANUAL EXAMPLES ON HOW IT IS USED Various formats, e.g. training and development needs assessment, interest inventory Formative, e.g. part of end-of course study/preparation for exam Diagnostic, e.g. towards the development of a learning plan 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 11 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

9 Types of Assessment Tools Collection of Evidence DESCRIPTION A collection of proof of learning. It will demonstrate an individual s knowledge, skills and abilities. THE MANUAL EXAMPLES ON HOW IT IS USED Various formats, e.g. log books, notations, product samples, skills passport 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 12 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

10 Types of Assessment Tools Credential or Qualification Evaluation DESCRIPTION An individual s education and training credentials, or professional qualifications are evaluated to determine authenticity and relevance to Canadian standards. This process compares the credential documents to Canadian standards for a course, program, professional license or employment expectations. THE MANUAL EXAMPLES ON HOW IT IS USED For example, a transcript that articulates credential equivalency for the specific application 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 13 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

11 COST / RESOURCE DEMANDS RIGOUR Plot of General Benefits, Limitations 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 14 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

12 Types of Assessment Tools/Strategies Further Examples of Assessment Tools Abstract Annotated bibliographies Assessment stations Journal articles Case studies Concept maps Critical incident accounts Debates, oral arguments, mock court Dissertation Essay Field report In-basket/tray exercises Lab report Learning logs/records Make or design something Multiple-choice questions Observation Online discussion boards Oral presentations Problem sets Portfolios Reflective diaries Research projects/group projects Role play Simulations Wiki / blogs 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 15 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

13 Diagnostic, Learning, Professional Development Self Assessment LOW STAKES Who is Doing the Assessing? (This linked to the purpose/objectives of the assessment) Diagnostic, Learning, PD, Performance Evaluation Peer Assessment Measure of achievement/ learning, compliance External, Third-Party Assessment Complexity, risk, consequence requires multiple evidence Multiple Forms of Assessment HIGH STAKES 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 16 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

14 What form of Assessment is the Right One? 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 17 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

15 Creating an Assessment Strategy Due Diligence 2017 Tourism HR Canada, All Rights Reserved STEC JUNE 2017 EDUCATORS FORUM Slide 19 of 84 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 18 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

16 Typical Assessment Plan Testing Specifications / Testing Blueprint Introduction, Statement of Purpose, Reason Pre-requisites, Required Conditions Assessment Process Structural Variables Content Weighting and Coverage Bloom s Taxonomy Situational and Contextual Variables Standard Setting and Scoring, Data Analysis Procedures Suggested Renewal/Maintenance Requirements Who Why What Where When How 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 19 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

17 The context, e.g.: Typical Testing Blueprint Content Purpose, e.g. formative, summative, instructional, guidance Intended uses and outcomes The assessment criteria that informs the assessment Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Structural Variables Compliance with regulations Content Weighting and Coverage High stakes: failure to perform Bloom s Taxonomy can lead to fatality Situational and Contextual Assessment Variables of the worker: the specific function they undertake Standard Setting and Scoring, as Data part Analysis of the supply Procedures chain Suggested Renewal/Maintenance Based Requirements on competencies that support compliance requirements 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 20 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

18 Typical Testing Blueprint Content Statement on minimum requirements of: The candidate (i.e. what qualifies them to be eligible for assessment or designation) The process, e.g. what must be completed or supplied in order to be eligible Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Structural Variables Content Weighting and Coverage Experience (type, quantity)? Bloom s Taxonomy Situational and Contextual Variables Education or training? Language proficiency? Standard Setting and Scoring, Data Analysis Procedures Suggested Renewal/Maintenance Requirements 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 21 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

19 Typical Testing Blueprint Content Selected methods and rationale, e.g. Type and length of exam Time to complete Administrative conditions Method of delivery Scoring procedures Assessment aids Order of operation Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Think about the Structural Variables conditions and constraints that inform the process Content Weighting and Coverage Employer is doing the Bloom s Taxonomy assessment Situational and Contextual Variables Assessing current employees Standard Setting and Scoring, on Data select Analysis functions/skills, Procedures unique to each worker Suggested Renewal/Maintenance Requirements Done while on the job in a normal work context 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 22 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

20 Typical Testing Blueprint Content Presentation/format of the assessment Length of the assessment Cognitive domains addressed Range and conditions, where applicable, e.g.: time constraints, location Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Structural Variables Content Weighting and Coverage Bloom s Taxonomy Situational and Contextual Variables Standard Setting and Scoring, Data Analysis Procedures Suggested Renewal/Maintenance Requirements 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 23 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

21 Explicit summary of content areas (from assessment criteria), showing the relative importance and weighting of content Typical Testing Blueprint Content Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Structural Variables Content Weighting and Coverage Bloom s Taxonomy Situational and Contextual Variables Standard Setting and Scoring, Data Analysis Procedures Suggested Renewal/Maintenance Requirements 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 24 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

22 Typical Testing Blueprint Content Statement on the cognitive abilities represented by the assessment criteria and how this is concordant to the assessment strategy Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Structural Variables Content Weighting and Coverage Bloom s Taxonomy Situational and Contextual Variables Standard Setting and Scoring, Data Analysis Procedures Suggested Renewal/Maintenance Requirements 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 25 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

23 Typical Testing Blueprint Content Typical job incumbent profile Job context and setting Other profession-related contexts Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Factors to consider: Structural Variables Worker profile, e.g. Content Weighting and Coverage Type of experience Level of education Bloom s Taxonomy Language proficiency Situational and Contextual Cultural Variables factors Job context, e.g. Standard Setting and Scoring, Environment Data Analysis Procedures Pace Suggested Renewal/Maintenance Requirements Level of supervision Required tools/aids used 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 26 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

24 Typical Testing Blueprint Content Scoring rules and method, and the methodology/ies used to establish protocol Where applicable, decision making rules for: handling multiple source of evidence determining authenticity, currency, sufficiency of evidence Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Structural Variables Content Weighting and Coverage Bloom s Taxonomy Situational and Contextual Variables Standard Setting and Scoring, Data Analysis Procedures Suggested Renewal/Maintenance Requirements 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 27 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

25 Typical Testing Blueprint Content Recommendations for maintenance of the credential, where applicable Description of recommended administrative practices to monitor assessment results and to make decisions on changes, e.g. address errata, make updates Testing Specifications Introduction, Statement of Purpose Pre-requisites, Required Conditions Assessment Process Structural Variables Content Weighting and Coverage Bloom s Taxonomy Situational and Contextual Variables Standard Setting and Scoring, Data Analysis Procedures Suggested Renewal/Maintenance Requirements 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 28 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

26 National Council of State Boards of Nursing, Institute for Credentialing Excellence Administratively feasible For example: centralized approach For example: supported by current LMS and staff resources Publicly credible Viewed as important to the public: should be up-to-date Increased trust in services rendered Professionally acceptable Provides competitive advantage (for employment, contracts) Meaningful approach and appropriate effort: value proposition Legally defensible Conforms to psychometric requirements, where necessary Lawful Economically affordable Cost effective Priced at a level the market can bear; sees as value, fair 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 29 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

27 Steps for Development of Effective Assessments Planning DETERMINE AND PREPARE ASSESSMENT STRATEGY AND SPECIFICATIONS Development ADAPT OR DEVELOP AND VALIDATE ASSESSMENT CONTENT, INSTRUMENTS AND SUPPORTING TOOLS Assess PREPARE AND CONDUCT ASSESSMENT; GATHER FURTHER EVIDENCE, AS NEEDED; INTERPRET RESULTS AND PREPARE REPORT Administration Candidate Support FEEDBACK TO CANDIDATE GUIDANCE, SUPPORT * This is an example of the phases, e.g. not all applies where standardized testing is in place Monitoring Results, Quality Assurance REVIEW OF TEST RESULTS, MISUSE ISSUES, SECURITY OF ASSESSMENTS, ENACTING REVISIONS 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 30 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

28 Putting it to the Test Using an Assessment Tool 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 31 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

29 1 Answer the six questions 5 MINUTES 2 Debrief as a Group A FEW TAKE-AWAYS Mini Multiple-Choice Exam 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 32 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

30 Professionalism, Interpersonal Skills Communication skills, dealing with difficult situations Plan assessment activities Develop / prepare assessment strategies, tools Role of Assessor Assess competence On-going professional development Knowledge Assessment tools, system, profession/domain Follow up from assessment process Technical, Legal and Ethical Responsibilities Contribute to quality assurance, review 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 33 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

31 OVERVIEW Rater/Assessor Bias Halo Effect, Halo Error Central Tendency Leniency Bias Strictness Bias Cross-Cultural Biases Personal Prejudice Recency Effect/Error Impaired Judgment Self-Rater Bias Time Delay, Waning Memory Structural / Systemic Bias Calibration Bias? A+ 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 34 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

32 DEFINITIONS ACCOMMODATION Adaptations to test format or administration that maintain the same construct and produce results that are comparable to those obtained by individuals who do not use accommodations MODIFICATION Adaptations to test format or administration that change the construct being measured in order to make it accessible for designated students while retaining as much of the original construct as possible 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 35 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

33 Guidance Accommodation Examples Scribe Assistance Technology Adapted Language Spoken Examination Administration Use of Interpreter Adjusted Physical Environment Recorded Responses Allotment of More Time 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 36 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

34 Safeguards Against Inappropriate Use Assessment Instruments, Results Explicit, published statement on what the assessment is for and where it should not be used Published results should be descriptive to help interpret meaning and application Ongoing evidence the assessment is performing for the intended purpose and audience Follow consistent and prescribed administrative protocol (no exceptions) Corroborating evidence when assessment is used for special purposes Educate/support users of the results, e.g. information, tools, guidance 5 SEP 2018 CHICAGO LABELMASTER DANGEROUS GOODS SYMPOSIUM SLIDE 37 OF 39 ˑ 2018 TOURISM HR CANADA ALL RIGHTS RESERVED

35 LEAD PRESENTER/FACILITATOR Philip Mondor, President Président Tourism HR Canada RH Tourisme Canada And Emerit Consulting 2018, Tourism HR Canada, All Rights Reserved 4 71 rue Bank Street Ottawa ON Canada K1P 5N2 pmondor@tourismhr.ca pmondor@emeritconsulting.ca Tourism HR Canada works on behalf of 1.8 million workers and 200,000 businesses across Canada, serving one of Canada s most prosperous and growing industries. Its main goals include improving the quality and mobility of the tourism workforce, and undertaking workforce development initiatives that help individual businesses and regions maintain the skilled workforce they need to prosper. Emerit Consulting is the business division of Tourism HR Canada that specializes in labour market projects. With over 25 years of experience, Emerit is a world leader in the development of competency-based training, people assessment (psychometric services), competency/ standards-setting, voluntary certification and designation programs, accreditation programs, and specialized labour market research. Emerit Consulting has a long history in working with Essential Employability Skills, Language Benchmarks, Occupational Standards, and other job analyses models, including the development of worldrecognized methodologies and managing the application or use of these products in a wide range of labour market products. The consultancy team has a strong understanding of a wide number of sectors and industry groups, with projects that span 18 sectors. Emerit Consulting has had the privilege of working for clients in 15 countries (and growing).