Metrics That Matter Evaluation Design Best Practices

Transcription

1 Metrics That Matter Evaluation Design Best Practices Considerations for Establishing a Standard Evaluation Approach

2 Contents Why Standards?... 1 Streamline Administration... 1 Consistent, Reliable Reporting... 1 Internal and External Benchmarks... 1 Why should your organization consider KnowledgeAdvisors Standard Forms?... 2 Balanced Content, Based on Industry Standard Methodologies... 2 Ease of Implementation... 3 What types of Evaluations are available in Metrics That Matter?... 3 Post Event Evaluation... 3 Follow Up Evaluation... 4 Manager Evaluation... 4 Instructor Evaluation... 4 Customization Guidance... 5 Updates to Question Text... 5 Additional Custom Questions... 5 Course-specific Custom Questions... 5 Course-specific Evaluation Forms... 5 Additional Considerations... 6 Appendix: Standard Form Walkthroughs... 7 Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - i

3 Introduction This document includes best practices and recommendations for getting started with a standard evaluation approach for your organization. Recommended measures, content, and tips for customization are discussed, as well as considerations for evaluation scale, scale headers, and timing of evaluation deployment. Why Standards? There are many benefits to maintaining a set of standard evaluation questions for use across all training delivered by your organization. Streamline Administration From a survey assignment and delivery standpoint, having a very small number of forms to manage for class assignment streamlines the administration process. Metrics That Matter is like a giant database, so many different classes can leverage the exact same survey form and the data will be easily reportable based on the unique class information; it is not necessary to create a new survey for each class. Your KA account team will work with you during the implementation process to establish a set of default evaluations that are assigned automatically whenever a class is set up to be evaluated via Metrics That Matter, and will provide training to designated individuals to enable ongoing self-sufficient maintenance. Consistent, Reliable Reporting Having at least some degree of standardization across all training events will allow you to leverage MTM s reporting module to easily trend and compare data, making it easy to see beyond individual class scores and manage by exception. Standard performance goals may be set and monitored to further define and reinforce expectations for instructors and courseware developers. Many organizations find that they can expand their measurement strategy to include all training instead of focusing on selected courses or programs, which provides an opportunity to manage the L&D operation more effectively than ever before. Having comparable data is a critical step toward running training like a business in your organization. Internal and External Benchmarks Collecting comparable data across the L&D function is a critical first step to allow for internal comparison. When your organization leverages KnowledgeAdvisors standard questions and question categories, you can also compare your organization s effectiveness against a variety of benchmarks down to the individual question level. The vast majority of organizations who contribute to our benchmarks leverage our standard evaluations with minimal modification, which makes our benchmarks very robust, valid, and reliable. Having an external measure against which to compare your organization s performance helps to provide additional context when determining where to invest, improve, or cut resources within the L&D operation. Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - 1

4 Why should your organization consider KnowledgeAdvisors Standard Forms? Balanced Content, Based on Industry Standard Methodologies The instruments were created in partnership with world-class corporate universities and reflect the methodologies of Kirkpatrick, Phillips, Brinkerhoff, and Bersin. The instruments have been statistically validated and extensive research has been conducted to understand linkage within the instruments and to additional methods of measuring the impact of learning on the job (see recommended white papers listed below) The instruments are designed to be more than a smile sheet ; you can predict impact, linkage to results, and financial and non-financial ROI immediately after students attend class. There are instruments designed for use with specific learning methodologies, such as Instructor Led, Self-Paced Web-Based, and Virtual/Online Facilitated training, in addition to standard instruments designed to allow for the collection of more or less granular information based on the depth of measurement needed for a specific course or program. The instruments are designed around core constructs (question categories) that link to industryaccepted learning measurement methodology. The instruments have been used on an international scale, across a wide variety of industries, and training types. There is a clear linkage between the Post Event (end of class survey), Follow Up (on the job, months later) survey, Manager (third party observation on the job survey). In the reporting system each of these may be compared side-by-side based on a set of key performance indicators computed from the instruments. For additional detail and research related to the validity and reliability of KnowledgeAdvisors Standard evaluation forms or information regarding the reliability and accuracy of self-reported student evaluation data, you may download and review one or more of white papers listed below from our public website, Metrics That Matter: Aligning with Training Evaluation Models (2012) o Describes how MTM and our standard evaluations align with industry-accepted training evaluation models. The Predictive Learning Impact Model (2009) o Demonstrates the statistical relationship of key measures on KA standard evaluations to one another. Highlights which measures on the evaluation serve as critical predictive indicators of impact and application of training on the job. Validating the Human Capital ROI Scorecard (2005) o Research study comparing impact results using 3 data sources: self-reported student evaluation data collected electronically, evaluation data collected electronically from Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - 2

5 managers, and face to face interview data from managers. Validates that studentreported impact data can be just as reliable as more expensive face to face manager interviews in reporting the impact training has had on the job. Ease of Implementation Organizations can spend months developing evaluation content in-house, gaining consensus and revising questions, only to find that they are still not getting reliable, actionable data that promotes effective decision making. By leveraging our standards, you can get started quickly and have confidence that your organization is collecting a balanced set of measures based on industry-accepted methodologies that will help you take action not just at the individual class level but from a broader management perspective. What types of Evaluations are available in Metrics That Matter? Metrics That Matter supports a variety of evaluation form types, each designed to collect specific information at specific points in time. An overview of the content and typical use of our four most typically used form types is provided below. Post Event Evaluation Post Event forms are designed to be deployed to students immediately after they experience a learning event. In addition to typical reaction/satisfaction questions, the Post Event evaluation should include questions that ask students to predict how they will be able to use what was learned and how they think it will increase their job performance. KnowledgeAdvisors offers Standard Post Event forms tailored to Instructor Led training, Virtual/Online Facilitated training, and Self-paced Web-based training. Post Event evaluations contain questions designed to measure the following: Quality and Satisfaction: The effectiveness of the instructor and/or online experience, the relevance of the courseware, and whether the learning environment was conducive to learning Learning: Did the student learn something new? Impact: Will the student be able to apply what was learned on the job? Results: When the student applies the training on the job, what will be the result or benefit to the business? Value: Was this training experience a worthwhile investment for their career development? Alignment: Is this training aligned to the goals and objectives of the organization? Support: Will there be adequate support back on the job so that the student can apply what was learned? Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - 3

6 Follow Up Evaluation Follow Up forms are designed to be deployed to students after they have had adequate time to apply what they learned in training to the job, in order to true up the predictions they made regarding Job Impact, Results, and Value on the Post Event evaluation with what has actually occurred. Typical timing is between days, however the timing may be sooner depending on the type of training offered. Organizations are encouraged to begin with a standard timing for deployment of Follow Up forms and then tailor deployment for specific strategic programs if needed. Follow Up evaluations are not tailored according to the learning method used, however they are most typically used with instructor-led training (face to face or virtual) due to the increased investment associated with live events. Follow Up evaluations should contain questions designed to measure the following: Job Impact: Has the student applied what was learned on the job? To what degree? How often do they use what was learned? How soon after training were they able to apply what they learned on the job? What barriers may have prevented application? Results: What impact has the application of the learning had on the business? How much has their job performance improved as a result of the training? In which areas? Value: Was this training a worthwhile investment in their career development? Support: Has the learner had adequate support from their manager to apply the training on the job? Manager Evaluation Manager evaluations are designed to capture the same types of measures as those included on the Follow Up evaluation from the perspective of the student s manager. They are typically deployed at the same time as the Follow Up evaluation is sent to students. If several of a manager s direct reports attend the same class, the manager will receive a single evaluation from MTM and have an opportunity to provide feedback on each direct report separately. Please note that manager feedback is designed to be analyzed in aggregate and is focused on how the training impacted the student s job performance. It is not intended to in any way review the student s job performance or serve as a 180 degree competency assessment. If you are interested in capturing student and manager feedback (and/or peer, direct report, etc.), talk with your KA account team about our 360 module for competency assessment. Instructor Evaluation Instructor evaluations are designed to be completed by the class instructor(s) immediately after they teach a class, in order to capture the instructor s point of view. The form contains questions about the level of readiness and engagement of the students, the course content, and class environment (including classroom technology). Use of this evaluation helps to provide instructors with an easy, consistent mechanism to provide their perspective. Typically, instructor feedback is used by courseware designers as they prepare to update course content. Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - 4

7 Customization Guidance Many of our clients leverage our standards with no customization. During the implementation process, KnowledgeAdvisors will configure an initial set of standard evaluation forms for your use and will train one or more of your organization s Administrative users on the use of our authoring tool, ActiveAuthoring, to enable self-sufficient form creation, editing, and maintenance. The appendix of this document includes links to form content walkthroughs of our Standard Post Event Instructor Led Training evaluation, Standard Follow Up evaluation, and Standard Manager survey. These content walkthroughs include the standard layout and questions on each form along with callouts that describe the objectives for each section of the form. Clients who customize their forms leverage one or more of the following approaches: Updates to Question Text The text of our standard evaluation questions may be modified to better fit the terms used in your organization (e.g. Facilitator instead of Instructor ). When this approach is used, the question text displayed to the learner is modified as desired and the original text of the question is maintained in reporting. If desired, the updated question text may be configured to display in reporting as well. Additional Custom Questions Your organization may elect to add custom questions designed for use with all courses as a part of your standard question set. KnowledgeAdvisors can assist you with mapping these question(s) to a standard question category to allow for benchmarking at the question category level, or a completely custom question category may be created. Course-specific Custom Questions Metrics That Matter allows for questions to be added to a standard evaluation form and then assigned to ONLY appear when the form is assigned to classes associated to one or more courses. This allows for a single standard form to be maintained and assigned to all courses for ease of administration, while also allowing for some evaluation content that is specific to one or more courses. The custom questions are typically related to learning transfer or job impact related to specific course objectives. Organizations who choose to use this approach typically allow their courseware developers to submit up to 3-5 course-specific questions to be added to the standard evaluation, and an MTM Administrator updates the standard survey accordingly. Course-specific Evaluation Forms In some cases, a course may be strategic or unique enough to warrant a dedicated evaluation that is much more customized than the course-specific question approach can easily support. In these cases, we recommend copying your organization s standard evaluation, renaming it and modifying it as desired, and configuring it to be assigned by default to classes associated with one or more specific courses. Organizations are encourage to leverage this approach sparingly in order to minimize the amount of form management and maintenance required in the future. Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - 5

8 Additional Considerations Form Scale Metrics That Matter can support evaluation scales from 1-3 to 1-10 and can be displayed with the high number on the left or the right. Data is normalized for reporting to support accuracy of benchmark data collected in a variety of scales. You may also choose to collect data via one scale and run reports in a different scale and the data will be converted accurately. Our standard evaluations display a 1-7 scale, reading from left to right. There is quite a bit of research in the broader market regarding the benefits or drawbacks of various scales, and while we are scale agnostic, we selected a 1-7 scale because it is possible to easily place a value of equal weight over each scale value (Strongly Agree, Agree, Slightly Agree, etc.) and we prefer the additional granularity the slightly scale items offer in reporting the data. The majority of our clients use either a 1-5 or 1-7 scale, however we recommend that you consider your organization s culture and any current scale standards you may have already established when determining the scale to present to your learners. It is recommended that a consistent scale be used throughout the evaluation form in order to provide a consistent experience for respondents. Scale Headers/Labels Metrics That Matter supports scale headers/labels over each scale value, over the high and low values only, and over the high, middle, and low values. Our standard evaluations typically leverage a label over each scale value. You may use one of many available standard sets of scale labels, or configure your own custom scale labels. If creating your own, consider each of the labels and whether they are giving equal weight to each value along the scale. It is recommended that a consistent set of scale labels be used throughout the evaluation form in order to provide a consistent experience for respondents. If more than one set of scale labels must be used, consider presenting the questions with the differing scale in a separate section of the form; do not mix the labels from one question to another. Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - 6

9 Appendix: Standard Form Walkthroughs Each of the links below will allow you to download a document that contains the standard form content along with callouts that describe the objectives for key sections and questions on the form. Form Content Overview: KA Standard Post Event Evaluation - Instructor Led Training Form Content Overview: KA Standard Follow Up Evaluation Form Content Overview: KA Standard Manager Evaluation Copyright 2012 KnowledgeAdvisors. All Rights Reserved. Page - 7