Metrics that Matter Training Portfolio Evaluation Model Formal Learning Interventions
Contents Introduction... 2 Portfolios... 3 Other Considerations:... 4 Duration... 4 Sampling... 4 Survey Form Types...5 Training Portfolio Management Model Formal Learning... 6 Reporting... 7 Implementation... 7 Implementation questions... 8 Copyright 2015 CEB. All Rights Reserved. P a g e 1
Introduction To support the continued use of a standardized evaluation strategy, while ensuring appropriate metrics are obtained according to the desired outcomes of specific training interventions, CEB recommends deploying surveys according to a Portfolio Model that considers the intended purpose and degree of impact to the business when determining the appropriate survey(s) to be deployed. The Portfolio Model leverages widely accepted methodologies for the collection of relevant, actionable data, and provides a framework for determining when and how to apply various measurement strategies across the breadth of an organization s learning interventions. There are many books dedicated to the measurement of the few high-profile, expensive programs that consume 10% of the training budget; this framework provides guidance for managing not only these programs, but also the hundreds or thousands of programs that serve as vital resources to your organization s Human Capital and absorb the other 90% of the learning budget. Just as individuals and companies maintain a diverse range of investments with varying degrees of expected return, learning and development organizations maintain a diverse range of learning interventions within the course catalog, which can be viewed as investments of company resources made in the individuals who experience them. Individuals and companies categorize their investments into portfolios to facilitate the management of numerous investments and assets, to first track performance at the portfolio level, then within each portfolio. Learning and development organizations can utilize this approach to segment or group learning investments such as specific courses, coaching, and other programs into portfolios based on the purpose and impact that each learning investment is expected to have on the business. Copyright 2015 CEB. All Rights Reserved. P a g e 2
Portfolios An example of the types of learning investments a typical learning and development organization may manage, including a brief description of each portfolio is listed below: Strategic: Programs that are broad-reaching and extremely strategic to the company's ongoing performance; measureable business outcomes have been tied to the program and those measures are more focused on improving overall company performance to differentiate the company from competitors. Example: company-wide customer service program to build service oriented culture Specialized: Programs that are designed to have a high impact on participants' jobs, such as specialized skill training. Training is focused on specific outcomes and focused on improving individual performance in areas critical to the business. Example: sales training, leadership training, new hire onboarding, critical and proprietary software or process training Core Curriculum: Programs that are designed to have a direct impact on participants' job performance, although the impact from individual to individual may vary significantly. For example, employees newer to the company or their role may report high impact, and employees who have been with the company or their role for some time and may be taking the training for review purposes may report lower impact. Employees may use courses in this portfolio to hone skills in areas that may be of interest but not directly tied to the current role or critical to the business. Example: time management, general IT skills, business writing Informational: Programs that are designed to provide information, but that may not be intended to have a direct impact to tasks that participants perform regularly on the job. Example: webinar delivered by an executive to communicate company strategy and key areas of focus for the upcoming year Compliance: Programs that are generally recurring and required in order to shield the company from risk or maintain specific legal requirements. Example: safety, harassment, some industry-specific regulatory requirements Copyright 2015 CEB. All Rights Reserved. P a g e 3
Other Considerations: Duration Sampling The duration, or time spent experiencing the learning intervention, is an important consideration in the overall evaluation strategy. It would likely not make sense, for example, to require a lengthy customized Post Event, Follow up, and Manager survey for a 1-hour informational webinar, as the time the participant and manager may invest in completing the surveys may be more than 25% of the class duration! While time in class may not directly correlate to business impact, good judgment should be used when considering the evaluation strategy. Sampling may also be included in the evaluation strategy. Survey sampling percentages may be set in Metrics That Matter at the Learning Provider, Student Demographic, Course, and Class levels. When considering sampling, ensure that the volume of the program will be sufficient to yield an adequate sample of surveys within the expected response rate. Course-level sampling is ideal for largevolume programs (500 or more participants per class, or large volume elearning courses) where it is appropriate to keep a pulse on results without over-surveying the population. Generally, the lower the intended impact to the business, the less risk is associated with the use of sampling to gather results. Copyright 2015 CEB. All Rights Reserved. P a g e 4
Survey Form Types Management of the performance of each of the portfolios outlined above may be effectively accomplished in a scalable, replicable manner through the application of appropriate survey instruments intended to capture metrics that will allow for consistent reporting. The Portfolio Model leverages multiple survey form types, each designed to collect specific information at specific points in time. An overview of the content and typical use of the four most typically used form types is provided below. Post Event: Post Event surveys are designed to be deployed to students immediately after they experience a learning event. In addition to typical reaction/satisfaction questions, the Post Event survey should include questions that ask students to predict how they will be able to use what was learned and how they think it will increase their job performance. Post Event surveys are typically tailored to the learning methodology (instructor led, virtual classroom, elearning, etc.). Follow Up: Follow Up surveys are designed to be deployed to students after they have had adequate time to apply what they learned in training to the job, in order to true up the predictions they made regarding Job Impact, Results, and Value on the Post Event survey with what has actually occurred. Typical timing is between 60-90 days, however the timing may be sooner depending on the type of training offered. Manager: Manager surveys are designed to capture the same types of measures as those included on the Follow Up surveys from the perspective of the student s manager. They are typically deployed at the same time as the Follow Up survey is sent to students. Instructor: Instructor surveys are designed to be completed by the class instructor(s) immediately after they teach a class, in order to capture the instructor s point of view. The survey contains questions about the level of readiness and engagement of the students, the course content, and class environment (including classroom technology). Use of this evaluation helps to provide instructors with an easy, consistent mechanism to provide their perspective. Copyright 2015 CEB. All Rights Reserved. P a g e 5
Training Portfolio Management Model Formal Learning Portfolio Purpose Typical Key Metrics Recommended Segment Duration Survey Deployment Compliance/ Compliance/ Less than Satisfaction, quality, Lite Post Event Informational Information 2 hours learning, alignment, transfer value (soft) Core Skill building or 2 hours - Satisfaction, quality, Standard Post Event, Curriculum maintenance 2 days learning, alignment, Follow Up, with variable job job impact, business Instructor impact results, value Strategic Broad-reach Depends Satisfaction, quality, Standard Post Event, programs on learning, alignment, Follow Up, designed to program; job impact, business Manager, provide a typically 1 results, value (for Instructor measurable day or specific business competitive more results), candidate for advantage impact study (Success Case, Phillips ROI, etc.) Specialized High job impact 1 day or Satisfaction, quality, Content-Specific specialized skill more learning, alignment, Post Event, building (i.e., job impact, business Follow Up, Leadership) results, value (for Manager, critical to the specific business Instructor business results) Copyright 2015 CEB. All Rights Reserved. P a g e 6
Reporting Data is useless until it is transformed into information via reporting and analysis, which leads to action. The evaluation strategy leads into the reporting strategy, as comparisons between courses within the same Portfolio may uncover different insights and actions than a more homogenous approach. Courses should be grouped according to Portfolio for reporting, which allows for micro-level analysis within each Portfolio and macro-level analysis across all Portfolios to ensure an appropriate investment mix across the training function. Micro-level (within a specific Portfolio) analysis examples: - Comparing courses within a Portfolio according to quality, alignment, impact, and perceived value to determine the highest and lowest performers - Reviewing the quality or impact results for the highest volume courses within a Portfolio - Trending metrics within a Portfolio over time to ensure consistency and/or to determine if investments made toward improvement are bearing fruit Macro-level (across multiple Portfolios) analysis examples: - Reviewing the volume and $ allocation of each Portfolio as it relates to all Portfolios as a whole provides insight to drive resource allocation - Compare consistent metrics across Portfolios and set performance goals according to historical performance and strategic planning Implementation To implement the Training Portfolio model, consider the following steps: 1. Determine which Portfolio groupings are appropriate for your organization 2. Assign each course to a Portfolio 3. Finalize the instruments for each Portfolio (this may be done in a staged approach) Copyright 2015 CEB. All Rights Reserved. P a g e 7
4. Determine the desired approach for survey assignment, to ensure that the appropriate surveys are assigned according to Portfolio 5. Configure a curriculum hierarchy in MTM that includes each Portfolio, and assign courses appropriately 5. As data is collected, begin reporting with a new perspective! Implementation questions - How many Portfolios should be tracked? o The Portfolios outlined in this document are meant to serve as a guide; your organization may choose to maintain more or fewer. When organizations first begin using Metrics That Matter, it is recommended to focus on measuring one Portfolio and then build and diversify the survey approach by adding additional Portfolios. - I already group my courses according to curriculum area; how is this different? Can I use both a Portfolio approach and a Curriculum-based approach? o Yes! Analyzing courses according to curriculum area can yield valuable insights into how the needs of learners are being met within specific content areas. The Portfolio approach focuses less on the specific content and more on the purpose or intended impact. It s certainly possible for both approaches to complement one another. - How do I ensure that the right surveys are assigned? o Metrics that Matter (MTM) SmartSheets are available in each of the form types outlined above and apply scientific rigor to gather the rights set of metrics to not only measure the business impact of learning, but to predict future impact with enough lead time to change and improve the results. It is recommended to leverage the SmartSheets when deploying the Portfolio Model in order to gather comprehensive feedback is gathered via the perspectives of 3 stakeholders (learners, managers and instructors) at 2 points in time across 15 key categories. Metrics that Matter (MTM) can be configured to assign surveys by portfolio and to individual offerings to align with your organization s evaluation plan. Ongoing assignment of Copyright 2015 CEB. All Rights Reserved. P a g e 8
surveys to portfolios and offerings can be done either automatically through an LMS integration or manually by an MTM administrator. - How can I easily group courses into Portfolios for reporting? o The Curriculum Hierarchy feature in the Administration module of Metrics That Matter allows courses to be grouped for reporting purposes either automatically via LMS integration or manually by an MTM Administrator. - What about my historical data? Can I report on it by Portfolio as well? o Yes! You may find that you have more or less detail than desired when analyzing data collected prior to the implementation of the Training Portfolio Model, but you will still be able to draw many new conclusions immediately after implementing this approach. Copyright 2015 CEB. All Rights Reserved. P a g e 9