Implementing Learning Scorecards: Tracking Activity, Efficiency, and Impact. elearning Assessment and Evaluation Approaches. August 18 & 19, 2011

Transcription

1 elearning Assessment and Evaluation Approaches 402 Implementing Learning Scorecards: Tracking Activity, Efficiency, and Impact Lynn Schmidt, Ph.D., Raytheon and Toni Hodges DeTuncq, THD & Company, Inc.

2 The elearning Guild s Online Forum August 19, 2011 Implementing Learning Scorecards: Tracking Activity, Efficiency and Impact Lynn Schmidt lynn_m_schmidt@raytheon.com Toni DeTuncq toni@thdandco.com Session Objectives Determine an organization s needs for a learning scorecard Develop a prototype of a learning scorecard Assess what data is required to populate a learning scorecard Session 402 Implementing Learning Scorecards: Tracking Page 1

3 Learning Scorecards Do You Know Your Learning Score? Score a record of points made by competitors in a game or match Scorecard a card for keeping score of a sports contest Learning Scorecards Provides macro and micro-level perspective of success and progress Serves as a brief report versus detailed studies Shows connection of training s contribution to business objectives Integrates various types of data Demonstrates alignment between programs, strategic objectives, and operating goals Session 402 Implementing Learning Scorecards: Tracking Page 2

4 Macro Learning Scorecard Tier 1 Data Elements: Roll-up Investment/Efficiency Data Activity Data Impact Data Tier 2 Data Elements: Each line of business Macro Scorecard Tier 1 Investment/Efficiency and Activity Data - how much, how often, and value to the organization Session 402 Implementing Learning Scorecards: Tracking Page 3

5 Macro Level Tier 1 Investment/Efficiency - Example Metric Result Value Data Source Detailed L&D Spend as % of overall Operating Expenses 4.0% We are funding L&D at the desired level LMS Displayed as graph and/or table Macro Scorecard Tier 1 Activity - Example Metric Result Value Data Source Detailed # person hours of training (monthly) 1,803 Number has dropped need to investigate: -surge gap -turnover PeopleSoft Graph/Table Session 402 Implementing Learning Scorecards: Tracking Page 4

6 Impact Data Tier 1 Overall Averages for each Line of Business (LOB) Level 1 Data Average score for all programs Level 2 Data Average % score Level 3 Data Average % score Level 4 Data Average % business objectives met score Level 5 Data Percent ROI Impact Data Tier 1 Overall Averages for each LOB LOB Reaction Learning Performance Impact ROI Sales % 85% 90% 400% Operations % 52% 43% 150% Leadership % 80% 76% 430% Mexico % 60% 50% N/A UK % 75% 65% N/A Session 402 Implementing Learning Scorecards: Tracking Page 5

7 Micro Learning Scorecards Completed for Each Impact Study Conducted Micro Learning Scorecard Program Title: Target Audience & Number Trained: Duration: Business Objectives: Satisfaction Learning Application Tangible Benefits Level 1 Level 2 Level 3 Levels 4 & 5 Intangible Benefits Technique to Isolate Effects of Program: Technique to Convert Data to Monetary Value: Fully-loaded Program Costs: Barriers to Application of Skills: Recommendations: Session 402 Implementing Learning Scorecards: Tracking Page 6

8 Learning Scorecard Case Study Installation Technician Workshop Workshop impact study was conducted with 24 technicians who participated in the pilot program Conducted Level 1 assessments Conducted Level 2 pre and post-tests Level 3 feedback was received from managers concerning improved quality assurance checks (not quantified) Level 4 & 5 data was extracted from the enterprise data warehouse approximately three months after the training - trouble call and install business impact data for the 24 employees that participated in the pilot program Level 4: Contribution to Business Participants felt the program had a high to very significant impact on the following business goals: Customer Satisfaction Productivity reduced repeat truck rolls reduced repeat trouble calls increased number of completed installations Session 402 Implementing Learning Scorecards: Tracking Page 7

9 Level 4: Business Impact, Conversion & Isolation Repeat truck rolls: saved truck rolls $80.00 per truck roll $16, annualized cost savings Repeat trouble calls to call center: saved calls $13.00 per call $4, annualized cost savings Level 4: Business Impact, Conversion & Isolation Productivity: 38 additional technician calls per month $14, annualized savings based on labor costs Total annualized benefit: $36, Total benefit attributed to training based on management estimation 50% $18, annualized benefit Session 402 Implementing Learning Scorecards: Tracking Page 8

10 Level 5: Benefits, Costs & ROI ROI Formula: benefits minus costs divided by costs x 100 Total Adjusted Benefit: $18, Total Costs: $13, Return on Investment: 39% Benefit/Cost Ratio: 1.39:1 (benefits divided by costs) Intangible Benefits Improved employee morale resulting in lower turnover; many employees expressed gratitude that the company cared enough to provide the training Increased customer satisfaction due to fewer trouble calls and more reliable service resulting in lower customer churn Better prepared employees to install and troubleshoot advanced services as they are launched Meeting the business critical commitment to assist employees in realizing their career growth potential Session 402 Implementing Learning Scorecards: Tracking Page 9

11 Micro Learning Scorecard Program Title: Installation Technician Workshop Target Audience & Number Trained: Technicians, 24 Duration: 3 Days Business Objectives: Customer Satisfaction, Increased Productivity Level 1 Level 2 Level 3 Levels 4/5 Intangible Benefits Skills: 4.5 Course: 4.6 Instructor: 4.8 Pre: 74% Post: 97% N/A $18, ROI: 39% BCR: 1.39:1 Improved Morale Increased Customer Satisfaction Case Study for Micro Learning Scorecard Read learning impact case study Fill out the micro learning scorecard template Place data from the case study in appropriate areas of the scorecard Take 5 minutes to complete Session 402 Implementing Learning Scorecards: Tracking Page 10

12 Tips for Implementing a Micro Learning Scorecard Start today don t wait for the request for it Begin with the end in mind plan in advance Keep the process simple use estimations Always account for the influence of other factors Be conservative when converting and isolating Involve management in the process Educate the training team Educate the management team Communicate your results Macro Learning Scorecard Impact Data for each LOB or Business Unit Session 402 Implementing Learning Scorecards: Tracking Page 11

13 Macro Scorecard Tier 2 For each LOB or Business Unit Investment/Efficiency Data same as Tier 1 Activity Data same as Tier 1 Impact Data recording Micro Scorecard Data with added business alignment information Macro Scorecard Tier 2 Impact Data Example Course Title Business Objectives Alignment with Organizational Strategy Average Level 1 Average Level 2 Average Level 3 Average Level 4 Average Level 5 Course #1 1.Reduce Turnover Retain intellectual capital % 65% Course #2 1.Increase Sales 2.Improve Customer Sat Increase profit and maintain shareholder value % 75% 65% 250% Course #3 1.Reduce Scrap 2.Reduce Overtime Improve employee efficiency % 45% Session 402 Implementing Learning Scorecards: Tracking Page 12

14 Macro Learning Scorecard Considerations The learning organization must be able to demonstrate how they are aligned with organizational strategy Each program must have business objectives Investment/Efficiency and Activity data will vary by organization but should be used to tell a story and could be used to benchmark Level 1 instrument must be standardized Level 2 instruments must be designed, validated and scored consistently Level 3, 4 and 5 data must be collected using industry-accepted standards Additional Resources Handbook of Training Evaluation and Measurement Methods, 3 rd Edition, Jack Phillips, ISBN ASTD in Action: Implementing Training Scorecards, Lynn Schmidt, ISBN The Leadership Scorecard, Jack Phillips and Lynn Schmidt, ISBN Linking Learning and Performance, T.K. Hodges, Butterworth Heinmann, 2002 Implementing Learning Scorecards Job Aid: elearning Guild Event Resources Page Session 402 Implementing Learning Scorecards: Tracking Page 13