Session 601. Success Case Evaluation: Making the Business Case for Learning. April 29 & 30, 2010 Practical and Effective Assessments in e-learning

Transcription

1 Practical and Effective Assessments in e-learning Session 601 Success Case Evaluation: Making the Business Case for Learning Timothy Mooney, Advantage Performance Group

2 Success Case Evaluation Method A Strategic Approach to Evaluation Tim Mooney Advantage Performance Group April 2010 & BTS All Rights Reserved Reproduction of these materials in any form is prohibited without prior written permission from the Advantage Performance Group Advantage Performance Group Advantage Performance Group is the world s largest and fastest growing performance consulting and training solutions network Leadership and Teams Sales and Service Supremacy Business Literacy Performance Management 2 Page 1

3 Purpose Share thoughts and concerns about current evaluation strategies Provide guidelines for a new direction Begin to prepare you to have a different type of discussion about development and evaluation with senior management Agenda The Criteria for Training Evaluation The Two Realities of Training Evaluation A Fresh Approach to Training Evaluation Next Steps Page 2

4 What Criteria Matter Most? Please choose your top THREE criteria: Scientifically rigorous Simple to do Objective Valid measures true impact Actionable gives insights on ways to improve results Provides hard data Credible and persuasive to senior management Provides a ROI calculation Our Criteria for Training Evaluation Credible Simple Valid Actionable Page 3

5 Coping with the Realities of Training There are Two Realities of training that evaluation must recognize and leverage Coping with the Realities of Training Reality #1... Exercise Page 4

6 Impact of Training Estimate Of 100,000 people who attended training, how many went back to work and Did not try it at all? Tried it a bit, had trouble, went back to old methods? Tried it and achieved concrete & valuable results? % = % = % = Impact of Training Estimate Of 100,000 people who attended training, how many went back to work and Did not try it at all? Tried it a bit, had trouble, went back to old methods? Tried it and achieved concrete & valuable results? % = % = % = In the chat pod, type your answer (e.g., 0%-0%-100%) Page 5

7 Predictable Results > 15% DID NOT TRY IT AT ALL 70% +/- TRIED IT TO SOME EXTENT BUT GAVE UP < 15% TRIED IT AND GOT POSITIVE RESULTS Predictable Results COSTS DID NOT TRY IT AT ALL TRIED IT TO SOME EXTENT BUT GAVE UP TRIED IT AND GOT POSITIVE RESULTS RETURN Page 6

8 The Tyranny of the Mean _ x Put Bill Gates in a room with 1,000 homeless people What is the average net worth of those 1,001 people? Page 7

9 If Bill Gates was a room with 1,000 homeless people... What is the average net worth of those 1,001 people? Type your estimate in chat pod The Misleading Mean _ x Page 8

10 A New Metric COSTS DID NOT TRY IT AT ALL TRIED IT TO SOME EXTENT BUT GAVE UP TRIED IT AND GOT POSITIVE RESULTS UNREALIZED VALUE RETURN An Example COSTS DID NOT TRY IT AT ALL TRIED IT TO SOME EXTENT BUT GAVE UP TRIED IT AND GOT POSITIVE RESULTS 90/100 did NOT use it or get results UNREALIZED VALUE $50,000 Total = $500,000 RETURN Page 9

11 An Example COSTS DID NOT TRY IT AT ALL TRIED IT TO SOME EXTENT BUT GAVE UP TRIED IT AND GOT POSITIVE RESULTS 90/100 did NOT use it or get results Unrealized value = $4.5 million! UNREALIZED VALUE $50,000 Total = $500,000 RETURN Our Goal COSTS 90%+ TRIED IT TO SOME EXTENT BUT TRIED IT AND GOT POSITIVE RESULTS GAVE UP RETURN Page 10

12 Reality #1 Reality #1: Training gets predictable results Lessons Avoid the Tyranny of the Mean Calculate Unrealized Value Aim to grow impact Coping with the Realities of Training Reality #2... Exercise What makes or breaks the ability of training to produce business impact? Page 11

13 The Learning-to-Performance Model Create Focus, Alignment & Intentionality Provide Quality Learning Interventions Support Performance Improvement BEFORE AFTER Results Learning to Performance Process Create Focus, Alignment & Intentionality Provide Quality Learning Interventions Support Performance Improvement RESULTS Greatest Opportunities for Improvement Leverage Page 12

14 Reality #2 Reality #2: Training alone never fully explains success or failure to get results Lessons Don t try to single out the sole effects of the training Theoretical and misdirected pursuits How many angels can dance on the head of pin? What percentage of the business impact was due to the training? Page 13

15 Reality #2 Reality #2: Training alone never fully explains success or failure to get results Lessons Don t try to single out the sole effects of the training Identify the performance system factors and manager behaviors that make the difference The Two Realities of Training Reality #1: Training gets predictable results Reality #2: Training alone never fully explains success or failure to get results So where do we go from here? Page 14

16 A Fresh Approach to Evaluation A Fresh Approach to Evaluation Don t try to single out the sole effects of the training Goal: Give credit to the partnership, not training Seek out the performance system factors that make the difference Pinpoint the key make or break manager behaviors Page 15

17 Traditional Training Evaluation Strategy Producing effective learning outcomes Area of Investigation: Learning event or activities Feedback on Training Content Estimate ROI Training Function Managers Senior Leaders Actions to Change Training Event Demonstrate Value of the Training Evaluation Strategy Investigation Focus: The Entire Learning-to-Performance Process Creating alignment, focus and intentionality Producing effective learning outcomes Supporting performance improvement Page 16

18 Evaluation Strategy Investigation Focus: The Entire Learning-to-Performance Process Creating alignment, focus and intentionality Producing effective learning outcomes Supporting performance improvement Feedback on Results: Measure Business Impact What s Working/What s Not & Why Evaluation Strategy Investigation Focus: The Entire Learning-to-Performance Process Creating alignment, focus and intentionality Producing effective learning outcomes Supporting performance improvement Feedback on Results: Measure Business Impact What s Working/What s Not & Why Training Function Managers Systems (e.g., HR) Owners Senior Leaders Actions to Change Conditions & Improve Processes Page 17

19 Evaluation Strategy Investigation Focus: The Entire Learning-to-Performance Process Creating alignment, focus and intentionality Producing effective learning outcomes Supporting performance improvement Feedback on Results: Measure Business Impact What s Working/What s Not & Why Training Function Managers Systems (e.g., HR) Owners Senior Leaders Actions to Change Conditions & Improve Processes Enhanced Organizational Capability to Leverage Business Results from Learning Investments Success Case Process Identify most and least successful trainees Page 18

20 Predictable Results?% DID NOT TRY IT AT ALL?% TRIED IT TO SOME EXTENT BUT GAVE UP?% TRIED IT AND GOT POSITIVE RESULTS Success Case Process Identify most and least successful trainees Conduct in-depth interviews with selected samples Page 19

21 A New Metric COSTS DID NOT TRY IT AT ALL TRIED IT TO SOME EXTENT BUT GAVE UP TRIED IT AND GOT POSITIVE RESULTS UNREALIZED VALUE RETURN Success Case Process Identify most and least successful trainees Conduct in-depth interviews with selected samples Credible and documented cases of success Knowledge of factors that drive impact Page 20

22 Success Case Process Identify most and least successful trainees Conduct in-depth interviews with selected samples Credible and documented cases of success Knowledge of factors that drive impact Tell the Story! Prove that training works! Applaud the success of the partnership! Trumpet the successes! Educate managers about their role in impact! Improve learning application processes! Some Recent Successes Office Systems Company Implemented major sales strategy Measured business impact & application of training Achieved 40% increase in this high priority sales area As a result of study, Sales VP changed process for how training was implemented Success led to mandatory manager preparation of trainees Page 21

23 Resources Telling Training s Story by Robert O. Brinkerhoff (Berrett Koehler, 2006) Courageous Training by Mooney & Brinkerhoff (Berrett Koehler, 2008) Using Evaluation to Build Organizational Performance and Learning Capability: A Strategy and a Method. R. Brinkerhoff and D. Dressler, Performance Improvement International Society for Performance Improvement, July For more info: contact Tim Mooney (630) tmooney@advantageperformance.com or Visit our website: Resources QUESTIONS??? Telling Training s Story by Robert O. Brinkerhoff (Berrett Koehler, 2006) Courageous Training by Mooney & Brinkerhoff (Berrett Koehler, 2008) Using Evaluation to Build Organizational Performance and Learning Capability: A Strategy and a Method. R. Brinkerhoff and D. Dressler, Performance Improvement International Society for Performance Improvement, July For more info: contact Tim Mooney (630) tmooney@advantageperformance.com or Visit our website: THANKS!!! Page 22