Using an ROI Approach to Better Manage Learning Investments and Reduce Scrap Learning KnowledgeAdvisors. All rights reserved.

Transcription

1 Using an ROI Approach to Better Manage Learning Investments and Reduce Scrap Learning 2007 KnowledgeAdvisors. All rights reserved.

2 Overview of KnowledgeAdvisors Human Capital Analytics Professional Services Benchmarking Benchmarking Technology We help organizations measure and improve the results of their people programs. Visit our website at Page 2

3 Should you calculate ROI? A We make choices based on: Reaction Knowledge/Attitude Utility Economics B Page 3

4 Drive Business Results Earnings Redirect poor-performing investments into high-impact programs Low Impact Programs High Impact Programs Time Page 4

5 Shift Paradigm Page 5

6 What CEO s Want revenue employees OR Revenue grows Human capital investment is flat revenue Revenue is flat Human capital investment declines employees Page 6

7 Learning Measurement Resource Allocation Today 81% of measurement resources focused on administration (collection, aggregation, filtering, reporting). < 20% is available for analysis and decision-making. Technology reverses the ratio: 80% on analysis, 20% on administration. Data analysis 19% Data collection 33% Data reporting 20% Data filtering 9% Source: KnowledgeAdvisors Learning Analytics Best Practices Research Study, Page 7 Data aggregation 19%

8 Reasonable Indicators There is an enormously high cost to data accuracy versus obtaining a reasonable indicator. In your organization, what best describes the level of accuracy that analytics must have in order to be perceived as useful for information decision-making purposes of senior management? Organizations should balance when to spend extra time and money to yield highly statistical results. Given how executives make decisions, reasonable data provided in a timely manner outweighs data with more precision delivered in a less timely manner. Reasonable, quantitative / qualitative indicators 78% Highly precise, statistically valid measures 11% Periodic "word of mouth" statements 11% Source: KnowledgeAdvisors Learning Analytics Best Practices Research Study, Page 8

9 Human Capital Impact Smart investments in learning drive future business results, which ultimately increase stakeholder value. Page 9

10 Why Invest in Learning? $$ Earnings with a strong corporate education program Time Earnings without a strong corporate education program Page 10

11 How do Shareholders Benefit? Market Value $$ Human Capital Value Time Book Value Page 11

12 Optimizing Training Investments Nobel Prize Winner, Gary Becker Human Capital Formula MP = Marginal product of trainees G = Present value of returns W = Wages paid i = Market discount rate C = Cost of training Page 12

13 The ROI Process Evaluation Instruments Evaluation Purposes Tabulate program costs Collect Data Isolate the effects of training Convert data to monetary value Calculate the return on investment Evaluation Timing Evaluation Levels Source: Return on Investment in Training and Performance Improvement Programs, Jack J. Phillips, Gulf Publishing Company, Houston Texas, 1997, p. 25 Identify tangible benefits Page 13

14 Estimate, Isolate, Adjust Estimate the change in performance (ex. How much did sales increase since the sales improvement initiative?) 20% Isolate the effect that training had on the change (ex. What % of sales increase was driven by the training vs. other factors?)=10% Adjust for confidence (or lack thereof) in estimates (ex. How confident are you in your estimates?)=50% Monetize the performance change related to training (ex. $100,000 is a 20% increase in sales) = $100,000 x.10 x.50 = $5,000 benefit from initiative Fully loaded costs to compare against the benefit (ex. Cost to rollout initiative is $1,000, travel is $500, lost work time is $800) =$2,300 Page 14

15 The Measurement Process Data Collection Define metrics (KPI s) Design instruments Determine population Disseminate instruments Collect data Data Storage Database creation Centralize data Data entry Data security Raw data access Data Processing Aggregation of data Filtering of data Convert into metrics Because data collection, processing and reporting is automated, you have more time to analyze the data and turn it into actionable business intelligence to: Data Reporting Standard reports (canned) Ad-hoc querying Trend-line analysis Statistical analysis Benchmark comparisons Transactional details Manager reports Executive summaries Improve your learning programs Demonstrate value to your stakeholders Page 15

16 After Training: Practical Indicators Reaction Level 1 Results Level 4 & Estimation Isolation Adjustment Learning Level 2 ROI/ROE Level 5 Job Impact Level 3 Page 16

17 ROI Models to Manage Investments Cost and Complexity Page 17

18 ROI The Cost Side Direct and measurable costs Analysis costs Design, development: over the life of the program Materials: workbooks, participant handouts Instructor prep and delivery Facilities Content acquisition Administrative and allocated overhead Evaluation costs Indirect or imputable costs Participant tuition Participant travel expenses Participant opportunity costs Conservatism is a best practice! Page 18

19 Human Capital ROI A Job Impact ROI Easy to use, practical, scaleable, replicable Provides a financial ROI relative to the improvement in human capital performance Uses principles of estimation, isolation, and adjustment Part of a regular balanced scorecard for learning measurement Assumption is that human capital s value (i.e. salary) is like any other asset that has expected value that can appreciate or depreciate over time Page 19

20 Human Capital ROI: Example Factor Item Amount Actual Fully loaded salary of participants (average) $50,000 Actual Training cost per participant $1,000 Estimate Performance improvement, including training 30% Isolate Performance improvement due to training 60% Isolate Performance relative to time spent performing 20% those skills on the job Adjust Bias, confidence, conservatism 65% Training impact on performance improvement, adjusted for bias: 30% x 60% x 20% x 65% = 2.34% Calculate monetized benefit = $50,000 x 2.34% = $1,170 Calculate ROI = $1,170 - $1,000 = $170 Calculate Benefit to Cost Ratio: $1,170 / $1,000 = 1.17 Page 20

21 Case Study: New Horizons Largest IT Training Company in the world Train in multiple certified IT channels Used by NH centers around the world Over 1.5 million surveys collected Collect data using multiple IT channel forms (Microsoft, Citrix, Cisco) Use the data to manage their operations quality and customer service Leverage the data for strategic marketing purposes (see PR) Page 21

22 New Horizons Highlights 38% of performance improvements were directly attributable and isolated to training 70% of participants apply training to job within 8 weeks 97% utilize/plan to utilize the training on the job 60% increase in skill/knowledge from training A 4:1 predicted benefit to cost ratio when completing the training Prior to our relationship with New Horizons we were not effectively providing a productivity baseline for our knowledge workers. -- Anthony Garreffi, Manager E-Learning / Technical Training, Iron Mountain Page 22

23 New Horizons Highlights 4/1 to 4/30 Iron Mountain ROI Data Iron Mountain Human Capital ROI Score Card Level 5 Return on Investment Post Event Costs (per person) Monetary Benefits Benefit to Cost Ratio (per person) ROI Percentage Payback Period (months) 1,000 3, % 3.43 This shows the predicted ROI results for Iron Mountain. It is calculated based on the improvement in human capital isolated to the training, adjusted for bias. Source: KnowledgeAdvisors Metrics that Matter Database Page 23

24 Business Results ROI A deeper Level 4 ROI Leverage technology to make practical and scaleable Provides a financial ROI relative to the change in the business result that prompted the training Uses principles of estimation, isolation, and adjustment specific to the business result Part of a regular balanced scorecard for learning measurement Page 24

25 Business Result ROI: Example Training cost per participant: $5,000 Business Result Significantly Impacted: Sales Estimate change in sales (pre vs. post training) = +20% or $20,000 (sales were $100,000 before training, now they are at $120,000)(annualize this) Isolate performance improvement due to training: 60% (of the 20% above) Adjust for bias, confidence, conservatism: 65% Training impact on sales adjusted for bias: Calculate monetized benefit = 20% x 60% x 65% = +7.8% Calculate monetized benefit = $100,000 x 7.8% = $7,800 Calculate ROI = $7,800 - $5,000 = $2,800 Calculate Benefit to Cost Ratio: $7,800 / $5,000 = $1.56 Page 25

26 Case Study: PeopleSoft Education Collected over 20,000 evaluations since September 2003 Require participants to predict and estimate training linkage to the following business results: Increased Sales Decreased Costs Increased Productivity Increased Quality Decreased Cycle Time Increased Customer Satisfaction Increased Employee Satisfaction Can run a Business Result Score Card by Client, course, program etc. to look at linkage to business results Page 26

27 PeopleSoft Education Highlights PR Issued March 22, % improvement in user productivity after completing PeopleSoft training class 24% improvement in cycle time 22% improvement in quality >90% feel the training was a worthwhile investment for their employers We see great value in being able to link PeopleSoft education courses to an increase in productivity. - -Joseph Cummings, IS Director, DePaul (client of PeopleSoft) Page 27

28 Impact Study A process whereby you can be certified to do it A formal step by step process for comprehensive impact study analysis A set of defined cost components for each impact study More comprehensive data collection instruments Business result analysis for each participant s business results A specific Phillips ROI Scorecard with defined metrics at each level of learning measurement Used for specific projects or programs where deep analysis is needed (about 5 to 10% of the time) Page 28

29 Impact Studies: Go in Depth to Prove Results Sales Training Program: Impact Study revealed that trained employees average retail weekly sales were $12,075 vs. $10,449 for similar employees who were not trained. Performance Management Program: A performance management program at a restaurant generated over $170,000 in benefits directly tied to the training yet the training only cost less than $50,000. Masters Degree Program for Federal Agency: Goal was to reduce high turnover. Turnover 1 year after the program was 34% for employees NOT in the program and 3% for employees IN the program. It yielded a benefit to cost ratio of over 2 to 1. Above data from the ROI Institute, Drs. Jack and Patti Phillips Page 29

30 Identify Program Benefits Hard Data Output Quality Cost Time Soft Data Customer Service Work Climate Work Habits Initiative Page 30 30

31 Scrap Learning 55% of learning is not applied to the 55% job.that.that s s Scrap How much learning is applied to the job? When is learning applied to the job? What are the barriers to impact? What can be done to reduce scrap? Page KnowledgeAdvisors. All rights reserved.

32 Top 10 Ways to Waste Learning 1. Investments not aligned with business objectives 2. Learning not applied to job 3. Poorly designed programs 4. Ineffective instructors 5. Poor learning environment 6. Insufficient support 7. Inappropriate course selection 8. Under-utilized resources 9. Costs outweigh benefits 10. Lower cost alternatives Page 32

33 Dashboards Definition: A convenient place for vital information organized into a graphical representation that is both easy to use and easy to understand. Page 33

34 Examples of Macro Learning Constructs Operational Activity constructs (how much we train) Performance Optimization constructs (how well we train) Financial Budget/Fiscal constructs (how much it costs) Employee / Cultural Supportive Environment (how conducive is it) Note: these constructs are part of KnowledgeAdvisors Metrics that Matter standards Page 34

35 Sample Dashboards to Analyze Metrics Operational Performance Source: Metrics that Matter A report showing actual results over time with a color coded analysis to compare variances against prior periods and goals/benchmarks Page 35

36 Case Study Example: Sprint Situation Six Learning Institutes Hundreds of programs Uses the Phillips / Kirkpatrick Model Requirements Tool that can forecast ROI and key metrics Populate the HR Scorecard for top management Analytics Solution: Each institute has own metrics account to manage their data Rollup account created for the CLO to view metrics MTM tactical tools for day to day improvements Benchmark against the database for external comparison Data from MTM goes directly to Nextel Scorecard Page 36

37 Metrics that Matter Report Card for Sprint Page 37

38 Sprint HRD Scorecard Example Human Resources Development Scorecard > 100% Goal Achievement = 85-99% < 85% Metric Goal Jan Feb Mar Apr May Jun Mo. Avg Metrics That Matter Captured Data Training Participant Satisfaction w ith Instructors Goal Achievement 106% 109% 108% 108% 105% 108% 107% Learning Effectiveness from Training Goal Achievement 100% 105% 107% 106% 107% 105% 105% Job Impact from Training Goal Achievement 100% 101% 107% 105% 106% 105% 104% Business Impact from Training Goal Achievement 96% 99% 110% 106% 108% 108% 105% HRD System Captured Data HRD 2004 Goals Values pulled from Metrics That Matter Total Participants for Instructor Led Training 1,607 1,636 2,103 2,909 3,831 2,615 2,450 Total Participants for Online Facilitated Training Total Participants for WBT/Self-Study 16,461 10,118 9,219 9,348 18,438 10,394 12,330 Total Participants Trained (All) 18,213 11,791 11,592 12,343 22,337 13,111 14,898 Participant Hours for Instructor Led Training 14,754 12,053 18,847 22,613 28,076 25,679 20,337 Participant Hours for Online Facilitated Training Participant Hours for WBT/Self-Study Training 23,951 10,442 10,215 10,587 22,477 12,000 14,945 Total Participant Hours of Training 39,095 22,606 29,861 33,482 50,682 37,922 35,608 % of Nextel Learning via ILT 38% 53% 63% 68% 55% 68% 57% % of Nextel Learning via Online Facilitation 1% 0% 3% 1% 0% 1% 1% % of Nextel Learning via WBT/Self Study 61% 46% 34% 32% 44% 32% 42% Number of ILT/Interw ise Evaluations Received ,155 1,322 1, Number of WBT/Self-Study Evaluations Received ,583 1,332 1,173 Total Post-Event Evaluations Received 1,348 1,323 1,690 2,149 3,863 2,145 2,086 Evaluation Response Rate - ILT/Interw ise 80% 75% 68% 71% 77% 75% 48% 69% Evaluation Response Rate - WBT 20% 5% 9% 6% 9% 14% 13% 9% 2004 Page 38

39 Page 39 39

40 Contact Information For further information on the information contact: Jeffrey Berk Chief Operating Officer (phone) (fax) Page 40