Measuring the ROI of Online Learning Patti Phillips, Ph.D. patti@roiinstitute.net
Objectives Describe the difference between the benefits and the costs of online learning Calculate the ROI for online learning Align your online learning initiatives with the business
Given that online learning is defined as distance learning over the internet, what is your primary reason for offering it? (Choose the one best answer.) A. We do it to save time. B. We do it to avoid travel cost. C. We do it to reduce delivery costs. D. We do it because it is the best channel of delivery for the content.
Why the interest in measurement of online learning? The increasing cost of human capital Consequences of improper or ineffective practices Linkage of human capital to strategic initiatives Increased accountability of all functions Top executive requirement for contribution and human capital ROI
The ROI calculation is simple. BCR = ROI = Program Benefits Program Costs Net Program Benefits Program Costs X 100
Try it! BCR = $750,000 $425,000 $750,000 - $425,000 ROI = X 100 $425,000
What is your answer? A. BCR = 1.76; ROI = 76% B. BCR = 2.76; ROI = 176% C. BCR = 76; ROI = 176% D. BCR = 3.19; ROI = 219%
The ROI Calculation BCR = $750,000 $425,000 = 1.76:1 $750,000 - $425,000 ROI = X 100= 76% $425,000
What Makes a Good ROI? 1. Set the value at the same level as other investments 15% 2. Set slightly above other investments 25% 3. Set at break even 0% 4. Set at client expectations While we don t strive for a negative ROI, negative is not always bad!
Levels of Evaluation Measurement Focus Typical Measures 0. Inputs and Indicators The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing 1. Reaction & Perceived Value Reaction to the project or program, including the perceived value Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action 2. Learning & Confidence Learning to use the content and materials, including the confidence to use what was learned Skills, Knowledge, Capacity, Competencies, Confidence, Contacts 3. Application & Implementation Use of content and materials in the work environment, including progress with actual items and implementation Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use 4. Impact and Consequences The consequences of the use of the content and materials expressed as business impact measures Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement 5. ROI Comparison of monetary benefits from program to program costs Benefit-Cost Ratio (BCR), ROI%, Payback Period
The levels serve three purposes! Needs Evaluation Objectives
Business Alignment Start Here End Here Payoff Needs 5 ROI Objectives 5 ROI Business Needs 4 Impact Objectives 4 Impact Performance Needs 3 Application Objectives 3 Application Learning Needs 2 Learning Objectives 2 Learning Initial Analysis Evaluation Preference Needs 1 Reaction Objectives 1 Reaction Input Needs 0 Input Objectives 0 Input Business Alignment and Forecasting Program The ROI Methodology
Payoff Needs 5 ROI Objectives 5 End Here ROI Business Needs 4 Impact Objectives 4 Impact Job Performance Needs 3 Application Objectives 3 Application Initial Analysis Learning Needs 2 Learning Objectives 2 Learning Measurement and Evaluation Preference Needs 1 Reaction Objectives 1 Reaction Business Alignment and Forecasting Project The ROI Process Model
ROI Methodology Stage 1 Evaluation Planning Stage 2 Data Collection Develop Objectives Of Solution Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Collect Data After Solution Implementation Level 1 Level 3 Level 2 Level 4
How do you know it was your program? Stage 3 Data Analysis Capture Costs Of Solution Stage 4 Communicate Results Isolate the Effects of Solution Convert Data to Monetary Value Calculate the Return On Investment Generate Impact Study Report Level 5 Identify Intangibles Measures Intangible Benefits
Operating Standards 1. Report the complete story 2. Conserve resources 3. Use the most credible sources 4. Choose the most conservative alternatives 5. Isolate the effects of the program 6. No data no improvement 7. Adjust estimates for error 8. Throw out the extreme and unsupported 9. Use first year benefits for short-term programs 10.Include fully-loaded costs 11.Report intangible benefits 12.Communicate results to all stakeholders 16
Inputs Reaction Learning Application Isolate the Effects of the Program Impact ROI Intangible Benefits aaaaaaaaaaaaaaaaa aaaaaaaaaaaa
Measuring ROI of an online sales program Setting Large Midwestern financial services organization Needs Develop proficiency in newly defined competencies Capitalize on cross-selling products Reduce the time sales people spend in training Audience 3,000 sales associates 200 countries Remote working environment
Program Objectives Learning Objective: To develop proficiency with seven competencies as indicated by 85% on mastery assessment General sales skills Technical knowledge of the products Customer focus skills Prospecting Negotiating Managing resistance Gaining business results
Program Objectives Application Objectives: Three months after training sales associates score 80% on proficient use of knowledge and skills One week after attending training, sales associates contact 10 new prospects and conduct needs assessments on those prospects Thirty days after training the number of new sales proposals increase based on needs assessments
Program Objectives Impact Objectives: Six months after the the program the following business impact should include: Increase in new accounts Revenue contribution from new accounts* Increase in monthly amount of dollars earned Increase customer retention Revenue contribution from cross-selling products* ROI Objective: 25%
Results Level 1 Reaction Positive results as measured by overall rating on post program online survey (90% satisfied) Level 2 Learning 100% of sales associates scored a minimum of 87% on mastery check Level 3 Application Self-assessment data gathered through online questionnaire indicated sales associates were using knowledge and skills at the level of proficiency desired (80%) Sales associates were contacting 10 new customers per week New sales proposals generated within 30 days increased 22 percent as compared to proposals generated within 30 days of previous academy
Results Level 4 Business Impact 32 percent increase in new accounts opened 13% increase in overall annual revenue 8% increase in customer retention $1,729,000 in revenue from new accounts $13,500,000 revenue from cross-selling *Isolation technique estimates adjusted for error Level 5 ROI Monetary Benefits $1,729,000 converted to profit (30%) = $518,700 $13,500,000 converted to profit (30%) = $4,050,000 Program Costs $4,995,800 Cost Categories: Upfront costs Recurring technology costs Analysis Development Marketing Delivery Evaluation
Results Level 5 ROI ($518,700 + $4,050,000) - $4,995,800 X 100 $4,995,800 Intangible Benefits New accounts increased Customers are retained ROI = - 8.5% Training hours decreased from 105 hours to 49 hours per associate for a savings of 56 hours training per associate
Initial steps to develop your strategy Assess your readiness Identify stakeholders and data needs Determine the purpose of your evaluation practice Identify programs suitable for ROI Develop capability