Isolating Program Effects The Ultimate Link Between Your Programs and Results Patti Phillips, Ph.D. patti@roiinstitute.net
Objectives Describe how the isolation steps fits into the business alignment process Explain why isolating program effects ensures credible business impact and ROI Apply three most commonly used approaches to isolate the effects of the program What causes results? A. Your programs B. Other internal factors C. Other external factors D. I can t explain it E. All of the above
Levels of Evaluation Measurement Focus Typical Measures 0. Inputs and Indicators The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing 1. Reaction & Perceived Value Reaction to the project or program, including the perceived value Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action 2. Learning & Confidence Learning to use the content and materials, including the confidence to use what was learned Skills, Knowledge, Capacity, Competencies, Confidence, Contacts 3. Application & Implementation Use of content and materials in the work environment, including progress with actual items and implementation Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use 4. Impact and Consequences The consequences of the use of the content and materials expressed as business impact measures Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement 5. ROI Comparison of monetary benefits from program to program costs Benefit-Cost Ratio (BCR), ROI%, Payback Period
The ROI calculation is simple. BCR = ROI = Program Benefits Program Costs Net Program Benefits Program Costs X 100
Try it! BCR = $750,000 $425,000 $750,000 - $425,000 ROI = X 100 $425,000
The ROI Calculation BCR = $750,000 $425,000 = 1.76:1 $750,000 - $425,000 ROI = X 100= 76% $425,000
The levels serve three purposes! Needs Evaluation Objectives
Business Alignment Start Here End Here Payoff Needs 5 ROI Objectives 5 ROI Business Needs 4 Impact Objectives 4 Impact Performance Needs 3 Application Objectives 3 Application Learning Needs 2 Learning Objectives 2 Learning Initial Analysis Evaluation Preference Needs 1 Reaction Objectives 1 Reaction Input Needs 0 Input Objectives 0 Input Business Alignment and Forecasting Program The ROI Methodology
Evaluation Process Stage 1 Evaluation Planning Stage 2 Data Collection Develop Objectives Of Solution Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Collect Data After Solution Implementation Level 1 Level 3 Level 2 Level 4
How do you know it was your program? Stage 3 Data Analysis Capture Costs Of Solution Stage 4 Communicate Results Isolate the Effects of Solution Convert Data to Monetary Value Calculate the Return On Investment Generate Impact Study Report Level 5 Identify Intangibles Measures Intangible Benefits
Principles to Remember At least one method must be used to isolate the effects of the solution. When collecting and analyzing data, use only the most credible sources. When analyzing data, choose the most conservative alternative for calculations. If no data are available, it is assumed that little or no improvement has occurred. Estimates of improvements should be adjusted for the potential error of the estimate.
Methods to Isolate Program Effects Use of a control group arrangement Trend line analysis of performance data Use of forecasting methods of performance data Participant s estimate of program impact (percent) Supervisor s estimate of program impact (percent) Manager s estimate of program impact Use of expert/previous studies Calculate/estimate the impact of other factors Customer input
Financial Services, Inc. Organization: Financial Services (800 branches) Issue: Turnover Rate (48% - 63%) Solution: Comprehensive selection system Recruiting strategies Interviewing guidelines Evaluation guidelines Individual feedback Isolation Technique: Control Group 1/3 of branches are trained in new selection system 2/3 of branches are not trained in new selection system Post implementation data collection: 6 months
Control Group Design Control Group (2/3) M1 M2 Experimental Group (1/3) M1 Solution M2
Poll: What are the major problems? A. Length of time to evaluation B. Contamination C. Too many groups involved D. All of the above E. None of the above
Keys to successful control group implementation include: Match control and experimental groups as closely as possible Randomly select participants when possible Select a limited a number of specific criteria* Gain client agreement on criteria Avoid contamination Limit the number of participants Limit the time between the pre-program measures and post program measures Limit the number of people who know there is a performance comparison between the control and experimental groups Take advantage of pilot programs and naturally occurring control groups *This is a practical approach; the scientific approach requires some type of randomization that considers all variables.
What if we have no pre-program measure? Control Group Measurement Experimental Group Solution Measurement
Micro Electronics, Inc. Organization: Electronics Components Manufacturer Issue: Reject Rate Emphasis on quality has created downward trend in reject rates One work unit wanted to improve this rate Solution: Continuous Process Improvement Isolation Technique: Trend Line Analysis
Micro Electronics, Inc. 2% REJECT RATE 1% 1.85% Pre Program Six-Month Average CPI Program Conducted Projected Average Using Pre Data as a Base1.45%.7% Post Program Six-Month Average J F M A M J J A S O N D J MONTHS
What is the difference in reject rates from the program? A..5% B..78% C..75% D. 1.15%
Two conditions must be met for the trend line technique to successful: 1. The trend that has developed prior to the solution is expected to continue if the solution had not been implemented. 2. No other new influences entered the process after the solution was implemented.
National Bank Organization: Financial Services Issue: Variety of Product Lines Solution: Sales Culture Sales Training Incentives Management Reinforcement Isolation Technique: Estimation Data collected via questionnaire to branch managers Branch employees provided input during meeting
Four influencing factors Sales Training Incentives Management Reinforcement Market Fluctuations
During carefully organized meeting, branch manager: Described the task and the process. Explained why the information was needed and how it will be used. Had employees discuss the linkage between each factor and the specific output measure. Provided employees with any additional information needed to estimate the contribution of each factor.
Asked employees to identify any other factors that may have contributed to the increase. Obtained the actual estimate of the contribution of each factor. The total must be 100%. Several consensus-reaching tools were offered. Obtained the confidence level from each employee for the estimate for each factor (100%=certainty; 0%=no confidence). The values are averaged for each factor.
Results for One Branch Monthly increase in credit card accounts: 175 (fact) Contributing Factors Consensus Impact (%) Average Confidence (%) Sales Training 32% 83% Incentives 41% 87% Management Reinforcement 14% 62% Market Fluctuations 11% 91% Other 2% 91% 100%
Poll: How many credit card accounts are due to the sales training program? A. 46.48 B. 56 C. 71.75 D. 14.43
The calculation is: Monthly increase in credit card accounts: 175 (fact) Impact (%) Confidence (%) Sales Training 175 X 32% = 56 X 83% = 46.48 46.48 credit card accounts are due to the sales training.
Here is what happens: Impact (%) Est. Number Confidence (%) Uncertainty (%) 175 X 32% = 56 X 83% 17% The branch employees estimate that 56 new credit card accounts are due to sales training; but, they are only 83% confident in their estimate -- this means they are 17% uncertain. This allows for a margin of error of: 56 x 17% = 9.52 (+/-)
Given that there is a margin of error: 65.52 + 9.52 56 46.48-9.52
Given that there is a margin of error: 65.52 + 9.52 56 46.48-9.52
Poll: Why do we go with the low number? A. At least one method must be used to isolate the effects of the solution. B. When collecting and analyzing data, use only the most credible sources. C. When analyzing data, choose the most conservative alternative for calculations. D. If no data are available, it is assumed that little or no improvement has occurred. E. Estimates of improvements should be adjusted for the potential error of the estimate.
Keys to successful use of estimations: Explain the task up front. Describe the rules Explain the importance of the process Share the series of questions with participants at the outset of solution implementation then again during the evaluation process Focus on one impact measure at a time Share the first measure and show improvement Identify the different factors that have contributed to performance Discus the linkage between each factor and the measure Allocate the improvement due to the program Provide confidence estimates
Results are accepted based on credibility. Reputation of the source Biases Source of data Source of study Motives of researcher Personal bias of audience Methodology of the study Assumptions made in the analysis Realism of the outcome data Type of data Scope of analysis
Poll: Of the three techniques introduced today, which one(s) will work for you? A. Control Group B. Trend Line Analysis C. Estimations D. All of them, depending on the solution and measures E. None of them, what are my other options?
Let us hear from you! For a copy of the ROI Methodology process model, email your name and mailing address to: info@roiinstitute.net