Demonstrating Value for Money: An Introduction to the ROI Methodology

Transcription

1 Demonstrating Value for Money: An Introduction to the ROI Methodology COMISS Network Annual Forum January 13, 2019 Presented by Patti P. Phillips, Ph.D. President & CEO, ROI Institute, Inc. Session Objectives Upon completion of the session, participants will perceive the content as Relevant to their roles Important to their current success Immediately applicable New to their understanding of ROI During the session, participants will learn What ROI is and what it is not How to ensure alignment exists between chaplaincy efforts and organization and individual needs Why value reported through the process is credible

2 The Challenge and Opportunity In July 2017, an unlikely group convened on the campus of Birmingham-Southern College to learn how to show the value of what they do. Although this task is not all that unusual, the audience may surprise you. The United Methodist Endorsing Agency (UMEA), a division of the General Board of Higher Education and Ministry (GBHEM), hosted this Return on Investment (ROI) Workshop for 57 senior chaplains, spiritual care managers, and faith community leaders. Here s an excerpt of an article about the program published by The United Methodist Church.1 Over the past several years, a major paradigm shift has occurred in how organizational systems understand and value spiritual care. For many organizations, value and relevance are primarily centered on the bottom line. Although monetary value is a critical concern, it is the comparison of this value with the program/project costs that captures the attention of stakeholders translating into ROI. Show me the money is the familiar response from many business leaders and investors. Sometimes this is an appropriate response. At other times, it is misguided, especially when it comes to understanding the impact and value of chaplaincy and spiritual care. As Jeffrey Parkkila, senior chaplain at Westminster Retirement Community, Winter Park, Florida, stated, I have struggled to find the language to communicate with the corporate world the needs of ministry and the value of spiritual care. This ROI training gives me a platform to communicate our value. Realizing that measures not subject to monetary conversion are also important, if not critical to most programming and projects, participants learned that a balanced profile of success is required, which must include qualitative and quantitative data as well as financial and non-financial outcomes. John Callanan, senior chaplain with United Methodist Homes of the Greater New Jersey Conference reflected, My CEO has been pleading with me to demonstrate value, now I have a process to demonstrate the impact of pastoral care and bring the pastoral care department into greater accountability. Stephen Brinkley, senior chaplain of the Trauma Center at Orlando Regional Medical Center commented, Business language is a new language for me yet, my health system is challenging me to translate the value of ministry in a new way. Failure to do so on my part jeopardizes the future of chaplaincy. Jack and Patti Phillips engaged the workshop participants in learning a new language by embracing the ROI Methodology. The methodology offers a balanced approach to measurement that captures five levels of outcome data. When chaplains and spiritual care leaders engage the process, the ROI model provides alignment, connecting needs assessment with evaluation thereby empowering the translation of ministry s value and impact within organizational systems. 2

3 Invigorated by what she was learning, Linda Stetter, director of Spiritual Care, St. Mary Corwin Medical Center, Pueblo, Colorado, stated, I can now quantify my ministry. This methodology empowers Spiritual Care to not be perceived as a cost center but an organizational contribution center! This is great news! Jeffrey Uhler, chaplain at the Aurora Medical Center, Milwaukee, Wisconsin, added, Last evening I received a message from my supervisor about a conversation she was having with management pertaining to the addition of chaplaincy staff. Management s feedback to her, you ll need an ROI plan. Today, chaplaincy received good news we have learned a methodology to give management just what they asked for, the ability to demonstrate ROI. I am excited! Bruce Fenner, endorsing agent for The United Methodist Church and director of endorsement at UMEA later noted that the evaluations this workshop were the highest of any program ever offered by the endorsing agency for its constituents. He attributes this to the outstanding leadership, clarity of vision, practical instruction, and applicability of the material for this time of our lives. We were fortunate to have Jack and Patti Phillips bring their leadership. They are world-renowned experts in the field of measurement, noted Fenner. If we, as clergy working in specialized ministry, are to be relevant in this increasingly secular culture and workforce, there is a pressing need to learn a new language in ministry the language of business. Business leaders do not typically understand the pastoral care world, nor are they going to learn our language. Rather, we must become bilingual if spiritual care is to have value and impact in broader organizational systems. There is nothing better suited to get us on our way than what we experienced from the ROI Institute. Questions for Discussion Why is the topic of ROI important to advancing the practice of chaplaincy? What are your key concerns about putting ROI to work for your practice? 3

4 What is ROI? Benefits/Cost Ratio = Program Benefits Program Costs Net Program Benefits ROI = x 100 Program Costs ROI Calculation Example Program Benefits from Participants (1st year) = $750,000 Costs per Program (25 participants) = $425,000 BCR = = ROI = = x 100 = ROI Target Options Set the value as with other investments: i.e., 15% Set slightly above other investments: i.e., 25% Set at break-even: 0% Set at client expectations What value would be appropriate? What is an acceptable ROI in your organization? 4

5 Five Critical Success Factors Evaluation Framework Measurement and analytics begin with a framework. The framework serves as the basis for answering questions that matter to various stakeholders. If your actions inspire others to you are a leader. John Quincy Adams 5

6 Evaluation Framework and Key Questions Levels of Evaluation Level 1: Reaction, and Planned Action Key Questions Answered Was the program relevant to participants jobs and mission? Was the program important to participants job/mission success? Did the program provide new information? Do participants intend to use what they learned? Would participants recommend it to others? Is there room for improvement with facilitation, materials, and the environment? Level 2: Learning and Confidence Do participants know what they are supposed to do with what they learned? Do participants know how to apply what they learned? Are participants confident in their ability to apply what they learned? Level 3: Application and Implementation Did participants gain new knowledge? How effectively are participants applying what they learned? How frequently are they applying what they learned? If they are applying what they learned, what is supporting them? If they are not applying what they learned, why not? Level 4: Business Impact So what? To what extent does participant application of what they learned improve the measures the program was intended to improve? How did the program impact output, quality, costs, time, customer satisfaction, employee satisfaction, work habits? What were the consequences of participants application of knowledge and skills acquired during the program, process, intervention, change? How do we know it was the program that improved those measures? Level 5: ROI Do the monetary benefits of the improvement in business impact measure outweigh the cost of the program? 6

7 The Alignment Process Start Here V Model End Here Payoff Needs 5 ROI Objectives 5 ROI Business Needs 4 Impact Objectives 4 Impact Performance Needs 3 Application Objectives 3 Application Learning Needs 2 Learning Objectives 2 Learning Initial Analysis Preference Need 1 Reaction Objectives 1 Reaction Measurement and Evaluation Alignment Questions Business Alignment and Forecasting Project The ROI Process Model 7

8 The Alignment Process V Model Example Start Here End Here Payoff Needs 5 ROI Objectives 5 ROI Absenteeism is ROI of 25% Calculate ROI costing $10,000 monthly. Business Needs 4 Impact Objectives 4 Impact Unexpected absenteeism Reduce absenteeism to 5% Monitor absenteeism is 9% and growing; six months after course records for six months benchmark data is at 5% Job Performance Needs 3 Application Objectives 3 Application Discussions between team Counseling discussions conducted in Follow-up questionnaire to member and supervisor are not 95% of situations when an participants to check occurring when there is an unexpected absence occurs frequency of discussions unplanned absence. three months Initial Analysis Alignment Questions Learning Needs 2 Learning Objectives 2 Learning Deficiency in counseling/ discussion skills. Be able to demonstrate the use of counseling skills Skills practice session during program Preference Need 1 Reaction Objectives 1 Reaction One-day counseling skills Program receives favorable Reaction questionnaire at workshop must provide rating of 4 out of 5 on need the end of program usable necessary and relevant for program, relevance of the skills; facilitator-led; program, and the participants are supervisors Project Measurement and Evaluation Business Alignment and Forecasting The ROI Process Model 8

9 Matching Evaluation Levels with Objectives Instructions: For each objective listed below, indicate the level of evaluation at which the objective is aimed. Level 1: Reaction Level 2: Learning Level 3: Application Level 4: Business Impact Level 5: Return on Investment Objective Evaluation Level After completing this program or project, participants should: 1. Improve work group productivity by 20%. 2. Use eight out of 10 leadership competencies each week. 3. Reduce student loan defaults by 30% in one year. 4. Use conflict resolution skills in 90% of conflict situations. 5. Achieve a 2:1 benefit to cost ratio one year after the new leadership program is implemented. 6. Be able to describe the four elements of the scholarship program. 7. Increase employee job satisfaction index by 25% in three months. 8. Address complaints with the 5-step process in 95% of complaint situations. 9. Perceive the absenteeism control policy to be fair. 10. Achieve a leadership simulation score average of 75 out of a possible Improve two impact measures by using the leadership competencies with these terms. 12. Provide a 4 out of 5 rating on appropriateness of new ethics policy. 13. Decrease the time to recruit new employees from 35 days to 20 days. 14. Complete steps in action plans in three months. 15. Perceive leadership program as important to their success. 9

10 ROI Methodology Process Model 10

11 Standards Guiding Principles 1. When conducting a higher-level evaluation, collect data at lower levels. 2. When planning a higher-level evaluation, the previous level of evaluation is not required to be comprehensive. 3. When collecting and analyzing data, use only the most credible sources. 4. When analyzing data, select the most conservative alternatives for calculations. 5. Use at least one method to isolate the effects of the program or project. 6. If no improvement data are available for a population or from a specific source, assume that no improvement has occurred. 7. Adjust estimates of improvements for the potential error of the estimates. 8. Avoid use of extreme data items and unsupported claims when calculating ROI calculations. 9. Use only the first year of annual benefits in the ROI analysis of short-term solutions. 10. Fully load all costs of the solution, project, or program when analyzing ROI. 11. Intangible measures are defined as measures that are purposely not converted to monetary values. 12. Communicate the results of the ROI Methodology to all key stakeholders. The Complete Story Inputs$ Reac(on$ Learning$ Applica(on$ Isolate$the$Effects$of$the$Program$ Impact$ ROI$ Intangible$Benefits$ aaaaaaaaaaaaaaaaa aaaaaaaaaaaa 11

12 Challenges Collecting Data Types of Data Method Surveys ü ü ü Questionnaires ü ü ü ü Observation ü ü Interviews ü ü ü Focus Groups ü ü ü Tests/Quizzes ü Demonstrations ü Simulations ü Action Planning/Improvement Plans ü ü Performance Contracting ü ü Performance Monitoring ü Factors to Consider When selecting methods, consider: Time required for participants Time required for supervisors Costs of methods Amount of disruption Accuracy Utility Culture/Philosophy When selecting sources of data, consider: Participants Supervisors Direct reports Peer groups Internal staff External sources Organizational records When determining timing, consider: Availability of data Ideal time for behavior change/application Ideal time for business impact Convenience of data collection Constraints on data collection 12

13 Isolating Program Effects 1. Use of control groups 2. Trend line analysis 3. Forecasting methods 4. Participant s estimate 5. Management s estimate of impact (percent) 6. Use of experts/previous studies 7. Calculate/estimate the impact of other factors 8. Customer input Method 1 Best Practice Use 2 1. Comparison Group Analysis 35% 2. Trend/Forecasting Analysis 20% 3. Expert Estimation 50% 4. Other 20% 1 Listed in order of credibility 2 Percentages exceed 100% Converting Data to Money Profit/savings from output (standard value) Cost of quality Employee time as compensation (standard value) Historical costs/savings from records Expert input External studies Linking with other measures Participant estimation Estimation from staff Credibility Resources Needed Standard values High Low Records / Reports analysis High High Databases Moderate Moderate Expert estimation Low Low 13

14 Tabulating Program Costs Direct Indirect Program materials Needs assessment Facilitator costs Program development Facilities Participant time Travel Administrative overhead Evaluation Intangible Benefits are Important Complaints Commitment Conflicts Stress Teamwork Job satisfaction Customer service Engagement Case Study: Participant ROI As the Director of Mission Integration and Spiritual Care at St. Mary-Corwin Medical Center in southeastern Colorado, Rev. Linda Stetter charged herself with measuring and calculating the ROI of the Farm Stand Prescription Pantry program. This program was developed to address high levels of morbidity due to obesity and diabetes in Pueblo County. Rev. Stetter discussed the importance of the farm stand program to those who benefit from it: This is a population that has been so marginalized and so shortchanged that they really don t know what questions to ask, or they don t feel comfortable asking questions or telling us what their needs are. They needed to know that they could trust us, that we would provide a safe environment throughout the course of the farm stand season, and that they could confide in us without any adverse consequences. Rev. Stetter highlighted the case of one individual, identified as TM, as he and his family went through the farm stand prescription pantry program. Aside from providing the family of six with food, those managing the program also provided counseling and even helped TM avoid an unnecessary surgery that he had been told he needed. Based on one unnecessary surgery prevented, she calculated the ROI: ROI = $10,235 $1,277 $1,277 x 100 = 701.5% Although the calculated ROI for this program, 701.5%, is very high and based only on one prevented surgery, she was able to get attention and make the point that even more value comes from the priceless intangibles Rev. Stetter identified. TM learned to trust the people associated with the farm stand, reframed taking charity into paying it forward through volunteering efforts the family is taking on, and became receptive to counseling. 14

15 Healthcare, Inc. Sexual Harassment Prevention Policy Change Launched with Executive Meeting Target Group: All supervisors and managers (655) with subsequent meetings with all employees (6,844) Data Collection (3) Self-Assessment Questionnaire 6 months after program (3) Employee Survey (25% sample) 6 months after program (4) Complaint and Turnover Records 12 months after program Isolating the Effects of the Program Complaints Trend line Analysis Turnover Forecasting Converting Data to Monetary Values Complaints Historical costs and input from experts (internal EEO/AA staff) Turnover External studies within industry Program Costs Fully loaded to include needs assessment, development, coordination, participant salaries and benefits, and evaluation Total Costs = $277,987 Intangible Benefits Job Satisfaction, absenteeism, stress reduction, community image, and recruiting ROI Calculation Monetary benefits from complaint reduction Value of one internal complaint = $24,343 Annual improvement related to program = 14.8 complaints (prevented) Monetary benefits from turnover reduction Value of one turnover statistic = $20,887 Annual improvement related to program = 136 turnovers (prevented) Calculate the ROI: BCR = Total Benefits Program Costs = ROI = Total Benefits Program Costs Program Costs X 100 = For more detail, see Proving the Value of HR: ROI Case Studies, 2 nd Edition, Jack J. Phillips, and Patti P. Phillips, ROI Institute, Inc. Birimingham, AL, p.1. 15

16 16