Defining, Measuring, and Improving Impact

Transcription

1 Defining, Measuring, and Improving Impact with Performance Management Sarah Buek, MSW, LCSW Insight Partners Consulting Agenda 6 Step Thought Process & Tools for designing programs and defining and measuring impact Asking the Right Questions Helpful Tools Characteristics of Learning Cultures Strategies & Competencies for Building Learning Cultures Resources 2 1

2 The Case for Performance Management Organizations that measure to learn often find that they re able to do more for their beneficiaries with less money, they re better able to adapt their programs to changing circumstances faster and more effectively; they also make better resource allocation decisions. Bridgespan Group (2011). Measurement as Learning: What Nonprofit CEOs, Board Members and Philanthropists Need to Know to Keep Improving. 3 The Extremes Know-Nothing Optimism Over-claiming and under-measuring Good intentions Conventional Wisdom Mind-Numbing Over-Measurement Counting what s easy to count Meaningless Burdensome Hunter, David. (2009) Yes We Can!: Performance Management in Nonprofit Human Services. Social Solutions. 4 2

3 Performance Management... A dynamic process that is designed to better understand program operations, monitor outcomes, and ultimately, help nonprofits become high performing organizations which produce positive outcomes (Castillo, 2012). Three components: Regular and accurate collection of information, which is then... Shared in usable formats, which is then... Used to make changes to improve something (services, programs, conditions, decision making, etc.) Castillo, I. and Chablani, A. (2012, August 14) Introduction to Performance Management. Retrieved from 5 Define Measure Learn Improve 6 3

4 Performance Management Evaluation Ongoing Frequency At large intervals or one-time Internal Staff Conducted by External evaluators Ever-evolving Managing toward goals or benchmarks Many (staff at all levels, Stakeholders, Boards) Research Questions Objectives Consumers High Tactical Utility Low Low Strategic Utility High Pre-determined Answer particular question (more/less, increase/decrease, yes/no) Few (mostly leaders or funders) Adapted from: Castillo, I. and Chablani, A. (2012, August 14) Introduction to Performance Management. Retrieved from Hunter, David. (2009) Yes We Can!: Performance Management in Nonprofit Human Services. Retrieved from 7 Asking the Right Questions 8 4

5 Defining, Measuring, and Improving Impact How does change happen? Theory of Change What change are we aiming for? Outcomes What does success look like? How Can We Tell? Indicators & Measures How will we make change? Logic Model What s our target? Did we succeed? Performance Management Plans So What? Now What? Quality Improvement Plans 9 Theory of Change Trial & Error Data Happen Upon Success Theory-based program design Predict Success Steps: Identify ultimate goals Identify preconditions backwards mapping Identify assumptions The quickest path to high-performance programs comes when an organization takes a theory-based approach to program development. (Jones, 137). 10 5

6 Theory of Change What has to be true in order for... What has to be true in order for Defining, Measuring, and Improving Impact How does change happen? Theory of Change What change are we aiming for? Outcomes What does success look like? How Can We Tell? Indicators & Measures How will we make change? Logic Model What s our target? Did we succeed? Performance Management Plans So What? Now What? Quality Improvement Plans 12 6

7 Making It Measurable Outcome: What are we trying to accomplish? Indicator: What will that look/sound/feel like? Evaluation Method/Measure: How will we know or track? 13 Examples Outcome Indicator Measure Participants will increase employment skills. Participants improve selfmanagement skills. Participants experience improved mental health. Participants gain knowledge of financial management skills. Increased score on the Employment Readiness Checklist. Decreased incidents of verbal or physical aggression and impulsivity. Decreased symptoms of Depression. Increased score on pre/post test. Employment Readiness Checklist Progress Notes Beck Depression Inventory Pre/Post Test 14 7

8 Defining, Measuring, and Improving Impact How does change happen? Theory of Change What change are we aiming for? Outcomes What does success look like? How Can We Tell? Indicators & Measures How will we make change? Logic Model What s our target? Did we succeed? Performance Management Plans So What? Now What? Quality Improvement Plans 15 Logic Model Based upon Theory of Change Activities that create preconditions necessary for outcomes Describes How Components: Inputs Activities Outputs Outcomes Basically, a logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve. ~Kellogg Foundation 16 8

9 If, Then Inputs (Resources) Activities (Services) Outputs Outcomes (Results) In order to do that (activities), we must have this (inputs). In order to provide that (outputs), we must do this (activities). To achieve these results (outcomes), we must provide this (outputs). 17 Logic Models & Theories of Change 18 9

10 How does change happen? Theory of Change Defining, Measuring, and Improving Impact What change are we aiming for? Outcomes What does success look like? How Can We Tell? Indicators & Measures How will we make change? Logic Model What s our target? Did we succeed? Performance Management Plans So What? Now What? Quality Improvement Plans 19 Maps Measurement & Evaluation Strategies Aligned with Logic Model Outcomes Performance Management Plans Communication & Grant Management Tool Components: Outputs, Outcomes, Quality Goals Indicators & Measures Data Source Targets Frequency Audience 20 10

11 Type Goal Statement Indicator/ Measure Performance Outcome Managemen t Plan 75% of clients who complete 10 counseling sessions will decrease stress. Decreased score on Perceived Stress Scale Output # of clients enrolled # of clients completing intake Quality 90% of clients are satisfied Satisfaction score of 3.0 or higher Data Source Outcomes spreadsheet Frequency Quarterly Audience Board Funder A Funder C Database Monthly Program Mgr. Surveys Semi-annual Staff, Mgr, Board 21 Defining, Measuring, and Improving Impact How does change happen? Theory of Change What change are we aiming for? Outcomes What does success look like? How Can We Tell? Indicators & Measures How will we make change? Logic Model What s our target? Did we succeed? Performance Management Plans So What? Now What? Quality Improvement Plans 22 11

12 Quality Improvement Plans All-Encompassing Compliance Program Performance Quality Process Management Organizational Strength Process Ex. Plan, Do, Study, Act Strategies 23 Quality Improvement Strategies Compliance Did we follow the rules? Did we do what we said we d do? Audits, Record Reviews Program Performance Did we achieve our goals? Did we make a difference? Performance Management Plans Quality Were our services high quality? Are our stakeholders satisfied? Surveys, Focus Groups, Grievances, Suggestion Boxes Process Management Did we work efficiently? Did we implement with fidelity? Process monitoring, milestones/objectives, fidelity audits Organizational Strength Is the organization healthy? Staff satisfaction, Risk Management, Turnover 24 12

13 Use It Or Lose It The most damaging thing an organizational leader can do is to design a well-crafted performance and outcomes system and communicate the results but fail to use the data to take corrective action (Jones, 217). 25 How NPO s Use Data 89% to report to Board 84% to report to Funders, design/revise programs 76% to plan/revise strategies 66% to communicate to stakeholders 53 & 58% to make staffing and resource allocation decisions (respectively) 48% to share best practices or lessons learned Innovation Network. State of Evaluation 2012: Evaluation Practice and Capacity in the Nonprofit Sector

14 Using Performance Data: Feedback Loops Who needs to know? Staff Managers Board of Directors Funders/Donors Clients What should they know? Scorecards Dashboards Communications Marketing How do you respond? Responsible person/entity Quality Improvement Process PDCA Data-Driven Decision Making For additional info see: Using Outcome Information: Making Data Pay Off from the Urban Institute. 27 Back to the Very Beginning... Does data support our Theory of Change? Adjust program design? Adjust measures, indicators, targets? 28 14

15 29 Formal Policies, Systems, and Practices Informal Practices and Symbolic Actions Beliefs, Values, and Attitudes 30 15

16 Characteristics of Learning Cultures Empowers Employees Provides Organizational Support for Learning Demonstrates Value Encourages Reflection Enables Knowledge Sharing Builds Trust 31 Readiness for Organization al Learning and Evaluation Culture Leadership Systems & Structures Communication and Information Teams Evaluation 32 16

17 Strategies for Culture Change Leadership Competencies and Action Selecting and Socializing Employees Creating Culturally Consistent Rewards Aligning Practices, Structures & Policies with Desired Culture Jones, S.C. (2014) Impact & Excellence: Data-Driven Strategies for Aligning Mission, Culture and Performance in Nonprofit and Government Organizations. San Francisco: Jossey-Bass. 33 Leadership Commitment Without a leader who is committed to measurement as a top priority, articulates how it enhances impact, and identifies someone within the organization to lead the change, organizations will not overcome the natural reluctance among staff to embrace what seems like such an overwhelming enterprise. The Bridgespan Group. Building Capacity to Measure and Manage Performance. Matthew Forti & Kathleen Yazbak

18 Gleicher s Formula for Change Change = D x V x F >R D = dissatisfaction with the current state V = desired future vision F = concrete first steps R = natural resistance to change 35 Key Leadership Competencies Cultivating Organizational Vision and Purpose Building Partnership and Alliances Directing and Measuring Work Enabling Constructive Change Encouraging Dialogue Jones, S.C. (2014) Impact & Excellence: Data-Driven Strategies for Aligning Mission, Culture and Performance in Nonprofit and Government Organizations. San Francisco: Jossey-Bass

19 Resources 37 Sarah J. Buek, MSW, LCSW