Performance Improvement Through Results Based Accountability (RBA)

Transcription

1 Performance Improvement Through Results Based Accountability (RBA) Cynthia McKenna, LCSW Lindsey Boudreau Lisa Pawlik, MA Catholic Charities Archdiocese of Hartford

2 Today s Presentation Results Based Accountability 101 Results Based Accountability in CT Catholic Charities, Inc. Archdiocese of Hartford and RBA Implementation Results Based Accountability and the PQI Standard

3 Results Based Accountability 101

4 What is RBA? Results Based Accountability or RBA was created by Mark Friedman At its core RBA is a method of thinking and taking action together that is simple and common sense, that uses plain language, produces minimum paper and is actually useful to community members, managers, and decision makers.

5 Trying Hard Is Not Good Enough

6 Results Based Accountability can measure two levels of results: Population Accountability about the well-being of WHOLE POPULATIONS For Communities Cities Counties States - Nations Performance Accountability about the well-being of CUSTOMER POPULATIONS For Programs Agencies and Service Systems

7 Other key RBA concepts Use data driven, transparent decision making Consider the result you want and then work backwards to find the means to achieve that result.

8 From Ends to Means From Talk to Action Performance Population RESULT or OUTCOME INDICATOR or BENCHMARK PERFORMANCE MEASURE Consumer result = Ends Service delivery = Means ENDS MEANS

9 The Language Trap Too many terms. Too few definitions. Too little discipline. Benchmark Outcome Result Indicator Modifiers Measurable Core Urgent Qualitative Priority Programmatic Targeted Performance Incremental Strategic Systemic Goal Measure Target Objective

10 DEFINITIONS POPULATION RESULT or OUTCOME A condition of well being for children, adults, families or communities Children ready for school, safe communities INDICATOR or BENCHMARK A measure which helps quantify the achievement of a result Percent ready at K entry, unemployment rate PERFORMANCE PERFORMANCE MEASURE A measure of how well a program, agency or service system is working Three types: 1. How much did we do? 2. How well did we do it? 3. Is anyone better off = Customer Results

11 Deciding on a Common Language

12 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 12

13 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 13

14 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT INDICATOR 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 14

15 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT INDICATOR PERF. MEASURE 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 15

16 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT INDICATOR PERF. MEASURE RESULT 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 16

17 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT INDICATOR PERF. MEASURE RESULT INDICATOR 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 17

18 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT INDICATOR PERF. MEASURE RESULT INDICATOR RESULT 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 18

19 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT INDICATOR PERF. MEASURE RESULT INDICATOR RESULT INDICATOR 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 19

20 IS IT A RESULT, INDICATOR, OR PERFORMANCE MEASURE? RESULT INDICATOR PERF. MEASURE RESULT INDICATOR RESULT INDICATOR PERF. MEASURE 1. Safe Community 2. Crime Rate 3. Average Police Dept response time 4. A community without graffiti 5. % of surveyed buildings without graffiti 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training program who get living wage jobs 20

21 Population Accountability For Whole Populations in a Geographic Area

22 The 7 Population Accountability Questions 1. What are the quality of life conditions we want for the children, adults, and families who live in our community? 2. What would these conditions look like if we could see them? 3. How can we measure these conditions? 4. How are we doing on the most important of these measures? 5. Who are the partners that have a role to play in doing better? 6. What works to do better, including low cost and no cost ideas? 7. What do we propose to do?

23 Population Level Results for Children, Families and Communities A Working List Healthy Births Healthy Children and Adults Children Ready for School Children Succeeding in School Young People Staying Out of Trouble People in Safe, Stable, Affordable Housing Families with Adequate Income Safe and Supportive Communities

24 Performance Accountability For Services, Agencies and Service Systems

25 Program Performance Measures Quantity Quality Output Input Effect Effort How much service did we deliver? How much change/effect did we produce? How well did we deliver it? What quality of change/effect did we produce?

26 Program Performance Measure Quantity Quality Effort How much did we do? How well did we do it? Effect Is anybody better off? # Is anybody better off %

27 Program Performance Measures

28 Cost Program Performance Measures Quantity PQI Quality Efficiency, Admin overhead, Unit cost, Staffing ratios, Staff turnover Staff morale, Access, Waiting time, Waiting lists, Worker safety Effort Customer Satisfaction (quality service delivery & customer benefit) Effect Cost / Benefit ratio Return on investment Client results or client outcomes Benefit value Effectiveness Value added Productivity

29 Program Performance Measures Example: Drug Treatment Program How much did we do? Number of persons treated How well did we do it? Unit cost of treatment Is anyone better off? Number of clients off alcohol/drugs Percent of clients off alcohol/ drugs - At exit - 12 months post- exit

30 Program Performance Measures Example: Education How much did we do? How well did we do it? Number of Students Student-Teacher Ratio Is anyone better off? Number of High School Graduates Percent of High School Graduates

31 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs staff turnover rate # participants who got jobs % of children reading at grade level cost per unit of service # applications processed % patients who fully recover

32 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs Better Off staff turnover rate # participants who got jobs % of children reading at grade level cost per unit of service # applications processed % patients who fully recover

33 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs staff turnover rate How Well Better Off # participants who got jobs % of children reading at grade level cost per unit of service # applications processed % patients who fully recover

34 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs staff turnover rate How Well # participants who got jobs Better Off Better Off % of children reading at grade level cost per unit of service # applications processed % patients who fully recover

35 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs staff turnover rate How Well # participants who got jobs Better Off Better Off % of children reading at grade level cost per unit of service # applications processed % patients who fully recover Better Off

36 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs staff turnover rate How Well # participants who got jobs Better Off % of children reading at grade level cost per unit of service How Well # applications processed % patients who fully recover Better Off Better Off

37 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs staff turnover rate How Well # participants who got jobs Better Off % of children reading at grade level cost per unit of service How Well # applications processed How Much % patients who fully recover Better Off Better Off

38 How Much, How Well, and is anybody better off: What goes where? % participants who got jobs staff turnover rate How Well # participants who got jobs Better Off % of children reading at grade level cost per unit of service How Well # applications processed How Much % patients who fully recover Better Off Better Off Better Off

39 Choosing Measures and Beginning the Turn the Curve Process The 7 Performance Accountability Questions 1. Who are our customers? 2. How can we measure if our customers are better off? 3. How can we measure if we are delivering services well? 4. How are we doing on the most important of these measures? 5. Who are our partners that have a role to play in doing better? 6. What works to do better, including low cost and no cost ideas? 7. What do we propose to do?

40 Choosing Measures and the Data Development Agenda How much did we do? Measure 1 Measure 2 Measure 3 Measure 4 Measure 5 Measure 6 How well did we do it? Measure 7 Measure 8 Measure 9 Measure 10 Measure 11 Measure 12 DDA Measure 13 Measure 14 Measure 15 Measure 16 Measure 17 Measure 18 Is anyone better off? Measure 13 Measure 14 Measure 15 Measure 16 Measure 17 Measure 18 DDA DDA

41

42 Implementing Performance Measurement 1. The first purpose of performance measurement is to improve performance 2. Avoid the performance measurement equals punishment trap Create a healthy organizational environment Start small Build bottom up and top down simultaneously

43 How performance accountability impacts population accountability

44 The Leaking Roof

45

46 Questions?

47 RBA in Connecticut

48 How RBA Is Being Used in Connecticut Connecticut legislature State agencies Communities Non-profits Funders Connecticut RBA Practitioners Network

49 State Legislature Questions Raised by Appropriations Committee during the Budget Process 2004 National Conference of State Legislators RBA session followed up by a legislative briefing where Mark Friedman was invited to present to the full Appropriations Committee and other members of the legislature Working group formed to determine whether RBA could be adopted by the Connecticut General Assembly. The working group became the Appropriations RBA Subcommittee charged with using a results focus in the appropriations process & engaging other subcommittees /executive branch agencies.

50 State Legislature By late 2005 the legislature joined with the governor's office to pilot RBA for policy, budgeting and oversight decisions. Agencies were invited to volunteer to participate in the pilot Over the next several years the legislature studied the process and in 2009 expanded it to include all state agencies. As required by Public Law , in 2010 the Office of Program Review and Investigations (PRI) used the RBA framework to assess Family Preservation and Support Programs administered by DCF. Several subsequent bills require the use of RBA

51 Beyond the Legislature JUDICIAL BRANCH Under the active leadership of the executive director, the Judiciary s Court Support Services Division (CSSD) used RBA in their strategic planning and results management work as well as with external partners. RBA is now used in contracts and in the agency s internal processes. RBA performance measures are incorporated in all aspects of the division s work, and are demonstrating the effect of services on customer outcomes and not just activities.

52 Results Developed by Governor s Non Profit Health and Human Services Cabinet All Connecticut residents live in safe families and communities All Connecticut residents are economically secure All Connecticut residents are developmentally, physically, and mentally healthy across the life span All Connecticut residents who are elderly (65+) or have disabilities live engaged lives in supportive environments of their choosing All Connecticut residents succeed in education and are prepared for careers, citizenship, and life All children grow up in a stable environment, safe, healthy, & ready to succeed (from CT Children s Report Card)

53 Connecticut Early Childhood Result Statements Ready By Five, Fine By Nine Goal 1: All Children Healthy and Ready For School Success at Entry To Kindergarten Goal 2: All Children Healthy and Achieving School Success By Age 9 All Infants and Very Young Children Achieve Optimal Health and Development In Safe, Nurturing Families and Environments

54

55

56

57 CT Department of Children and Families

58 CT Department of Children and Families

59 Catholic Charities Inc. Archdiocese of Hartford (CCAOH) Organizational Structure RBA implementation

60

61 Serves 20,000 clients annually 35 locations across three counties 5 Major Program Areas with 75+ Programs 500 Employees $27 M Budget 65+ Funding Sources Federal, State, Local, Foundations, United Ways, Donors

62 The Department of Planning and Organizational Excellence Program Development was combined with Performance and Quality Improvement Outcomes and Accreditation strengthen the quality of services Improved ability to fund and sustain quality programs that demonstrate how clients are better off

63 RBA Implementation at CCAOH Senior Managers completed Training of Trainers by Mark Friedman Piloted RBA Report Cards Provided RBA Training to all managers and Supervisors Coaching and TA sessions with specific programs to select measures and implement Empowering Families

64 RBA Implementation at CCAOH Successes Programs focused on impacting Customer Results Consistent with Funders RBA expectations in contracts Increased buy in and understanding from direct service staff Challenges Change is always challenging Implementation across 60 programs and 34 sites RBA development and review with line staff in all programs Data and reporting from changing and new electronic records

65 CCAOH RBA Report Card

66 CCAOH RBA Report Card

67 CCAOH RBA Report Card

68 CCAOH RBA Report Card

69 CCAOH RBA Report Card

70 CCAOH RBA Report Card

71 RBA Staff Meeting Agenda 1. New data 2. New story behind the baselines 3. New partners 4. New information on what works 5. Changes to action plan

72 RBA and Supervision The 7 Performance Accountability Questions 1. Who are our customers? 2. How can we measure if our customers are better off? 3. How can we measure if we are delivering services well? 4. How are we doing on the most important of these measures? 5. Who are our partners that have a role to play in doing better? 6. What works to do better, including low-cost and no-cost ideas? 7. What do we propose to do?

73 Empowering Families

74 Empowering Families Brighter Futures Family Centers Brighter Futures Family Centers serve as community hubs. They are places where families can stay connected long after immediate crises or services have been delivered. The Centers emphasize family development and underscore the importance of families being their child s first teachers. They also serve as innovation hubs for new programs and services being developed by agencies and government. ~Richard Sussman, PhD, Director, Early Childhood Investments, Hartford Foundation for Public Giving

75 Empowering Families Community Cafés a series of structured small group conversations that bring parents together to discuss issues important to them. The goal is to directly engage parents in building the protective factors needed to promote healthy outcomes for children. Based on the Strengthening Families Protective Factors Framework, the careful structuring of the questions helps synthesize knowledge across a range of individual experience. Community Cafés also help those parents trained as hosts feel that they have an area of expertise and skill base for other leadership roles

76 Early Development Instrument (EDI)

77 Early Development Instrument (EDI)

78

79 Asylum Hill Neighborhood EDI Results January 2015 Percentage of children on track in each domain, by neighborhood: Communication Skills Emotional Maturity Language & Cognition Physical Health Social Competence Asylum Hill 50% 66% 45% 64% 58%

80 As a Result of the Community Cafés In the Language & Cognition domain, parents identified the literacy and numeracy sub domains as areas for action. Signature Dinner and a Story program has evolved. The program has become more purposeful and stories that incorporate numeracy have been highlighted. Literacy Pledge

81 Asylum Hill Family Center Family Survey Results Tell us about your neighborhood % who agree or strongly agree with the following statements Dec 2015 N=43 Mar 2016 N= 70 My neighborhood has safe places for children to play 49% 65% My neighborhood is a great place to raise a family 44% 64% I work with others to make my neighborhood a better place 33% 71% I am active in my child s school 47% 74% I feel welcome in my child s school 50% 74%

82 Asylum Hill Family Center Family Survey Results Tell us about your neighborhood % who responded frequently or always to the following statements Dec 2015 N=43 Mar 2016 N= 70 I have someone I can turn to for day to day emotional help with parenthood 46% 77% I have someone I feel comfortable talking to about personal problems 59% 72% I am flexible even when things in life don t go as planned 68% 83%

83 Asylum Hill Family Center Family Survey Results Let us know how we can help % who responded frequently or always to the following statements Dec 2015 N=43 Mar 2016 N= 70 I have enough money to take care of my family s basic needs (housing, 40% 74% food, clothing) I would know where to turn if my family needed food or housing 60% 83% If I needed help funding a job, I would know where to go for help 59% 80%

84 As a Result Intentional Introductions of Newly Enrolled Families Match Parent Ambassadors with Newly Enrolled Parents

85 Parent Partnerships Leadership and Empowerment Involving stakeholders and parents in performance and population change is essential (Dudley Street Boston, MA; Amani Neighborhood, Milwaukee, WI) Successful community change often relies on participation of community members (Parents in the case of Family Centers) Family Center core principles Parent Empowerment Embed neighborhood residents in design & implementation of FC operations Places parents in the Drivers Seat creating direction and ownership for change

86 Parent Leadership Beyond the Family Center Mind in the Making Trainer Dallas, TX HFPG Brighter Futures Replication Population Change Workgroup

87 RBA and the PQI Standard

88 RBA and PQI PQI 1: Culture of improvement RBA facilitates discussions about quality at every level of an organization from leadership to staff The 7 questions of performance accountability and RBA report cards allow programs and leadership to gather and review information on performance and make decisions based on that data

89 RBA and PQI PQI 2: Infrastructure RBA can be used by every program and department to integrate data into decision making RBA can help programs clearly identify areas for improvement RBA can help organizations identify overall trends. For example: Several clinical sites are struggling with staff turnover as evidenced by the How Well on report cards. The agency may need to review salaries or supervisory structures. Review of RBA report cards as part of PQI schedule of activities

90 RBA and PQI PQI 3: Roles and Responsibilities The basic questions of How Much, How Well, and Is Anybody Better Off guide supervisors and program directors to involve direct service providers in developing outcomes and indicators for their programs. Baseline data allows programs to clearly see their performance and consider ways to turn the curve RBA operates with the assumption that everybody can contribute to improvement if they are involved in reviewing program data and participate in decision making to Turn the Curve The 7 questions of performance accountability can be used in every supervisory relationship

91 RBA and PQI PQI 4: Performance and Outcomes Measures RBA involves partners (stakeholders) in data, performance targets and results Quality How Well Did We Do It? Impact Is Anybody Better Off? Data used to strengthen and build capacity, measure progress, evaluate and identify and assess risks

92 PQI 5: Case Record Review How Well measure RBA and PQI

93 RBA and PQI PQI 6: Analyzing and Reporting Information 7 performance accountability questions RBA report cards using performance dashboards

94 RBA and PQI PQI 7: Using Data Provides data to build capacity, improve programs, and have a positive impact on persons served Programs review RBA data together and collectively decide on actions to turn the curve RBA is a continuous cycle

95 If you do what you always did, You will get what you always got. Kenneth W. Jenkins President, Yonkers NY NAACP

96 References Friedman, M. Trying Hard Is Not Good Enough 2005 RBA 101 Powerpoint. Accessed at: materials/ Lee, A.F. Results Based Accountability (RBA) & Connecticut State Government. 2013: Results Based Public Policy in Action. Accessed at: Based Accountability RBA Connecticut State Government.pdf Hartford Foundation for Public Giving. EDI Report January Accessed at: report2015 final.pdf