Using Program Data to Improve APS Performance

Transcription

1 Using Program Data to Improve APS Performance Or How I Learned to Be A Geek and Enjoy It Karl Urban Director of Performance and Policy Development Texas Adult Protective Services

2 What are the Objectives of This Presentation? Participants will learn about quantitative and qualitative measures of APS performance. Participants will learn how to use APS program data for employee accountability and performance improvement. Participants will learn about the technological support systems for using APS program data.

3 What Will You Get Out of This Presentation? This session will: provide a review and discussion of data for measuring APS performance; explain how employees and managers are encouraged and empowered to use data at the individual, unit and program level; and demonstrate the technology systems and tools for accessing and using data. Participants will not become geeks but they may just become better managers and decisionmakers.

4 Why Am I Qualified to Talk About This? I am a policy wonk by education and disposition My prior experience with performance management using data Texas Performance Based Planning and Budgeting System State Unit on Aging/AoA/GPRA My current job in Texas

5 Why Is This Topic Important for APS Programs? The GAO just recommended that the federal government explore collecting performance information from APS systems. Our experience in Texas suggests APS programs are a potentially data rich environment that can be used for accountability and program improvement.

6 What Are My Biases on Managing With Data? It is a means, not the ends. Data helps you to ask the right questions; rarely does it give definitive answers to policy or operational questions. The rewards of managing by data are usually worth the effort, but you need to pay attention to potential unintended consequences.

7 What is the Status of APS Programs Today? It was the best of times; it was the worst of times Excitement and a little apprehension around EJA and attendant national focus How do we help the federal government know what is important to measure? How do you actually do it across disparate state and county-based systems? Most state and local programs are hurting financially How do we use performance information to help do more with less? What else do you think is critical in the APS world these days?

8 Texas is really, really big, very culturally diverse, and an extremely red state. and Jerry Jones is actually from Arkansas

9

10 What Does the Texas APS Inhome Program Do? APS investigates allegations of abuse, neglect and exploitation of vulnerable adults in the community age 65 or older or people with disabilities age APS clients live in their own home, family homes or unlicensed congregate settings such as personal care homes or room and boards. APS conducted 82,802 in-home investigations in FY If needed, APS arranges for or pays for services to alleviate abuse and neglect, such as respite, and emergency assistance with food and shelter. APS arranged or paid for services in 42,926 cases in FY APS also makes referrals to law enforcement and to guardianship programs. Employs around over direct delivery staff with a total budget of almost $45 million APS is the safety net for the safety net in Texas 10

11 What Does It Mean to Be the Safety Net for the Safety Net? Sexual Abuse 0.1% Suicidal Threat 0.7% Emotional Abuse 4.4% Exploitation 2.4% Medical Neglect 18.2% Physical Neglect 61.8% Mental Health Neglect 9.7% Physical Abuse 2.6% FY 2010 Cases By Type of ANE

12 What Do Texas APS MH&IDD Investigations Do? APS investigates abuse, neglect, and exploitation of clients receiving services in the following stateoperated and/or contracted settings: State Supported Living Centers State Hospitals State Centers Community mental health/mental retardation centers Privately-operated Intermediate Care Facilities for Persons with Mental Retardation Mental retardation waiver providers APS completed 9,922 MH&IDD related investigations in FY 10. Employs about 150 staff with a total budget slightly over $8 million 12

13 What Does the APS Program Look Like from a Metric Perspective?

14 Department of Family and Protective Services Adult Protective Services 14

15 Department of Family and Protective Services Adult Protective Services 15

16 Department of Family and Protective Services Adult Protective Services 16

17 Department of Family and Protective Services Adult Protective Services 17

18 Department of Family and Protective Services Adult Protective Services 18

19 Department of Family and Protective Services Adult Protective Services 19

20 Department of Family and Protective Services Adult Protective Services 20

21 Department of Family and Protective Services Adult Protective Services 21

22 Texas has a robust system of using qualitative (case reading) and quantitative (performance) data for employee accountability and program improvement. So?

23 What is Qualitative Data? Based on case reading by 7 quality assurance specialists 2 (closed) cases per worker per month Scored against standards based on policy Examples: Was the case initiated in accordance with policy? Was the client assessment tool completed appropriately? Did the investigator collect all the necessary evidence? Rolls up into 3 scales Investigation: how good was the investigation? Outcome: was the client protected and services delivered? Process: did we cross our Is and dot our Ts? Rolls up into worker, unit, region, statewide data

24 What is Quantitative Data? Based on documentation in case management system by workers Extensive system of available reports from MRS (Management Reporting System) Examples: Timeliness of documentation, face-to-face contacts Case durations There are multiple levels of reports: Twelve month summaries that are used for evaluation purposes and Monthly reports that provide a case detail list for the months summarized on the twelve timeliness reports There are also quarterly, weekly and even daily reports

25 How Does APS Use Program Data? Provide data to inform employee and program accountability. Provide data for managing and enhancing the program. Provide data to assess and enhance quality of casework and investigations.

26 How Does APS Use Data for Employee Accountability? Provide data to inform employee and program accountability. Achieve client-centered outcomes (intake, investigations, risk assessment, delivery of services). Inform job performance standards. Establish method of accountability, including awareness and procedures for monitoring. Provide information for conducting periodic employee performance reviews.

27 Task 1: Case Initiation (Quantitative) Task 2: Investigation (Quantitative) Task 4: Investigation (Qualitative) Task 6: Service Delivery (Qualitative) Task 7: Process Compliance (Qualitative) Task Performance Standard Data Source Caseworker will initiate each case within 24 hours. Caseworker will make face-to-face contacts within policy timeframes for the priority. Caseworker will fully investigate allegations and all factors that present a threat to the client's health or safety. Caseworker will work to alleviate the client's state of abuse/ neglect/ exploitation and remove threats to client's health and safety by designing and implementing, a service plan that addresses problems identified during the investigation. Caseworker will meet deadlines, perform consultations and notifications, and document the casework according to policy. Average for Timeliness in INV Stage Closed Investigations - 24 Contact during the review period (5) Distinguished = Above 98% (4) Commendable = 96%-98% (3) Competent = 94%-95% (2) Needs Improvement = 92%-93% (1) Needs Major Improvement = Below 92% Average for Timeliness in INV Stage Closed Investigations - Face to Face Contact during the review period (5) Distinguished = Above 94% (4) 92%-94% = Commendable (3) Competent = 89%-91% (2) Needs Improvement = 80%-88% (1) Needs Major Improvement = Below 80% Investigation Scale average for the sample of cases read during the review period. (5) Distinguished = Above 93% (4) Commendable = 87%-93% (3) Competent = 80%-86% (2) Needs Improvement = 73%-79% (1) Needs Major Improvement = Below 73% Client Outcome Scale average for the sample of cases read during the review period. (5) Distinguished = Above 93% (4) Commendable = 87%-93% (3) Competent = 80%-86% (2) Needs Improvement = 73%-79% (1) Needs Major Improvement = Below 73% Process Compliance Scale average for the sample of cases read during the review period. (5) Distinguished = Above 93% (4) Commendable = 87%-93% (3) Competent = 80%-86% (2) Needs Improvement = 73%-79% (1) Needs Major Improvement = Below 73% (See Data Warehouse Report -> APS -> Employee Development -> INV Timeliness in INV Stages Closed (12 Month) - INV APS 11 % 24 Contact Met) (See Data Warehouse Report -> APS -> Employee Development -> INV Timeliness in INV Stages Closed (12 Month) - INV APS 11 % FTF Contact Met) (See APS Case Reading System -> Reports -> Three Scales Report -> Investigation Scale Investigation Scale Score - %) (See APS Case Reading System -> Reports -> Three Scales Report -> Client Outcome Scale Client Outcome Scale Score - %) (See APS Case Reading System -> Reports -> Three Scales Report -> Process Compliance Scale Process Compliance Scale Score - %)

28 How Does APS Use Data for Managing and Enhancing the Program? Systemic/periodic data collection (management reporting system/tools). Enhance employee performance. Measure the operationalization of certain policies. Regional/unit/worker development.

29 FY 2010 In-Home Program Performance Dashboard Region Investigation Outcome Process Compliance *Case Initiation *FTF Contact *Monthly Status Contact Doc. Time. **** Inv Duration Mean SD Duration Mean Sum INV + SD Duration Validation Rate Supervisor Rejection Rate Rapid Closure Rate % Inv Progress Svc 1 94% 98% 93% 97% 95% 91% 87% % 13% 6% 73% % 99% 96% 97% 97% 92% 96% % 6% 10% 56% % 94% 90% 95% 89% 87% 87% % 13% 15% 34% % 96% 92% 97% 95% 90% 86% % 3% 9% 61% % 94% 91% 94% 90% 80% 76% % 8% 11% 58% % 93% 89% 96% 91% 77% 83% % 8% 15% 49% % 96% 93% 94% 95% 85% 90% % 13% 14% 37% % 87% 84% 90% 87% 63% 70% % 5% 12% 47% % 98% 95% 97% 95% 89% 93% % 12% 9% 61% % 92% 88% 91% 92% 84% 79% % 11% 11% 48% % 96% 94% 98% 95% 91% 87% % 6% 11% 53% 37 State 91% 94% 91% 95% 92% 76% 85% % 9% 12% 49% 36 *** Caseload Aver Daily State FY10 91% 94% 91% 95% 92% 76% 85% % 9% 12% 49% 36 State FY09 91% 92% 88% 95% 91% 71% 80% % 11% 12% 52% 32 State FY08 93% 91% 86% 94% 90% 70% NA % 11% 11% 51% 30 State FY07 91% 89% 85% 94% 87% 58% NA % 11%** 12% 51% 36 State FY06 90% 90% 88% 95% 84% 52% NA % 12% 52% 51 Casereading Database HHSC Data Data Warehouse

30 In-home Performance Summary: Region 3 Compared to State Measure Number of Intakes Region 13,605 13,016 14,307 15,817 18,888 State 82,029 77,238 83,605 89, ,401 Average Investigation Duration (Days) Region State Averaged Service Delivery Duration (Days) Region State Timeliness of Documentation Rate (%) Region NA NA 79% 84% 87% State NA NA 83% 83% 84% Average % of Investigations Pending 60+ (Days) Region State Rapid Closure Rate Region 14% 15% 14% 14% 15% State 11% 12% 12% 12% 12% Monthly Status Contacts Rate Region 47% 55% 64% 72% 87% State 48% 58% 70% 71% 82% Face-to-Face Initiation Rate Region 80% 84% 87% 87% 89% State 95% 94% 94% 95% 92% Overall Case Reading Scores Investigation 3 Scale Region 88% 89% 93% 90% 90% State 90% 91% 93% 91% 91% Outcome 3 Scale Region 89% 87% 90% 91% 94% State 90% 89% 91% 92% 94%

31 Monthly Summary of Pending APS State Supported Living Center Investigations DATE Under 10 Days 10 Days or More 10+ Days with Extension 10+ Days without Extension Percent 10+ Days Percent 10+ Days without Extension 1-May-2011 Sunday 2-May % 2.22% 3-May % 0.00% 4-May % 2.08% 5-May % 2.13% 6-May % 0.00%

32 How does APS Use Data to Assess and Enhance Quality of Casework and Investigations? Appropriate notifications to law enforcement and/or other entities. Capture critical components of casework. Establish clear investigation performance standards. Ensure compliance with documentation requirements. Establish process for reviewing staff decisions. Reporting to external and/or internal Stakeholders.

33 Where Does All This Data Come From? The intranet, of course Case reading database MRS data warehouse Accessible to: EVERYONE!

34

35

36

37

38

39

40

41

42

43

44

45 For Discussion: What are the consequences - positive and negative of using data to drive accountability and performance?

46 What are the consequences - positive and negative - of using data to drive accountability and performance? Provide data to inform employee and program accountability Objective, 3 rd party, consistent, measures compliance with the most important policy Valuable resource for conversations between supervisors and employees about employee s development When used in evaluations, QA data is perceived as punitive, not constructive Case reading sample size is small at employee level What should determine employee performance: subjective, inconsistent supervisors or biased, inconsistent case readers?

47 What are the consequences - positive and negative - of using data to drive accountability and performance? Provide data for managing and enhancing the program What gets measured gets moved All management cares about is the numbers

48 What are the consequences - positive and negative - of using data to drive accountability and performance? Provide data to assess and enhance quality of casework and investigations. Zero in on cause and effect by disaggregating and examining Track and trend key issues Data must be accurate and reliable ( faceplate game )

49 Karl Urban Director of Performance and Policy Development Texas Adult Protective Services