Quality Assurance in VR

Transcription

1 Quality Assurance in VR Pathways to Performance: Making a Difference on Program Evaluation Summit, September 21, 2010

2 Overview of panel presentation 1. Susan Foley (ICI): Overview of the NIDRR RTAC on VR Program Management 2. Sue Stoddard (): Findings from research activities on VR QA initiatives 3. Allison Fleming (Michigan State): Research on QA in VR findings from MSU 4. Heike Boeltzig (ICI): Next steps for the RTAC

3 1. Overview of the NIDRR RTAC on VR Program Management

4 RTAC on VR Program Management Funded by NIDRR October 1, 2009 to September 30, 2014 Develop and implement a VR Program Management Model in 5 to 7 state VR agencies Components must include Quality Assurance, Strategic Planning, and Human Resources

5 Goals of the RTAC Create a management model that includes QA, strategic planning, and HR components Import, where possible, ideas from other organizations Evaluate if the use of the management model leads to better performance.

6 Major activities Literature Reviews of QA, SP, and HR Content analysis of monitoring reports Case studies of management practices Develop a management practices model through demonstration in five to seven state agencies Evaluation of model implementation Broad dissemination

7 Management Practices What do we mean by Management Practices? QA Example: Technical Craft of QA and Program Evaluation versus the Use of QA Information What information is needed to make decisions to steer the ship?

8 What do we mean by model? Models, Frameworks, Concepts, and Theories. We are looking to create a coherent blueprint that helps VR leaders navigate through the interrelationships of QA, HR, and strategic planning decisions Blueprint means flexible.

9 What do we mean by better performance? Ultimately to improve employment outcomes Chain of outcomes that logically lead to improved employment (i.e., time in status) Other goals: Employer of Choice, Partner of Choice, Provider of Choice

10 Relationship to QA and Program Evaluation CoP Highlighting use of QA information by managers. RTAC includes technical assistance activities that may be of interest. Upcoming activities and partnership opportunities

11 2. Findings from Research Activities on VR QA Initiatives

12 Overview Concept of continuous improvement QA in other related agencies Findings from RSA Monitoring Reports Next steps

13 VR and QA VR operates within Federal and State rules and regulations For a long time, QA = Case review San Diego Case Review and its impact Recent expansion of interest in QA State-wide Quality Management initiatives Monitoring reports and RSA feedback Summit and community of practice RTAC on management models including QA, HR and SP 2010 IRI

14 QA as part of overall VR management How does QA inform management decisions at all levels? How does QA inform strategic planning? How do we measure results? How do we ensure continuous improvement?

15 Continuous improvement Plan-do-check-act PDCA Continuous cycle Not a new idea: Shewhart in 1930s Underlying most QA and Quality Management models and approaches Developed in a manufacturing setting; but this thinking can work in any organization

16 Classic Shewhart PDCA cycle Continuous process: doing and learning Plan Do Act Check

17 As modified by US DEd Emphasis on measuring and analyzing

18 Plan to improve What is the overall strategic plan? How will we track success of the plan? How will we identify what is working? What can be improved? How are we assessing need? Planning for the changing environment? How does QA fit into the overall management? Is QA a part of the plan cycle? How does QA contribute to the quality of the planning and to the success of the plan?

19 Do Make a change with the intent to track the results How do we use available knowledge to support implementation? What do we want to accomplish? How can we measure the results? When appropriate, try new approach on a small scale first (demo, pilot).

20 Check Check whether the changes are achieving the desired result or not. Continuously check to ensure that you know results at all times, and to identify any unanticipated results. Use continuous QA measures and processes, plus ad hoc studies, program evaluation Measure performance to inform decisions

21 Act Adjust approach based on findings, as needed Make use of measures of performance and results Recommend continuation or expansion, or recommend modifications Involve all necessary stakeholders In the case of demos or small scale tests, transfer knowledge and approach.

22 What contributes to quality? Leadership committed to continuous improvement, quality and results Motivated and engaged workforce Informed customers: VR consumers Employers Referral sources and collaborators: Schools, rehab centers, CRPs, etc. Effective service delivery system

23 Influences: federal and state agency quality initiatives Federal: Standards and indicators, PART State government initiatives in Quality Management Baldrige awards Sterling awards/system Other performance or efficiency initiatives

24 Methods used in other QA systems Look at other related service organizations to identify the range of tools used in QA Consider utility of these tools as part of VR QA All 7 systems we reviewed relied in part on review of files, records and reports In addition, many other quantitative and qualitative information sources Many-faceted view of how things are going

25 QA information sources in related QA systems Interviews with management, staff and participants, family members, external stakeholders Direct observation and site visits Customer satisfaction surveys, exit interviews, followup Licensing and certification QA standards and performance indicators Grievances, complaints Focus groups Audits, including legislative audits

26 Findings from Monitoring Reports RSA s monitoring reports include discussions of QA practices and results Reports published on the RSA MIS Have reviewed these reports to identify themes and practices Have prepared a summary of findings and observations on the current status of QA in VR

27 VR QA strengths and challenges Source: RSA monitoring reports Review conducted by in late 2009 Qualitative analysis of 51 agencies with available public reports as of 11/09 Review identified strengths and challenges as cited in the areas of QA, HR and Strategic Planning

28 Some limits on this method Only covers about 2/3 of VR agencies (reports available by 11/09) Not comprehensive about everything that is in place, but was mentioned in the reports Not a shapshot of current status; covers visits in 2007, 2008, and 2009 Protocol and experience of the monitoring teams has changed over time

29 Case file review QA areas Case management system including data system CRPs/vendors Fiscal management Other

30 Identified QA strengths and challenges Strengths or promising practices Gaps or areas to strengthen For our purposes, indicates possibilities for strengthening agency capacities or tools available

31 Case file reviews: strengths Dedicated QA staff lead/conduct reviews 8 Process identifies staff training needs 6 Process involves local staff as reviewers 4 Process includes feedback loop 4 Overall well-defined process 3 Process identifies promising practices 2 Reviews includes all consumer case files 2

32 Case file review: challenges No formal process or instrument used 2 Only includes successful cases 2 Focuses only on compliance 2 Reviews conducted infrequently 1 Poor reliability of reviews 1

33 Case mgt. systems: strengths System facilitates caseload management 9 Produces required RSA reports 7 System includes edit checks 6 System includes ticklers/reminders of due dates 5 Integrates programmatic and fiscal data 5 Allows regular and ad hoc management reports 3 Frequently updated 2 Allows direct input from counselors 1

34 Case mgt. systems: challenges Antiquated-needs updating/replacing 15 Limited ability to generate reports 3 Lacks structure to help caseload management 3 Data errors identified in reports 3 Not fully accessible to persons with disabilities 2 System maintained by DSA 2 Lacks ability to depict data graphically 1 Lacks remote access 1

35 CRP/vendor mgt.: strengths Good overall relationship 8 Regular monitoring/oversight of CRPs 6 Use of outcome-based contracts 6 Able to track vendor performance and costs 4 Require monthly performance report from vendors 3 Use of state waivers to expedite service purchases 1 Implement revised contract model 1

36 CRP/vendor mgt.: challenges General lack of accountability 18 Limited number of vendors lack of consumer choice CRP staff require training 3 16 Lack performance-based contracts 2

37 Fiscal mgt.: strengths Overall effective fiscal management system Good coordination with DSA / Independent agency Integrated financial and programmatic data forecasting Fiscal reporting Low or reduced administrative costs 2

38 Fiscal mgt.: challenges Fiscal data tracking and/or controls need improvement Inaccurate reporting of fiscal data 9 Amount of carryover is excessive 8 Fiscal staff/data not coordinated with program staff/data Under-match of federal grant award 6 Resource allocation needs review 6 Need better forecasting/oos management Other (policies, costs, timeliness,3rd party agreements, sub-granting, etc.) 14

39 Other QA strengths Overall QA system is well defined 6 Performance on standards and indicators 1

40 Technical Assistance Fiscal management 35 Case management systems 15 Overall quality assurance systems 13 CRP management and relations 12 Use of data in decision making 6 Reporting of RSA 911 data 6 Implementation, Order of Selection (OOS) 5 Case record review systems 3

41 TA continued Establishing goals 2 Use of 3rd-party cooperative agreements 2 Standards and indicators 1

42 Take-aways Use of monitoring reports as one approach to understanding status of QA in VR agencies Process not designed as a research process Process has changed over time Some differences in what is expected as QA At the RSA level, the monitoring itself is QA Given some limitations, what have we learned?

43 Most mentioned QA themes Encouraged commitment to improvement, overall QA system CRP accountability Capacity of CRPs/vendors Fiscal management policy/practice Fiscal data tracking/controls Data systems Case management systems

44 Going forward Transfer tools and practices that strengthen QA Develop management practices that use QA information for program improvement Compliance is necessary but not sufficient How to think outside the box in planning? How does good QA support innovative thinking? How best assure that QA is agency-wide?

45 Use performance information for management decisions From practices To Uses To Improved results Source: GAO, MANAGING FOR RESULTS: Enhancing Agency Use of Performance Information for Management Decision-Making: GAO

46 Practices Demonstrating management commitment Aligning agency-wide goals, objectives, and measures Improving the usefulness of performance information Developing capacity to use performance information Communicating performance information frequently and effectively

47 From practices to uses Identify problems and take corrective actions Develop strategy and allocate resources Recognize and reward performance Identify and share effective approaches

48 From uses to results Continuous improvement Learning from the system Connecting performance measurement to decisions and actions Checking to see what works, what s on track Providing incentives and feedback

49 It s all about quality Quality is everybody s job Quality is not an optional bolt-on accessory... It is integral to all management planning and decisionmaking Michael Fox, Quality Assurance Management

50 3. Research on QA in VR Findings from MSU

51 Quality Assurance, Program Evaluation, and Rehabilitation Settings

52 Importance of QA for rehabilitation providers Increased levels of accountability Increased competition for funding Need for improved service delivery/know what works Bellini & Rumrill, 1999, Leahy et al., 2009

53 Context Quality assurance and program evaluation are essential elements of continuous improvement efforts in rehabilitation services QA: systematic process designed to identify, analyze and eliminate any variation (defects) in processes and outcomes Directed at processes, not people Leahy et al., 2009

54 External evaluator: CARF Access Convenience of location, satisfaction with timeliness of services Process Satisfaction with services, appropriateness to personal needs, fidelity (following protocol/policy/best practices), health and safety Outcomes Labor market participation, earnings (including Δ), inclusion, independence/self-sufficiency, esteem/self worth, & selfadvocacy Wilkerson, Shen, & Duhaime, 1998

55 Community rehab provider Development of the QA process Goal setting Staff roles Stakeholder involvement

56 External evaluator: RSA Outcomes-based RSA performance indicators Ex. Change in number of employment outcomes (improvement) Ex. Percentage employed (effectiveness) Ex. Competitively employed (effectiveness/quality) Ex. Significant disability (access) Ex. Earnings ratio (effectiveness/quality) Ex. Self-support (effectiveness/quality) Ex. Minority ratio (access)

57 VR agency Collaboration, long-standing relationship Staff involvement Focus on data/data integrity Plan-do-check-act

58 Key elements of QA: GAO Evaluation culture Data quality Analytic skills and experience Valued by leadership Utility, usefulness in informing policy and practice Direct, conceptual, or persuasive

59 4. Next steps for the RTAC

60 VR baseline data on QA Linkages with other NIDRR-funded VR projects: VR-RRTC National Survey of State VR Agencies scheduled for fall/winter 2010 Purpose to develop state-by-state maps of employment service constellation and VR s role within constellation Additional survey items on QA integrated to collect baseline data that complement information from other sources (e.g. monitoring reports, state plans, RSA data) Two additional waves of state VR surveying planned for 2011

61 Case studies of mgt. practices 5-8 case studies of VR agencies doing innovative work in at least two of the areas (QA, SP and HRM) Purpose, goals, and objectives of case studies In process of identifying and selecting potential practices and sites Sources (monitoring reports, state plans, RTAC Advisory Panel, other stakeholders incl. TACE Centers, etc.) Procedures (lit review, expert input, nomination process, etc.) Criteria for selecting practices and sites Case study design and implementation, fall/winter 2010

62 Mgt. practices laboratory Continuous updating of conceptual framework through integration of data from various research efforts Translating framework into action: in process of Developing a template for implementation Developing a matrix of sites for sampling (N=7 intervention) Refining sampling, outreach and recruitment procedures Utilization-focused research implementation through continuous engagement of RTAC Advisory Panels (VR staff, researchers)