Evaluating Organizational and Systems Change. National Child Welfare Evaluation Summit May th, 2009 Washington, DC

Transcription

1 Evaluating Organizational and Systems Change National Child Welfare Evaluation Summit May th, 2009 Washington, DC

2 Karen A. Blase,, Ph.D. Senior Scientist National Implementation Research Network Crystal Collins-Camargo Camargo,, MSW, Ph.D. Director, National Quality Improvement Center on the Privatization of Child Welfare Services Assistant Professor, University of Louisville, Kent School of Social Work Paul Vincent, MSW, Child Welfare Policy and Practice Group Raymond L. Crowel, Psy.D.. (Moderator) Vice President for Human Service Systems ICF International

3 1. What are the key differences between organizational change and systems change and how are they related? Systems change and organizational change are sometimes used interchangeably. Do they mean the same thing in human service systems? 2. What methods show the most promise in evaluating organizational/systems change? 3. What are the challenges inherent in the evaluation process? 4. What outcomes should be measured in organizational or systems change? How do you measure those outcomes? 5. How do you address the needs of other organizations in the CW agency s effort to promote and evaluate systems change? 6. How do we link systems change to outcomes to the outcomes expected by CFSR? Critical Questions

4 Evaluating Organizational and Systems Change: An Implementation Perspective National Child Welfare Evaluation Summit May 27, 2009 Washington, DC Karen A. Blase, PhD, Senior Scientist Dean L. Fixsen, PhD, Senior Scientist Frank Porter Graham Child Development Institute

5 "All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get." R. Spencer Darling Business Expert A Sobering Observation National Implementation Research Network 2009

6 Setting the Stage A context for organizational and systems change?...from what, to what? Definitions of organizational and systems change Cascading Logic Models to clarify evaluation methods and measures National Implementation Research Network 2009

7 Context.from what, to what? Organizational and Systems Change are in service to the new or improved future state The organization changes so that. The system needs to change so that National Implementation Research Network 2009

8 Pre-Requisite: Defining IT From what current state to what future state? The it must be operationalized whether it is: An Evidence-Based Practice or Program A Best Practice Initiative A Systems Change Initiative Operationalize Part of Speech: verb Definition: to define a concept or variable so that it can be measured or expressed quantitatively Webster's New Millennium Dictionary of English, Preview Edition (v 0.9.7) Copyright Lexico Publishing Group, LLC National Implementation Research Network 2009

9 Change in order to. Functional Organizational Change: Purposeful and iterative change in an individual agency s internal structures and procedures and in its external relationships in order to support, sustain, and renew effective services. Strategic Systems Change: Purposeful and iterative change among interdependent service, regulatory, funding, and decisionmaking bodies in order to create and maintain hospitable conditions for effective service delivery. National Implementation Research Network 2009

10 Organizations and Systems don t change people do (Jim Wotring) How do we identify and measure changes in people s behavior, knowledge, attitudes that relate to better outcomes for children and families? How do we measure the products they create to support the service or system change? One measure two purposes Independent variable or process measure at one level Dependent variable or outcome measure at the next level Cascading Logic Model National Implementation Research Network 2009

11 Cascading Logic Model Population Intervention Strategies and Measures (WHAT) Intervention Outcomes Young Children at Risk of Child Abuse Triple P (fidelity measures) Project Safe Care (fidelity measures) PCIT (fidelity measures) Reduced incidence of child abuse and neglect Improved safety at home Positive disciplinary practices National Implementation Research Network 2009

12 Cascading Logic Model Population Intervention Strategies and Measures (WHAT) Intervention Outcomes Young Children at Risk of Child Abuse Populations Front-line Practitioners and Case Managers Triple P (fidelity measures) Project Safe Care (fidelity measures) PCIT (fidelity measures) Organizational and Systems Change Strategies (HOW) Provision of skillful, timely training, coaching, performance assessments in supportive administrative environments (pre/post training scores, % of staff trained before delivering service, % of coaching services delivered on time, % of fidelity assessments done on time, changes in policies and procedures to facilitate) Reduced incidence of child abuse and neglect Improved safety at home Positive disciplinary practices Change-Related Outcomes Competent use of selected Interventions: Triple P (fidelity measures) Project Safe Care (fidelity measures) PCIT (fidelity measures) National Implementation Research Network 2009

13 Cascading Logic Model Population Intervention Strategies and Measures (WHAT) Intervention Outcomes Young Children at Risk of Child Abuse Populations Front-line Practitioners and Case Managers Triple P (fidelity measures) Project Safe Care (fidelity measures) PCIT (fidelity measures) Organizational and Systems Change Strategies (HOW) Provision of skillful, timely training, coaching, performance assessments in supportive administrative environments (pre/post training scores, % of staff trained before delivering service, % of coaching services delivered on time, % of fidelity assessments done on time) Reduced incidence of child abuse and neglect Improved safety at home Positive disciplinary practices Change-Related Outcomes Competent use of selected Interventions: Triple P (fidelity measures) Project Safe Care (fidelity measures) PCIT (fidelity measures) Personnel in Child Welfare Training Systems, Supervisors, Data System Managers, Agency Administrators Agreements with Program Developers Training and coaching for Trainers Release time for Supervisors to learn to coach Installation of fidelity monitoring systems Skillful, timely training, coaching, performance assessments in supportive administrative environments National Implementation Research Network 2009

14 Population Cascading Logic Model Intervention Strategies and Measures (WHAT) Intervention Outcomes Young Children at Risk of Child Abuse Populations Front-line Practitioners and Case Managers Triple P (fidelity measures) Project Safe Care (fidelity measures) PCIT (fidelity measures) Organizational and Systems Change Strategies (HOW) Provision of skillful, timely training, coaching, performance assessments in supportive administrative environments (pre/post training scores, % of staff trained before delivering service, % of coaching services delivered on time, % of fidelity assessments done on time) Reduced incidence of child abuse and neglect Improved safety at home Positive disciplinary practices Change-Related Outcomes Competent use of selected Interventions: Triple P (fidelity measures) Project Safe Care (fidelity measures) PCIT (fidelity measures) Personnel in Child Welfare Training Systems, Supervisors, Data System Managers, Agency Administrators Program and Agency Administrators, Child Welfare Agency Personnel at the State/County/Local level Agreements with Program Developers Training and coaching for Trainers Release time for Supervisors to learn to coach Installation of fidelity monitoring systems Common mission, vision developed Linked Implementation Teams developed Changes in Policies and Procedures Changes in Funding Streams support new functions and relationships Decision-Support Data Systems Collaboration among providers to organize services; pool resources for infrastructure Skillful, timely training, coaching, performance assessments in supportive administrative environments Relationships with Program Developers formalized Training and coaching for Trainers (pre/post) Release time for Supervisors to learn to coach Fidelity monitoring systems exist and are used Outcome measures exist and are used

15 Organization & System Change Requirements Alignment of Policies and Procedures with new ways of work Multiple Feedback Loops Policy to practice to policy Robust, Reliable, Effective Infrastructure Selection, Training, Coaching, Fidelity, Outcomes, Decision-support Data Systems Stage Appropriate Activity Leadership Technical and Adaptive Collaboration.so that Financing (planning, service, infrastructure) National Implementation Research Network 2009

16 The State Health Authority Yardstick (SHAY) Finnerty, M.T, Rapp,C.A., Bond, G.R., Lynde, D.W., Ganju, V., Goldman, H.H. (2009). The State Health Authority Yardstick, Journal of Community Mental Health, Springer Science & Business Media, LLC, March EBP Specific Systems Level Facilitating Factors at the State Level EBP Plan Financing Training Leadership Policy and Regulations Quality Improvement Stakeholders Some Interesting Measures

17 A mechanism is in place for collecting and using client outcome data characterized by the following: (Use boxes to indicate criteria met.) Note: Client outcomes must be appropriate for the EBP, e.g. Supported employment outcome is persons in competitive employment, and excludes prevocational work, transitional employment, and shelter workshops. If outcome measurement is variable among sites, consider typical site. 1) Outcome measures, or indicators are standardized statewide, AND the outcome measures have documented reliability/validity, or indicators are nationally developed/recognized 2) Client outcomes are measured every 6 months at a minimum 3) Client outcome data is used routinely to develop reports on agency performance 4) Client specific outcome data are given to programs and practitioners to support clinical decision making and treatment planning 5) Agency performance data are given to programs and used for purposes of quality improvement 6) Agency performance data are reviewed by the SMHA +/- local MHA 7) The SMHA routinely uses agency performance data for purposes of quality improvement; performance data trigger state action. Client outcome data is used as a mechanism for identification and response to high and low performers (e.g. recognition of high performers, or for low performers develop corrective action plan, training & consultation, or financial consequences, etc.). 8) The agency performance data is made public (e.g. website, published in newspaper, etc.) Scores: 1. 0 components components components components 5. All 8 components Quality Improvement: Client Outcomes Sample Question from the SHAY National Implementation Research Network 2009

18 More Methods and Measures See work by: Phyllis Panzano Implementation Climate Charles Glisson Organizational Culture and Climate National Implementation Research Network Mary Cramer ICE Internal Coalition Effectiveness Scale Collaboration Factors Inventory (Fieldstone Alliance) m?sku= National Implementation Research Network 2009

19 Change is Great!..You go first!

20 For More Information Karen A. Blase, Ph.D Dean L. Fixsen, Ph.D At the Frank Porter Graham Child Development Institute University of North Carolina Chapel Hill, NC National Implementation Research Network 2009

21 For More Information Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: tions/monograph/ To order the monograph go to: National Implementation Research Network 2009

22

23 Evaluating Organizational and Systems Change: Challenges and Practical Solutions Crystal Collins-Camargo, MSW PhD University of Louisville Kent School of Social Work National Quality Improvement Center on the Privatization of Child Welfare Services

24 Key Objectives for the QICs as Stated by the Children s Bureau: Promote the development of evidence-based knowledge about effective child welfare practices Ensure the dissemination of this information in a manner that informs and alters practice at the direct service level Five year multi-site research initiatives Lessons Learned from Two Quality Improvement Centers Multi-Site Research

25 Southern Regional QIC Impact of clinical casework supervision on staff retention, practice, organizational and client outcomes in four states Organizational change: implementation of new in supervisory and middle management practice Involved shift in organizational culture which was important to measure QIC on the Privatization of Child Welfare Services Effectiveness of innovative performance based contracting and quality assurance systems in promoting practice, organizational and client outcomes Systems change across the public/private partnership Involved development of shared vision and inclusive collaborative strategy which was important to measure Both Involve Organizational Change One Targeted Systems Change

26 Traditional research models focusing on direct causality may not take into account the reality of public child welfare which is subject to rampant change and complicated contextual influences (Solomon, 2002). Agency/system may be implementing multiple changes concurrently Pressure to rollout pilots pre-maturely, contaminate comparison sites through introducing promising practices Changes in leadership influence commitment Challenges in Evaluating Systems and Organizational Change in Child Welfare

27 Challenges Continued

28 Measure interaction between the intervention & its prevailing contextual condition as well as the norms, values and interrelationships which may influence the efficacy of the program (Pawson & Tilly, 1997, p 26). CW agency, and other system representatives as active partners in the design and conduct of the evaluation, and problem-solving related to research challenges Building value of evidence-based practice Cross-disciplinary writing/presentation teams Resist pressure to cherry pick intervention sites or focus on areas of greatest need Promising Approaches

29 Articulate the central role of research & relationship to client/systemic outcomes at every opportunity Tendency to focus on the intervention Build into reporting structure, meeting agenda, site visits Allow for collaborative decision-making on instrumentation, procedures, etc. Emphasis on collaborative relationship makes partners accountable to each other and the greater program Build in as much flexibility as you can in the participatory research process multiple data sources/collection points/methodologies as insurance Promising Approaches

30 Organizational/ systemic culture Collaborative process Shared vision Shared resources Communication Trust Fidelity to the intervention protocols Intervention drivers Training supervisory coaching Internal facilitative structures/processes data feedback loop Contextual factors impacting the intervention and it s outcomes Measurement in Addition to Intervention-Specific Outcomes

31 The Key to Linking Systems Change to CFSR Outcomes is in the Theory of Change Child Welfare Children/Families System Community PBC/QA Planning Implementation Evolution

32

33 Changing the Child Welfare Practice Culture: The QSR Paul Vincent, MSW, LCSW The Child Welfare Policy and Practice Group A Nonprofit Organization Committed to Improving Outcomes by Improving Practice The Child Welfare Policy and Practice Group

34 Create a Practice Framework/Model of Practice Practice the values, principles, relationships, approaches and techniques used at the system and casework practitioner level to enable children and families to achieve the goals of safety, stability, permanency and well-being. Framework a structure to hold together or support something; an underlying set of ideas: a set of ideas, principles, agreements or rules that provides the basis or outline for something intended to be more fully developed at a later stage. The Foundation of an Effective Practice Culture The Child Welfare Policy and Practice Group

35 Trusting relationships foster acceptance of help Families pursue plans they help design Affirming strengths builds confidence in taking the risks of change Assessments are more accurate when based on underlying needs Children in placement do better connected with home and neighborhood Core Practice Model Principles The Child Welfare Policy and Practice Group

36 Needs based, individualized plans are more likely to produce change Informal supports are essential to sustaining change Family team based decisions are most effective Coordination is most effective through team conferencing Services should be flexible Core Practice Model Principles The Child Welfare Policy and Practice Group

37 Record Audit Is there a current permanency plan? Was the CPS investigation completed on time? Does the child have an IEP? Did the mom complete parenting classes? Was there a mental health evaluation? Qualitative Review Is the child making progress toward permanency? Is the child safe? Is the child performing at grade level? Is mom now able to parent capably? Is the child functioning adequately at home and at school? Measuring Practice Quality The QSR The Child Welfare Policy and Practice Group

38 Record Audit Did the parent sign the permanency plan? Did the team meet about the plan? Is there a formal family assessment? Were services provided in a timely manner? Did the parent complete the case plan tasks? Qualitative Review Did the parent feel meaningfully involved in decision making? Does the team have the needed composition and function as a unified support? Does the team understand the underlying needs of the family? Were services individualized and matched to need? Is the parent prepared to provide adequate caregiving independent of agency supervision? Measuring Practice Quality The QSR The Child Welfare Policy and Practice Group

39 Safety Stability Progress Toward Permanency Living Arrangement Health Emotional/Behavioral Well-Being Learning and Development Family Functioning Measuring Child and Family Status The Child Welfare Policy and Practice Group

40 Engagement Team Formation and Functioning Coordination Assessment Planning Long-Term View Resource Availability Successful Transitions Tracking and Modification Measuring System Performance The Child Welfare Policy and Practice Group

41 Results The Child Welfare Policy and Practice Group

42 Evaluating Organizational and Systems Change National Child Welfare Evaluation Summit May th, 2009 Washington, DC