IMPLEMENTING CONTINUOUS QUALITY IMPROVEMENT PROCESSES FOR SAFETY ORGANIZED PRACTICE (SOP): PRACTICE PROFILES

Transcription

1 IMPLEMENTING CONTINUOUS QUALITY IMPROVEMENT PROCESSES FOR SAFETY ORGANIZED PRACTICE (SOP): PRACTICE PROFILES National Human Services Training Evaluation Symposium May 22, 2014 Holly Bowers, PhD, Melanie Schindell and Susan Brooks University of California Davis 1

2 Project Contributors California Department of Social Services Casey Family Programs Children s Research Center University of California Davis, Northern California Training Academy Connected Families SOP Practice Profiles Development Workgroup Members Holly Bowers, Melanie Schindell, Sophia Chin, Mike Caslor, Peter Pecora, David Plassman, Heather Meitner, and Susan Brooks. Also field tested practice coaches, Nancy Goodman and Geri Wilson. 2

3 Purpose of Presentation Provide a brief overview of Safety Organized Practice; Discuss the development of the Practice Profiles; Explain pilot project Having supervisors who volunteered to utilize the tool; Use of the case file review tool Group Feedback How to create a win-win? (Benefit of use to supervisors beyond data collection for evaluation and research) Is there a statistical analysis that can be used to help inform the validity and increase reliability of the tools? 3

4 Background: Safety Organized Practice Advocacy/ Communication Evaluation Safety Organized Practice Engagement/ Involvement Assessment (Critical Thinking) Cultural Humility 4

5 Safety-Organized Practice GOOD WORKING RELATIONSHIPS Solution-focused conversations Strategies for talking with children CRITICAL THINKING Mapping Structured Decision Making (SDM) assessments ENHANCING SAFETY Harm and danger statements Well-formed goals Building safety networks Collaborative planning 5

6 Current Evaluation Efforts Pre/post in class surveys, six month follow up Over 1,000 practitioners have been trained since 2010 throughout California Practice leaders (coaches) feedback via focus group and on-line surveys Collecting learner feedback of external coaching experiences Use of Goal Attainment Rating Scales 6

7 Some Highlighted Survey Results January January

8 Completely Agree PARTICIPANTS' LEVEL OF SATISFACTION WITH THE TRAINING Neutral Completely Disagree 1.00 The training was well organized. The training provided what I needed to begin using an integrated approach. I learned new way to practice that I will begin using immediately. The trainers presented the information in ways that were clear and understandable. The traing met my expectations Northern (N=141) Bay Area (N=30) Central (N=169) 8

9 Know very well 5 KNOWLEDGE OF SOP PRACTICES AND TOOLS (N=338) Know enough to begin using Never head of Prior to Training After the Training 9

10 KNOWLEDGE OF RISK ASSESSMENT TOOLS Know very well Know enough to begin using Never head of 1 SDM Knowledge (N=305) CAT Knowledge (N=33) Prior to Training After the Training 10

11 Participants' Perceived Level of Integrating SOP into Everyday Work (N=343) Belief that Organization and Supervisor Support Integration of SOP 4.22 Belief that Risk Assessment Tools will be Integrated into Own Work 4.03 Belief that Safety Organized Practices and Tool will be Integrated into Own Work Completely Disagree Neutral Completely Agree 11

12 Northern California County Coaching Satisfaction (n=6 counties receiving coaching for SOP, responding in October 2013) Strongly Agree Uncertain Strongly Disagree 1.00 We typically received the assistance we needed from our coach to enhance the quality of case decisions. In my office, we are committed to participating in coaching. In my office, we are committed to the improvement of practice in addition to compliance. Supervisor(s) provides visible, ongoing support for implementing Safety Organized Practice (SOP) Manager(s) provides visible, ongoing support for implementing Safety Organized Practice. Coaching has improved staff s skills and/or SOP related behaviors. 12

13 Using Practice Profiles to Help with Transfer of Learning and Inform Continuous Quality Improvement Efforts Well-defined practice model includes: a philosophy, values and principles, core components, core activities associated with each core component practical assessments of fidelity (Ogden & Fixsen, 2014). 13

14 Why? Provide a fully operational practice model Facilitate development of effective training protocols, coaching strategies and fidelity assessments Promote continuous improvement strategies and datadriven decision making Increase the ability of the program or practice model to be replicated in new settings and contexts Refine organizational and system supports that facilitate consistent, effective practice Ensure outcomes can be accurately interpreted 14

15 What are Practice Profiles? 15

16 Practice Profiles Identifies critical components of a particular practice, program, role, etc. Defines a linear and gradual progression of skill acquisition (though the process may not be linear) Defines characteristics of skill rather than frequency Clearly articulates the practice(s): Emergent Practice Accomplished Practice Distinguished Practice 16 16

17 Purpose of Practice Profiles for SOP The SOP Practice Profiles were designed to outline characteristics of various skill levels rather than the frequency in which tasks associated with the practice are completed. To account for the gradual nature of skill development the practice level definitions have been written to showcase what practice would look like at the highest point of each level. Workers will generally make the most progress if then focus on a few key skill aspects at a time, and then reflect on their practice to select the next areas to improve. They are an attempt to describe a set of complex and dynamic skills. It is important to hold on to the fact that there is a multidimensional and cumulative relationship to these profiles

18 Fidelity How do we know we are using the SOP approach as intended? How do we know how we are doing in our SOP work as an agency? How do we know which areas of SOP need improvement? What are the critical observable elements of SOP? Practice Profiles Identify critical elements of SOP that are OBSERVED Identify critical elements that can be measured Used to help evaluate program implementation related to fidelity study questions Critical Elements Activities, materials, and skills that are observed Are discrete Are specific to SOP Consist of structural and Process Structural & Process Structural: What services are delivered Specific tools, specific behaviors Process: How services are delivered 18 18

19 SOP Practice Profiles To date there are six practice profiles Safety Planning Incorporating the Child/Youth Perspective Harm and Risk/Danger Statements Safety Networks Safety Mapping Safety Goals 19 19

20 Dialogue Structure for Rating the PP Purpose: The depth of knowledge behind the reason for why the practice is used. PRACTICE AREA: HARM AND DANGER STATEMENTS Emergent Practice Accomplished Practice Distinguished Practice Describes Harm and Danger Statements as key elements of Safety Mapping that are about what happened to the child that brought them to the attention of CPS and about what might happen in the future. Describes Harm and Danger Statements as key elements of Safety Mapping that have clear and specific statements about the harm to the child in the care of the parents. Describes the Harm Statement as having the following 3 distinct components: 1) Who reported ( or it was reported that ) 2) What happened to the child, where, when, etc. 3) The impact it had on the child. Describes the Danger Statement as having the following 3 distinct components: 1) CPS is worried that (or the SW and other professionals or other s in the family network are worried that ) 2) Behavior/action of the parents (what parents might do) 3) Possible impact on the child (what are we worried may happen to the child, or what may happen to the child again) Describes the development of Harm and Danger Statements as important to clearly identify the caregiver actions that have a harmful effect on the child so that everyone understands the worries about danger and the reason for CPS involvement. Describes Harm and Danger Statements as key elements of Safety Mapping that have the most meaning when they are developed in the Safety Mapping process with the family, network and other professionals to achieve a shared understanding of the worries about danger and the reason for CPS involvement with the family

21 PRACTICE AREA: HARM AND DANGER STATEMENTS Dialogue Structure for Rating the PP Emergent Practice Accomplished Practice Distinguished Practice Stakeholders: CPS actions to engage individuals or organizations with an investment in incorporating the child s perspective Develops Harm and Danger Statements as an individual activity or in consultation with a supervisor to identify harm and danger. Shares statements with caregivers and checks for understanding. Works with the family to identify stakeholders in the child s safety. Develops a plan to share the Harm and Danger Statements with the family, safety network and professionals. Everything in accomplished practice, plus: Ensures the voice of the child is clear, and when possible actively involves the child in the development of Harm and Danger Statements. Ensures the safety of the group by anticipating possible conflict among participants and setting up alternative ways of participating, i.e., telephone access

22 Champions of Safety Organized Practice Pilot Project: Supervisors 22

23 Design of Pilot Project Total of 20 Supervisors are participating across the state of California Each supervisor will complete ONE practice profile set with at least 2 workers for the month of May, June, and July 2014 Same practice profile rated three separate times Potential 40 ratings, follow up ratings Supervisor and worker have unique ID (may be linked to in class trainings) Participate in group webinar Instructional Video: Participate in monthly calls, County coaches of SOP asked to support supervisors Attend in person meeting end of July 23

24 Questions asked of Supervisors Demographics Number of people supervise How long has been a supervisor County work in Attended 3 day In Class Training (asked about supervisor and worker) 24

25 Worker Practice Questions Please think of the person you supervise and completed the practice profile ratings. Then rate this person on how YOU think this person currently practices their day to day work. The person I supervise respects service recipients as people worth doing business with. The person I supervise is consistent in his/her work with families. The person I supervise uses the most reliable and valid assessments available. The person I supervise maintains a focus on safety. The person I supervise pays attention to what's working within the family in the care and safety of children. 25

26 Written Feedback Were the written directions for the Practice Profile helpful? Do you have suggestions for ways the directions can be improved? Were the questions adequate for you to complete the form? Should anything be added or removed? Please provide feedback on the definitions of the practice areas Were there any that you think should be revised and if so in what way? Do you have any additional suggestions or comments about the SOP Practice Profiles? 26

27 Supervisor Characteristics Range in skill level and experience using Safety Organized Practice On a scale of 1-10, 1=beginner to 10=expert, average about a 5.5 for the group Some supervisors recently attended three day in class Foundational SOP Training, some attended 2 years ago Some supervisors have used the practice profiles previously (6 of 20), most have not 27

28 Discussion Questions How to create a win-win? (Benefit of use to supervisors beyond data collection for evaluation and research) Is there a statistical analysis that can be used to help inform the validity and increase reliability of the tools? 28

29 Thank you! 29

30 Selected References Durlak, J. A., & DuPre, E.P. (2008). Implementation Matters: A Review on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation. American Journal of Community Psychology, 41, Ogden, T., & Fixsen, D. L. (2014). Implementation science: A brief overview and a look ahead. Zeitschrift für Psychologie, 222, Turnell, A. and Edwards S. (1999). Signs of Safety. New York: Norton