A step by step process for the implementation of the Standardized Program Evaluation Protocol (SPEP) in Pennsylvania

Transcription

1 PA Juvenile Justice System Enhancement Strategy (JJSES) Stage 3: Behavioral Change SPEP Task Matrix A step by step process for the implementation of the Standardized Program Evaluation Protocol (SPEP) in Pennsylvania Developed by the EPISCenter at PSU, in partnership with the Juvenile Court Judges Commission and the PA Council of Chief Juvenile Probation Officers The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Bennett Pierce Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the PA Department of Human Services. This resource was developed by the EPISCenter through PCCD grant VP-ST

2 The SPEP Task Matrix is intended to provide information to county juvenile probation departments on the process for SPEP implementation. Each objective is necessary (or strongly recommended) as counties consider the evaluation of services impacting recidivism reduction for juvenile justice involved youth. Under each objective listed in the steps below are tasks needed to complete the objective, person/agency responsible for task completion, timeframe for task completion, resources necessary, and notes intended to provide information helpful in the completion of each task. This document is not designed to replace training on the SPEP; it is a to do list to ensure the reader has an understanding of the tasks associated with a successful implementation of the Standardized Program Evaluation Protocol. Steps for Implementation of the SPEP in a County Juvenile Probation Department Readiness and Stakeholder Engagement Page 3 Train juvenile court personnel on evidence based practices [1] Train county juvenile probation officer to become SPEP Specialist [2] Develop a Continuum of Services (list of contracted services and programs) [2] Communication and outreach: Key Leader Orientation and planning [3, 4] Kickoff meeting: Community Stakeholder Orientation and information session(s) [5] Collect county specific data [6] The SPEP Process (ongoing) Page 7 Prioritize services/programs for going through the SPEP process [7] Prepare provider for the SPEP process [8, 9] Service categorization & quality of service interviews [10] Compile interview findings, complete program profile [11] Duration, dosage, and risk data collection and analysis [12] Develop and deliver feedback report, discuss implications for performance improvement [13, 14, 15] System Improvement Page 12 Service provider and juvenile probation collectively determine improvement priorities [16] Develop performance improvement plan [16] Identify needed resources, obtain approvals, implement plan [17] Quarterly contact for updates and technical assistance [17] SPEP Re-assessment [18] 2

3 Readiness and Stakeholder Engagement 3

4 Objective: 1 Train Juvenile Court Personnel on Evidenced-Based Programs and Practices Identify the trainer(s) that will be involved in the Prior to a commitment to Training at location to be determined introduction of evidence based programs and practices to all or any juvenile court personnel that has yet to receive the training through the JJSES implementation. participate in SPEP Ensure that the Youth Level of Service Inventory (YLS) risk assessment is utilized and the results are shared with service providers. Complete organizational readiness assessment as part of JJSES Stage One activities. Objective: 2 Train Juvenile Probation Officer(s) in County Participating in the SPEP This training is for those learning to SPEP community-based and residential services. Identify the juvenile probation officer (JPO)/representative(s) that will be involved in the introduction and implementation of the SPEP process. Establish timeframe for key leader and kickoff meetings. Schedule the SPEP training for the JPO and other identified staff (if appropriate). This includes orientation to the research and how to administer SPEP, SPEP interview observation, cofacilitating a SPEP interview, facilitating a SPEP interview, understand and facilitate SPEP score calculation and how to write a feedback report. Identify and schedule the SPEP with a service provider for JPO observation/shadowing. Create the Continuum of Services where possible (this is a list of all contracted service providers and types of services offered). The continuum helps to identify which services should be prioritized for the SPEP process; it also assists with determining which service providers are utilized most frequently by the county juvenile probation department weeks after a commitment is made to participate in SPEP Training at location to be determined 2014 SPEP User's Guide Dr. Lipsey s research, SPEP Score Template SPEP Service Type Fact Sheets, Quality of Service Assessment EPISCenter website, Performance Improvement Guide and Template JPO administrators should communicate with program administrators to determine if provider is willing and able to participate in the SPEP process. For more information on this refer to Objective 7. 4

5 Objective: 3 Develop a Rollout Communication Plan The plan should be a guide for SPEP rollout. The intent is to educate leaders and stakeholders on the SPEP process and includes training, the dissemination of written materials, and the development of the plan for meetings of key leaders and local stakeholders. The communication plan should be tailored to suit the needs of each county participating. Develop planning committee (includes stakeholders representing all parties impacted by the SPEP process). Meeting to discuss process for rollout (include providers in this process who have gone through SPEP for either presentation or consultation). Establish timeframe for key leader orientation (KLO) and kickoff meetings Progress updates should occur at pre-determined intervals among those responsible for the roll-out. This cannot be overemphasized. Objective: weeks after a commitment is made to participate in SPEP Copies of documents off of the EPISCenter website related to SPEP (available under the Juvenile Justice tab) Sample PowerPoints from past key leader and kickoff presentations SPEP webinars (EPISCenter website) Provide a copy of the Task Matrix Prepare for and hold the Key Leader Orientation (KLO) This is the initial meeting of local stakeholders that explains what all involved (leadership included) need to know and understand about the SPEP process. The intent is to increase understanding that the process is collaborative and focuses on performance improvement. This objective also serves as a meeting in which stakeholders can begin to prepare for the SPEP Kickoff Meeting. Schedule meeting (time and location should be suitable to stakeholders) Establish meeting structure (agenda, PowerPoint, Within 10 business days of initial communication plan meeting with juvenile probation office personnel Meeting to review structure of Kickoff & KLO meetings Discuss stakeholders; provide copies of sample agendas for *EPISCenter staff will develop standard packets meeting objectives) KLO meetings, PowerPoint examples for KLO meetings, and Identify presenters for the KLO meeting other relevant documents*: SPEP User's Guide Identify local SPEP leadership or stakeholders to 4-6 weeks following the invitation Dr. Lipsey s research, SPEP Score Template, be invited to the KLO meeting SPEP Service Type Fact Sheets, Quality of Service Assessment (JPO, Quality Assurance Director, Deputy Chief JPO, Data collection template Supervisors, JJSES Coordinator, etc.) EPISCenter website, Performance Improvement Guide Deliver a formal KLO presentation (if preferred); and Template Include question and answer session KLO sign in sheets and evaluation forms Begin preparation for the Kickoff Meeting Identify role of those involved in Kickoff Meeting (who will present, what will be presented, time frames) Determine venue and desired structure for kickoff meeting Two half day meetings (same information, different attendees) or one larger meeting for all attendees Review continuum and identify initial service provider to participate in the SPEP process.???? As part of KLO Agenda and copies of: Dr. Lipsey s research, SPEP Score Template, SPEP Service Type Fact Sheets, Quality of Service Assessment, EPISCenter Website, Performance Improvement Life Cycle, PowerPoint w/notes pages) Distribute KLO meeting sign in sheets and evaluations 5

6 Objective: 5 Prepare for and hold the SPEP Kickoff Meeting The purpose of this meeting is to increase knowledge and create buy-in of the SPEP process to participants at Kickoff meeting. The intent or focus of the meeting should emphasize the partnership & performance improvement process. Send invitation to all stakeholders: county juvenile court personnel, human service administrators (if deemed appropriate), service providers, etc. Within a reasonable time frame to be determined by the juvenile probation department (preferably EPISCenter staff will provide assistance with agenda items and recommended participants Generate list of handouts and compile master copies of all handouts. Collect and create county specific data presentation - Obtain Juvenile Case Management System (JCMS) Data Reports from the Center for Juvenile Justice Training and Research (CJJT&R). Refer to objective 6 for more details. Deliver SPEP presentation to Kickoff participants; include question and answer session. Distribute Kickoff sign in sheets and evaluations. Objective: 6 Collect County Specific Data within 4-8 weeks from the KLO) 4-6 weeks prior to the Kickoff meeting During the meeting. Establish RSVP date for invited participants Research overview, documents related to the SPEP process, Pre- Visit Checklist, background readings, etc., Sign in sheet/attendee list, and Kickoff participant evaluation Presentation on county specific preparation (PowerPoint, data etc) Agenda & handouts (include PowerPoint w/notes pages) Give consideration to providing refreshments Compile YLS summary reports Prior to Kickoff meeting preferably There are several reports related to the YLS within the Juvenile Case Management System (JCMS). Some examples include: YLS Detailed Scoring by Date Range (for trends and prevalence of domains); YLS by Disposition; YLS by Supervision Type; Serious, Violent and Chronic Offender data; and Reports related to Length of Stay (LOS), etc. The data provides context within the JJSES as to why SPEP is valuable and provides a snapshot of serious, violent & chronic offender (SVCO) data (can be included in kickoff meeting if available/desired). 6

7 The SPEP Process 7

8 Objective: 7 Identify Agencies and Programs to be Prioritized for SPEP Prioritize service providers based on: 4 weeks prior to SPEP interview JPO administrators should communicate with Program complexity (typically, community program administrators to determine if provider is based programs are less complex) willing and able to participate in the SPEP process. Number of youth served (prioritize It is recommended that this occur as part of the engagement frequently used services) process. Service type, if known Complete the Continuum of Services where Risk level of youth served possible in preparation to select service provider for Data collection system the SPEP (this is a list of all contracted service providers Collaborative working relationship with and types of services offered). the juvenile court Provider knowledge and capacity for SPEP participation. Objective: 8 Coordinate with Service Provider All Parties to Prepare for SPEP Process Pre-SPEP visits and/or calls to include presentations, tours, information sharing and/or question and answer sessions). This includes those who are facilitating the SPEP, as well as service provider representatives responsible for decision making as it relates to SPEP. Prior to initial SPEP interview and as needed Minimum of 2 weeks prior to meeting Send letter introducing SPEP, confirming meeting date and time. Include the Task Matrix and information on the materials requested at SPEP interview (see confirmation letter template). Also refer to For programs with multiple services, more than one interview may be necessary. There will be at least one interview for the service type identification and an additional interview(s) for the quality of service. Quality of service interview(s) occur after the service(s) to go through the SPEP process are identified. Schedule the SPEP interview (EPISCenter JJSIS or consultant will assist as needed or requested) Include Pre-Visit Checklist PA SPEP Guide and data collection template Objective: 9 Compile Documentation Materials and Identify and Confirm Staff to be Interviewed Compile required materials in preparation for SPEP interview(s) To be completed prior to SPEP interview(s) As stated in Objective 8, refer to list of items to have available for review during the SPEP Identify staff to be present for SPEP interview(s) and explain the interview process to each staff member. See resources/materials interview; also review Pre-Visit Checklist Multiple levels of provider agency personnel represented at SPEP interview (administration, supervisors, line staff) 8

9 Objective: 10 Conduct Interview(s) with Providers Part 1: Service Categorization (known as unpacking at a residential SPEP) & Part 2: Quality of Service, which may require several meetings Complete a Full Program Profile: obtain as much information as possible on the service(s): what it is, how it is delivered, daily schedule; information on staff hiring, education and training; how staff are supervised and evaluated. Interview agenda (optional) Distribute evaluations Occurs during interview Review of policies, procedures, protocols, manuals etc. provided by agency personnel SPEP interview worksheet, SPEP score sheet, SPEP quality measures, SPEP fact sheets 2014 SPEP User s Guide Full Program Profile template SPEP interview evaluation Objective: 11 Compile Interview Findings, Complete and Present Full Program Profile SPEP interviewers reach consensus on service Notes from interview; SPEP score sheet; quality measures type(s) and assessment of quality of service. checklist (include in SPEP interview packet); Full Program Profile. Identify all services (including those that are ineligible for the SPEP process) within the Full Program Profile. Review of Full Program Profile with juvenile probation and then service provider. This meeting may occur in person, by teleconference or virtually. Accurate descriptions of the services will ensure content information is accurate for Feedback Reports. * As soon as possible following the SPEP interview to speed up the process of developing Feedback Reports. For community-based services, this typically occurs the same day. For residential services multiple meetings may be required to debrief and reach consensus. The Full Program Profile is a dynamic document, subject to change as services evolve. Service type needs to be identified in order to accurately establish cohort and cohort timeframes. In some cases such as Aggression Replacement Training (ART), the service type will be known in advance. *Juvenile probation personnel may choose to limit their role in developing the Full Program Profile and opt to have the EPISCenter JJSIS take the lead. 9

10 Objective: 12 Collect and Analyze Cohort Data Obtain cohort information: Preferably prior to interview for Data to be obtained internally through local county generated Based on identified service categorization, community based services and reports or requested from JCMS; service provider may also establish cohort data timeframe following the service type provide data (preferably no more than 6-12 months identification of residential prior to the SPEP interview) services (refer to 11 ) For each youth in cohort obtain name(s)*, service start and end dates, amount of hours of service received and YLS risk level Minimum of 10 youth necessary for full SPEP score (preferably more) Cohort includes only youth no longer receiving the service For cohorts with youth from counties not currently participating in the SPEP process, it is recommended that the lead juvenile probation department contact those counties to provide information on the SPEP process and to request any missing cohort data. Analyze cohort data: Determine which type of score and report can be generated** Calculate proportion of youth receiving targeted duration (# of weeks) Calculate proportion of youth receiving targeted dosage (# of hours) Proportion of youth with moderate, high and very high risk level (per YLS) Proportion of youth with high to very high risk level (per YLS). In other cases, cohort identification cannot begin until after the service classification meeting/service identification. Youth Level of Service (YLS) results for youth in cohort, identifying the risk level as low, moderate, high or very high Data should include the number of hours and weeks a youth received the service (refer to the EPISCenter website for a template available for download). Vanderbilt Guide to Scoreable Services & Vanderbilt Guide to What is a SPEPable Service (Full, Interim and Advisory Scoring refer to notes in Objective 11) The closer the dates of the cohort selection to the SPEP interview date(s), the more accurate the depiction of the service(s) as currently delivered. Refer to note in Objective 11 *note: DO NOT send identifiers (youth names) via **cohort data will determine appropriate SPEP score Objective: 13 Calculate SPEP Score Identify SPEP service type(s); complete quality of service worksheet; review data related to duration and contact hours; review risk level data of all As soon as possible, typically within 7 days post interview for community based services. For 2014 SPEP Users Guide and other training materials; review of information gathered during interview; gathering of additional information (post interview) may be necessary. *A full SPEP score cannot be generated for instances where the cohort is fewer than 10 youth (known as an interim score); and also youth in cohort; complete the SPEP score sheet, residential services this could *A full SPEP score cannot be generated when less than 80% and include the Program Optimization Percentage take several months. Cohort data of the cohort of youth have known risk levels per the Youth (POP)* See notes SPEP Score Calculation Instructions and Worksheet Level of Service (YLS) (known as an advisory score) *If service is identified as a Blueprints for Healthy Youth Development Model Program, duration and dosage is calculated at 80% of developer s recommendation for both. Vanderbilt Guide to Scoreable Services Vanderbilt Guide to What is a SPEPable Service 10

11 Objective: 14 Prepare Feedback Report(s) Prepare narrative report that includes information collected through review of website and other resources and during the SPEP interview(s). The report describes the agency, program, and services provided, identifies the SPEP service type(s), quality of service delivery, calculation of duration, dosage and risk data, basic and POP score and performance improvement recommendation(s). As soon as possible post interview Feedback report template; SPEP score sheet and SPEP Score Calculation Instructions and Worksheet; notes from interviews; cohort data; review of website and documentation; review of quality of service assessment. Full Program Profile completed by JJSIS at SPEP Interview. Programs that offer multiple services may also have multiple feedback reports. Objective: 15 Conduct Feedback Report Review Meeting(s) and Begin Discussion on Performance Improvement Planning Schedule meeting to include EPISCenter JJSIS, juvenile probation and service provider to review feedback report(s), answer questions and discuss performance improvement For community based services, within 30 days from date of SPEP interview. SPEP Users Guide; notes from interviews; SPEP score sheet and feedback report & supporting material (Service Type Fact Sheets, or other documentation from research to support findings). Facilitate the meeting, keeping track of initial ideas and thoughts about report findings and possible performance improvement activities PDF version of Feedback Report(s) provided electronically to service provider within 48 hours of Feedback Review meeting. Reasonable amount of copies of the Feedback Report (color copies recommended). SPEP Performance Improvement Process Guide and Template. Provide information and available reference materials that are applicable for performance improvement. Distribute Feedback Report Review sign in sheets and evaluations Feedback Report review meeting evaluations and sign in sheets. 11

12 System Improvement 12

13 Objective: 16 Develop and Finalize Performance Improvement Plan Juvenile probation and service provider collaborate on goals for performance Community based services: 1-30 days from the date of the Performance Improvement Guide for Juvenile Justice Stakeholders and Concepts of Performance Improvement improvement guided by Feedback Report findings Feedback Report Review Performance Improvement Template and recommendations Meeting. Continuous Quality Improvement (CQI- from Qii initiative) Guide Development of the Performance Improvement Residential services: timeframe SPEP Service Type Fact Sheets Plan will vary, but usually can be SMART Plan Goals completed within 30 days from Review and approval of plan by all stakeholders the date of the Feedback Questionnaire and Template?? (can include program administrators, board members, Report Review Meeting, for Sample templates, examples of existing plans, and other juvenile court, etc.). each service. materials as needed Plan includes timeframes to achieve each goal and a schedule for progress update meeting/call. *The agency may use an existing performance improvement template or create their own. See the Appendix in the Performance Improvement Guide for assistance in incorporating all of the essential information. Implement and Achieve the Goals of Performance Improvement Plan Objective: 17 Service provider and juvenile probation staff Performance Improvement Resources will vary based upon the recommendations in the implement the plan through the objectives Feedback Report and established goals for the Performance identified in the Performance Improvement Plan. Improvement Plan. Progress Update Meetings/Calls Achievement of performance improvement goals Plan implementation begins shortly after (approximately 30 days) the Feedback Report Review Meeting, and may last as long as a year or more Phone calls or meetings will occur on a quarterly basis and as necessary Substantial completion of the performance improvement goals should occur within 12 months (this will vary based on individual circumstance) Technical assistance will be provided as needed through the EPISCenter or others with expertise in continuous quality improvement. Data can then be collected to measure improvements. 13

14 Objective: 18 Prepare for SPEP Re-assessment Determine if there has been a substantial completion of the performance improvement goals to warrant SPEP re-assessment. Verify that the new cohort has received the improved service months following achievement of performance improvement goals (goals in plan completed or near completion) Refer to Objective H: Resources. Data collection mechanism to track new cohort (duration, dosage and risk level). New cohort data collection should begin after sufficient progress has been made on the performance improvement goals. If sufficient progress has not been made, then work on goals should continue. Establish tentative timeframes for data collection of the new cohort Re-visit the prioritized services that have previously gone through the SPEP process to determine if other services can benefit from the SPEP process. Determine if the service should receive a SPEP-re assessment (based on the two previous tasks)*. Objective: 19 Youth in the new cohort should no longer be receiving the service (ie: the youth will have successfully completed the service or were removed from the service per order of juvenile court). Provide Pre-visit Checklist if needed 6-24 months (allows time for a sufficient cohort size of youth to receive the improved service) To be determined Review the Full Program Profile and update as needed. There must be a sufficient cohort size (minimum of ten youth) and available data collected to support/warrant a subsequent SPEP Reassessment. If the cohort size is fewer than 10 youth, continue to collect data or consider alternative scoring options (the advisory score; refer to Objective 13 for definition). To be determined *If it is decided to not re-assess a service that went through the SPEP process, other therapeutic services in the program /agency are still eligible for the SPEP process, provided there is sufficient data available (eg: cohort size, amount of service and risk level data). Refer to Objective 12 and proceed with SPEP process 14