A step by step process for the implementation of the Standardized Program Evaluation Protocol (SPEP) in Pennsylvania

Transcription

1 PA Juvenile Justice System Enhancement Strategy Stage 3: Behavioral Change SPEP Task Matrix A step by step process for the implementation of the Standardized Program Evaluation Protocol (SPEP) in Pennsylvania Developed by the EPISCenter at PSU, in partnership with the Juvenile Court Judges Commission and the PA Council of Chief Juvenile Probation Officers The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Bennett Pierce Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the PA Department of Human Services. This resource was developed by the EPISCenter through PCCD grant VP-ST Draft

2 Steps for Implementation of the SPEP in a County Juvenile Probation Department Training Page 3 Research and implementation of SPEP [A] Communication and outreach: Key Leader orientation and planning [B, C, D] Kickoff meeting: community stakeholder orientation and information session(s) [E] The SPEP Process (ongoing) Page 8 Develop a Continuum of Services (list of contracted services and programs) [E, F] Prioritize services/programs for going through the SPEP process [F] Prepare provider for the SPEP process [G, H] Service categorization & quality of service interviews [I] Compile interview findings, complete program profile [I, J] Duration, dosage, and risk data collection and analysis [K] Develop and deliver feedback report, discuss implications for performance improvement [L, M, N] System Improvement Page 13 Service provider and juvenile probation collectively determine improvement priorities [O] Develop performance improvement plan [O] Identify needed resources, obtain approvals, implement plan [P] Quarterly contact for updates and technical assistance [U, V] Last Revised

3 Training Last Revised

4 Objective: Train Juvenile Probation Officer(s) in County Participating in the SPEP This training is for those learning to SPEP community-based and residential services. Identify the juvenile probation officer (JPO)/representative(s) that will be involved in the introduction and implementation of the SPEP process Schedule SPEP training for the JPO and other identified staff (if appropriate) This includes orientation to the research and how to administer SPEP, SPEP interview observation, co-facilitating a SPEP interview, facilitating a SPEP interview and how to write a feedback report Identify and schedule a SPEP observation/shadowing for JPO Objective: B 2 4 weeks after a commitment is made to participate in SPEP Training at location to be determined 2014 SPEP User's Guide Dr. Lipsey s research, SPEP Score Template SPEP Service Type Fact Sheets, Quality of Service Measures EPISCenter website, Performance Improvement Life Cycle Develop a Rollout Communication Plan The plan should be a guide for SPEP roll out. The intent is to educate leaders and stakeholders on the SPEP process and includes training, the dissemination of written materials, and the development of the plan for meetings of key leaders and local stakeholders. The communication plan should be tailored to suit the needs of each county participating. Meeting to discuss process for rollout 2 4 weeks after a commitment is made to participate in SPEP Copies of documents off of the EPISCenter website related to SPEP (available under the Juvenile Justice tab) Establish timeframe for key leader and kickoff meetings Sample Power Points from past key leader and kickoff presentations SPEP webinars (EPISCenter website) Determine desired structure for kickoff meeting Provide a copy of Task Matrix Two half day meetings (same information, different attendees) or one larger meeting for all attendees with a break for lunch Last Revised

5 Objective: C Hold the Key Leader Orientation (KLO) This is the initial meeting of local stakeholders that explains what all involved (leadership included) need to know and understand about the SPEP process. The intent is to increase understanding of all involved that the process is collaborative and focuses on performance improvement. Schedule meeting (time and location should be suitable to stakeholders) Establish meeting structure (agenda, PowerPoint, meeting objectives) Within 10 business days of initial communication plan meeting with juvenile probation office personnel Identify presenters for the KLO meeting Identify local key SPEP leadership to be invited to meeting (JPO, Quality Assurance, Director, Deputy Chief JPO, Supervisors, JJSES Coordinator, etc.) Include question and answer session Distribute evaluations Identify role of those involved in Kickoff Meeting (who will present, what will be presented, time frames) Identify initial service provider to participate in the SPEP process 4-6 weeks following the invitation Just prior to conclusion of KLO Meeting to review structure of Kickoff & KLO meetings Discuss stakeholders; provide copies of sample agendas for KLO meetings, PowerPoint examples for KLO meetings, and other relevant documents*: SPEP User's Guide Dr. Lipsey s research, SPEP Score Template, SPEP Service Type Fact Sheets, Quality of Service Measures EPISCenter website, Performance Improvement Life Cycle KLO evaluation form Agenda and copies of: Dr. Lipsey s research, SPEP Score Template, SPEP Service Type Fact Sheets, Quality of Service Measures, EPISCenter Website, Performance Improvement Life Cycle, PowerPoint w/notes pages) *EPIS staff will develop standard packets Objective: D Determine Logistics for and carry out SPEP Kickoff Meeting Choose format: Prior to KLO meeting EPIS staff will provide assistance with agenda items and recommended participants Two half day meetings (same information, different attendees) or One larger meeting for all attendees with a break for lunch Establish RSVP date for invited participants EPIS staff will provide assistance with agenda items and recommended participants The data provides context within the JJSES as to why SPEP is valuable and provides a snapshot of serious, violent & chronic offender (SVCO) data (can be included in kickoff meeting if available/desired). Within 90 days from initial meeting where the rollout communication was discussed (refer to Objective B) Research overview, SPEP process documents, Pre-Visit Checklist, background readings, etc., Sign in sheet/attendee list, and Kickoff participant evaluation Presentation on county specific preparation (PowerPoint, data etc) 4-6 weeks prior to the Kickoff meeting Agenda & handouts (include PowerPoint w/notes pages 2 weeks prior to meeting Give consideration to providing refreshments Last Revised

6 Identify meeting venue. If necessary reserve space to accommodate participants (sufficient size to accommodate number of invitees) Send invitation to all stakeholders: county juvenile court personnel, human service administrators (if deemed appropriate), service providers, etc. Generate list of handouts and compile master copies of all handouts Collect and create county specific data presentation - Obtain Juvenile Case Management System (JCMS) Data Reports from the Center for Juvenile Justice Training and Research (CJJT&R). Refer to objective E for more details Complete the Continuum of Services where possible (this is a list of all contracted service providers and types of services offered) Deliver SPEP presentation to Kickoff participants Increase knowledge of the SPEP process to participants at Kickoff meeting (focus emphasis on partnership & performance improvement) Include question and answer session Objective: E Collect County Specific Data Retrieve YLS summary reports from JCMS Prior to kickoff meeting preferably Request the following (if available) from the CJJT&R Help Desk contact person: YLS by Disposition YLS by Supervision Type Serious, violent and chronic offender data Last Revised

7 The SPEP Process Last Revised

8 Objective: F Identify Agencies and Programs to be Prioritized for SPEP Prioritize service providers based on: 4 weeks prior to SPEP interview JPO administrators should communicate with program Community based initially; residential administrators to determine if provider is willing and able to services follow participate in the SPEP process Number of youth served Service type, if known Risk level of youth served Data collection system Continuum of Services *determine if data and cohort size are sufficient for participation Strong relationship with the juvenile court Provider knowledge and capacity for SPEP participation Objective: G Establish Date(s) with Provider for SPEP Schedule SPEP interview (EPISCenter JJSIS or consultant will assist as needed or requested) Minimum of 2 weeks prior to meeting Send letter introducing SPEP, confirming meeting date and time. Include the task matrix and information on the materials requested at interview (see confirmation letter template) Include Pre-visit Checklist For residential services, more than one interview may be necessary, depending on the number of services identified. There will be at least one interview for the service type identification and an additional interview(s) for quality assessment, once service(s) to go through SPEP process are identified. Residential SPEP Process Guide and data collection spreadsheets Objective: H Compile Documentation Materials and Identify and Confirm Staff to be Interviewed Compiles required materials in preparation for SPEP interview(s) Identify staff to be present for SPEP interview(s) and explain the interview process to each staff member To be completed prior to interview As stated in Objective G, refer to list of items to have available for review during the interview; also review Pre-visit Checklist Multiple levels of provider agency represented at interview (administration, supervisors, line staff) Last Revised

9 Objective: I Conduct Interview(s) with Providers Part 1: Service Categorization (known as unpacking in a residential SPEP) & Part 2: Quality of Service, which may require several meetings Complete a SPEP program profile: obtain as much information as possible on the service: what it is, how it is delivered, daily schedule; information on staff hiring, education and training; how staff are supervised and evaluated Interview agenda (optional) Occurs during interview Review of policies, procedures, protocols, manuals etc. provided by agency personnel SPEP Interview Worksheet (SPEP score sheet, SPEP quality measures, SPEP fact sheets) 2014 SPEP User s Guide Distribute evaluations Program profile template SPEP interview evaluation Objective: J Compile Interview Findings Interviewers reach consensus on service type and As soon as possible following Notes from interview; SPEP score sheet; quality measures quality measures the SPEP interview. For checklist (included in SPEP interview packet); program profile community-based services typically the same day. For residential services multiple meetings may be required to debrief and reach consensus. Objective: K Collect and Analyze Cohort Data Obtain cohort information: Preferably prior to interview for Data to be obtained internally through local county generated Based on identified service categorization, community based services and reports or requested from JCMS; service provider may also establish cohort data timeframe following the service type provide data (preferably no more than 6-12 months identification of residential prior to the SPEP interview) services (refer to L ) Youth Level of Service (YLS) results for youth in cohort, For each youth in cohort obtain identifying the risk level as low, moderate, high or very high name(s)*, service start and end dates, Service type needs to be amount of hours of service received and identified in order to accurately Data should include the number of hours and weeks a youth YLS risk level establish cohort and cohort received the service Minimum of 10 youth necessary for full timeframes. In some cases such SPEP assessment (preferably more) as Aggression Replacement Training (ART), the service type **SPEP Scoring Guidelines (Full, Provisional and Advisory Cohort includes only youth no longer will be known in advance. In Scoring refer to notes in Objective L) receiving the service other cases, cohort identification Analyze cohort data: cannot begin until after the Determine which type of score and service classification report can be generated** meeting/service identification. Calculate proportion of youth receiving targeted duration (# of weeks) Calculate proportion of youth receiving targeted dosage (# of hours) Proportion of youth with moderate to very high risk level (per YLS) Proportion of youth with high to very high risk level (per YLS) The program profile is a dynamic document, subject to change as services evolve. The closer the dates of the cohort selection to the SPEP interview date(s), the more accurate the depiction of the service(s) as currently delivered. *note: DO NOT send identifiers (youth names) via **cohort data will determine appropriate SPEP score to be developed (refer to SPEP Scoring Guidelines) Last Revised

10 Objective: L Calculate SPEP Score Identify SPEP service type(s); complete quality measures worksheet; review data related to duration and contact hours; review risk level data of all youth in cohort; complete the SPEP score sheet, and include the program optimization percentage* See notes As soon as possible, typically within 7 days post interview for community based services. For residential services this could take several months SPEP Users Guide and other training materials; review of information gathered during interview; gathering of additional information (post interview) may be necessary *If service is identified as a Blueprints for Healthy Youth Development Model Program, duration and dosage is calculated at 80% of developer s recommendation for both *A full SPEP score cannot be generated for instances where the cohort is fewer than 10 youth (known as a provisional score) *A full SPEP score cannot be generated when less than 80% of the cohort of youth have known risk levels per the Youth Level of Service (YLS) (known as an advisory score) Commented [S1]: Should we add that matrix of score types from Vanderbilt? Identify title of Vandy documents for what s in the status notes section (Don t have them with me to reference) Objective: M Prepare Feedback Report(s) Residential programs in particular may have multiple feedback reports. Prepare narrative report that includes information collected during the SPEP interview(s) that describes the program, agency and services provided, identifies the SPEP service type(s), quality measures, calculation of duration, dosage and risk data, basic and POP scores and performance improvement recommendations. As soon as possible post interview Feedback report template; SPEP score sheet; notes from interviews; review of documentation; review of quality measures checklist 4.0 Objective: N Conduct Feedback Report Review Meeting(s) and Begin Discussion on Performance Improvement Planning Schedule meeting to include SPEP interviewers, juvenile probation and service provider to review feedback report(s), answer questions and discuss performance improvement For community based services, within 30 days from date of SPEP interview SPEP Users Guide; notes from interviews; SPEP score sheet and feedback report & supporting material (Service Type Fact Sheets, or other documentation from research to support findings) Facilitate the meeting, keeping track of initial ideas and thoughts about report findings and possible performance improvement activities Distribute feedback report meeting evaluation For residential based services, the timeframe is from the initial SPEP interview until performance improvement commences SPEP Performance Improvement Process (includes an optional performance improvement template) Provide information and available reference materials that are applicable for performance improvement Feedback report meeting evaluation Last Revised

11 System Improvement Last Revised

12 Objective: O Task Juvenile probation and service provider collaborate on goals for performance improvement guided by feedback report findings Development of the performance improvement plan Review and approval of plan by all stakeholders (can include program administrators, board members, juvenile court, etc.). Plan includes timeframes to achieve each goal and a schedule for progress calls Develop and Finalize Performance Improvement Plan EPIS JPO Provide Targeted Timeframe Resources/Materials Status/Notes r Community based services: 1-30 days from the date of the feedback review meeting Residential services: timeframe will vary Performance Improvement Guide for Juvenile Justice Stakeholders to include Concepts of Performance Improvement Performance Improvement Template Continuous Quality Improvement (CQI- from Qii initiative) Guide SPEP Service Type Fact Sheets SMART Plan Goals Questionnaire and Template Sample templates, examples of existing plans, and other materials as needed *The agency may use an existing performance improvement template or create their own. See Page 11 and Appendix A in the Performance Improvement Guide for assistance in incorporating all of the essential information. Implement and Achieve the Goals of Performance Improvement Plan Objective: P Service provider and juvenile probation staff Resources will vary based upon the recommendations in the implement the plan through the objectives Feedback Report and established goals for the Performance identified Improvement Plan. Progress Update Meetings/Calls Performance improvement plan implementation begins shortly after (approximately 30 days) the review of the feedback report, and may last as long as a year or more Phone calls or meetings will occur on a quarterly basis and as necessary Technical assistance will be provided as needed through the EPISCenter or others with expertise in continuous quality improvement. Achievement of performance improvement goals Substantial completion of the Data can then be collected to measure improvements. performance improvement goals should occur within 12 months (this will vary based on individual circumstance) Objective: Q Prepare for SPEP Re-assessment Determine if there has been a substantial months following Data collection mechanism to track new cohort (duration, dosage completion of the performance improvement implementation of performance and risk level). goals to warrant SPEP re-assessment. Verify that improvement goals (goals in plan the new cohort has received the improved service. completed or near completion) New cohort data collection should begin after sufficient progress has been made on the performance improvement goals. If sufficient progress has not been made, then work on goals should continue. Objective: Q cont. Important: youth in the new cohort should no longer be receiving the improved service (ie: the youth will have successfully completed the service or were removed from the service per order of juvenile court). Prepare for SPEP Re-assessment cont. Last Revised

13 Establish tentative timeframes for data collection Provide pre-visit checklist if needed of the new cohort Evaluate services that have not previously gone through the SPEP process to ensure all services can benefit from the SPEP process months (allows time for a sufficient cohort size of youth to receive the improved service) Review the program profile and update as needed There must be a sufficient cohort size (minimum of ten youth) and available data collected to support/warrant a subsequent SPEP. If the cohort size is fewer than 10 youth, continue to collect data or consider alternative scoring options (the advisory score; refer to Objective L for definition) Determine if the service should receive a SPEP-re assessment (based on the two previous tasks)* Objective: R *If it is decided to not re-assess a service that went through the SPEP process, other therapeutic services in the program/agency are still eligible for the SPEP process, provided there is sufficient data available (eg: cohort size, amount of service and risk level data) Refer to Objective G and proceed with SPEP process Last Revised