Introduction. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 1

Transcription

1

2 Introduction PBIS Assessment ( is a web application designed to assist in high-fidelity, sustained implementation of schoolwide positive behavioral interventions and supports (SWPBIS). The site includes tools and instruments documenting PBIS fidelity of implementation. Reports are available for each of the instruments administered on the site, including Total Score Report, Subscale Report, Individual Items Report, and a Data Download. The Total Score and Subscale reports are represented by the use of percentages. This document provides an overview about each of the survey tools and information about how the percentages for the reports are calculated. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 1

3 Table of Contents Instrument Page Benchmarks for Advanced Tiers (BAT) 3-4 Benchmarks of Quality (BoQ) 5-6 Early Childhood Program-Wide Benchmarks of Quality (ECBoQ) 7-8 Monitoring Advanced Tiers Tool (MATT) 9-10 School-wide Evaluation Tool (SET) Self-Assessment Survey (SAS) School Safety Survey (SSS) Team Implementation Checklist (TIC) Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 2

4 Benchmarks for Advanced Tiers Documentation The Benchmark of Advanced Tiers (BAT) is an annual assessment evaluating the implementation of Tier II and Tier III PBIS practices within a school. Each of the BAT items is scored using a 0-2 scale (0 = Not Yet Started, 1 = Partially in Place, 2 = Fully in Place). The BAT is typically used annually after a team has been trained in Tier II and tier III supports and has reached 80% on the Total Scores for the MATT. The BAT serves to promote sustainability and continued improvement in Tier II and III supports. The results of the BAT guide development of an action plan that defines what specific steps will be taken to improve implementation of Tier II and III supports. The BAT provides four reports within PBIS Assessment: Total Score, Subscale, Items, and Download. The BAT Total Score and Subscale reports are based on a percentage of points scored on the various items of those reports. The table below shows the calculations for the Total Score (Summary Score for Foundations, Tier II, and Tier III) and the Subscale (Feature Areas within each of the Summary Scores). Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 3

5 Total Score: Subscale Score: The Items report provides each of the BAT item scores the respondent submitted. The Download report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Benchmark Criterion for implementing with fidelity for the BAT is 80% in the Total Score for Foundations, Tier II, and Tier III. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 4

6 Benchmarks of Quality Documentation The Benchmarks of Quality (BoQ) is an annual assessment for school teams to use to identify areas of success and areas for improvement in PBIS implementation. The BoQ includes 53 items which are scored using a 0-3 scale. The information for scoring each of the items is outlined in the BoQ Scoring Guide. The BoQ is typically used after teams have scored 80% or above on the TIC or with initial implementation of PBIS. The results of the BoQ are used for action planning for improving Tier I PBIS implementation. The BoQ provides four reports within PBIS Assessment: Total Score, Subscale, Items, and Download. The BoQ Total Score and Subscale reports are based on a percentage of points scored on the various items in those reports. The Total Score represents the total percentage score. The total score is calculated by: 1) Sum of all the responses 2) Divided by the total number possible. The Subscale report provides the percentage of points within each of the ten elements in the Benchmarks of Quality. The following elements are included in the BoQ: PBIS Team, Faculty Commitment, Discipline Procedures, Data Entry & Analysis, Expectations, Recognition, Teaching, Implementation Plan, Classroom Systems, and Evaluation. The Subscales are calculated by: 1) Sum of the responses of the questions in each element. 2) Divided by the sum of the total possible points within each element. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 5

7 The Items report provides each of the BoQ item scores the respondent submitted. The Download report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Benchmark Criterion for implementing with fidelity for the BoQ is 70% in the Total Score. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 6

8 Early Childhood Benchmarks of Quality Documentation The Early Childhood Benchmarks of Quality (ECBoQ) is an annual assessment of Tier I PBIS implementation in early childhood settings. The results of the ECBoQ are used to create an action plan for the continued growth and support for PBIS within the early childhood setting. Each of the ECBoQ items is scored using a 0-2 scale (0 = Not in Place, 1 = Partially in Place, 2 = In Place). The ECBoQ provides four reports within PBIS Assessment: Total Score, Subscale, Items, and Download. The ECBoQ Total Score and Subscale reports are based on a percentage of points scored on the various items in those reports. The Total Score represents the total percentage score. This is calculated by: 1) Sum of all the responses. 2) Divided by the total number possible. The Subscale report provides the percentage of points within each of the ten elements in the ECBoQ. The following elements are included in the ECBoQ: Leadership Team, Staff Buy-In, Family Involvement, Program-Wide Expectations, Strategies, Teaching Pyramid, Procedures, Staff Support, and Monitoring. The subscales are calculated by: 1) Sum of the responses of the questions in each element. 2) Divided by the sum of the total possible points within each element. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 7

9 The Items report provides each of the ECBoQ item scores the respondent submitted. The Download report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Benchmark Criterion for implementing with fidelity for the ECBoQ is 80% for the Total Score. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 8

10 Monitoring Advanced Tiers Tool Documentation The Monitoring Advanced Tiers Tool (MATT) is a progress-monitoring tool used to assess the status and implementation of Tier II and III PBIS Systems. The MATT is typically used quarterly to assess progress and to continue to update a team s Tier II and III action plan. Each of the Monitoring Advanced Tiers Tool (MATT) items is scored using a 0-2 scale (0 = not started, 1 = in progress, 2 = fully in place). The MATT provides four reports within PBIS Assessment: Total Score, Subscale, Items, and Download. The MATT produces two Total Scores representing the percentage of possible points for (a) Tier II practices, and (b) Tier III practices to assess implementation progress across time. The Total Score provides a score for Tier II and a score for Tier III. The table below shows the calculations for the Total Score (Summary Score for Tier II and Summary Score for Tier III) and the Subscale Score (Feature Areas within each of the Summary Scores). Total Scores: 1) The Total Score for Tier II includes Tier I Critical Elements, Tier II Organizational Elements, and Tier II Critical Elements 2) The Total Score for Tier III includes Tier I Critical Elements, Tier III Organizational Elements, and Tier III Critical Elements. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 9

11 Subscale: 1) Five Subscale scores are provided in the MATT: Tier I Critical Element, Tier II Organizational Elements, Tier III Organizational Elements, Tier II Critical Elements, and Tier III Critical Elements. The Items report provides each of the MATT item scores the respondent submitted. The Download report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Benchmark Criterion for implementing with fidelity for the MATT is 80% in each of the Total Score areas. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 10

12 School-wide Evaluation Tool Documentation The School-wide Evaluation Tool (SET) is a research tool that is used to assess the implementation of Tier I PBIS in schools. The SET is typically administered annually, with some sites completing both a fall and spring SET during the first year as a Pre/Post. The SET is administered by an outside evaluator who interviews the administrator, team members, staff members, and students; reviews PBIS written documentation; and tours the school site. The SET is scored on a 0-2 scale (0 = Not Started, 1 = In Progress, 2 = Fully in Place). Each item is scored using the rubric within the SET scoring guide. The SET provides three reports within PBIS Assessment: Subscale, Items, and Download. The Subscale report includes the SET Total Score, represented by the Implementation Average. The SET includes seven research-based features for effective PBIS implementation: Expectations Defined, Expectations Taught, Reward System, Violations System, Decision-Making, Management, and District Support. The Subscale report provides a percentage of implementation for each of the features, as well as a percentage of overall implementation shown as the Implementation Average (Total Score). Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 11

13 Calculation for the Subscale scores and the Total Score (Implementation Average) are described below: N = Number of points possible within each subscale. A. Expectations Defined (N/4) Implementation Average: B. Expectations Taught (N/10) Sum of the percentages of C. Reward System (N/6) each of the Subscale features, D. Violations System (N/8) divided by 7. E. Decision Making (N/8) F. Management (N/16) G. District Support (N/4) The Items report provides each of the SET item scores the respondent submitted. The Download report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Benchmark Criterion for implementing with fidelity for the SET is reaching 80% in Expectations Taught AND 80% in Implementation Average (80/80). Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 12

14 Self-Assessment Survey Documentation The Self-assessment Survey (SAS) is used by school staff for initial and annual assessment of PBIS implementation. All staff members are encouraged to complete the SAS. The SAS includes four systems: School-wide, Non-Classroom, Classroom and Individual Students. Each item within the SAS is scored on a 0-2 scale (0 = Not in Place, 1 = Partially in Place, 2 = In Place) for the Current Status and a 0-2 scale (0 = Low, 1 = Medium, 2 = High) for Priority for Improvement. Non responses are not included in the counts or calculations. The SAS provides four reports within PBIS Assessment: Total Score, Subscale, Items, and Download. The Total Score reports the percentage of respondents who answered In Place, Partially In Place, and Not In Place for the Current Status. It also reports the percentage of respondents who answered High, Medium, or Low for the Improvement Priority. The Total Score is reported for each of the four systems: School-wide, Classroom, Non-Classroom, and Individual Student. The Total Score is calculated by: 1) Summing the number of answers within each system for Current Status and Improvement Priority. 2) Dividing that number by the total number of respondents. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 13

15 The Subscale report provides a report for the percentage of items in place for each of the elements within the School-wide System. Subscale reports are not currently available for Classroom, Non Classroom, and Individual Student Systems. The elements within the Subscale report include: Expectations Defined, Expectations Taught, Reward System, Violations System, Monitoring, Management, and District Support. In addition, the Subscale report displays an Implementation Average demonstrating the overall average (Total Score) for all elements. The Subscale score is calculated by: 1) N is the number of responses, Q is the number of questions for a subscale (e.g., Q is 1 for Expectations defined and 4 for Monitoring ), IP is the number of In Place responses for subscale questions, and P is the number of Partial responses for the subscale questions. 2) The formula for the subscale is: (2*IP + P) / (2*N*Q). Multiply the total number of In Place responses by 2, add the number of Partial responses, and divide this figure by the maximum possible score ( In Place for all questions). As always, NULL values are completely excluded (not included in either numerator or denominator). Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 14

16 The Items report provides a percentage score for each of the items for Current Status ("In Place", "Partially in Place", and "Not in Place") as well as the Improvement Priority ("High", "Medium", and "Low"). The Items report is calculated by: 1) For each system area, calculate a total summary (S) by counting the total number of responses (R) for a column (e.g., In Place: ) and dividing that number by the total number of responses for the row (e.g., In Place + Partial + Not in Place). The Items report provides an automatic default to color code the individual items. (The report can also be generated without color). The colors are determined by the percentage of In Place points in the Current Status. The colors (white/yellow/red) indicate the following: White: % of implementation points In Place; Yellow: 51-79% of implementation points In Place; Red: 0-50% of implementation points In Place. The Download Report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Benchmark Criterion for implementing with fidelity for the SAS is an overall Implementation Average of 80%. For individual subscale element items, 80% In Place meets the benchmark for that element. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 15

17 School Safety Survey Documentation The School Safety Survey (SSS) scores are based on a 0-3 scale (0 = Not at All, 1 = Minimally, 2 = Moderately, 3 = Extensively). Non responses, Don t Know, and Not Applicable responses are not included in the counts or calculations. The SSS provides four reports within PBIS Assessment: Subscale, Item, Download, and Other. The Subscale report provides the Risk Ratio and the Protection Ratio derived from the submitted responses. The Risk Ratio is calculated by: 1) Sum of the Risk Total Values in the Risk Section from the individual surveys. 2) Dividing the sum by the Risk Answered Values from the individual surveys. 3) The number is then divided by 3 to calculate the Risk Ratio. The Protection Ratio is calculated by: 1) Sum of Protection Total Values in the Protection Section from the individual surveys. 2) Dividing the sum by the Protection Answered Values from the individual surveys. 3) The number is then divided by 3 to calculate the Protection Ratio. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 16

18 The Items report provides the Risk Items average and the Protection Items average. The Risk Items average is calculated by: 1) Adding the sum of Risk Total item values. 2) Dividing the sum by the number of responses per item. The Protection Items average is calculated by: 1) Adding the sum of Protection Total item values. 2) Dividing the sum by the number of responses per item. The Download report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Other report offers a collection of the textual responses entered by respondents in Part 3 of the SSS. The SSS is an outcome measure. The goal is for the Total Risk Score to decrease and the Protection Total Score to increase over time. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 17

19 Team Implementation Checklist Documentation The Team Implementation Checklist (TIC) is a Tier I progress-monitoring tool typically completed at least quarterly by the school team. The results for the TIC are used to plan activities to support and improve the implementation of SWPBIS. The TIC Scores are based on a percentage of the items scored on a 0-2 scale (0 = Not Yet Started, 1 = In Progress, 2 = Achieved). The TIC provides four reports within PBIS Assessment: Total Score, Subscale, Item, and Download. The Total Score is the percentage of items Fully Implemented and Partially Implemented. To calculate the Total Score: 1) Sum the items Fully Implemented (FI = 2) and divide the sum by the total possible points (44) for the survey. 2) Sum the items Partially Implemented (PI = 1) and divide the sum by the total possible points (44) for the survey. 3) Sum the items Not Implemented (NI = 0) and divide the sum by the total possible points (44) for the survey. Percentage of Fully Implemented items = (FI / 44) Percentage of Partially Implemented items = (PI / 44) Percentage of Not Implemented items = (N / 44) Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 18

20 The Subscale report presents the percentage of points scored in each subscale area. The following subscales and corresponding TIC questions are: Establish Commitment (1-2); Establish and Maintain Team (3-5); Self-Assessment (6-8); Define Expectations (9-11); Teach Expectations (12); Reward Expectations (13); Consequences (14); Classroom (15-17); Establish Information System (18-19); and Building Capacity for Function-Based Support (20-22). To calculate the Subscale scores: 1) Sum the points for each subscale area (FI = 2, PI = 1, NI = 0). 2) Divide by the total points possible for each subscale area. The Items report provides each of the TIC item scores the respondent submitted. The Download report provides the choice of a pdf or an Excel (csv) file of the survey responses. The Benchmark for the TIC is 80% fully implemented in the Total Score. Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 19

21 For more information, please contact: Rossetto Dickey, C., Eliason, B. (2013). PBIS Assessment Instrument Documentation. Page 20