The New Accountability in Teacher Preparation: Making It Work for Your State and Programs

Transcription

1 The New Accountability in Teacher Preparation: Making It Work for Your State and Programs Webinar with Michael Allen, Teacher Preparation Analytics Ashley Bencan, New Jersey Department of Education May 25, :30 a.m. 1:00 p.m.

2 Agenda Welcome and Introduction The New Accountability in Teacher Preparation Michael Allen A State s Perspective Ashley Bencan Q & A for Both Presenters Wrap-up and Evaluation 2

3 Housekeeping Phones are muted. Please use chat box for questions and comments Recording of webinar will be posted online Please complete the evaluation survey at the end of the webinar For assistance with technical issues please 3

4 About the WestEd is a non-profit research, development and service agency WestEd has been leading the Mid-Atlantic Comprehensive Center (MACC@WestEd), funded by the Office of Elementary and Secondary Education, U.S. Department of Education, since October 2012 MACC@WestEd works with Delaware, Maryland, New Jersey, Pennsylvania, and the District of Columbia 4

5 Comprehensive Center Priorities Standards and Assessment Educator and Leader Effectiveness School Turnaround Early Childhood School Readiness College and Career Readiness Innovations Scale Up Effective Data Use 5

6 Presenters Michael Allen Founding Partner Teacher Preparation Analytics Ashley Bencan Deputy Director Office of Recruitment and Preparation Division of Talent and Performance New Jersey Department of Education 6

7 The New Accountability in Teacher Preparation: Making it Work for Your State and Programs Michael Allen Teacher Preparation Analytics

8 Four Questions What s changing in accountability for teacher preparation? What are the potential benefits for states and preparation programs? What are the limitations and challenges to new approaches? What decisions have to be made in moving forward? 8

9 The Back Story Teaching in U.S. longstanding low-status profession Skepticism about quality of teacher preparation NCATE Accreditation (1954) A Nation At Risk (1983) Focus on Outcomes CAEP Accreditation (revised traditional approach) New Accountability 9

10 Traditional Accountability: Compliance with Professional Program Standards Standards focus on program capacities, including capacity to evaluate candidate strengths States and national accrediting body (now CAEP) make determination every 7 years Self-reports, site visits Includes review/approval of individual licensure programs EPPs submit annual or biennial reports as evidence for 7-year re-approval decision 10

11 Emerging Accountability: Dashboards and Report Cards on Program Performance Moving towards Continuous Improvement Not only periodic evaluation and correction Individual licensure programs more central focus Not only EPP in aggregate Annual program reports given greater weight May trigger intervention independent of 7-year re-approval Program assessment data more quantitative Not rubric-based judgments about meeting standards Accountability focus on candidate and completer outcomes ideally success in teaching e.g., edtpa, impact on P-12 students 11

12 Why Dashboards & Report Cards? Annual report can provide greater transparency Data more accessible to various stakeholders Focus on outcomes is more compelling Quantitative data more objective, comparable Program level is where the rubber meets road Reduces documentation burden for EPPs and SEAs Annual review facilitates continuous improvement as SEA/EPP partnership 12

13 What Are The Challenges? Deficient data in many states Deficient (or missing) assessments in the field Getting buy-in, cooperation of key stakeholders Adequate time for development, implementation Moving forward with humility Balancing accountability, program improvement, public information Funding and sustaining the effort 13

14 Measuring Program Performance: What Must States Decide? Goals for the system: Program improvement, accountability, public information? Indicators and measures Implementation timeline (including pilot period) Program performance benchmarks Weighting indicators, assigning performance levels? Basis for program review and intervention Nature of annual report and public access Role of CAEP and multi-year approval? System evaluation and improvement process 14

15 Teacher Preparation Impact Web 15

16 Ideal Program Outcomes Great majority of candidates complete the program All completers have solid teaching skills, content knowledge Almost all completers enter teaching within 1-2 years All completers effective classroom teachers from day one Most completers remain in education at least 5-10 years Many completers teach in high-need schools and subjects 16

17 For What Are Programs Accountable? Programs are primarily accountable for Selecting candidates who are likely to succeed Ensuring completers have strong knowledge and skill Preparing candidates for high-need subjects, schools Programs are jointly accountable for Ensuring candidates complete program Ensuring completers are successful as teachers Programs are partially accountable for Ensuring completers remain in the profession 17

18 Four Domains of Outcomes Measures Candidate Selection Profile Knowledge and Skills for Teaching Performance as Classroom Teachers Contribution to State Needs 18

19 12 Suggested Performance Indicators Candidate Selection Profile Academic Strength Teaching Promise Candidate/Completer Diversity Performance as Classroom Teachers Impact on K-12 Student Learning Demonstrated Teaching Skill K-12 Student Perceptions Knowledge and Skills for Teaching Content Knowledge Pedagogical Content Knowledge Teaching Skill Completer Rating of Program Contribution to State Needs Entry/Persistence in Teaching Placement/Persistence in High- Need Subjects/Schools 19

20 Developing Valid, Comparable Measures Use valid, reliable, and standardized assessments Ensure data are accurate, reliable, and uniform Use measures that are Consistent with research Information rich Reflective of candidate differences within programs Sensitive to complex factors Maximize individual program-level measures Work continuously to improve the measures 20

21 Effectiveness of Measures Less Effective Percentage of candidates and of completers by race/ethnicity Assessment Pass Rate Principal s rating of program preparation based on survey of new teacher performance More Effective Admitted cohort completion % by race/ethnicity/at-risk Assessment mean and % in top and bottom third of distribution Supervisor s assessment of new teacher practice based on multiple observations 21

22 The 2020 Key Effectiveness Indicatorsv

23 Using Measures for Program Improvement Provide EPPs with reports on all specific measures at the individual program level Identify high, moderate, and low performance Give EPPs guidance for using report data for improvement Facilitate exchange of information between highperforming and low-performing programs 23

24 Using Measures for Accountability Keep eyes on the prize: Program Improvement Decide relationship between annual report and 7-year re-approval/accreditation decision Establish program performance benchmarks Decide about weighting, performance levels Establish rules, values for review triggers Decide what report information will be public 24

25 The Basis for Intervention High-Stakes System Composite program score or assigned performance level (rating or ranking) triggers state intervention Medium Stakes System Very low or repeated low scores on any indicator trigger further inquiry and possible action Low-Stakes System Annual report provides evidence of program strength or progress for 7-year approval decision 25

26 The Dashboard... REMEMBER:... Is Not the Engine 26

27 Development and Implementation Timelines Balance urgency and need for buy-in Use piloting process to work out effective measures, benchmarks, and weightings Decide when program scores begin to count Assess data capacity/needs start with lowhanging fruit Delay implementation of features that may be most controversial or problematic 27

28 Requirements for Success Laser-like focus on program improvement Robust data system Buy-in and support of multiple stakeholders Convincing and effective performance measures Using measures to signal appropriate intervention Effective plan to address program weaknesses Considering consequences of program sanctions Commitment to long-term development process Adequate SEA and program capacity 28

29 For Further Information Building an Evidence-Based System for Teacher Preparation (Michael Allen, Charles Coble, Edward Crowe) Contact Michael Allen (202)

30 SUPPLEMENTAL SLIDES

31 Some Measure Challenges & Possible Fixes Small n-size Small candidate sample or small population? Combine candidate scores in related programs Measure performance of latest 3 cohorts Waning Influence of Program In multi-year sample, weight more recent higher? Try to account for diminishing dose in regression equations? e.g., Kentucky s proportional accountability 31

32 Setting Performance Benchmarks Set benchmarks for Goldilocks Especially in the beginning, err on the side of setting them too high Set benchmarks for score distribution, not only score average A piloting phase is a good opportunity to calibrate benchmarks 32

33 Weighting the Indicators A few states do, most states don t Are low-weight indicators truly important? Weighting can maximize or minimize program differences Some programs may not have data on all measures (especially small programs) Indicators conundrum: The measures for the most compelling indicators may be the least representative and reliable 33

34 Evaluating and Improving the System Have programs improved their performance outcomes 3-5 years into implementation? Should performance benchmarks be raised or lowered? Do weightings and performance levels need to be adjusted or reconsidered? Are programs using the performance data to make program changes? Do some indicators or measures need to be improved or jettisoned? Are there new indicators or measures ready for implementation? 34

35 Accountability Impact Questions?? What Impact? How Much? How Long? How? 35

36 Basic Performance Hypothesis Given a large enough number of candidates and completers to account for anomalous situations and controlling for biasing factors, if Program A s candidates and completers score well on a measure and Program B s score poorly, then Program A performs better than Program B in relation to that particular measure. 36

37 A State s Perspective Ashley Bencan Deputy Director Office of Recruitment and Preparation Division of Talent And Performance New Jersey Department of Education 37

38 Deliberative and Confidential New Jersey DEPARTMENT OF EDUCATION New Jersey Department of Education: Educator Preparation Accountability Framework Spring 2016

39 Deliberative and Confidential 39 New Jersey DEPARTMENT OF EDUCATION Vision for Data Use Program Approval - Programs apply to DOE to begin operating - Method of collecting qualitative info Utilize a variety of data points to tell a complete picture of a provider; Embed state-provided outcome data into our accreditation process Annual Performance Review - EPPPR contain a mix of outputs and inputs for all of the programs at an EPP - CAEP Annual Report (EPPAR) submitted by Providers - Title II Federal Reporting Requirement Periodic Review - NJDOE reviews all available data and information to determine continued approval (e.g. accreditation report, and all Annual Performance Data) Accreditation - Accreditation team collects and reviews institutional data, program specific data and institutional context - Self-study, on-site visit, data/outcomes

40 Deliberative and Confidential 40 New Jersey DEPARTMENT OF EDUCATION Current Data We annually produce EPP Performance Reports (EPPPR), that link multiple data systems and report on placement, performance and persistence Data System Description Sample Data Teacher Cert. Info. System (TCIS) NJ SMART Eval. State Cert. Tool (ESCT) NJ SURE Houses all certification information; Needs to be upgraded LEA reporting system houses all public school employment information; not dynamic Application allows each LEA to review and certify their evaluation submission to the state Higher Ed office (separate from NJDOE) maintains a data collection for IPEDS Content Assessments, edtpa, certification area, Survey data Salary, position, years of experience, location, persistence Summative Rating, msgp, Teacher Practice Score, Student Growth Objective scores GPA, Transfers, State of residence, Area of Study Limitations: report at the provider level, limited to traditional-route teacher programs, not yet using data for accountability, static reports, not aligned with accreditation and Title II

41 Deliberative and Confidential 41 New Jersey DEPARTMENT OF EDUCATION Biggest Challenge Appropriate use of data to determine state actions Our periodic review will determine continued approval for our programs o What data elements (and by how much) should define an effective program? o Should we create a tiered system? o How do we measure the impact of this work?

42 Questions & Answers 42

43 Closing Summary of presentations What did you learn and did you make any connections to other State leaders? Questions, Comments, or Feedback? 43

44 Contact Information Michael Allen Ashley Bencan Mid-Atlantic Comprehensive WestEd 44

45 Feedback Survey URL: 45