Florida Association of Professors of Educational Leadership (FAPEL) Florida Educational Leadership Examination (FELE) Spring Meeting January 19, 2016 Tallahassee, Florida 1
Postsecondary Assessment Team for Today Phil Canto, Bureau Chief Dr. Suleyman Olgar, Scoring and Reporting Supervisor Lauren White, Scoring and Reporting Team Dan Moore, Test Development Team 2
Today s Postsecondary Assessment Presentation Present status of current FELE program Discuss status of research for Next Gen FELE Review 2015 data (critical implementation year) Review closely the FELE holistic scoring process Provide sample materials and inform Q & A 3
Educator Quality BHAG Every Florida teacher and leader is prepared, developed, supported, and supervised by educators to make teaching better. 4
FELE Program 411 No planned substantive changes to current FELE content, structure, passing scores, or examination fees in 2015 or 2016; Ongoing enhancement and expansion of M/C item pools for all three subtests We constantly monitor to ensure quality control; With recent SBE changes to school grades, there are no impacts to FELE items and prompts. 5
FLDOE Research for Next Generation FELE FLDOE is evaluating several options for performance based or enhanced assessment(s) May impact FELE and/or other exams in the future (current or new) End of 2016 to 2017 before pilot project could begin with statewide inputs Normal test development process is at least 2 years Biggest question is, What is the current FELE not giving us that a performance assessment might? 6
FELE 3.0 Program Changes January 1, 2014 (New Test) FELE 3.0 Examination Subtest 1: Leadership for Student Learning Subtest 2: Organizational Development Subtest 3: Systems Leadership Effective January 1, 2015 (New Format & Passing Scores) Subtest 3: Systems Leadership Multiple Choice Section Written Performance Section (Examinees must pass both sections independently to pass FELE Subtest 3) 7
FELE Recommended Passing Scores, Projected Pass Rates vs. 2015 Operational Pass Rates FELE Subtest Recommended Number Correct Recommended Percentage Correct Projected Passing Rates Operational Pass Rates* N % Subtest 1: Leadership for Student Learning 48/70 69% 58% 1,617 63% Subtest 2: Organizational Development 48/69 70% 55% 1,568 58% Subtest 3: System Leadership: Multiple- Choice Section 36/55 66% 1,620 50% 34% Subtest 3: System Leadership: Written- Performance Section 7/12 59% 1,618 46% * Operational pass rates are calculated for the first-time examinees taking FELE subtests from January 2, 2015 to December 31, 2015 with the approved passing scores at the November 2014 SBE meeting which became effective by January 1, 2015. Note: FELE Subtest 3 written performance assessment is holistically scored by two human raters using a scoring rubric of 1 6 range. FELE Subtest 3 written performance assessment score is the sum of the two ratings. 8
FELE 3.0 First-Time Examinee Pass Rates (2014 vs. 2015) FELE 3.0 2014 2015 N Pass Rate Mean Score N Pass Rate Mean Score FELE Subtest 1: Leadership for Student Learning 1,495 84% 219 1,617 63% 202 FELE Subtest 2: Organizational Development 1,485 90% 225 1,568 58% 200 FELE Subtest 3: System Leadership (Multiple-Choice Section) 1,485 82% 221 1,620 50% 197 FELE Subtest 3: System Leadership (Written Performance Section) N/A N/A N/A 1,618 46% 7 FELE Assessment Level 1,475 73% N/A 1,523 22% N/A 9
FELE 3.0 Best Attempt Examinee Pass Rates (2014 vs. 2015) FELE 3.0 2014 2015 N Pass Rate Mean Score N Pass Rate Mean Score FELE Subtest 1: Leadership for Student Learning 1,495 87% 219 1,617 70% 202 FELE Subtest 2: Organizational Development 1,485 93% 226 1,568 69% 200 FELE Subtest 3: System Leadership (Multiple-Choice Section) 1,485 86% 222 1,620 63% 197 FELE Subtest 3: System Leadership (Written Performance Section) N/A N/A N/A 1,618 59% 7 FELE Assessment Level 1,475 81% N/A 1,523 42% N/A 10
FELE Performance by Competency (2014 2015) Test Name Competency Name 2014 2015 Florida Educational Leadership Examination (FELE) N % Correct N % Correct FELE Subtest 1: Leadership for Student Learning FELE Subtest 2: Organizational Development 01 (17-18) Facilitate student learning goals 69.7% 69.8% 02 (18) Prioritize student learning via leadership 70.6% 74.6% 03 (15-17) Develop/implement instructional 1,495 1,572 71.5% 71.6% framework 04 (17) School environment for student learning 68.4% 70.5% 01 (11-12) Recruitment and induction practices 72.2% 74.1% 02 (22-23) Faculty/staff development and retention 69.5% 71.6% 1,485 1,522 03 (16-17) Develop leadership w/in the organization 70.9% 71.9% FELE Subtest 3: System Leadership (Multiple-Choice Section) 04 (17-18) Personal/professional behavior practices 70.3% 71.0% 01 (13-14) Decision-making processes 62.7% 67.2% 02 (14) School operation management practices 69.8% 68.5% 1,485 1,574 03 (14) Financial and resource management 64.4% 66.9% 04 (13) School legal practices and applications 64.8% 66.2% FELE Subtest 3: System Leadership 05 (1) Effective communication practices N/A N/A 1,471 62.6% (Written Performance Section) 11
Score Verification Score verification eligibility: A non passing score of 190 or higher Written Performance component FELE Number of Sessions Number of Items Reviewed Number of Status Changes 2014 2015 2014 2015 2014 2015 Subtest 1: Leadership for Student Learning 11 25 119 197 0 0 Subtest 2: Organizational Development 3 22 59 163 1 0 Subtest 3: Systems Leadership Multiple-Choice Section Subtest 3: Systems Leadership Written Performance Section 23 23 254 (14) 212 1 1 0 42 0 45 0 0 12
FELE Written Performance Assessment (WPA) & Holistic Scoring Interpretation of data and writing ability as it relates to the data analysis and target audience; Organization, sentence structure, and mechanics, as well as the accuracy of data interpretation An examinee response is judged for its total effect; Rater makes an overall evaluation taking multiple performance factors into consideration Responses are rated independently by two raters and sent through a discrepancy resolution process if rater scores vary by more than 1 point. 13
FELE WPA Responses & Holistic Scoring Process Scoring Rubric identifies performance characteristics to be judged, includes the criteria for evaluating those features, and describes how performance varies across the scoring scale Anchor Set is used to calibrate raters to the rubric Supplemental Rating Criteria lists the critical elements of data use by score points Calibration/Practice Set includes pre scored responses and used for rater calibration prior to operational scoring 14
FELE WPA Rubric & Score Scale SCORE 6 5 4 3 2 1 CRITERIA The writing sample has a clearly established topic that the writer fully develops with specific details and examples, including accurate and thorough data interpretation, explanation, and application. The writer clearly describes the purpose of the communication and reflects an understanding of the target audience with plans for change appropriately aligned with data findings, all relevant implications, and analysis of trends. Point of view is consistently maintained. Organization is notably logical and coherent. The writer demonstrates superior facility in the use of proper grammar and syntax. The writing sample has a clearly established topic that is adequately developed and recognizable through specific details and/or examples, including mostly accurate and thorough data interpretation, explanation, and application with minor errors in details not affecting conclusions. The writer adequately describes the purpose of the communication and reflects an understanding of the target audience with plans for change appropriately aligned with data findings, relevant implications, and analysis of trends. Point of view is mostly maintained. Organization follows a logical and coherent pattern. The writer demonstrates infrequent errors in the use of proper grammar and syntax. The writing sample has an adequately stated topic that is developed with some specific details and/or examples, including some components of data interpretation, explanation, and application. The writer describes the purpose of the communication and reflects a basic understanding of the target audience with some unexplained terms or references. Plans for change may omit a few necessary details or relevant implications in aligning data findings and analysis of trends. Point of view is somewhat maintained. Organization is mostly logical and coherent. The writer demonstrates satisfactory use of proper grammar and syntax. The writing sample states a topic that is developed with generalizations, with some accurate components of data interpretation, explanation, and application. The writer somewhat describes the purpose of the communication and reflects a marginal understanding of the target audience with several unexplained terms or references. Plans for change omit some necessary details or relevant implications in aligning data findings and analysis of trends. Explanations of some data elements may be faulty, and placement of data-related details may not be effective. Point of view is ambiguous. Organization is occasionally illogical or incoherent. The writer demonstrates some errors in the use of proper grammar and syntax that do not detract from the overall effect. The writing sample presents an incomplete or ambiguous topic, with most components of data interpretation, explanation, and application omitted or inaccurate. The writer poorly describes the purpose of the communication and reflects little understanding of the target audience with numerous unexplained terms or references. Support is developed with generalizations with little description of plans for change or relevant implications in aligning data findings and analysis of trends. Explanations of data elements are faulty, and placement of data-related details is ineffective. Point of view is confusing and distracting. Organization is frequently illogical and incoherent. The writer demonstrates serious and frequent errors in proper grammar and syntax. The writing sample has no evident topic. Numerous components of data interpretation, explanation, and application are missing or inaccurate. The writer fails to describe the purpose of the communication and does not address the target audience. Plans for change and/or relevant implications in aligning data findings and analysis of trends are minimally addressed. Development is inadequate and/or irrelevant. Point of view has not been established. Organization is illogical and/or incoherent. The writer demonstrates severe and pervasive flaws in the use of proper grammar and syntax. 15
Sample FELE WPA Prompt #1 (High School ABC) 16
Sample SRC for FELE WPA Prompt #1 (High School ABC) 17
Sample FELE WPA Prompt #2 (Middle School CDE) 18
Sample SRC for FELE WPA Prompt #2 (Middle School CDE) 19
Questions Call 850 245 0513 Visit http://www.fldoe.org/accountability/assessments/posts econdary assessment 20
21