UNDERSTANDING AND USING PREDICTED TARGETS FOR REGENTS EXAMS AND SLOS

Transcription

1 UNDERSTANDING AND USING PREDICTED TARGETS FOR REGENTS EXAMS AND SLOS DATAG DECEMBER 5, 2014 Department of Curriculum, Instruction and Technology

2 Longitudinal SLO Project Team 2 Brandi Bakshi Strategic Data Project Fellow Jane Boyd IDW Supervisor Jeffrey Davis Data Analyst Pradeep Sahay Cognos Consultant Pat Reinhardt DW Technical Manager Dr. Elaine Zseller, Ph.D., Program Supervisor

3 Overview 3 Need for Formulas Why did Nassau BOCES start this project? Fundamentals of Regression What are the fundamentals of regression? Application of Regression to NYSED historical data How could regression be applied to historical NYSED Regents assessments for courses ending in a Regent? Applying the Equation to the SLO How do we use the equations for writing SLOs? LPS Download in the Nassau BOCES Instructional Data Warehouse How were the formulas implemented within a reporting framework? Comparison of predicted to actual 2014 results Did at least 70% of the students in the county meet or exceed their target for 2014? SLO Formula Updates What is new for ?

4 Summer of Two Cataclysmic Events Earthquake hits NYC. State Education Department (SED) introduces SLOs as part of the new APPR plan.

5 As introduced, SLOs needed. 5 A reliable and valid baseline assessment A sound set of learning objectives A predicted target score for each student A reliable and valid final assessment to measure learning

6 SED s Early Recommendation: Pre-tests 6 SLOs were needed for courses ending in Regents. Often, a prior Regent was used as the pre-test. In year one of SLO implementation, some students spent multiple periods during the first days of classes failing three or more exams. For example, a student taking a Chemistry Regents at the beginning of Chemistry class resulted in scores of 15 or 20. Improvement from that baseline was inevitable. Instructional time was lost administering these exams.

7 Nassau BOCES Solution 7 Is a student s prior test history a better predictor of a student s performance on a Regents exam than an end of course Regent taken in September? District Opinion The formulas are the best indicator of student and teacher success that they have used. We believe that the SLO targets are a more fair and equitable system for creating SLOs than using baseline and summative assessments.

8 An Alternative Solution 8 Use statistics to predict growth Regression More consistent Among students Among teachers More objective Not dependent upon opinion

9 9 Fundamentals of Regression: Section Overview Review correlation and regression Understand Line of Best Fit Observe the effect of manipulating an equation

10 10 Fundamentals of Regression: Correlation Correlation is used to see if a relationship exists between two variables Is there a relationship between students past performance on multiple assessments and success on the current assessment?

11 Fundamentals of Regression 11 How do we use the relationship between variables to make predictions? Example of a prediction question: Is there is a positive relationship between hours of study and regents score?

12 Fundamentals of Regression: Model 12 The reason for collecting data and plotting the collected values is usually to try to find an equation that can be used to model a relationship between variables. In multiple linear regression, there will be more than one input variable. In our project, we used two assessments and three demographics to predict the outcome variable. The process of taking data points and coming up with an equation is called "regression."

13 13 Fundamentals of Regression: Model The graph of the "regression equation" is called "the regression line". This line is also known as the trend line or the line of best fit. Regression works by trying to fit a straight line between the data points (scatter plot), so that the overall distance between points and line is minimized.

14 Fundamentals of Regression: Equation 14 Linear equation, y = a + bx y is the outcome variable, the target a indicates the point on the y-axis crossed by the line and is referred to as the constant b is the multiplier (coefficient) of input variable x is the input variable; the data being used to predict In the following example, y is the predicted Physics target and x is Algebra II/Trig

15 Physics 15 Fundamentals of Regression: Scatter Plot with Trend Line Physics13 Linear (Physics13) Algebra II & Trigonometry

16 Physics 16 Fundamentals of Regression: Begin with the line of best fit (trend line). Approximately 50% of the points are above the line. Write an equation of the line Software is used for this. Linear Regression Physics13 Linear (Physics13) Algebra II & Trigonometry

17 17 Regression Applied: Overview What does regression look like when applied to students historical assessment data and how does it effectively predict a target? Correlation: Predictor assessments (SLO baseline assessments) have a moderate to strong alignment with the target assessment (SLO summative assessment). Longitudinal: Two prior assessments are used to predict to the target year test scores.

18 18 Regression Applied: Strength of Correlation Relationship Correlation Value 0.0 x <x <x <x <x 1.0 Correlation Strength weak modest moderate strong very strong The closer correlation values are to 1(or to 1), the better a fit your regression equation is to the data values

19 19 Regression Applied: How tightly do NYSED Assessments Align? Global Studies 2014 option Global Studies 2012 ELA Grade 8 CONVERTED 2013 Earth Science 2013 Geometry 2014 Global Studies Pearson.683 **.709 **.703 ** 1 Correlation Sig. (2-tailed) N 13,551 5,573 6,357 16,424 Number of students who took 2013 Earth Science and 2013 Geometry: N = 1,518

20 20 Regression Applied: How tightly do NYSED Assessments Align? Global Studies 2014 option Global Studies 2012 ELA Grade 8 CONVERTED 2013 Earth Science 2013 Integrated Algebra 2014 Global Studies Pearson.683 **.709 **.657 ** 1 Correlation Sig. (2-tailed) N 13,551 5,573 7,872 16,424 Number of students who took 2012 ELA Grade 8 (converted score) and 2013 Earth Science: N = 4,997

21 21 Regression Applied: Choosing Predictor Assessments Discussion with educators Assessments closer in time and/or content to the target Correlation coefficient over 0.5 Combination of Predictor Assessments for 2015 Global Studies Predictor Assessment 1 ELA 8 (2013) Living Environment (2014) ELA 8 (2013) Earth Science (2014) Predictor Assessment 2 Living Environment (2014) Geometry (2014) Earth Science (2014) Geometry (2014) Target Test Year Global Studies (2015) Living Environment (2013) Chemistry (2014)

22 Regression Applied: Longitudinal 22 Input variables - predictor variables Historical data - SLO baseline assessments Demographics Output variable - target Assessment score Constant NYSED Assessment 1 NYSED Assessment 2 3 Demographic Variables Target Assessment Score

23 Regression Applied: Prediction 23 The equation is developed using known data. Earth Science Geometry Poverty + LEP + Challenge = Global Studies 2014 predicted score The equation is then used to predict to the unknown. Earth Science Geometry Poverty + LEP + Challenge = Global Studies 2015

24 Adjusting the Regression Equation: Section Overview 24 Purpose of Target Adjustments Target Considerations Tweaking the Formula Formula Results Sample Prediction Equation

25 Physics 25 Adjusting the Regression Equation: Purpose of Target Adjustments The original equation shows about 50% of students above the black line. How can we change this to have a higher percent of students above the line? Physics13 Linear (Physics13) Algebra II & Trigonometry

26 Physics 26 Adjusting the Regression Equation: Purpose of Target Adjustments Purpose adjust formula so that 80% of all students score at or above the predicted target score (or above the line ) % Physics13 Linear (Physics13) Algebra II & Trigonometry

27 27 Adjusting the Regression Equation: Target Considerations The equation could predict scores that were over 100. The highest predicted score was set to 95. This was capped at 95 to benefit teachers of gifted students. The equation also produced target scores that were not whole numbers. Round predicted score down to the nearest whole number This will benefit the teachers.

28 Adjusting the Regression Equation: Comply with SLO Criteria 28 The equation must work for students on free and reduced lunch change the coefficient of the economic variable. The equation must work for students with disabilities change the coefficient of the challenge variable. The equation must work for limited English proficient students change the coefficient of the LEP variable.

29 29 Adjusting the Regression Equation: Tweaking the Formula through a Model

30 30 Adjusting the Regression Equation: Adjust the Formula Tweaking the Formula Are the students meeting or exceeding the 80% target? Meet or Exceed Target (# and %) All Students Demo = 0 Poverty = 1 Challenge = 1 LEP = 1 # 3,459 2,106 1, N 4,324 2,632 1, % 80% 80% 80% 80% 80% Predicted Global Studies Score for 2014 (Adjusted) = (Living Environment Score 2013 * 0. 82) + (Geometry Score 2013 * 0.20)+ (Poverty * -2.98) + (Challenge * ) + (LEP * )

31 Comparison of Unadjusted 2014 to 2015 Global Studies Formula Unadjusted.23 + (Living Environment 2013 *.82) + (Geometry 2013 *.20) + (Poverty * -1.27) + (Challenge * -1.98) + (LEP * -.73) 2015 Adjusted (Living Environment 2014 * 0. 82) + (Geometry 2014 * 0.20)+ (Poverty * -2.98) + (Challenge * -6.65) + (LEP * -4.68)

32 LPS Download Overview 32 Nassau BOCES IDW (Cognos 10) Excel Download Source and Target (Predicted) Student Scores Multiple pathways (handouts, 2014 & 2015)

33 LPS Download Options 33 Student List by District Student List Grouped by Building Student List Grouped by Building and Teacher* * Teachers with Courses Ending in Targeted Assessments

34 State Course Codes (2015) Grade 8 Science English/Language Arts III 01003CC - ELA III (Common Core) Algebra I 02052CC - Algebra I (Common Core) Geometry Trigonometry/Algebra Biology 03051L - Regents Biology Lab Chemistry 03101L - Regents Chemistry Lab Earth Science 03001L - Regents Earth Science Lab Physics 03151L - Regents Physics Lab U.S. History Comprehensive World History and Geography AP Eng. Language and Composition AP Eng. Literature and Composition AP Physics AP European History AP World History AP U.S. History

35 Caveats for Users 35 Students Must Have Two Source Assessments to Receive a Predicted Score Only Students with Source Assessment Scores in Their CURRENT District Receive a Predicted Score Current/Accurate Demographics Are VITAL! Staff-Student-Course Records Must Be Pushed Up to Level 1 Data Warehouse Students May Appear Multiple Times on the Teacher Version

36 Caveats for Developers 36 Formulas Must Be Edited for Use with Existing Reporting Model Thirty-Three Formulas in 2015 Each Formula Checked for Accuracy Up to 132 Possible Checks (Main Formula Plus Three Demographic Flags) Compare Download Results Against Original Spreadsheet Update Formulas for the Current Year Be Aware of Changing Scales!

37 37 Sample LPS Download (2015)

38 38 Sample LPS Download (2015)

39 39 Sample LPS Download (2015)

40 40 Sample LPS Download (2015)

41 41 Sample LPS Download (2015)

42 42 Sample LPS Download (2015)

43 43 Sample LPS Download (2015)

44 44 Sample LPS Download (2015)

45 45 Sample LPS Download (2015)

46 46 Applying the Equation to the SLO: SLO Roster Ms. Smith - Global Student Name Student ID Baseline Living Environment Geometry Score 2014 Score 2014 Target Global Studies Regents Percent of Students - Yes HEDI Score Actual Student Score Met or Exceeded Target Yes or No

47 47 Nassau BOCES Comparison of Predicted 2014 to Actual 2014 The county percent of students meeting or exceeding the target for each assessment ranged from 75.4% to 84.8% except for one of the three Integrated Algebra pathways (65.4%). Within each assessment, there was variation among districts of the percent of students meeting or exceeding their target, 0% to 100%. Within each district their was variation of the percent of students meeting or exceeding their target (see handout). Most districts set HEDI scale at 70% or 75% for effective.

48 48 Nassau BOCES Comparison of Predicted 2014 to Actual 2014 Target Assessment Path N County Percent of Students Target District Variation Minimum District Variation Maximum Global Studies Path 1 3, % 25.0% 100% Global Studies Path 2 4, % 31.3% 100% Global Studies Path 3 4, % 33.3% 100% Global Studies Path 4 1, % 8.3% 100% This table is available for all of the 2014 paths in Appendix C of the report. The report will be available on the DATAG website.

49 49 Nassau BOCES Comparison of Predicted 2014 to Actual 2014 Target Assessment Path N County Percent of Students Target District Variation Minimum District Variation Maximum Earth Science Path 1 3, % 33.3% 100% Earth Science Path 2 2, % 50.0% 100% Earth Science Path 3 3, % 26.7% 98.2% This table is available for all of the 2014 paths in Appendix C of the report. The report will be available on the DATAG website.

50 50 WNYRIC Comparison of Predicted 2014 to Actual 2014 Assessment # of Students Met Target Total Students Percent Met Target Regents Algebra II/Trigonometry - Jun 10,158 13,556 75% Regents ELA - Jun 12,271 16,152 76% Regents Geometry - Jun 15,580 19,983 78% Regents Global History - Jun 16,321 23,757 69% Regents Integrated Algebra - Jun 6,201 8,940 69% Regents Living Environment - Jun 21,004 24,701 85% Regents Physical Setting/Chemistry - Jun 10,141 14,162 72% Regents Physical Setting/Earth Science - Jun 19,571 24,556 80% Regents Physical Setting/Physics - Jun 4,048 5,018 81% Regents US History & Gov't - Jun 18,658 24,172 77% Total 133, ,997 77% This table is available for all of the 2014 paths in Appendix D of the report. The report will be available on the DATAG website.

51 51 Assessment Variation Within Sample District Regent Nassau County % Met or Exceeded District sample % Met or Exceeded Difference Global Studies Global Studies Global Studies Global Studies Earth Science Earth Science Earth Science

52 52 Pathway Summary for 2014 Regent Scores Pathway Prediction Assessment 1 Prediction Assessment 2 Global Studies Pathway 1 Living Environment 2013 Integrated Algebra 2013 Global Studies Pathway 2 Living Environment 2013 Geometry 2013 Global Studies Pathway 3 Earth Science 2013 Integrated Algebra 2013 Global Studies Pathway 4 Earth Science 2013 Geometry 2013 Earth Science Pathway 1 Math Science Earth Science Pathway 2 Science Living Environment 2013 Earth Science Pathway 3 Math Math This table is available for all of the 2014 paths in Appendix B of the report. The report will be available on the DATAG website.

53 53 Pathways for 2014 & Pathways 2015 Pathways 1 - Algebra II/Trigonometry 1 - Algebra II/Trigonometry 3 - Integrated Algebra 3 - Common Core Algebra 2 - Chemistry 2 - Chemistry 1 - Common Core ELA 2 - Comprehensive English 6 - Comprehensive English (Jan (3); Jun (3)) 3 - Earth Science 4 - Earth Science 2 - Geometry 2 - Geometry 4 - Global Studies 5 - Global Studies N/A 1 - Science Grade Living Environment 4 - Living Environment 2 - Physics 2 - Physics 2 - US History 2 - US History 23 Formulas Total 33 Formulas Total

54 54 Technical Report Available on DATAG Website Building a Multiple Regression Model to Set Student Learning Objective Targets for High School Courses Ending in a NYSED Regent Purpose Significance of the study Theoretical framework Problem Solution Methodology Results Appendix A: 2013 Pearson Correlation of Predictor Variables to Target Variable Appendix B: Pathway Summary for Predicted Regents Exam Scores Appendix C: Comparison of Predicted to 2014 Actual Scores Appendix D: WNYRIC Performance Accuracy of the SLO Prediction Formula

55 Contact Information 55 Brandi Bakshi, Harvard Fellow Dr. Elaine Zseller