Board of Education Executive Summary Accountability Office: Center for Research and Evaluation June 26, 2012

Transcription

1 I. History: Overview of Center for Research and Evaluation: The Charlotte-Mecklenburg Schools Board of Education adopted Research and Evaluation Policy IL in May It states that the Superintendent will ensure that the administration develops and maintains the ability to conduct expert and reliable research and evaluation of selected programs, including programs already in place in the district as well as those proposed for purchase. The Center for Research and Evaluation (CRE), which started in the summer of 2007, is largely responsible for carrying out the stipulations of this policy. This department falls under the Research, Analysis and Data Utilization Department within the Office of Accountability. The CRE currently employs a director, and two full-time senior analysts. The CRE provides results that support the continuation, termination or modification of programs, initiatives, or policies to ensure every child receives a quality education. The end goal for CRE staff is to positively impact the achievement of all CMS students through the appropriate use of research and evaluation. In , a project management process was implemented in an effort to organize the flow of work in and through the CRE and to ensure that evaluation projects are prioritized and completed, with results disseminated in a timely manner. This process was revisited and refined this year as part of the ISO 9001 certification process. As a result of this, three CRE processes were identified and individually mapped Evaluation Process, Annual Survey Process and Outside Research Process. Depending on the project, the CRE performs interim or end-of-program evaluations. Interim evaluations are conducted to provide feedback on programs for enhancement purposes while end-of-program evaluations are conducted to determine program effectiveness. Literature reviews are written for all program evaluation projects, but also when needed by the superintendent or other CMS staff. Evaluation priorities are determined by the superintendent s cabinet. The CRE completes separate annual district surveys of students, parents, teachers and principals. These surveys consist of core items that provide a basis for annual comparisons as well as other items as needed. The CRE also conducts needs assessments and product evaluations as requested. Center for Research and Evaluation 1

2 Outside research requests are addressed by the CRE through a proposal review process. This is the process that researchers follow if they desire to conduct research within CMS and that is not CMS sponsored. To do their study, these researchers must complete an application, pay a processing fee (sometimes waived), provide an approval letter from an Institutional Review Board, and sign a Memorandum of Understanding with CMS if the research is approved by the Chief Accountability Officer. For its evaluations and several other projects, the CRE identifies external researchers to collaborate with. This can involve the outside researcher taking the lead and CRE doing quality control checks or vice versa. As a result, CRE staff work diligently to build and maintain relationships with professors at local universities. Additionally, measures are taken with respect to CRE staff to ensure continuity and quality assurance. CRE staff members are cross-trained to ensure that there is not a dependence on one staff member to be able to perform a given task. The benefits of this are to lessen the impact in the event of the loss of CRE staff members and to allow for flexibility in task assignments. This also increases our capacity for internal quality assurance. II. Driving Governance: Board of Education Policy Code IL: Research and Evaluation 1. Research and Evaluation A. General parameters The Charlotte-Mecklenburg Board of Education believes that all programs, initiatives and products ( programs ) in place within the school district should contribute to the mission of Charlotte-Mecklenburg Schools: to maximize academic achievement by every student in every school. Therefore, the Superintendent shall ensure that the administration develops and maintains the ability to conduct expert and reliable research and evaluation of selected programs, including programs already in place in the district as well as those proposed for purchase. III. Budget: Funding for the CRE goes directly to fund three full-time employee positions. The positions funded include a director and two senior analysts. The total salaries and benefits for these positions is $271,844. Center for Research and Evaluation 2

3 IV. Goals, Objectives and Measures: Intermediate Goals Supply data to inform policy decisions Implement effective programs Support budgetary decisions Target professional development Long-Term Goal Increase student achievement Objectives Conduct program evaluations to determine the level of implementation and what works in CMS schools. Meet all deadlines on assigned program evaluations (provided priorities in projects stay the same). Implement and analyze all annual CMS survey data (parent, teacher, principal, and student). Process completed outside research applications within 30 business days of submission. Build collaborative efforts with individuals, universities and/or vendors to assist with aspects of assigned program evaluations. Measures Project management process implemented in Quality control processes implemented in ISO 9001 processes implemented in Continued refinement of these processes to ensure goal attainment V. Major Initiatives: Evaluation Reports Completed (by June 30) o Data Wise o K-3 Intensive Reading o Small Schools - Final In Progress o Pre-K 8 o Post Secondary Success Feasibility o Risk Factor Scorecard o Math Forward Center for Research and Evaluation 3

4 o Bright Beginnings (contracted) o The New Teacher Project (working with TNTP staff) 2. Outside Research All research studies conducted within CMS must go through the CRE. Researchers submit an application outlining their research. This application is reviewed by an ad-hoc committee formed of CMS staff members. Prior to being approved by CMS all projects must be approved by an Institutional Review Board. If the project is approved, the CRE develops a Memorandum of Understanding (MOU) with the researcher that outlines the researcher s rights and responsibilities. The CRE processes approximately 100 outside research applications each year. 3. Literature Reviews and Data Requests Examples of Literature Reviews o Best Practices in Elementary School Scheduling o Differentiated Staffing in High Poverty Schools o Impact of Bright Beginnings Programs o K-8 Education Models: A Literature Review o Relationship between District Funding and Student Achievement Examples of Data Requests o New Leaders for New Schools (RAND) o Mathematica/Optimal Solutions Group o Mathematica/Branch Associates o The New Teachers Project o Communities in Schools o YMCA o A Child s Place o No Easy Walk o Schools and Internal Offices 4. Surveys Survey Projects o Teacher Survey o Student Survey o Parent Survey o Principal Survey Center for Research and Evaluation 4

5 5. Collaboration As mentioned above, the CRE identifies an outside researcher to collaborate with on every project. In addition, the CRE assists other partner organization with their research requests and data needs. These collaborative partnerships include: o The UNC Charlotte Institute for Social Capital The CRE has representatives on the Board, Data and Research Oversight Committee and the Data Review Committee o University Partners UNC Charlotte Harvard University Queens University University of South Carolina o Non-profit Organizations The New Teachers Project New Leaders for New Schools Citizen Schools Communities in Schools Teach for America A Child s Place VI. Results: District leadership continues to depend on the CRE for not only large scale program evaluations, but also smaller projects such as literature reviews, data analyses, brief research studies and product evaluations. Results from the CRE have been used in a variety of districtlevel programmatic decisions. In the past, Read 180 was eliminated, Midwood High was continued, and modifications were made to DIBELS and Teach for America. Currently, because of the Data Wise evaluation, the Data Tools team was realigned so that there is a representative devoted to a specific Zone. VII. Lessons Learned/Next Steps: The CRE has policies and procedures in place that have increased the efficiency and effectiveness of processes and systems, and increased the accuracy of the results presented. However, there are far more requests than our current staffing structure are able to provide. Therefore, it is imperative that we devise innovative ways to expand our services to cover the need. Lastly, going through the process of ISO certification has greatly enhanced our ability to onboard new employees because of the detailed documentation of processes and procedures that now exists. Center for Research and Evaluation 5