Augmentation: An Implementation Strategy for the No Child Left Behind Act of

Transcription

1 policy report Augmentation: An Implementation Strategy for the No Child Left Behind Act of 2001 Stacy Hicks-Herr Jenny Hoffmann (based on a paper by Margaret Jorgensen, PhD) June 2003

2 Augmentation: An Implementation Strategy for the No Child Left Behind Act of 2001 Introduction With the passage of the No Child Left Behind Act of 2001 (NCLB), states have the opportunity to redefine and enhance their assessment programs to support the diagnostic needs of students, educators, and parents Harcourt Educational Measurement (Harcourt) has developed a strategy that combines the best features of norm-referenced and standards-based test designed to ensure that states comply with the annual reporting requirements of NCLB The augmentation strategy described in this report allows Harcourt s Stanford Achievement Test Series, Tenth Edition (Stanford 10) to be powered up to exactly meet a particular state s NCLB-driven requirements as well as measure standardsbased performance and precisely identify targeted instruction opportunities Harcourt s strategy gives states a variety of options to consider such as the development of customized test items, tests, and forms as they map out short- and long-term goals for educational achievement and accountability Increasingly, a state s assessment program must provide information about student performance as compared to a national reference group and as measured to state-specific instructional standards The various approaches to a successful augmentation strategy must be examined in terms of the concerns of various partners in education State Policymakers: Members of the Legislature, State Superintendents, and other State Officials These individuals are focused on developing a mechanism to evaluate academic performance and progress from the school level to the state level Data generated by a state s assessment program should assist in the accountability and reporting requirements of NCLB District Administrators: Local Superintendents, Principals, and other District Officials These individuals require a combination of diagnostic and accountability information to improve teaching and learning 2

3 Teachers Teachers need timely and targeted information to better meet the instructional needs of their students Parents Parents need information that they can understand and use to help their children learn Members of the Community These individuals, whether or not they have children in school, are likely to place a high value on the quality of education in their locality The challenge is to build an assessment program with the depth, breadth, and stability necessary to: (1) help states fully comply with NCLB; (2) meet the needs of all stakeholders; (3) produce both norm-referenced and standards-based scores; and (4) maximize learning for all students Development of an Augmented Stanford 10 Methodology and Workflow The development of a customized assessment requires the constant consideration of each of the following steps Each step in the workflow informs the next 1 Review of state standards 2 Development of the test blueprint 3 Development of item specifications 4 Identification and assembly of suitable content 5 Test item review by Harcourt 6 Test item review by state 7 Test form construction 8 Final review and approval by Harcourt and state Each of these steps is discussed in detail below Step 1 Review of State Standards Harcourt begins the development of a standards-based assessment with a thorough and rigorous study of the state s instructional standards and curriculum framework (Note: Because the terminology used by states in their content standards documentation 3

4 varies, the terms instructional standard or standard as used in this report are interchangeable with other similar terms such as content standard, objective, benchmark, indicator, etc) All involved must have a clear understanding of what each instructional standard is intended to cover with regard to both content and level of cognitive complexity This includes understanding the underlying pedagogical and curriculum philosophy A thorough understanding of the state s instructional standards is the foundation on which the real work of applying Harcourt s augmentation strategy can begin Step 2 Development of the Test Blueprint A test blueprint is a guide for test construction that details the content areas and instructional standards covered by the test It also shows which standards are assigned to each reporting category (content area) as well as the types and numbers of items in each reporting category Since the passage of NCLB, the creation of test blueprints for standards-based testing programs has become more challenging because many states have more instructional standards than can be reasonably measured in an on-demand multiple-choice test setting Of course, most states also have standards that are not appropriate to measure in on-demand testing situations, such as standards that assess oral fluency or require the reading of books from a reading list However, the standards that states must consider when applying an augmented assessment strategy are only those suitable to an on-demand multiple-choice test setting States must first select those standards that can and should be the focus of a multiplechoice augmented assessment program The next step in the process of blueprint creation is for the state to decide which of its standards should be assigned to each reporting category With Harcourt s assistance, states will review their options and make these decisions from a content, psychometric, and instructional utility perspective Harcourt s experience has shown that in the majority of cases, states will devise their test blueprints using one, or a combination of, the following three approaches: 1 Measure and report on every standard This approach would be useful in terms of supporting targeted instruction and student learning However, for most states this would result in extremely long testing times, even at a minimal rate of only four to six items per standard 2 Identify superordinate standards The assessment would encompass all standards but with only the selected superordinate standards being reported This strategy would result in reasonable testing times but not provide the level of detail necessary to target instruction and student learning 4

5 3 Identify a critical core of standards This assessment would consistently encompass and report on a selected core of critical standards, while other, less critical standards would be tested and reported on periodically This approach would result in reasonable testing times as well as support targeted instruction and student learning Once the state has arrived at a suitable approach to the construction and use of an augmented assessment program, the test blueprint can be finalized The result is a plan for what will be measured, how it will be measured, and how the results will be reported Step 3 Development of Item Specifications The next step in applying the augmentation strategy is for the state to review existing (or develop new) item specifications that define item-specific properties such as format, content, number of options, etc The item specifications should include: Descriptions of eligible and ineligible content Sample test items with rationales for correct and incorrect answers Links to the state s existing curriculum framework These specifications would then be used later in the augmentation process to guide the development of new test items and evaluate all test content for appropriateness Step 4 Identification and Assembly of Suitable Content States that want to develop an augmented assessment program capable of producing norm-referenced scores as well as standards-based information must begin with the items in each Stanford 10 subtest identified by Harcourt as core items This core, rooted in the integrity and 80-year history of the Stanford Achievement Test Series, provides the continuity and national comparability that states require It also serves as a valid, compact foundation upon which a tailored assessment program can be built Beginning with the Stanford 10 core items, the state, with assistance from Harcourt, must perform an initial standards match or content gap analysis to identify areas where augmentation will be required The test blueprint and item specifications developed in steps 1, 2, and 3 should be used to guide this process Although it is desirable to have as many of the Stanford 10 core items as possible match the state s blueprint and item specifications, it is likely that some of these existing items may not be suitable (Note: Core items deemed not suitable cannot be eliminated if the state wishes to produce norm-referenced scores) Furthermore, it is almost certain that many gaps will be identified where the test must be augmented with additional items to cover all aspects of the state s blueprint These gaps will be filled using existing items from Harcourt s Augmentation Item Bank and, if necessary, by creating new items 5

6 The Harcourt Augmentation Item Bank All items currently in the Harcourt Augmentation Item Bank have undergone a thorough review by Harcourt content specialists, a bias and sensitivity review by our bias review panel, and a technical review by the Harcourt content review committee Following are some of the points considered in examining these items: Does the item measure the standard it intends to measure? Does the item measure the content as taught by expert teachers? Does the item conform to best practices in item design? Is the content of the item accurate? Is the item unique to the item bank? Is the language of the item clear and concise? The initial reviews are followed by a review by a senior content specialist who analyzes the item for accuracy and best testing practices The items are then reviewed by editors who make sure that age-appropriate vocabulary as well as proper spelling and grammar are used Editorial specialists also correct items that do not conform to the Stanford style guide (eg, correct use of italics, underlining, indentation, etc) At this point, Harcourt would select the additional content required from the Harcourt Augmentation Item Bank in accordance with the state s finalized test blueprint and item specifications If the available content pool does not contain items that match every state specification, Harcourt would create new items as necessary to match the state s blueprint and item specifications Slightly more items than specified would be selected/created by Harcourt to allow for possible rejection when the items are reviewed at the state level (see Step 6 below) An ideal assessment program has the breadth and depth required to produce scores that are meaningful while keeping testing times reasonable The focus of Harcourt s assistance to the state during the augmentation process is to optimize this return on investment from the state s perspective Small adjustments made by the state during the development of the blueprint and item specifications can dramatically increase the suitability of existing Stanford items and reduce the need to develop new content Step 5 Test Item Review by Harcourt In order to identify items that would function properly for the state s assessment program, the content selected in Step 4 above would be reviewed by Harcourt assessment specialists to ensure that: each item s p-value fell within the specified range; 6

7 the distracters functioned appropriately; the items represented an appropriate range of different p-values; and the items were not biased against any group The assessment specialist ensures that the items considered for inclusion in the state s test forms contain suitable content and exhibit appropriate psychometric qualities The assessment specialist documents the review of all content by preparing an item review checklist The reviewed content and checklists are then provided to the state for review Step 6 Test Item Review by State The test items assembled and reviewed by Harcourt are typically reviewed at the state level by a committee convened for this purpose Harcourt frequently travels to the state and serves as a facilitator during these committee meetings The item review committee is usually made up of school administrators, curriculum and instruction experts, and representatives of various groups and populations enrolled in the school system Representatives of the state s education agency may attend to clarify policy issues and to answer questions about the assessment program The goal of the external review process is for the state to review all content item by item and verify that all is acceptable Some items may require minor revisions before they are accepted A few may be rejected or require replacement Additional committees may conduct reviews to ensure that content is not biased based on gender, socioeconomic status, ethnicity, or special condition Step 7 Test Form Construction Using the items that are accepted by the state during the external review process, Harcourt assessment specialists and psychometricians build the test forms Each form is built to accurately reflect the test blueprint for both content and psychometric requirements In addition, special care is taken to ensure that answer keys are balanced and that clueing is not introduced Step 8 Final Review and Approval by Harcourt and State Once the final test forms are assembled for the state by Harcourt, a final internal review is conducted, verifying the integrity of the technical characteristics of each form The internally approved test forms are then sent to the state for final review and approval At this point, Harcourt would accommodate any changes the state may have until the final test forms are approved by both Harcourt and the state for publication 7

8 Conclusion Although the No Child Left Behind Act of 2001 sets forth requirements related to the testing of specified content, collection of student information, and reporting of scores, each state will take a different approach to the development of a comprehensive, long-range assessment program Furthermore, compliance with federal and state testing requirements may mean that states will still have a range of options from which to choose as they configure their testing programs Harcourt will assist states as they navigate these decision-making complexities States should contact Harcourt for information about developing a fully compliant and valid assessment system Additional copies of this document are available from: Harcourt Educational Measurement Bulverde Rd San Antonio, TX (customer service) (fax) 8