NOCTI Score Reports. Our goal is to provide useful scoring information for students, teachers, administrators, and parents.

Transcription

1 i

2 NOCTI Score Reports NOCTI s assessment package includes a comprehensive score report. Our standard score report summarizes each participant s written and/or performance assessment results, and provides both norm and criterion-referenced scores at a national, state and local level. Educators can use this data to recognize student achievement, promote continuous program and instructional improvement, and meet regulatory requirements. Score reports are available in functional formats, including downloads that allow schools to easily make use of the data. An overview and sample reports are provided in this publication. We hope that you will find this information useful as you implement a new assessment program or continue to manage an existing program. In addition to this guide, existing clients can access numerous resources available at the Client Services Center, and our customer service specialists are available by phone or online to clarify any additional reporting questions you may have. Our goal is to provide useful scoring information for students, teachers, administrators, and parents. ii

3 Table of Contents Score Interpretation Guide Overview...1 Individual Score Report...2 Figure 1 Sample Individual Score Report...2 Group Score Report...3 Figure 2 Sample Group Score Report...3 Analysis of Scores...4 Figure 3 Sample Analysis of Scores...4 Specialized and Customized Reports...5 Figure 4 Sample from a Task Link Report...5 Figure 5 Sample Demographic Analysis Report...6 Figure 6 Sample Comparison to Cut Score Report...6 Figure 7 Sample Year to Year Comparison Report...7 Figure 8 Sample Integrated Academics Report...8 General Information about NOCTI Score Reports...9 Scoring Terminology...10 iii

4 Score Interpretation Guide Overview The next several pages present examples of the score reports included in NOCTI s standard assessment package. A percentage score for each duty area and/or performance job as well as total scores are provided for every participant. Comparative data is also provided for the group, site, state, and nation. Below is an explanation of each data category. Pre-test/Post-test A total post-test score is reported for each participant as well as for each duty and/or performance job. Pre-test scores are reported, when applicable, and the percentage change from pre-test to post-test administration is included at the duty level and for the total score. % of Change Compares the pre-test score to the post-test score. This is the percentage of gain or loss for the named participant. If a pre-test was not administered, the pre-test and percent of change columns will both be blank. Group Group scores reflect the average of the entire testing group for the test code and test date referenced. Site Cumulative The site cumulative data is the average of all participants who have taken an assessment at a particular location since the assessment was released for use and includes the current testing group. State State data represents the average score for all participants in a particular state who have taken an assessment since it was released for use.* Nation This data refers to the average score for all participants who took this assessment anywhere in the nation.* *NOCTI computes state and national normative data in January and July so these scores represent the average of all participants who have taken this assessment prior to January or July of the current year. For example, score reports generated and viewed in May of the year will reflect computed averages from January and will remain the same until the averages are computed in July. 1

5 Individual Score Report Score reports are provided for each participant (Figure 1) and include scores for each duty area/performance job as well as a total score. Participants that complete both a pre- and post-test will receive pre- and post-test score information along with the percentage of change. Pre/post comparative data is a powerful tool for teachers and school administrators to assist in identifying and correcting the factors that impact student learning and performance. This report can be shared with participants and their parents and can also be included in a student portfolio. Sample Individual Score Report Figure 1 2

6 Group Score Report The group score report (Figure 2) is a summary of the assessment results for all participants within the particular test group. Participants are listed alphabetically with individual scores displayed for duty areas/performance jobs and a total score. Group averages can be used to identify strength and gap areas in the program. Sample Group Score Report Figure 2 3

7 Analysis of Scores The analysis of scores (Figure 3) provides useful comparative data that can be helpful in making inferences about the observed assessment results. Group, site, state, and national averages are displayed for both total assessment and duty areas/performance jobs. State and national averages are updated in January and July each year. Group and site averages are updated with every assessment administration. Sample Analysis of Scores Figure 3 4

8 Specialized and Customized Reports NOCTI offers both specialized and customized reports that provide additional or more detailed information. Examples of the types of specialized and customized reports that can be created are listed below. Costs of the reports vary by the type of information requested. Task Link Report Provides in-depth information on an individual test taker s performance on the various tasks within a duty. By utilizing a Task Link Report (Figure 4), teachers can easily identify strengths and areas for improvement by task for the individual test takers. Task Link Reports are available for all Job Ready assessments. Sample from a Task Link Report Figure 4 5

9 Demographic Analysis Report (Figure 5) - Prepared based on NOCTI s demographic questions which include gender, ethnic category, age, and grade point average. Additional demographic questions can be added to meet the needs of a particular school such as designations for students involved in Tech Prep, articulation, dual credit, Career and Technical Student Organizations (CTSOs) and work-based learning options. These reports can also incorporate comparison to cut score by demographic group if desired. Sample Demographic Analysis Report Figure 5 Comparison to Cut Score Report (Figure 6) - Provides the percentage of test taker scores meeting the cut score if a cut score has been determined for an assessment. The data can be displayed by the test title at a specific school, district, or state level. This report is customized to specifically meet the needs of the school, district or, state. Sample Comparison to Cut Score Report Figure 6 6

10 Year-to-Year Comparison Report (Figure 7) - Being able to measure year-to-year gains is an important piece of an assessment program. Below is an example of a two-year comparison by test title and school. State and national averages are also provided for comparative purposes. Introductory text, tables and charts can to be designed around the specific needs of the testing program. NOCTI works with a number of states that request year-end reporting. Reports are modified annually to include the data considered most important and include comparisons of instructional programs, test titles, regions, educational reference groups, and demographic designations. Sample Year-to-Year Comparison Report Figure Annual Averages Annual Averages Accounting Accounting Assessment Year 1 Assessment Year 2 Year Year 1vs. 1 Year School 1 School 2 School 3 School 4 Statew ide Overall National Overall School 1 School 2 School 3 School 4 Statew Average ide Overall National Average Overall Average Average Year 1 Year 1 Year 2 Year 2 7

11 Integrated Academics Report (Figure 8) - The Integrated Academics Report shows when students respond correctly to items that are matched to academic standards. This information provides clues to develop strategies for formative assistance to students. It is intended as a guide to point out areas of overlap so that concepts can be reinforced, and as a guide for upcoming instruction. The example below shows the student performance on the ten items in the Carpentry assessment which have been identified as addressing the Math standard referenced. The number of items answered correctly is displayed, as well as the percentage correct of those possible, and the average for the testing group. One conclusion based on this example could be that more emphasis may need to be placed on Geometry concepts. Sample Integrated Academics Report- Figure 8 8

12 General Information about NOCTI Score Reports Score reports become available approximately two weeks from the time online tests are released to NOCTI, or answer sheets are received at the NOCTI headquarters. Score reports remain in the Client Services Center for one year from the test date. Saving score reports to your local drive ensures that the data will be available after the year has lapsed. NOCTI score reports are designed to provide valuable data. Each score report has both a View option and an Excel option. The View option provides a pdf document that can be printed or saved to your local drive or other media. Selecting the Excel option opens the score report in a spreadsheet format, which allows for manipulation of data as needed. 9

13 Scoring Terminology This glossary provides definitions of terms used in this text. For many of the terms, multiple definitions can be found in the literature. Please note that technical usage may differ from common usage. Accountability - Reporting of student, program or institutional data for the purpose of program improvement. Assessment Blueprint - Assessment blueprints provide a list of the competencies for each assessment with a summary of the skill areas covered, sample questions from the written test, a sample job from the performance test and a percentage breakdown of the test content. The assessment blueprints are available for download at Assessment (Summative) - An assessment measure of achievement at the end of an instructional unit, course of study, or program. Authentic (Performance) Assessment - Assessment technique that involves the gathering of data through systematic observation of a behavior, process or product and the evaluation of those based on a clearly articulated set of performance criteria, which serves as the basis for evaluation judgments. Cognitive (Multiple Choice) Assessment - The process of systematically gathering test scores and related data in order to make judgments about an individual s ability to perform various mental activities involved in the processing, acquisition, retention, conceptualization, and organization of sensory, perceptual, verbal, spatial and psychomotor information. Also written test. Competency - Demonstration of the ability to perform a specific task or achieve specified criteria. Competency Listing - (Also critical competencies, core competencies, standards, duties and tasks.) Terms used interchangeably on NOCTI web site, referring to the broad categories and particular skills covered on a specific assessment. Criteria - The set of standards used to evaluate or rate an individual s performance of a process or the resulting product. With respect to NOCTI performance assessments, criteria are provided with defined levels of expected performance. Criterion-Referenced Assessment - An assessment that measures student progress toward specific curriculum goals or standards. Scores are reported as comparisons to predefined acceptable levels of performance rather than through comparisons to other students. 10

14 Scoring Terminology (Continued) Cut Score - An established score which an individual needs to achieve on an assessment to demonstrate minimal competency. Demographics - Characteristics of human populations (e.g., gender, ethnicity, etc.). Duty - This is an arbitrary division of a job (or occupational area) consisting of related tasks that are grouped for descriptive purposes. Duties are usually stated as general areas of responsibility, with action words ending with ing. Item - (Also question ) a term used to refer to a question on a multiple choice assessment or a job and task in a performance assessment. N - On NOCTI assessment reports, stands for number, and equals the number of participants included in each of the groups. For example on Figure 3, for the group, N=10, and for the Site Cumulative, N=25. Performance (Psychomotor) Assessment - Assessment technique that involves the gathering of data through systematic observation of a behavior, process or product and the evaluation of those based on a clearly articulated set of performance criteria, which serves as the basis for evaluation judgments. Spatial and psychomotor skills are evaluated as they pertain to the criteria. Also called hands-on or practical assessment. Post-Test - An assessment measurement administered after program completion, usually for the purpose of documenting attainment of program-related objectives or comparing the score to an earlier pre-test measure on the same content. Pre-Test - An assessment measurement administered prior to course initiation, usually for the purpose of identifying existing skills and/or knowledge or for comparing post-test measurement of the same content. Standard Deviation - The square root of the deviance, used as a measure of dispersion or spread among a group of scores. The number of points an individual scored above or below the national average. Standard Error of Measurement - An estimate of the amount of variation to be expected in a particular score on a particular test. This statistic provides a range of scores within which a test subject's true score is likely to fall. Task - A meaningful unit of work activity generally performed on the job by one worker within a limited period of time. A task must be logical and necessary to achieving a single objective or output. Task Link - Links the individual s performance to the various tasks within a competency that is used to evaluate specific strengths and weaknesses. 11