Global Regional National MICS EGRA/EGMA (11(nationally Total (Grade 2 and 3) Total (Grade 2,3, and

Transcription

1 Goal 4 Target number: 4.1 Indicator Number and Name: Proportion of children and young people achieving at least a minimum proficiency level in (i) reading and (ii) mathematics in grades two and three Agency: UNESCO Institute for Statistics (UIS) 1. Has work for the development of this indicator begun? Yes. Since early 2016 the UIS is coordinating methodological work in two main lines of action some specific to Early Grades but the other (proficiency framework, definition of minimum level and linking methodology) are in need for all three points of measurement (end of primary and end of lower secondary) of indicator Methodological development The reporting format aims to communicate two pieces of information: a. the percentage of students meeting minimum proficiency standards for the relevant domains (mathematics and reading) for grades 2/3; and b. when different programs can be considered comparable and the conditions under which the percentage can be considered comparable to the percentage reported from another country. This requires the following inputs to frame the indicator: a. What contents should be measured and what is the percentage of coverage to be covered by a given assessment to be comparable to others?; b. What procedures are good enough to ensure quality of the data collected?; c. A proficiency scale where all assessments could be informed (and its conversion function or the linking procedure); and d. A definition of the minimum level for each domain that would allow the estimation of the percentage of students achieving the minimum proficiency level Data collection The sources of information for reporting on this indicator are learning assessments classified in various categories: a. School-based: Cross-National assessments (global and regional) and National Assessments b. Population based: Foundation skills assessments and Non- Official Citizen-Led assessments To date, between the Periods , according to UIS database, the most representative group with data are national and regional assessment programs. With some flexibility to accept, reporting on one grade above (Grade 4) global large-scale assessments could be included (PIRLS and TIMSS) as shown in table below: Global Regional National MICS EGRA/EGMA (11(nationally Total (Grade 2 and 3) representative) Total (Grade 2,3, and (11(nationally 4) representative) 2. Who are the entities, including national and international experts, directly involved and consulted in developing the methodology/and or data collection tools? The Education 2030 Framework for Action (EFA) ratified the mandate of the UNESCO Institute for Statistics (UIS) to remain the official source of cross-nationally comparable data on education. Moreover, it also proposes that: In addition to collecting data, the UIS will work with partners to develop new indicators, statistical approaches and monitoring tools to better assess progress across the targets related to UNESCO s mandate, working in coordination with the Education 2030 Steering Committee. In this capacity, the UIS has established two technical groups to discuss and frame indicators composed by countries (in a fair distribution), multilateral agencies and relevant stakeholders. These two groups are:

2 a) Technical Co-operation Group (TCG) to Education 2030 in charge of developing the 43 thematic indicators in education, and providing inputs for indicator reporting; b) Global Alliance to Monitor Learning (GAML) in charge of developing all the learning outcomes and skills indicators in the SDG4. Both groups encompass a variety of stakeholders from cross-national entities such as RTI (EGRA and EGMA), UNICEF (MICS), LLECE/ERCE, CONFEMEN/PASEC, PILNA, SACMEQ, SEAMEAO and IEA (PIRLS, TIMSS. Other institutions consulted include DFID, DFAT, the International Commission on Financing Global Education Opportunity, Education International, FHI 360, GPE, Inter-American Development Bank, and Norwegian agency of International development. 3. What is the involvement of or how do you plan to involve National Statistical Systems in the development of the methodology? The methodological plan was presented to the GAML 4th meeting in November in Madrid and endorsed by the TCG on the Indicators for SDG 4-Education 2030 during its fourth meeting in Dubai in January TCG member states, from all SDG regions, are in most cases represented by the National Statistical Offices. 4. Please briefly describe the process of developing the methodology for the indicator The UIS is coordinating the development of a process to support international consistency in reporting against Indicator to provide the following: 4.1. Conceptual Framework Activities 1. Map national assessments and international assessments in reading and mathematics covering both domains and all points of measurement; 2. Draft a Global Framework for Reference for each domain; 3. Conduct a consultation with experts and countries to finalize a framework proposal; and 4. Elaborate criteria to define satisfactory coverage of any assessment system so that it can be compared to others through the framework Expected Output and deliverables A global assessment framework with a definition of Intended contents and skills for each point and domain; A definition of the satisfactory coverage of the framework for each domain for reporting; and A tool to map the coverage of a given assessment onto that framework Methodological Framework Activities 1. Elaborate on guidelines of good practices including Manual of Good Practices; 2. Develop a tool to assess the quality of procedures; and the definition of a minimum or good enough level of procedures 3. Consult, pilot and validate the tool with countries Expected Output and deliverables A set of guidelines of good practices that ensure procedural consistency (manuals and guides); A definition of the satisfactory coverage of the framework for each domain for reporting; and A tool to map the degree of compliance with the good practices Reporting framework Activities 1. Define an interim reporting strategy that lies within the long terms vision of the UIS reporting strategy and, hence, footnotes will help to guide countries in their efforts; 2. Collect and analyse existent data from cross-national assessments, national assessments, and their characteristics through UIS existing Catalogue of Learning Assessments; 3. Map all proficiency levels of existent cross national assessments with their descriptors, aligned in a continuum for lower to higher grades and levels of proficiency; 4. Map what are the points in each assessment that define the minimum proficiency level and its content descriptors; 5. Based on previous mapping define a proficiency framework including the proposed minimum proficiency level and preliminary performance level descriptors; and 6. Elaborate reporting guidelines for countries Expected Output and deliverables

3 A proficiency reporting framework based on current assessments proficiency descriptors, and to include a common view of the proficiency descriptors; A decision on the minimum proficiency level for each point of measurement and domain and its proficiency definition; and A linking methodology for assessment to report on this proficiency framework through a pedagogical and psychometric linking process that could allow assessment programs to report in a consistent way Consensus building Activities 1. Submit a proficiency framework and minimum level proposals for validation; 2. Submit Reporting Framework and minimum proficiency level proposals for GAML and TCG validation; 3. Inform countries about agreement through the UNESCO permanent delegations; and 4. Submit the developed methodology for a Tier re-classification request to the IAEG-SDGs in November Expected outputs Endorsed Reporting Strategy including the definition of a proficiency framework and the minimum level 7. Please indicate new international standards that will need to be proposed and approved by an intergovernmental process (such as UNSC) for this methodology. The intergovernmental process involved is the one validated by The Framework for Action for Education 2030, the Technical Cooperation Group to education 2030 and the Global Alliance to Monitor Learning. As part of this process the following guidelines and standards will be produced: Manual of Good Practices in Learning Assessments and related quick guides; Global Reference Contents for Reading and Mathematics to guide assessment elaboration; Global Reporting Framework for indicator 4.1.1; and Data Alignment tool to inform on coverage and manual of procedures. 8. When do you expect the methodological work on this indicator to be completed? March Activity 4.1 is finished and all the activities in bold are in the last steps to be completed in the second part of Activities 4.7 will be carried out in the first quarter of Are data and metadata already being collected from the National Statistical System for one or more components of this indicator? Yes. If yes, please describe: Data and metadata are collected through the Catalogue of Learning assessments from national systems that involve both results of the assessments (Module 2) and their characteristics in terms of frequency and procedures (Module 1). Cross-National Learning Assessments will be informed by the organizations that administer the assessment. These include: Programme d'analyse des systèmes éducatifs de la CONFEMEN (PASEC), Progress in International Reading Literacy Study (PIRLS), Tercer Estudio Regional Comparativo y Explicativo (TERCE), Trends in International Mathematics and Science Study (TIMSS), National-level Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA), PAL Network Citizen- Led Assessments, and Multiple Indicator Cluster Survey (MICS). Data on countries alignment in terms of content coverage of Global Content for Reference of reading and mathematics and the Global Reporting frameworks will be assessed through a procedural alignment tool. 10. How do you plan to collect the data? As describe in point 7 above: Direct submission from countries through the Catalogue of Learning Assessments; and Submission from International Organizations, which administer cross-national initiatives. If the indicator involves multiple components from different data sources, please describe how each individual component of the indicator will be collected here. For a given country, the indicator will come from same data source unless the country specifies and justifies two sources of reporting. To date this has not been the case.

4 However, the sources of information are not identical as per detail above. The data could come from organizations administering cross-national assessments or from countries themselves, as detailed above. With what frequency is data expected to be collected? Data collection is annual, but reporting for monitoring will be every three to four years pending last decisions of the TCG about periodicity. 11. Is there a process of data validation by countries in place or planned for this indicator? Yes. If yes, please briefly describe: The produced indicators based on both direct data collection by countries, and by organizations, running the crossnational assessments are being shared with each of the participating countries for their review and validation as per usual UNESCO institute for Statistics procedures. The organizations administering cross-national assessments have a validation process with countries before publishing the data. Despite that mechanism being in place, the UIS is consulting to reassure the countries are aware they have information for SDG4 reporting. Through this process, the Education Ministries take the final decision on their data to be reported. The validation process occurs before each data release and data refresh (September of each year and January/February of the following). 12. If you have any additional comments that you believe would be helpful to IAEG-SDG members in analysing the work plan and methodological development of the indicator, please provide them here: 1. Allowing some flexibility in reporting: plus one minus one grade subject to country s validation. The map below shows the assessment coverage and how it would significantly maximize reporting if flexibility is allowed: 2. Coverage by world population is more relevant than by counting countries as per the table below mainly due to the presence of China and India: Number of Countries Population share World Lower Share Lower World Primary (%) Primary Share (%) Sub-Saharan Africa , North Africa & West Asia South Asia ,869 1, East & South East Asia ,273 1, Caucasus & Central Asia Latin America & Caribbean Pacific Europe & North America ,102 1,018 92

5 (as of July/August 2018)