ADDRESSING CHALLENGES FROM LACK OF DATA IN DISBURSEMENT OF AID TO EDUCATION TO LOW INCOME COUNTRIES EVALUATION OF PRIMARY COMPLETION RATES ESTIMATES

Transcription

1 ADDRESSING CHALLENGES FROM LACK OF DATA IN DISBURSEMENT OF AID TO EDUCATION TO LOW INCOME COUNTRIES EVALUATION OF PRIMARY COMPLETION RATES ESTIMATES A Thesis submitted to the Faculty of the Graduate School of Arts and Sciences of Georgetown University in partial fulfillment of the requirements for the degree of Master of Public Policy in Public Policy By Aneta O. Popiel, M.A. Washington, DC April 19, 2013

2 Copyright 2013 by Aneta O. Popiel All Rights Reserved ii

3 ADDRESSING CHALLENGES FROM LACK OF DATA IN DISBURSEMENT OF AID TO EDUCATION TO LOW-INCOME COUNTRIES EVALUATION OF PRIMARY COMPLETION RATES ESTIMATES Aneta O. Popiel, M.A. Thesis Advisor: Thomas Wei, Ph.D. ABSTRACT Aid to education may impact the country s progress towards achieving universal primary education. Yet, the data which donor institutions use to allocate aid is often missing or miscalculated. That is likely to result in misallocation of aid and impact country s ability to achieve universal primary education. This paper addresses a challenge of the missing Primary Completion Rate data, which Global Partnership for Education (GPE) uses to estimate aid to education allocations. It compares three methods of estimating missing Primary Completion Rate: imputation from average PCR levels (Method 1); inference from the past level of PCR (Method 2), and imputation from country characteristics (Method 3). The efficiency of the methods is evaluated by comparing the magnitude of the estimated error term obtained using each of the methods and reported PCR levels for 44 low income counties. Results show that inference from the past level of PCR outperforms the other two methods. Simulations using the estimated results also reveal that the average deviation from the true allocation may reach 86 percent and 71 percent for Methods 1 and 3, and 33 percent for Method 2. iii

4 Table Of Contents Introduction Impact of foreign aid on Primary Completion Rate Needs and Performance Framework (NPF) Needs for quantitative aid allocation formula Elements of the NPF Challenges in applying NPF missing Primary Completion Rate (PCR) Addressing missing data on PCR Research structure Objective Hypothesis Data Empirical strategy Results Method Method Method Comparison of the three methods Implication for aid allocation Limitations and potential for further research Conclusions Appendix Bibliography iv

5 Introduction The Global Partnership for Education (GPE) is an organization within the United Nations (UN) system that focuses on supporting over 50 low and lower middle income countries in reaching Education for All (EFA) goals, which were established during Education for All Conferences in 1990 in Jomtien, Thailand, and in 2000 in Dakar, Senegal. During these summits, the international community committed itself to achieving six education goals: providing free and compulsory primary education for all, increasing adult literacy by 50 percent, reaching gender equality in access to education, expanding early childhood care and education, promote learning and life skills for young people and adults, and improving the quality of education (Education for All International, 2008). The donors pledged to supply developing countries with relevant financial aid to meet those objectives. The role of the GPE is to mobilize and coordinate financial aid to education, which includes allocating funds to selected countries. One of the biggest challenges that GPE faces, is the scarcity of funds, which may hinder achieving universal primary education before the 2015 target year. To address this problem, as well as, to improve transparency and accountability in funds allocation, GPE adopted the Needs and Performance Framework (NPF) a quantitative formula for allocating funds. The formula includes nine factors that are hypothesized to impact a country s performance in education (e.g., size of the country, poverty level, gender parity etc.). However, some factors are frequently not reported by recipient countries, which can distort how the funds are distributed. Given that the amount of aid to education might impact a country s ability to make progress towards the EFA goals, the objective of this paper is to examine various ways of estimating, when missing, 1

6 one of the nine factors in the allocation formula Primary Completion Rate (PCR) to identify the most accurate approach. This paper is organized as follows: Section 1 shows how progress towards EFA goals in developing countries may depend partially on the amount of aid to education those countries receive. Section 2 presents the challenges that the GPE faces in allocating aid to education because of miscalculated or missing PCR data. It also identifies weaknesses in common methods used to estimate PCR when it is missing. These methods typically involve substituting the PCR with other indicators. I then consider three alternative approaches that directly impute PCR, examining how they compare to each other in terms of generating accurate estimates of PCR (Section 3). In Section 4, I discuss the results of the three approaches and their potential impact on disbursement of aid to education by GPE. Finally, I lay out the limitations of this paper and potential for further investigations (Section 5) and draw final conclusions (Section 6). 1. Impact of foreign aid on Primary Completion Rate Primary Completion Rate is believed to best reflect a country s progress towards universal primary education one of the main EFA goals (Cameron, 2005; Bloom, 2006). It is defined as a total number of students successfully completing (or graduating from) the last year of primary school in a given year, by the total number of children of official graduation age in the population (Bruns et al. 2003; Cameron 2005). Measured in that way, PCR may exceed 100% since it includes overage children who started school earlier or repeated grades 2

7 (Bruns et al., 2003). The accurate estimation of PCR, which translates to accurate disbursement of aid to education by donor agencies like GPE, is only relevant if these resources impact a country s progress towards achieving universal primary education. After the World Education Conference in Dakar, aid targeted at primary education increased from 1.48 billion dollars in 1999 to 3.73 billion in 2004 (Aiglepierre and Wagner, 2010). This increased funding coincided with a decrease in the number of primary school- aged children that were out of school from 108 million in 1999 to 61 million in 2010, and with an increase in the global primary net enrollment ratio from 84% in 1999 to 91% in 2010 (UNESCO, 2012). These trends suggest that there has been movement towards universal primary education and that the increased funding may have played a role. However, research on the effectiveness of aid on education is scare and methodologically fragile, due to limitations of sector- specific aid data and education outcomes (Aiglepierre and Wagner, 2010). Dreher et al. (2008) attempted to estimate the impact of aid to education on PCR for 96 low and lower middle income countries. They did not find a significant impact, although their data was based on a limited sample of countries. Estimated impacts of aid to education on the enrollment ratio revealed that each additional dollar of aid per capita, increases enrollment by 0.26 percent. If aid is measured as a percentage of GDP, increasing aid to education by 1 percent of a recipient country s GDP implies an increase in primary enrollment by percentage points. The findings also suggest that foreign aid to 3

8 education is more effective at raising the enrollment rates than domestic expenditures (Dreher et al., 2008). Similarly, a study by Michaelowa and Weber (2007) provides evidence of the impact of aid on primary, secondary and tertiary education. They find smaller and less robust effects than Dreher et al. Controlling for a number of independent variables (expenditure on education as a share of GDP, pupil teacher ratio, share of children aged 0 14 of total population, and good governance), they found that an increase in aid to education by 1 percent of a recipient s GDP implies an increase in PCR by 2.5 percentage point (Michaelowa and Weber, 2007). Finally, Wagner and Aiglepierre (2010), estimate the impact of aid to primary education with an instrumental variables approach. The instrument for aid used in the model is country s participation in the Fast Track Initiative (FTI), which gave rise to the Global Partnership for Education. Authors used the year of endorsement of a particular country for FTI instead of aid variable because they found that it positively and significantly impacts the amount of aid allocated to primary education without interaction with other variables, such as good governance. They found that participation in FTI, which includes receiving aid to education, positively impacts a receiving country s PCR. An increase of one dollar per capita in FIT funds, increased the estimated PCR by 14.5 percent (Wagner and Aiglepierre, 2010). These findings seem to confirm suggestions made in literature, that the 4

9 impact of aid on education is higher if aid to education, instead of total aid, is used as the explanatory variable (see Michaelowa and Weber, 2007 and Roberts, 2003). In sum, existing research suggests that there is a positive impact of aid on achieving universal education, although the magnitude of the impact varies depending on the methodology and data used. As a consequence, donor institutions, like GPE, which allocate funds specifically for education purposes, may substantially impact the progress towards universal primary education. The methodology they use to allocate funds could therefore be an important instrument in achieving EFA goals in these countries. 2. Needs and Performance Framework (NPF) 2.1. Need for quantitative aid allocation formula GPE currently allocates aid to education among over 50 low and lower middle income countries from a US$2 billion trust fund. Prior to the mid- 2000s, the aid to education was allocated according to a country s policy plans for achieving EFA goals. The evaluation of the plan and the amount each country received were subject to GPE discretion. This encouraged eligible countries to lobby for larger aid amounts. It also privileged those countries that applied for aid first, as there were no procedures for proportional aid allocation, nor a cap on the amount a particular country might have received (interviews with GPE employee, 2012). The new millennium spurred discussions about the effectiveness of foreign aid, which challenged GPE s approach to allocating aid to education. The Paris Declaration in

10 provided a road map to improve the quality of aid and its impact on development, which also included principles of obtaining and measuring development results, and accountability of donors and recipients for these results (The Paris Declaration, 2005). As a consequence, GPE adapted a quantitative formula in 2007 called the Needs and Performance Framework (NPF) to determine the allocation of aid for each country. 2.2 Elements of the NPF NPF is a quantitative tool designed to determine allocations of GPE s aid to education to eligible countries. The NPF consists of The Needs Index and The Performance Index, each of which contains a set of quantifiable variables. The specific elasticity for each factor (the percentage change in a country s GPE allocation brought about by a 1% change in the given factor) is determined by precedence (for example, in those cases where donors have already agreed on a comparable coefficient in IDA s Performance Based Allocations used by the International Association for Development). The table below presents the nine factors in the NPF. The elasticity of each factors is in brackets. The Needs Index contains six factors. The fragility of the country is a measure adopted from the World Bank and indicates countries that score 3.2 or lower on the Country Policy and Institutional Assessment (CPIA), which is the primary tool used to assess the quality of countries policies. The Gender Parity Index in school completion (GPIC) is the ratio of girls to boys who accomplish primary education. The other factors are: distance from the EFA goal (measured by how far is the country from 100% PCR), the size of the country, its 6

11 income and current external financing of education. The Performance Index includes three factors: whether a country implements policies likely to impact education (expressed by Policy and Institution Indicator PII), whether a country demonstrates a financial commitment to education (expressed by the share of GDP allocated to education), and whether a country exhibits progress towards universal primary education (as expressed by changes in PCR for a given country). 7

12 TABLE 1. The quantifiable variables in the GPE s Needs and Performance Framework Equation Needs Index (NI) Performance Index (PI) Variable Measurement Variable Measurement The fragility of the country 1 (0.15) 0-1 variable (FRAG) Policies which are likely to have impact on education development (1) The Policy and Institutional Indicator (PII) The gender parity in school completion (- 0.35) GPIC Country s willingness to commit resources, domestic and foreign, to education sector. (0.3) The Public Educational Expenditure as a share of GDP (E/GDP) The distance from the EFA goal (0.15) Primary Completion Rate (PCR) Progress made by the country in meeting the universal primary education in 2015 (0.3) PCR Progress Indicator (PPI) The size of the country (0.9) Number of children of school age (CSA) Income of the country ( ) Per capita Income (PCI) The volume of current external financing for education (- 0.1) EXT Source: author s interpretation, Global Partnership for Education, 2012 This quantitative framework does not replace qualitative judgment on a case- by- case basis. However, its adoption reflects the need for consistency and transparency agreed upon by the international community. It reduced the transaction costs for the recipient countries, which no longer need to lobby heavily for larger aid amounts. It is also more equitable 1 GPE uses the World Bank definition of the fragile state, which are countries which scoring 3.2 or lower on the Country Policy and Institutional Assessment (CPIA), which is the primary tool used to assess the quality of countries policies. 8

13 since it does not grant a premium to countries who submit their proposal earlier than others (GPE, 2012). 2.3 Challenges in applying the NPF missing Primary Completion Rate (PCR) One of the nine factors in the NPF formula, which recipient countries often do not collect nor report, is the Primary Completion Rate. The main source of the PCR data is UNESCO s Institute for Statistics (UIS), which compiles data provided by the governments and other national authorities. UIS has historically faced problems collecting reliable data from all countries since quantitative education indicators are likely to be misinterpreted or corrupted to a greater degree than indicators in any other discipline (Bloom, 2006). In the case of PCR, many low income countries report only enrollment at the beginning of the year, whereas reporting the PCR also requires the collection of end of year enrollment data (Bruns et al., 2003). As a result, the data measuring progress towards universal primary education is not available for many developing countries. Analysis of global data on PCR (FIGURE 1) shows that among 90 low and lower income countries in years , on average, 44 percent were missing PCR in a given year. The highest proportion of missing PCR data among all countries was reported in 1997 (72 percent), and the lowest in 2009 (27 percent). In 1999 and beyond, missing PCR is much lower than prior to This shift appears to most likely be attributed to the establishment of The UNESCO Institute for Statistics in November

14 FIGURE 1. Proportion of countries missing data on PCR, Source: World Bank Edstats Query FIGURE 3 summarizes missing PCR data across low and lower income countries. From 1990 to 2010 there are countries that have complete data on PCR (e.g. Morocco and Syrian Arab Republic), and countries that have no data on PCR (e.g. Zimbabwe, South Sudan, Somalia, Kosovo and Korean Democratic Republic). 10

15 FIGURE 2. Number of years in which data on PCR is missing,

16 2.4 Addressing missing data on PCR Since the enactment of the EFA goals, the problem of missing data on education has attracted increased attention of scholars and policy makers. A number of studies have tried to address this issue. One approach is to substitute the missing PCR with an alternative measurement (Filmer et al., 2006). Bruns et al. (2003) used the following proxy primary completion rate formula: Proxy PCR = (!!!!"!#$!"#$%&!"!"#$%&"!!"!!!!"#$%!"#$!"!"#$%"&!"!!!"!!"#"$%"!&) (!"!#$!"#$%&!"!!!"#$%&!"!""#$#%&!"#$%#&'()!"#!"!!!!"!#$%&'"() Adopted from: Bruns et al., 2003 Such a strategy entails several weaknesses. First, countries that do not report PCR data often also do not report the data necessary to calculate the proxy. Second, the proxy indicator does not take into account the dropout rates in the final year of primary education, which is not available for most countries. Finally, this formula requires age specific population estimates, which carry a high risk of error especially in countries without a national census due to war or mass migration (Bruns et al., 2003). Barro and Less (2000) produced a measure of education attainment for 142 countries from using adult primary completion rate. They used data on the population aged over 15 and 25, reporting the percentage of youth or adults, who completed primary school. As a result, they estimated a status indicator of PCR, which differs from the PCR 12

17 defined by UNESCO, which is an annual event. The disadvantage of this method is that it does not reflect a country s progress towards achieving universal primary education and does not address the issue of missing data on education (Cameroon, 2005). Finally, the study by Cohen Soto (2005) adopted a strategy of backward or forward extrapolation of primary completion rates from the adult literacy rate. The limitation of such a strategy is that it does not account for variance in mortality rates among high and low educated people. As a result, the backward procedure could overestimate schooling while the forward procedure could underestimate it (Cohen Soto, 2005). None of the above methods the proxy for primary completion rate, adult literacy rate or extrapolation, fully addresses challenges in measuring the PCR. They result in indicators that are static, do not account for demographic changes in the society, and which are calculated with data that is often missing. As result, they do not solve the problem of missing PCR in allocation of aid to education. 3. Research structure 3.1. Objective The objective of this research is to examine various ways of estimating one of the nine factors in the NPF formula Primary Completion Rate to identify the most accurate approach. The findings will help GPE allocate their funds in a way that is closest to the intended allocation based country s needs and performance. 13

18 This paper provides new evidence on how efficient different methods of estimating PCR are by comparing the estimated results with the reported PCR for countries with available PCR data. The paper compares three methods of estimating PCR data: imputation from average PCR level of country that report PCR; inference from the past level of PCR, and imputation from country characteristics (e.g. GDP per capita, size of population of school- age, etc.). The main difference between this paper s approach and earlier measures resides in the effort to assess the accuracy of various methods by comparing their estimated results with the reported PCR through a series of simulations. 3.2 Hypothesis This paper tests the following hypotheses: for countries being targeted by GPE, when data on the Primary Completion Rate are missing, the most accurate way to impute the missing data is to impute based on past data. But if past data are not available, the next most accurate method is to impute based on socio economic characteristics of the country. Either of these methods is more accurate than imputing based on the raw group average. 3.3 Data The primary data set for this research is a database of over 2,500 internationally comparable education indicators Edstats. This data set has been recently compiled by the World Bank drawing from the UNESO Institute for Statistics, Organization for Economic 14

19 Cooperation and Development, and other international agencies. The data set encompasses country level data for 216 countries as far back as the 1970s. The analysis is on 36 low and 54 lower middle income countries (see list of countries Annex 1) in the period Empirical strategy In this study I assess the accuracy of estimating missing PCR data with three different methodologies. The sample I use in this study includes 44 countries that have at least 3 PCR data points available, including the PCR in The logic for the empirical strategy is to simulate real world circumstances, in which the PCR is estimated using one of the three methods. To achieve that, a different random sub sample of 22 countries is drawn 1000 times from the full sample of 44 countries. Each random sub sample of 22 countries is treated as if they were missing the PCR in 2011 ( PCR missing countries ). The sub sample size was selected to mirror the proportion of countries that actually did not report PCR in 2011 (~50 percent). For each random sub sample, the other half of the sample not selected are treated as PCR reporting countries. Then one of the methods below is performed on the selected samples. Method 1. Calculate the average PCR for the PCR reporting countries. This average is the estimated value for each of the PCR missing countries. 15

20 Method 2. Estimate PCR for PCR missing countries based on their past PCR levels. The estimating equation takes the following form: y = β0 + β1x1 Where the dependent variable y is PCR, and the independent variable is a year dummy (X1). The coefficient β1 is the estimated change in PCR that corresponds to the passage of one year. This equation is estimated separately for each PCR missing country and the estimated coefficients are used to predict the 2011 PCR for each country. Method 3. Estimate the PCR based on country characteristics. The cross sectional country level equation for 2011 takes the following form: y = β0 + β1x1 + β2x2 + β3x3 + β4x4 Where the dependent variable y is PCR, and the independent variables are GDP per capita (X1), over- age enrollment (X2), pupil teacher ratio ( X3), Gender Parity Index (X4). The equation is estimated separately for each of the PCR reporting countries and the estimated coefficients are used to predict the 2011 PCR for PCR missing countries. All three methods result in an estimate of PCR in 2011 for the 22 randomly selected sub sample of countries, which is repeated 1000 times for each method. For each method, I compare the resulting estimated PCR points to the reported PCR for those countries by calculating the absolute value of the deviation between the estimated PCR and reported PCR. Then for each method, I examine the distribution of the deviations, as well as 16

21 measures of central tendency (mean, median). This provides a standard way to compare the accuracy of each of the three methods. Finally, I use the NPF formula to present the deviation in aid allocation that results from applying the three methods of estimating PCR. 4. Results 4.1. Method 1 Estimating PCR based on the average PCR of PCR reporting countries could be accurate if countries missing and reporting PCR were similar in terms of other characteristics related to PCR. However, the analysis of characteristics of those countries shows that they differ substantially in several regards. FIGURE 5 shows the percentage of countries missing PCR data on every continent in The highest percentage of PCR missing countries is in East Asia & Pacific and South Asia (over 70 percent), and the lowest is in Europe & Central Asia (over 20 percent). In absolute terms, Sub Saharan Africa has the largest number of countries with missing PCR (17 out of 40). 17

22 FIGURE 3. Percentage of PCR missing and reporting countries in 2011, by region. Source: Source: World Bank Edstats Query; number is parenthesis indicates number of countries in the region as defined by the World Bank. TABLE2 2 compares the characteristics of PCR reporting and PCR missing countries. The first group have lower GDP per capita than those countries missing the data in However, the proportion of education expenditure allocated to primary education is higher in PCR reporting countries than. They also have a lower Gender Parity Index and higher over age enrollment, on average. The data did not reveal significant differences between PCR missing and PCR reporting countries with regard to per pupil spending, drop out rate in primary education and pupil teacher ratio. 18

23 TABLE 2. Comparison of PCR missing and reporting countries in Characteristics of countries ( ) Group I Countries missing PCR Group II Countries not missing PCR Proportion of years missing PCR (%) * GDP per capita (constant US $, 2000) * Per pupil spending as % of GDP per capita Expenditure on primary education as % total education expenditure * Gender Parity Index for gross enrollment * Over age enrollment ratio in primary education (%) * Drop out rate in primary education Pupil teacher ratio Total sample size = 90. Countries missing PCR in 2011 = 45. Countries not missing PCR in 2011 = 45 Significant difference between the means of the two groups at 5% significance level are marked with asterisk * Gender Parity Index (GPI) for gross enrollment: gross enrollment in primary education is the ratio of the female- to- male values of the gross enrollment ratio in primary education. A GPI of 1 indicates parity between sexes. ** Over age enrollment ratio: the sum of the number of pupils (total, male, female) in each grade of primary school who are one or more years older than the official age for that grade, expressed as percentage of the number of pupils (total, male, female) attending primary school. *** Drop out rate in primary education: the proportion of pupils (total, male, female) in any grade of primary in a given school year and who no longer attend school the following school year. ****Pupil teacher ratio: the number of pupils enrolled in primary school divided by the number of primary school teachers. Source: Author compilation based on Edstats; Definition of indicators: UNESCO Institute of Statistics As suggested in TABLE 2, PCR missing and reporting countries appear to differ on several characteristics that may be related to PCR. The examination of PCR levels for 2011 also shows that the variation in PCR among countries that report PCR data is substantial. In 2011, the average PCR for the entire sample was 78 percent, and its minimum and 19

24 maximum values were 38 and Assuming that the variability in PCR amongst PCR missing countries is somehow comparable to the variability in the PCR amongst PCR reporting countries, imputing with a group average could lead to substantial over or underestimation of the true PCR for each of the PCR missing country. This result may lead to aid allocations that do not reflect the real need and performance of these countries. The process of imputing the average PCR of the PCR reporting countries to PCR missing countries over 1000 simulations resulted in the following results. TABLE 3. Method 1 summary statistics for the absolute value of deviations across the 1000 random samples of 22 countries. Average Median Standard deviation The average 4.98 percentage points is the average of the absolute value of deviations of the average PCR of the PCR reporting countries from the reported PCR of the PCR missing countries. The median a standard deviation from the reported PCR are 4.26 and 3.84 percentage points. 2 Although PCR is a ratio, its value can exceed 100%, which can be a symptom of late entry or grade repetition. In the sample, 24 countries have PCR which exceeds 100 percent in at last one year in the considered period: Albania, Armenia, Belize, Bolivia, Cape Verde, Fiji, Georgia, Guyana, Indonesia, Kiribati, Kyrgyz Republic, Marshall Islands, Mongolia, Myanmar, Philippines, Sri Lanka, Samoa, Syrian Arab Republic, Tajikistan, Tonga, Uzbekistan, Vanuatu, Vietnam, and Zambia. 20

25 For example, the reported PCR for Moldova in 2011 is percent. As a result of Method 1 simulation, Moldova was assigned PCR which varies from to percent, which is an overestimation in comparison to its reported PCR. For Lao PDR, PCR estimations are also consistently higher than the reported PCR (92.61 percent) they vary from to percent. In case of Cote d Ivoire, all simulation rounds of Method 1 result in PCR estimations lower than the percent reported PCR (they vary from to percent), showing the underestimation of PCR which resulted from Method 1. The distribution across the 1000 simulations of the average of the absolute value of deviations for each sample of 22 countries is presented in the figure below (FIGURE 4). Note the min and max of the deviations across the 1000 simulations. FIGURE 4. Distribution of the average of the absolute values of deviations from the reported PCR in 1000 simulations. Frequency Average of the absolute value of deviations 21

26 4.2 Method 2 The weakness of the first method lies in its cross country character, where average PCR of a group of countries is ascribed to other countries that have potentially different characteristics. The alternative approach is to apply a linear extrapolation to predict the PCR for a given country for year 2011, based on its past PCR. Such estimation will provide good results only if the trend in PCR is linear and stable over time. FIGURE 5 below plots PCR over time for selected countries. With a few exceptions, they suggest that the linearity and stability assumptions may be reasonable, in which case one might expect that this method will provide accurate estimations of PCR. FIGURE 5. PCR in selected countries,

27 Using a single regression for each country with PCR as the dependent variable and year as the independent variable, I estimated a time trend for each country based on its pre PCR data. Using the time trend estimate, I extrapolated PCR to 2011 for each of the 22 countries in each randomly selected sub sample. After repeating the process 1000 times, I compared the estimated PCR with the reported PCR of the PCR missing countries and obtained the following results. 23

28 TABLE 4. Method 2 summary statistics for the absolute value of deviations across the 1000 random samples of 22 countries. Average Median Standard deviation The average of the absolute values of deviations of estimated PCR from the reported PCR is 1.74, the median is 1.54, and the standard deviation is This summary statistics show that the estimated PCR I obtained in Method 2 should be closer to the reported PCR for individual countries in the sample, than once obtained in Method 1. For example, Moldova s reported PCR in 2011 is 91.09, and PCR estimated with Method 2 is percent (the PCR estimations with Method 1 varies from to percent). For Cote d Ivoire, the reported and estimated PCR obtained with Method 2 are and percent (Method 1 estimates varies from to percent). Lao PDR s reported PCR for 2011 is 92.61, whereas Method 2 estimate is percent (Method 1 estimates varies from to percent). In case of all three countries, the distance between the estimated PCR and reported PCR is smaller for Method 2 than for Method 1. The distribution across the 1000 simulations of the average of the absolute value of deviations for each sample of 22 countries is presented in the figure below (FIGURE 6). Note the min and max of the deviations across the 1000 simulations. 24

29 FIGURE 6. Distribution of the average of the absolute values of deviations from the reported PCR in 1000 simulations. Frequency Average of the absolute value of deviations 4.3 Method 3 The proposed third method for estimating PCR is doing so, based on country characteristics related to education. Using the cross sectional country level regression for 2011 and PCR as dependent variable I tested several configurations of the independent variables to see which ones best explained the variation in PCR in The regression results are summarized in TABLE 5. 25

30 TABLE 5. Regression analysis of the key parameters driving PCR in Dependent Variable: Primary Completion Rate in 2011 (1) (2) (3) GDP per capita (Constant US dollars) ** (.004) Over- age enrollment (.183) Pupil- teacher ratio ** (.004) (.163) (.179) ** (.003) (.153) (.186) Gender Parity ** (30.830) Constant ** (5.420) ** (8.797) (32.567) Observations R- square Adjusted R- square For all specifications the dependent variable is the Primary Completion Rate in All the education indicators concern primary education. Standard errors are reported below each estimate. *significant at 10%, **significant at 5%. According to the results above, the model that explains the most variation in PCR is as follows: y = β0 + β1x1 + β2x2 + β3x3 + β4x4 Where the dependent variable y is PCR, and the control variables are GDP per capita (X1), over age enrollment (X2), pupil teacher ratio ( X3) and Gender Parity Index (X4 ). I used 26

31 the above model separately for each of the PCR reporting countries and used the estimated coefficient to predict the 2011 PCR for PCR missing countries. The process of estimating the PCR based on country s characteristics over 1000 simulations gave the following results. TABLE 6. Method 3 summary statistics for the absolute value of deviations across the 1000 random samples of 22 countries. Average Median Standard deviation The 6.17 percentage point is the average of the absolute value of deviations of the reported PCR from the true of PCR missing countries. The median a standard deviation are 5.15 and 4.83 percentage points. If Method 3 was applied to estimate the missing PCR for Moldova, the country might have been assigned PCR varying from to percent, whereas its reported PCR is Similar disparities between the reported PCR and the estimated PCR are true for Cote d Ivoire and Lao PDR. Cote d Ivoire s reported PCR in 2011 is and the estimates from Method 3 indicate to PCR from to percent; in case of Lao PDR the estimates varies from to percent (the reported PCR is 92.61). 27

32 The distribution across the 1000 simulations of the average of the absolute value of deviations for each sample of 22 countries is presented in the figure below (FUGURE 7). Note the min and max of the deviations across the 1000 simulations. FIGURE 7. Distribution of the average of the absolute values of deviations from the reported PCR in 1000 simulations. Frequency Average of the absolute value of deviations 4.4 Comparison of the three methods TABLE 7 contains comparison of summary results I obtained executing three methods of estimating PCR and comparing it to the reported PCR of randomly selected countries in the sample. 28

33 TABLE 7. Summary statistics for the three methods. Average Median Standard deviation Method Method Method According to the reported values of average, median and standard deviation, the method which estimates PCR that is likely to be the closest to the reported PCR is estimation based on past PCR data for a given country (Method 2). The average (1.74), median (1.54) and standard deviation (1.20) of the average of the absolute value of deviations for Method 2 have values lower than those obtained in other two methods. These results are confirmed by overlaid kernel density curves of the distribution of the average of the absolute value of deviations in three methods (FIGURE 8). The absolute value of deviations from the reported PCR is most highly condensed around zero for Method 2. 29

34 FIGURE 8. Comparison of kernel density curves of the distribution of the average of the absolute values of deviations from three methodologies. Density Method 1 Method 2 Method Absolute value of standard deviation Note the simulation results for selected countries in TABLE 8 that summarizes information about the reported PCR for Moldova, Cote d Ivoire and Lao PDR as well as the minimum and maximum difference between the reported and estimated PCR. 30

35 TABLE 8. Simulation results for three methods for selected countries. Country Moldova Cote d Ivoire Lao PDR Reported PCR (%) Difference between reported PCR Estimated PCR (percentage points) Method 1 Min * Max Method 2 Difference Method 3 Min Max * Negative number indicates overestimation in relation to the reported PCR. The higher the number, the larger the overestimation. Estimates for three selected countries indicate that Method 1 and 3 predict PCR which is over- or underestimated in comparison to the reported PCR. The maximum difference between the true and reported PCR vary from percentage points (Method 1, Lao PDR) to percentage points (Method 3, Moldova). The minimum difference vary from percentage points (Method 1, Moldova) to percentage points (Method 3, Lao PDR). Differences obtained with Method 2 for the three countries are smaller than the minimum differences obtained in Method 1 and 3: 2.49 (Moldova), (Cote d Ivoire), and (Lao PDR). This confirms that Method 2 is the most accurate in estimating PCR closest to the reported PCR. 31

36 4.5 Implications for aid allocation The PCR estimates obtained in three methods allow a comparison of the difference in the amount of aid each country may receive depending on the method that is applied to estimate its missing PCR. In order to show the marginal effect of reported PCR and estimated PCR on the disparity in amount of allocated aid I used the NPF formula. The formula, instead of raw PCR, uses distance to universal primary education, which is a difference between 1 (indicating 100% universal primary education) and PCR (expressed as a proportion). I calculated distance to universal primary education for 44 countries in the sample using reported PCR and estimated PCR and multiplied that estimate by 0.15, the elasticity of distance to universal primary education in the NPF formula. To obtain the marginal effect of the distance to primary education in aid allocation, I assumed that all other factors in the formula are constant. I also assumed that the potential amount of aid is US$100,000,000, which is a maximum grant amount a country may receive from GPE. TABLE 8 summarizes how application of reported PCR and estimated PCR to the NPF formula, may impact, on the margin, the amount of aid countries receive. 32

37 TABLE 9. Deviation between the amount of actual allocated aid and imputed aid allocations. Average of actual allocated aid for 44 countries (US$ millions) Absolute value of deviations between the actual allocated aid and imputed aid allocations (US$ millions) Average Median Standard Deviation Method 1 3,234,012 2,770,452 2,443,425 1,582,462 Method 2 3,234,012 1,065, , , Method 3 3,234,012 2,294,513 2,027,175 2,027,175 According to TABLE 8, Method 2 results in aid allocation which is closest to the actual allocated aid allocation, where reported PCR is used. The average, median and the standard deviation of the absolute value of deviations of the amount of aid are the smallest when Method 2 is used. I come to the same conclusion by comparing the distribution of imputed aid allocation and the actual grant allocation for the 44 countries in the sample. FIGURE 9 shows the plots of each allocation against the 45 degree line, which corresponds to all points where the imputed and actual aid allocations are equal. 33

38 FIGURE 9. Imputed aid allocation vs. the actual aid allocation, by country. Method 1. Imputed aid allocation (US$ 000) countries 45 degree line Actual aid allocation (US$ 000) Method 2. Imputed aid allocation (US$ 000) countries 45 degree line Actual aid allocation (US$ 000) 34

39 Method 3. Imputed aid allocation (US$ 000) countries 45 degree line Actual aid allocation (US$ 000) Aid allocations for the 44 countries are mostly clustered around the 45 degree line in Method 2, confirming that the difference between the actual aid amount and the imputed aid amount are likely to be the smallest when Method 2 is applied. 5. Limitations and potential for further research This research presents several limitations. First of all, it has not been able to account for the measurement error present in UIS and World Bank Edstats data. The estimation of the missing PCR were obtained using existing PCR data, with an assumption that reported PCR is accurate. However, as literature suggests, PCR is likely to be misinterpreted or corrupted (Bruns et al., 2003; Bloom, 2006). 35

40 Second, each method can only results in accurate estimates, under specific conditions. Method 1 is likely to produce accurate results, if PCR missing and PCR reporting countries are comparable. The analysis of these two groups revealed that they differ significantly with regard to several characteristics (e.g. GPP per capita, expenditure on primary education as % total education expenditure, Gender parity Index and over age enrollment ratio in primary education). Method 2 is likely to be an accurate approach only if PCR level over time is stable for a given country. As data for selected countries showed, this condition is likely to be met within sample used for this research. Finally, data and sample size allowed only for a limited multivariate regression model for the estimation of PCR based on country s other characteristics (Method 3). It is likely, that with a larger sample size and richer set of covariates, the model would produce more accurate PCR estimates. It would also be desirable to access the efficiency of a hybrid model, which would combine Method 2 and 3. In such hybrid model past PCR and other characteristics of the country could be used to predict current PCR level. 6. Conclusions This research was motivated by the incidence of missing education data, i.e. Primary Completion Rate, which is used by Global Partnership for Education to disburse aid to education to low income countries. I compared three methods of estimating missing PCR: imputation from average PCR levels, inference from the past level of PCR, and imputation from country characteristics. I performed the analysis on 44 PCR reporting countries, out of 36

41 which half were treated as PCR missing countries to compare the reported PCR to the estimated PCR obtain with the three methods. The results reveal that imputing PCR based on country past PCR levels, rather than average PCR of PCR reporting countries or country characteristics, is likely to produce an estimate which is closest to reported PCR. Estimating PCR based on past trends and applying it to GPE s fund allocation formula results also in amount of aid allocation closest to the actual amount of aid, estimated with the reported PCR. In conclusion, Global Partnership for Education has made a substantial progress in addressing challenges of transparency and fairness by adopting Needs and Performance Framework in allocation of aid to education. However, the challenge of missing data used in the NPF formula may impact the amount of grant an individual country receives. Considering that aid directed specifically to education is likely to increase progress towards universal primary education (Aiglepierre and Wagner, 2010; Dreher et al. 2008; Michaelowa and Weber 2007), it should be a priority of grant donors, like GPE, to increase effort of collecting accurate data from low income countries and apply most accurate methods of estimating missing data. 37

42 Appendix 1. Low and lower middle income countries according to the World Bank classification. 3 Countries in bold receive aid from GPE. Low income countries Afghanistan Gambia, The Mozambique Bangladesh Guinea Myanmar Benin Guinea Bissau Nepal Burkina Faso Haiti Nigeria Burundi Kenya Rwanda Cambodia Korea, Dem. Rep. Sierra Leona Central African Republic Kyrgyz Republic Somalia Chad Liberia Tajikistan Comoros Madagascar Tanzania Congo, Dem. Rep. Malawi Togo Eritrea Mali Uganda Ethiopia Mauretania Zimbabwe Lower middle income countries Albania India Samoa Armenia Indonesia Soa Tome and Principe Belize Iraq Senegal Bhutan Kiribati Salomon Islands Bolivia Kosovo South Sudan Cameroon Lao PDR Sri Lanka Cape Verde Lesotho Sudan Congo, Rep. Marshall Islands Swaziland Cote d Ivoire Micronesia, Fed. Sts. Syrian Arab Republic Djibouti Moldova Timor Leste Egypt, Arab Rep. Mongolia Tonga El Salvador Morocco Ukraine Fiji Nicaragua Uzbekistan Georgia Nigeria Vanuatu Ghana Pakistan Vietnam Guatemala Papua New Guinea West Bank of Gaza Guyana Paraguay Yemen, Rep. Honduras Philippines Zambia Source: World Bank Development Indicators, Angola is the only upper middle income country which is a recipient of aid from GPE. 38

43 Bibliography Bloom E. D. (2006). Measuring Global Education Progress. Project on Universal Basic and Secondary Education: Cambridge, Mass: American Academy of Arts and Science Bruns B., Mingat A., Rakotomalala R. (2003). Achieving Universal Primary Education by 2013 A Chance for every Child. Washington D.C: World Bank Cameron L. (2005). Primary Completion Rates, Technical Paper WP 09 01, Education Policy and Data Center, Washington, D.C. Dreher A., Nunnenkamp P., Thiele R. (2008). Does Aid for Education Educate Children? Evidence from Panel Data, The World Bank Economic Review, 22 (2), , doi: /wber/lhn003 Heyneman S. P. (2005). Foreign Aid to Education: Recent U.S. Initiatives Background, Risks, and Prospects, Peabody Journal of Education, 80(1), Education International (2008). Education for All by Education International s response to the Global Monitoring Report Brussels: Education International Global Partnership for Education (2012). Revised Needs and Performance Network, BOD/2012/01 DOC 01, Washington D.C. UNESCO (2012). Education for All. Global Monitoring Report Youth and Skills. Putting Education to Work, Paris UNICEF (2001). Delamonica E., Mehrotra S., Vandemoortele J. Is EFA Affordable? Establishing the Global Minimum Cost of Education For All, Siena 39