Review of the World Bank's Portfolio of Public Sector Management Projects. Findings on PSM Project Performance. Working Draft for Consultation

Transcription

1 Review of the World Bank's Portfolio of Public Sector Management Projects Findings on PSM Project Performance Working Draft for Consultation 6 June 2011

2 Contents I. Introduction... 3 II. Key Findings... 4 A. Question 1. Governance and PSM Project Performance Methodology Key Results... 5 B. Question 2. PSM Project Performance across Thematic Areas Methodology Results... 6 C. Question 3. Project Design Features and (PSM) Project Performance Delays in Project Preparation Frequency and Number of Team Changes... 7 III. Data Annexes... 9 A. Annex I: Governance and PSM Project Performance Correlations between different Measures of Civil Liberties and Political Rights Freedom House Civil Liberties and Political Rights Indicators B. Annex II. PSM Project Performance by Theme C. Annex III. Project Design Features and (PSM) Project Performance Time between Project Milestones Frequency of Team Changes... 24

3 I. INTRODUCTION This note summarizes selected preliminary findings from a review of the World Bank s Public Sector Management (PSM) portfolio conducted to inform the PSM Approach for 2010 to It focuses on three questions related to project-performance: Question 1.a. Do PSM projects perform less well than non-psm projects? To explore this question, the review constructs a measure of success rate differentials between PSM and non-psm projects for each relevant client country. Question 1.b. Does the governance context explain any performance differential? More specifically, do PSM projects perform distinctively worse than projects in other sectors in countries with poor governance? Evidence on how the governance context affects the riskiness of investments in PSM reforms is important for informing a theory of PSM change that should underlie the PSM Approach (see the companion note). This note tests whether countries with high differentials in success rates are also poorly governed, using data on civil liberties and political rights (Freedom House and Polity IV) as a proxy measure of (one aspect of) governance. Question 2. Are there noteworthy differences in project performance between different PSM reform areas, i.e. between civil service, public financial management and tax projects? Specifically, are performance differences across these areas as measured by IEG outcome ratings consistent with those found by the IEG review 1 of the Bank s PSM portfolio in 2008, based on CPIA changes? Compared to CPIA ratings, IEG outcome ratings as a measure of success have the advantage that they are clearly attributable to the project intervention, but the disadvantage that the degree of progress they measure is not clear. Question 3. Which project design features predict project performance? Specifically, the note focuses on (i) project team changes and (ii) long project preparation times as predictors of project performance. The first question is relevant, as many TTLs of PSM projects have highlighted frequent team changes as detrimental to project performance in the staff survey conducted for the PSM Approach (see the companion note). The portfolio review sought to test whether these subjective views can be supported with objective data. The second question is relevant as long delays in project preparation may be proxy indicators of limited client commitment and could serve as a useful early warning sign or risk predictor for the Bank s risk management approach. Each of the following three sections summarizes the key findings for these three questions, respectively, and highlights key methodological issues. All findings reported are preliminary and subject to further verification and review. The project sample underlying this review comprises all projects approved between FY 1990 and 2010 with available IEG outcome data. A narrow and a broad definition of PSM projects are employed. The board definition comprises all investment and development policy lending / adjustment projects that 1 Word Bank (2008) Public Sector Reform: What Works and Why? An IEG Evaluation of World Bank Support, The World Bank, Washington D.C. 3

4 include at least a 25 percent component relating to public sector themes 2 or sectors 3. This broad definition comprises a large number of downstream PSM reform projects. The narrow definition refers to the subset of these projects that are mapped to the Public Sector Governance Board. The narrow definition thus mostly includes upstream PSM reform projects, i.e. projects that focus on PSM reforms at the center of government. Project performance is consistently measured based on IEG outcome ratings. 4 II. KEY FINDINGS A. Question 1. Governance and PSM Project Performance 1. METHODOLOGY The review proceeds in two steps to identify whether the governance context has a distinctive impact on PSM projects. First, the average performance differential between PSM projects and similar non-psm projects is measured within each country. To this end, each PSM project is matched with three similar 5 non-psm projects in the same country. The average success rate differential is calculated for each country by subtracting the weighted average of PSM project performance in the respective country from the weighted average of the matched non-psm projects performance. Figure 1 shows the resulting success rate differentials plotted on a World Map, employing the narrow definition of PSM projects. It is important to note that reported success rate differentials within each country are not statistically significant, given the small number of PSM projects per country in the project sample. Second, countries are grouped to test whether success rate differentials between PSM and non-psm projects differ if aggregated among countries with high and low indicators of civil liberties and political rights. Countries are grouped into free, partially free and non-free countries, based on the Freedom House country classification. 6 Within each of these groups, the success rate differential between PSM and non-psm projects is estimated. More precisely, the review employs a (nonparametric) nearest-neighbor matching identification strategy with exact matching at the country level. Compared to a cross-country regression that controls for observable country characteristics, this strategy effectively mitigates the risk of biased estimates due to unobserved country-level characteristics (such as growth). However, it is still vulnerable to bias because of omitted projectlevel characteristics (such as prior analytical work, team quality, counterpart commitment etc). 2 Public sector themes include theme codes 25-30, i.e. administrative and civil service reform, decentralization, public expenditure, financial management, and procurement, tax policy and administration, other accountability/anti-corruption, other public sector governance. 3 Public sector sectors include sector codes BC, BH and BZ, i.e. central government administration, sub-national government administration and general public administration. 4 Projects are classified as successful if their outcomes were evaluated as highly satisfactory, satisfactory or moderately satisfactory by IEG. Projects are classified as unsuccessful if their outcomes were evaluated as moderately unsatisfactory, unsatisfactory or highly unsatisfactory. 5 The matching criteria used to identify similar projects to PSM projects within the same country were (i) the approval FY, (ii) the committed amount and (iii) the lending instrument type. 6 Civil Liberties and Political Rights are hard to measure. The above findings based on Freedom House data are therefore cross-checked against an alternative measure of a similar concept, the Polity IV Index, yielding broadly consistent findings. For brevity, results using the Polity IV data are not reported here. 4

5 2. KEY RESULTS The review finds that in particular upstream PSM projects (narrow definition) do perform distinctively worse than non-psm Bank projects. As shown in Table 7, upstream PSM projects perform on average 6-7 percentage points worse than similar non-psm projects in the same country. This finding holds at the 90 percent confidence-level. The review s findings suggest that this performance differential may to a significant extent be explicable by the distinctive riskiness of PSM projects in countries with strongly limited civil liberties and political rights. Upstream PSM projects do perform worse in non-free countries where political rights and civil liberties are strongly limited 7. But they do not perform distinctively worse in partially free and free countries. Matching estimates suggest that that success rate of mostly upstream PSM projects in nonfree countries is on average about 20 percentage points lower than of similar non-psm projects in the same countries (see Table 8). This finding holds at the 90 percent confidence level. By contrast, in partially free and free countries, the success rate differentials are much smaller and statistically insignificant. 8 Results from matching estimates are corroborated by linear cross-country regression estimates, when comparing countries with similar growth rates, income level, ODA dependency, secondary school enrollment and from the same region (see Table 4 and Table 5). 9 The plot of success rate differentials on the World Map in Figure 1 provides some intuitive support for this finding. Many of the countries with high success rate differentials between non-psm and PSM projects are non-free based on the Freedom House classification (see Table 1 and Table 2). Regression results suggest that poor performance in non-free contexts is a distinctive particularity of PSM projects. As shown in Table 4 10, Public Sector Governance Board-mapped projects stand out as performing distinctly worse in non-free contexts than any other group of projects mapped to other Sector Boards. In sum, the findings suggest that investing in upstream PSM reforms in countries with strongly limited civil liberties and political rights is distinctly more risky than investing in other sectors in these countries. This finding is consistent with theory (see companion note) and case study evidence which suggest that undertaking performance-oriented reforms of the machinery of government is distinctly difficult in contexts where political incentives for such reforms are weak due to limited accountability to citizens, while opposing vested interests may be strong. 7 The grouping of countries is based on the Freedom House country classification into three groups of non-free, partially free and free. 8 In partially free countries, the success rate of PSM projects is only 5 percentage points below other projects and in free countries it is 1 percent above. Yet, non of these latter findings are statistically significant at standard levels. 9 Undertaking the same analysis based on a broader definition of PSM projects that comprises both upstream and downstream (sector) PSM projects yields similar (same sign), but statistically insignificant results (see Table 6). That these findings are statistically insignificant may be due to lower average PSM component share in these projects or reflect that upstream PSM reforms are more affected by civil/political liberties than downstream PSM reforms. Conducting the same analysis using Polity IV Index as an alternative measure of political rights and civil liberties yields results that point in a similar direction (same sign), but coefficients are smaller and not statistically significant (results not reported). 10 See the large and highly significant coefficient for PSGB-mapped projects in Table 4. Other projects with large & significant coefficients are projects mapped to the Economic Policy Board (which often comprise large public sector components) and projects mapped to the Health Sector Board. 5

6 B. Question 2. PSM Project Performance across Thematic Areas 1. METHODOLOGY Performance differences among PSM projects focusing on civil service, public financial management and tax reform are reported without controlling for any other project or country-level characteristics. The findings should thus be interpreted as merely descriptive statistics without causal implications. Reported differences in success rates across PSM thematic areas may, but need not reflect the specific nature of the respective reform area. As PSM projects typically comprise multiple PSM themes, two alternative inclusion thresholds are employed a low (standard) threshold of 25 percent and a high threshold of 50 percent. For example, a project is only regarded as a PFM project if it comprises at least a 25 or, respectively, 50 percent PFM component. Anti-Corruption is not separately considered as a PSM thematic area, as the sample of projects with large enough anti-corruption components is small. 2. RESULTS The review finds that the average success rate of tax reform projects stands out as the highest. Projects with large PFM and decentralization components perform similarly well, while the success rate for civil service projects is slightly lower. Based on the high 50 percent inclusion threshold, the average success rate for tax is 91 percent, for PFM 68 percent and for civil service projects 62 percent (see Figure 3). Based on the low 25 percent inclusion threshold, the average success rate for tax is 87 percent, for PFM 74 percent, for civil service projects 70 percent and for decentralization projects 76 percent (see Figure 2). These findings are broadly consistent with the findings of the IEG review of 2008, based on CPIA ratings. C. Question 3. Project Design Features and (PSM) Project Performance Regarding the impact of project design features on project performance, the review finds that both: (i) long delays between project preparation milestones and (ii) a high frequency of team changes 11 in projects predict below average performance of Investment Lending projects. 12 The analysis of these project design features focuses on Investment Lending (IL) projects only. It covers IL projects across the Bank, not just for Public Sector Projects, as, due to data availability constraints on these design features, the sample of PSM IL projects alone is too small to yield statistically significant results. 1. DELAYS IN PROJECT PREPARATION a) Methodology Similar to the analysis on question 1, matching 13 and cross-country regression identification strategies are used to estimate whether delays in project preparation predict project performance while holding constant other country and project level characteristics. Two delays in project preparation are distinguished the time elapse between the Project Concept Note review meeting and Project Approval and between Project Approval and Project Effectiveness. It is important to note that the analysis does not have causal 11 Team changes are identified based on data from the World Bank s Time Recording System (TRS). A team change is counted when the two team members who, on average, spent most time on a project for 6 months are both replaced by other team members. 12 Development Policy Lending / Adjustment Lending Projects are excluded from the analysis. 13 Matching estimates match projects with high and low preparation times within the same country that are similar in key observable characteristics. 6

7 implications. Delays in project preparation are unlikely to affect project performance themselves, but may rather be rough proxy indicators of other, unobserved factors that do causally affect project performance. b) Results Long delays in Investment Lending project preparation predict below-average project success rates. Based on matching estimates with exact matching by country, projects with high time elapse between PCN and Approval have a success rate that is on average 6 percentage points lower than projects with low time elapse (see Table 11). Strikingly, projects with high time elapse between Approval and Effectiveness have a success rate that is on average 15 percentage points lower than projects with low 14 time elapse (see Table 11). Both findings hold at the 95 percent confidence level. These findings are supported by linear regression estimates, controlling for other observable country- and project-level characteristics. As shown in Table 9, long delays between PCN and Approval and between Approval and Effectiveness predict lower average success ratings, at the 99 percent confidence level (without controls, see Table 9 and Table 10). Estimated coefficients for these delays decline when controlling for other project- and country level characteristics, but remain statistically significant at least for some specifications (column 8 in Table 9, column 2 and columns 5-8 in Table 10). Table 12 illustrates how average project performance ratings decline with increasing delays in project preparation. The finding that delays in project preparation predict significant variation in project success rates suggests that they may reflect unobserved performance drivers other than country- and project-level characteristics controlled for. In particular, it could seem plausible that delays between project Approval and Effectiveness reflect low client government commitment to the envisaged reform, which, in turn, protracts the project preparation process. If this can be supported by qualitative evidence, delays in project preparation might be useful as an easily available and meaningful early risk predictors as part of the Bank s risk management approach. 2. FREQUENCY AND NUMBER OF TEAM CHANGES a) Methodology Similar to the above analysis, matching 15 and cross-country regression identification strategies are employed to estimate whether the number and frequency of team changes predict project performance while holding constant other country and project level characteristics. 16 Drawing on data from the Bank s 14 The time elapse between PCN review and project Approval is defined as high if it is greater than 1000 days and smaller than 2000 days. It is defined as low if it is smaller than 1000 days. The time elapse between project Approval and Effectiveness is defined as high if it is greater than 332 days and smaller than 500 days. It is defined as low if it is smaller than 332 days. 15 Matching estimates match projects with high and low numbers of team changes within the same country that are similar in key observable characteristics. 16 The sample of projects with available data on the frequency of team changes is small. While most projects that have been closed and have available IEG outcome ratings have been approved prior to the year 2001, data from the World Bank s Time Recording System (TRS) is only available since February Figure 4 illustrates how this constraint affects sample size. While, in principle, it would be desirable to only include investment lending projects in the sample for which Team Change data is available for the entire implementation period (between Effectiveness and revised closing date), this results in a sample that is too small for estimating significant effects. To increase sample size, projects are included in the sample if team change data is at least available for three years of supervision. Table 13 contrasts the predictions for the narrow and broad project sample. While the predictions for both samples show the same tendency, the second sample is biased towards team changes towards the end of project implementation. It may therefore not yield representative results. 7

8 Time Recording System (TRS) 17, team changes for a given project are measured by two indicators by the (absolute) number of team changes over the project duration (controlling for the latter) and by the average time elapse [in days] between these team changes. It is important to note that frequent team changes may be both a cause and a consequence of poor project performance. 18 It seems plausible that a project performs poorly, staff have weaker incentives to stay on the project than if they are responsible for a well performing project. By lack of a credible instrument, the portfolio review cannot disentangle both directions of causality. The findings can thus be interpreted both ways. b) Results Findings suggest that a higher number or frequency of team changes predict below-average performance ratings, holding constant other project and country-level characteristics. Matching estimates suggest that the success rate of projects with a high (6-8) number of team changes is on average 13 percentage points lower than for projects with a low (<6) number of team changes, at the 90 percent confidence level (see Table 18). 19 This finding is supported by liner (OLS) and probit regression results (Table 16 and Table 17). These finding provide objective support to subjective Bank staff views that frequent TTL changes are harmful to project performance in PSM projects (see the companion note on staff survey results). Table 13 and Table 14 illustrate the association between the frequency of team changes and project success rates. 17 See Figure 5 for an illustration of how TRPS data indicates team changes. 18 This distinguishes team changes from project preparation times, which are mere symptom of other drivers of project performance. 19 By contrast, the implementation time (between effectiveness and revised closing) of IL projects is not associated with project performance. 8

9 III. DATA ANNEXES A. Annex I: Governance and PSM Project Performance Figure 1. Success-rate Differentials between Non-PSM and PSM Projects by Country Note: The map shows how much better non-psm projects do than PSM projects in the same country. The difference is largest in the countries highlighted in deep red (marking countries where the success rate of PSM projects is at least 20 percentage points lower than for non-psm projects). It is smallest in the countries in light red, where the success rate of PSM projects is either equal to or higher than for non-psm projects. Data is unavailable in the project sample for the countries highlighted in grey. 9

10 Polity IV Polity IV 1. CORRELATIONS BETWEEN DIFFERENT MEASURES OF CIVIL LIBERTIES AND POLITICAL RIGHTS Table 1. Discrepancies in Country Coverage between Freedom House ( Not Free ) and Polity (score <0). Country Count. Freedom House Countries never classified as not free between 1990 and 2010 Countries classified as Not Free at least for one year between 1990 and 2010 Total Countries never classified lower than 0 on the Polity 2 Score Countries classified at least once lower than 0 on the Polity 2 Score Total Table 2. Discrepancies in Country Coverage between Freedom House ( Not Free ) and Polity (score <0). Country Names. Freedom House Countries never classified as not free between 1990 and 2010 Countries classified as Not Free at least for one year between 1990 and 2010 Total Countries never classified lower than 0 on the Polity 2 Score 73 countries [not listed] Afghanistan, Ethiopia, Iraq, Lebanon, Maldives, Russian Federation, Somalia 80 Countries classified at least once lower than 0 on the Polity 2 Score Armenia, Bangladesh, Comoros, Croatia, Guinea- Bissau, Guyana, Lesotho, Morocco, Peru, Senegal, Venezuela Algeria, Angola, Azerbaijan, Belarus, Bhutan, Burkina Faso, Burundi, Cambodia, Cameroon, Central African Republic, Chad, China, Congo, (Democratic Republic of), Congo (Republic of), Cote d'ivoire, Djibouti, Egypt, Arab Republic of, Equatorial Guinea, Eritrea, Gabon, The Gambia, Ghana, Guinea, Haiti, Indonesia, Iran (Islamic Republic of), Jordan, Kazakhstan, Kenya, Kyrgyz Republic, Lao People's Democratic Republic, Malawi, Mauritania, Mozambique, Nepal, Niger, Nigeria, Pakistan, Rwanda, Sierra Leone, Sudan, Swaziland, Syrian Arab Republic, Tajikistan, Tanzania, Thailand, Togo, Tunisia, Turkmenistan, Uganda, Uzbekistan,Vietnam, Yemen (Republic of), Zimbabwe 65 Total

11 2. FREEDOM HOUSE CIVIL LIBERTIES AND POLITICAL RIGHTS INDICATORS a) Whole Bank Table 3. Probit Regression Estimates Across Sectors (Freedom House) VARIABLES (1) (2) (3) IEG Outcome Rating [binary] IEG Outcome Rating [binary] IEG Outcome Rating [binary] status==not Free ** (0.0696) (0.0981) (0.110) status==partially Free *** (0.0609) (0.0844) (0.0920) leninstrtype==investment * ** (0.0673) (0.0896) (0.0919) Committed Amount *** ** ( ) ( ) ( ) GDP per Capita Growth [annual %] *** ** ( ) ( ) GDP per Capita [const. PPP 2005 USD] 5.01e e-06 (1.40e-05) (1.61e-05) Net ODA received [% of GNI] ( ) ( ) Secondary School Enrollment [% gross] *** ( ) ( ) region==afr ** (0.149) region==eap (0.134) region==eca (0.132) region==mna *** (0.136) region==sar (0.151) Constant 0.841*** 0.508*** 0.827*** (0.0796) (0.148) (0.225) Observations 3,140 1,942 1,942 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 11

12 b) By Sector Board Table 4. Probit Regression Estimates by Sector Board (Freedom House) (1) VARIABLES IEG Outcome Rating [binary] Sectorboard==ARD (0.222) Sectorboard==EMT (0.236) Sectorboard==ENV (0.240) Sectorboard==EP (0.249) Sectorboard==FPD (0.220) Sectorboard==HE (0.238) Sectorboard==Other (0.345) Sectorboard==PS (0.236) Sectorboard==SP (0.275) Sectorboard==TR (0.249) Sectorboard==UD (0.275) Sectorboard==WAT (0.283) status==not Free (0.237) status==partially Free (0.196) Sectorboard==ARD & status==not Free (0.304) Sectorboard==ARD & status==partially Free (0.264) Sectorboard==EMT & status==not Free (0.334) Sectorboard==EMT & status==partially Free (0.287) Sectorboard==ENV & status==not Free (0.372) Sectorboard==ENV & status==partially Free (0.305) Sectorboard==EP & status==not Free * (0.338) Sectorboard==EP & status==partially Free (0.290) Sectorboard==FPD & status==not Free

13 (0.334) Sectorboard==FPD & status==partially Free (0.265) Sectorboard==HE & status==not Free * (0.339) Sectorboard==HE & status==partially Free (0.285) Sectorboard==Other & status==not Free (0.464) Sectorboard==Other & status==partially Free (0.407) Sectorboard==PS & status==not Free *** (0.350) Sectorboard==PS & status==partially Free (0.284) Sectorboard==SP & status==not Free (0.402) Sectorboard==SP & status==partially Free (0.328) Sectorboard==TR & status==not Free (0.344) Sectorboard==TR & status==partially Free (0.317) Sectorboard==UD & status==not Free (0.377) Sectorboard==UD & status==partially Free 0.605* (0.346) Sectorboard==WAT & status==not Free (0.399) Sectorboard==WAT & status==partially Free (0.354) leninstrtype==investment * (0.0899) Committed Amount ** ( ) Constant 0.931*** (0.189) Observations 3,140 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 13

14 c) Narrow PSM Project Definition Table 5. Probit and Linear OLS Regression Estimates for the Narrow PSM Universe (Freedom House) VARIABLES IEG Outcome Rating [binary] Probit Estimates Linear OLS Estimates (1) (2) (3) (4) (5) (6) IEG IEG Outcome Outcome IEG Outcome IEG Outcome Rating Rating Rating Rating [binary] [binary] [scalar] [scalar] IEG Outcome Rating [scalar] PSMNarrow== (0.196) (0.276) (0.278) (0.144) (0.177) (0.177) status==not Free ** *** (0.0719) (0.100) (0.113) (0.0604) (0.0810) (0.0930) status==partially Free *** *** (0.0632) (0.0869) (0.0944) (0.0508) (0.0673) (0.0740) PSMNarrow==1 & status==not Free ** ** ** *** ** ** (0.300) (0.392) (0.392) (0.274) (0.324) (0.325) PSMNarrow==1 & status==partially Free (0.239) (0.334) (0.335) (0.184) (0.227) (0.227) leninstrtype==investment * * ** * (0.0682) (0.0905) (0.0930) (0.0544) (0.0664) (0.0682) Committed Amount ** ** *** *** *** ( ) ( ) ( ) ( ) ( ) ( ) GDP per Capita Growth [annual %] *** * *** *** ( ) ( ) ( ) ( ) GDP per Capita [const. PPP 2005 USD] 4.72e e e e-06 (1.39e-05) (1.61e-05) (1.04e-05) (1.18e-05) Net ODA received [% of GNI] ( ) ( ) ( ) ( ) Secondary School Enrollment [% gross] *** *** ( ) ( ) ( ) ( ) region==afr ** *** (0.149) (0.122) region==eap (0.134) (0.106) region==eca (0.133) (0.0999) region==mna *** *** (0.135) (0.117) region==sar (0.151) (0.114) Constant 0.851*** 0.503*** 0.839*** 4.237*** 3.886*** 4.206*** (0.0822) (0.149) (0.226) (0.0651) (0.115) (0.177) Observations 3,140 1,942 1,942 3,140 1,942 1,942 R-squared Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 14

15 d) Broad PSM Project Definition Table 6. Probit Regression Estimates with Broad PSM Project Definition (Freedom House) VARIABLES IEG Outcome Rating [binary] Probit Estimates Linear (OLS) Estimates (1) (2) (3) (4) (5) (6) IEG Outcome IEG Outcome IEG Outcome IEG Outcome Rating [binary] Rating [binary] Rating [scalar] Rating [scalar] IEG Outcome Rating [scalar] Odef== (0.111) (0.144) (0.145) (0.0843) (0.103) (0.103) status==not Free (0.0826) (0.114) (0.127) (0.0683) (0.0908) (0.103) status==partially Free ** ** (0.0732) (0.101) (0.107) (0.0597) (0.0797) (0.0859) Odef==1 & status==not Free *** *** (0.154) (0.198) (0.200) (0.136) (0.163) (0.164) Odef==1 & status==partially Free (0.132) (0.173) (0.174) (0.104) (0.126) (0.126) leninstrtype==investment * * ** (0.0711) (0.0942) (0.0960) (0.0568) (0.0691) (0.0708) Committed Amount ** ** *** *** *** ( ) ( ) ( ) ( ) ( ) ( ) GDP per Capita Growth [annual %] *** * *** *** ( ) ( ) ( ) ( ) GDP per Capita [const. PPP 2005 USD] 4.95e e e e-07 (1.40e-05) (1.62e-05) (1.04e-05) (1.18e-05) Net ODA received [% of GNI] ( ) ( ) ( ) ( ) Secondary School Enrollment [% gross] *** *** ( ) ( ) ( ) ( ) region==afr ** *** (0.150) (0.122) region==eap (0.134) (0.106) region==eca (0.133) (0.0995) region==mna ** *** (0.136) (0.117) region==sar (0.153) (0.114) Constant 0.869*** 0.501*** 0.820*** 4.244*** 3.878*** 4.184*** (0.0932) (0.161) (0.232) (0.0743) (0.123) (0.182) Observations 3,140 1,942 1,942 3,140 1,942 1,942 R-squared Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 15

16 e) Matching Estimates Table 7. Nearest Neighbor Matching Estimates with exact matching by country All countries PSM definition Narrow Broad (1) (2) VARIABLES IEG Outcome Rating [binary] IEG Outcome Rating [binary] SATT * (0.0358) (0.0200) Observations 3,301 3,301 Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 Table 8. Nearest Neighbor Matching Estimates with exact matching by country (based on Freedom House country grouping) Non Free Partially Free Free PSM definition Narrow Broad Narrow Broad Narrow Broad VARIABLES (1) (2) (3) (4) (5) (6) IEG Outcome IEG Outcome IEG Outcome IEG Outcome Rating [binary] Rating [binary] Rating [binary] Rating [binary] IEG Outcome Rating [binary] IEG Outcome Rating [binary] SATT * (0.107) (0.0487) (0.0494) (0.0274) (0.0604) (0.0363) Observations ,546 1, Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 16

17 B. Annex II. PSM Project Performance by Theme Figure 2. Project Success rates by Theme (25 percent inclusion threshold) Number of Observations Tax Administration Public Financial Management Civil Service and Administrative Reform Decentralization

18 Figure 3. Project Success rates by Theme (50 percent inclusion threshold) Number of observations Tax Administration Public Financial Management Civil Service and Administrative Reform

19 C. Annex III. Project Design Features and (PSM) Project Performance 1. TIME BETWEEN PROJECT MILESTONES Table 9. Probit and Linear Regression Estimates for Whole Bank with Scalar Time Variables VARIABLES IEG Outcome Rating [binary] Probit Estimates Linear (OLS) Regression Estimates (1) (2) (3) (4) (5) (6) (7) (8) IEG Outcome IEG Outcome IEG Outcome IEG Outcome Rating Rating Rating IEG Outcome IEG Outcome Rating [binary] [binary] [binary] [scalar] Rating [scalar] Rating [scalar] IEG Outcome Rating [scalar] Days from PCN Review to Approval *** ** * *** *** *** * (7.27e-05) (9.51e-05) (9.55e-05) ( ) (6.48e-05) (8.13e-05) (8.15e-05) (9.33e-05) Days Approval to Effectiveness *** e *** ** ** * ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) Days Effectiveness to Rev. Closing 2.72e e e e e e e e-06 (4.80e-05) (6.41e-05) (6.52e-05) (7.48e-05) (4.10e-05) (5.28e-05) (5.33e-05) (6.13e-05) Average Days till Team Change * 9.99e-05* (7.90e-05) (5.32e-05) Committed Amount ** ** * * ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) status==not Free * (0.118) (0.134) (0.152) (0.0912) (0.106) (0.118) status==partially Free (0.104) (0.112) (0.127) (0.0757) (0.0833) (0.0934) GDP per Capita Growth [annual %] *** ** * *** *** *** ( ) ( ) (0.0106) ( ) ( ) ( ) GDP per Capita [const. PPP 2005 USD] 9.24e e e e e e-06 (1.69e-05) (1.96e-05) (2.23e-05) (1.26e-05) (1.44e-05) (1.54e-05) Net ODA received [% of GNI] ( ) ( ) ( ) ( ) ( ) ( ) Secondary School Enrollment [% gross] * ( ) ( ) ( ) ( ) ( ) ( ) region==afr * (0.165) (0.189) (0.133) (0.145) region==eap ** 0.228* (0.155) (0.179) (0.121) (0.133) region==eca (0.206) (0.235) (0.163) (0.173) region==lcr ** (0.185) (0.211) (0.141) (0.156) region==mna (0.189) (0.217) (0.167) (0.177) Constant 0.927*** 0.507** 0.692*** 0.506* 4.468*** 4.141*** 4.204*** 4.057*** (0.106) (0.217) (0.254) (0.285) (0.0905) (0.174) (0.198) (0.217) Observations 2,264 1,377 1,377 1,086 2,264 1,377 1,377 1,086 19

20 R-squared Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 20

21 Table 10. Probit and Linear Regression Estimates for Whole Bank with Dummy Time Variables VARIABLES IEG Outcome Rating [binary] Probit Estimates Linear (OLS) Regression Estimates (1) (2) (3) (4) (5) (6) (7) (8) IEG Outcome IEG Outcome IEG Outcome IEG Outcome IEG Outcome Rating Rating Rating IEG Outcome Rating Rating [binary] [binary] [binary] Rating [scalar] [scalar] [scalar] IEG Outcome Rating [scalar] tpcnapprovman==high *** ** ** *** *** *** * (0.0755) (0.0992) (0.0996) (0.119) (0.0702) (0.0867) (0.0866) (0.0988) tapproveffman==high *** * *** ** ** * (0.0946) (0.130) (0.134) (0.154) (0.0875) (0.110) (0.113) (0.124) tapproveffman==medium ** * *** ** ** * (0.0619) (0.0816) (0.0830) (0.0942) (0.0517) (0.0649) (0.0661) (0.0733) teffrevclosngbin== ** 0.150* (0.0606) (0.0789) (0.0799) (0.0927) (0.0516) (0.0651) (0.0654) (0.0744) Average Days till Team Change * 9.22e-05* (7.89e-05) (5.37e-05) Committed Amount ** * * * ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) status==not Free * (0.118) (0.134) (0.152) (0.0910) (0.106) (0.118) status==partially Free (0.104) (0.113) (0.127) (0.0758) (0.0830) (0.0930) GDP per Capita Growth [annual %] *** ** * *** *** ** ( ) ( ) (0.0106) ( ) ( ) ( ) GDP per Capita [const. PPP 2005 USD] 1.07e e e e e e-06 (1.70e-05) (1.97e-05) (2.22e-05) (1.27e-05) (1.45e-05) (1.55e-05) Net ODA received [% of GNI] ( ) ( ) ( ) ( ) ( ) ( ) Secondary School Enrollment [% gross] e ( ) ( ) ( ) ( ) ( ) ( ) region==afr (0.164) (0.188) (0.132) (0.144) region==eap ** 0.230* (0.155) (0.179) (0.120) (0.133) region==eca (0.205) (0.235) (0.163) (0.174) region==lcr ** (0.183) (0.210) (0.140) (0.156) region==mna (0.189) (0.218) (0.168) (0.179) Constant 0.776*** 0.525*** 0.676*** 0.519** 4.239*** 4.036*** 4.076*** 3.948*** (0.0544) (0.171) (0.214) (0.239) (0.0456) (0.135) (0.164) (0.179) Observations 2,263 1,376 1,376 1,085 2,263 1,376 1,376 1,085 R-squared Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 21

22 Table 11. Nearest Neighbor Matching Estimates with exact matching by country For time from PCN to Approval (high vs. low) For time from Approval to Effectiveness (high vs. low) (1) (2) VARIABLES IEG Outcome Rating [binary] IEG Outcome Rating [binary] SATT ** *** (0.0294) (0.0388) Observations 2,264 1,314 Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 22

23 Histogram of Projects included IEG Outcome Rating Success-rate Kernel-weighted local mean smooths Table 12. Does the Time Elapse between Project Milestones predict (Investment Lending) Project Performance? From Concept Note Review to Project Approval From Project Approval to Effectiveness From Effectiveness to Rev. Closing Kernel half-width= 200 days Kernel half-width= 50 days Kernel half-width= 300 days 23

24 2. FREQUENCY OF TEAM CHANGES Figure 4. Availability of Team Change Data for IEG Evaluated Investment Lending Projects 24

25 Histogram IEG Outcome Rating Mean Success-rate Kernel-weighted local mean smooths Table 13. Do Frequent Project Team Changes predict Investment Lending Project Performance? Narrow sample Only including projects with Team Change Data available at least since Project Effectiveness Broad sample Including projects with Team Change Data available for at least three years prior to the revised project closing date. \ Kernel half-width=200 Kernel half-width=200 Sample size= Sample size= 25

26 Histogram IEG Outcome Rating Mean Success-rate by Number of Team changes Table 14. Does the number of Team Changes on Investment Lending Projects predict average project Success Rates? Note: Due to few observations with 9 or 10 team changes, performance for these is excluded from the graph. Sample Size= 26

27 Table 15. Probit Regression Estimates for Whole Bank VARIABLES IEG Outcome Rating [binary] Probit Estimation Linear OLS Regression (1) (2) (3) (4) (5) (6) IEG Outcome IEG Outcome IEG Outcome IEG Outcome Rating [binary] Rating [binary] Rating [scalar] Rating [scalar] IEG Outcome Rating [scalar] Average Days till Team Change *** ** * *** * * ( ) ( ) ( ) (9.81e-05) ( ) ( ) Project Duration -7.47e ** *** ** *** *** (7.16e-05) (9.23e-05) (9.98e-05) (5.81e-05) (7.15e-05) (7.14e-05) Committed Amount ** *** *** ** ( ) ( ) ( ) ( ) ( ) ( ) Days from PCN Review to Project Approval -6.34e e-05 ( ) ( ) Days Approval to Effectiveness ( ) ( ) Days Effectiveness to Rev. Closing e-05 ( ) (7.99e-05) status==not Free (0.172) (0.197) (0.117) (0.142) status==partially Free (0.149) (0.166) (0.102) (0.114) GDP per Capita Growth [annual %] ** ** (0.0160) (0.0167) (0.0116) (0.0122) GDP per Capita [const. PPP 2005 USD] -2.64e e e e-05 (2.60e-05) (3.07e-05) (1.79e-05) (2.16e-05) Net ODA received [% of GNI] * * ( ) ( ) ( ) ( ) Secondary School Enrollment [% gross] *** * *** * ( ) ( ) ( ) ( ) region==afr (0.240) (0.178) region==eap (0.254) (0.178) region==eca (0.332) (0.239) region==lcr (0.302) (0.222) region==mna (0.304) (0.237) Constant 0.496** *** 3.931*** 3.934*** (0.198) (0.335) (0.434) (0.169) (0.259) (0.335) Observations R-squared Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 27

28 Table 16. Probit and Linear (OLS) Regression Estimates for whole Bank (with dummy variable for high versus low number of average days till team change) VARIABLES IEG Outcome Rating [binary] Probit Estimation Linear OLS Regression (1) (2) (3) (4) (5) (6) IEG Outcome IEG Outcome IEG Outcome Rating Rating Rating IEG Outcome [binary] [binary] [scalar] Rating [scalar] IEG Outcome Rating [scalar] Average Number of Days till Team Change==low * ** * * (0.111) (0.147) (0.149) (0.0842) (0.103) (0.105) Project Duration -5.80e ** *** ** *** *** (7.16e-05) (9.28e-05) (9.97e-05) (5.85e-05) (7.16e-05) (7.10e-05) Committed Amount ** *** *** ** ( ) ( ) ( ) ( ) ( ) ( ) Days from PCN Review to Project Approval 3.56e e-05 ( ) ( ) Days Approval to Effectiveness ( ) ( ) Days Effectiveness to Rev. Closing e-05 ( ) (7.92e-05) status==not Free (0.171) (0.197) (0.118) (0.142) status==partially Free (0.148) (0.166) (0.102) (0.114) GDP per Capita Growth [annual %] ** ** (0.0161) (0.0168) (0.0116) (0.0122) GDP per Capita [const. PPP 2005 USD] -2.72e e e e-05 (2.62e-05) (3.09e-05) (1.80e-05) (2.17e-05) Net ODA received [% of GNI] * * ( ) ( ) ( ) ( ) Secondary School Enrollment [% gross] *** * *** * ( ) ( ) ( ) ( ) region==afr (0.240) (0.177) region==eap (0.253) (0.178) region==eca (0.333) (0.240) region==lcr (0.303) (0.222) region==mna (0.304) (0.237) Constant 0.894*** 0.769** 0.742* 4.425*** 4.244*** 4.228*** (0.201) (0.347) (0.450) (0.164) (0.257) (0.341) Observations R-squared Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 28

29 Table 17. Probit and Linear (OLS) Regression Estimates for Whole Bank (with dummy variable for high versus low number of team changes per project) VARIABLES IEG Outcome Rating [binary] Probit Estimation Linear OLS Regression (1) (2) (3) (4) (5) (6) IEG Outcome IEG Outcome IEG Outcome Rating Rating Rating IEG Outcome [binary] [binary] [scalar] Rating [scalar] IEG Outcome Rating [scalar] changecountman==low 0.383** 0.489** 0.470* (0.184) (0.244) (0.244) (0.165) (0.226) (0.226) Project Duration -2.40e * ** * *** *** (7.27e-05) (9.36e-05) ( ) (6.04e-05) (7.39e-05) (7.32e-05) Committed Amount ** * *** *** *** ( ) ( ) ( ) ( ) ( ) ( ) Days from PCN Review to Project Approval -1.50e e-06 ( ) ( ) Days Approval to Effectiveness ( ) ( ) Days Effectiveness to Rev. Closing e-05 ( ) (7.95e-05) status==not Free (0.172) (0.197) (0.118) (0.141) status==partially Free (0.149) (0.166) (0.102) (0.115) GDP per Capita Growth [annual %] ** ** (0.0163) (0.0169) (0.0118) (0.0122) GDP per Capita [const. PPP 2005 USD] -2.81e e e e-05 (2.62e-05) (3.11e-05) (1.80e-05) (2.19e-05) Net ODA received [% of GNI] * * ( ) ( ) ( ) ( ) Secondary School Enrollment [% gross] *** ** *** * ( ) ( ) ( ) ( ) region==afr (0.242) (0.178) region==eap (0.254) (0.179) region==eca (0.332) (0.238) region==lcr (0.305) (0.221) region==mna (0.304) (0.236) Constant *** 3.812*** 3.771*** (0.271) (0.416) (0.507) (0.243) (0.348) (0.417) Observations R-squared Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 29

30 Table 18. Nearest Neighbor Matching Estimates with exact matching by country Changedays (treated= low, untreated= high ) Number of team changes (treated= high, untreated= low ) (1) (2) VARIABLES IEG Outcome Rating [binary] IEG Outcome Rating [binary] SATT * (0.0354) (0.0725) Observations Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 30

31 Figure 5. Example of Project Time Recording System Data Note: Each color represents time billed to the project by a Bank staff member. 31