Summary Ex-Post Evaluation Sustainability of Regional Rural Development Programmes (RRD)

Transcription

1 B M Z E v a l u a t i o n R E p o R t s Summary Ex-Post Evaluation Sustainability of Regional Rural Development Programmes (RRD) synthesis Report of Four Country Case studies

2 Summary Ex-Post Evaluation Regional Rural Development Programmes

3 Summary Ex-Post Evaluation Regional Rural Development Programmes Preface This study has been commissioned by the Federal Ministry for Economic Cooperation and Development (BMZ). It constitutes a synthesis of four individual evaluations, or country case studies, that were carried out by local independent evaluators, namely: l l l l Indonesia: PPA Consultants in ass. with PT Profik Mitra Abadi, Jakarta Zambia: Newton Lungu & Associates in association with Bowanda Services Ltd., Lusaka Sri Lanka: Centre for Poverty Reduction, CEPA, Colombo Tanzania: Stoas International, Dar es Salaam The present report was authored by independent evaluators Lutz Meyer and Kirsten Vorwerk, who also served in an advisory capacity throughout the entire process. The ex-post evaluations on which the synthesis is based were carried out in 2003 and 2004 on the basis of selected questions and methods. Their main goal was to learn more about the sustainability of the impacts of German contributions to Regional Rural Development projects that had already been completed. This was one of the most complex project types in the 1970s to 1990s. The idea was to learn about these impacts from the perspective of local experts. In addition to this synthesis report, the four case studies are also available (see imprint for contact data). The views presented in this study do not necessarily coincide with those of the BMZ. Rather, they are opinions held by the said independent external experts. Division for Evaluation of Development Cooperation

4 Summary Ex-Post Evaluation Regional Rural Development Programmes Summary of Evaluation The views presented in this study are opinions held by the independent external experts. 1. in 2003 the BMZ commissioned a series of ex-post evaluations of finalised development cooperation projects and programmes in the field of Regional Rural Development (RRD) so as to obtain more information on the relationship of cause and effects in Germany s development cooperation and to further fine-tune the methodical evaluation concepts developed so far for impact assessment. The aim of the evaluation was to obtain an independent and external assessment of the results (with a focus on impacts), relevance and sustainability of long-term development activities and an investigation into the underlying success factors. In this context, the identification of unintended results and a further improvement of the methodical investigation concepts for impact assessment developed so far were also of interest. 2. this serial evaluation is characterised by a number of distinctive features. The implementation of the evaluation was assigned to local organisations/consultants so as to effect a change in perspectives, create more ownership and contribute to strengthening capacity development in the field of impact evaluation. Advisory Groups staffed equally by both the local and the German side were established in the four participating countries to ensure adequate communication and counselling of the evaluators. In Germany, two backstoppers were contracted who accompanied the whole process, advised the BMZ and acted for it here and there. 3. the methodical procedure was based on three key analytical steps: (i) assessment of the situation at three points in time/time frequency; (ii) funnel approach (recording of all changes with subsequent investigation of their relationship with project interventions; (iii) observation of three impact fields: impact on the part of the target groups, impact on the sector and/or cross-sectoral results, and impact on the part of state executing organisations. The type and scope of data collection was conferred to the respective Consultants and varied from project to project. 4. Despite sometimes substantial differences in quality between the studies they were, taken together, considered as good. The studies were implemented on the basis of the DAC quality standards. Based on the BMZ s analysis parameters, all four teams carried out comprehensive analyses of documents, drew up detailed research designs and put them up for discussion in comprehensive inception reports. Special importance was attached to the compilation of primary data within the framework of field phases. Accordingly, between 100 and more than 850 interviews were held, depending upon the country concerned. All in all, a broad range of mutually complementary date collection instruments were used in all the projects (multi-method approach).

5 Summary Ex-Post Evaluation Regional Rural Development Programmes 5. the selection of the RRD projects to be investigated was done by the BMZ. Criteria were: a comprehensive project concept, a long project duration and the project having been finalised five years ago. The following projects were selected: l Indonesia: Area Development Project West Pasaman; , German contribution: 32 million Target population 200,000 urban and rural people, area 4,250 square kilometres, GDP growth/per capita : 4.2%, poverty 2001: 17% of households l Sri Lanka: Regional Rural Development Project (RRDP) Kandy; , German contribution: 8.1 million Target population 200,000 rural smallscale producers, area 2,000 square kilometres, GDP growth/per capita : 3.4%, poverty 2000: 26% of households l Tanzania: Tanga Integrated Rural Development Program (TIRDEP); , German contribution: 75 million Target population 700,000 rural smallscale producers, area 27,300 square kilometres, GDP growth/per capita : 0.6%, poverty % of households l Zambia: Integrated Rural Development Program (IRDP), Kabompo; , German contribution: 30 million Target population 65,000 rural small-scale producers, area 50,000 square kilometres, GDP growth/per capita : -2.1, poverty 2004: 75% of households 6. the projects evaluated had been implemented in regions with very different underlying general conditions, an aspect which is of crucial importance when drawing conclusions on the breadth of the identified impact. Differences exist not only with regard to the size of the intervention areas and the target population, but also regarding the general standard of living, the general economic development and the population growth in the respective countries or project regions. 7. The four evaluations have identified a great number of changes that have occurred in the course of the projects. In their analysis the evaluators have demonstrated that a number of identified changes are highly plausible to have been effected by the projects. 8. all in all it can be stated that the living conditions of the target groups have improved in all four project regions. The four case studies came to the conclusion that a significant part of these improvements was the result of project interventions. Sustainable project results can be substantiated especially in the health and education sector, food security, increase in income and employment and the ensuing rise in the standard of living. Project-induced changes are mostly the result of an improvement of the infrastructure, enhancing the (private sector) economy, and innovations in agriculture. 9. Significant results were achieved also at the sectoral and cross-sectoral level by the projects. They consist mainly of innovations in agriculture/diversification, development of new technologies, extension services, an expansion of the economic and social infrastructure (roads, health facilities, schools) and the promotion of the private sector. These components were essential factors in bringing about an improvement in the standard of living. However, only some of the self-help groups and cooperatives established still continue to be active. The rate of success is highest where emphasis is placed on the private sector aspect. 10. Results with regard to a positive regional development are discernible in all four cases. In Indonesia and Sri Lanka, the regional

6 Summary Ex-Post Evaluation Regional Rural Development Programmes development plans established under the projects are still being pursued. In Tanzania and Zambia, the implementation structures for regional development that had been put into place have ceased to exist. The great number of rather uncoordinated single measures in Tanzania had a tendency of harming the success of the project, moreover. 11. In the long term, however, there are factors which jeopardise the sustainability of the positive results achieved. In particular, maintenance of the infrastructure should be mentioned in this context. Also, in some cases the intensification of economic activities has adverse ecological effects such as acidification of the soil or overfishing. 12. Sustainability at the level of state executing organisations is low in all the projects. Although it is true that the innovations introduced in the projects have substantially strengthened the planning and management capacities at the individual level, the efforts to sustainably establish these innovations in state institutions have failed, the reasons mostly being inadequacy of funds, inefficient organisational structures and lack of coordination. And although various instruments developed in the projects are still used, they are not extrapolated or deployed adequately. 13. Far-reaching and fundamental reforms have been enforced after termination of the projects or even during their duration in the project regions investigated, reforms concerning especially a liberalisation of the economy and a decentralisation of state authority. These developments, but also other factors of influence such as e.g. the activities of other donors or international agreements, have had a decisive impact on the projects and the changes identified in the project regions. 14. The country case studies have identified a number of factors which in the evaluators views have enhanced the success of the projects. Thus, the projects excel in their good objective systems and a relatively high acceptance of the objectives on the part of the participating stakeholders, even though planning had been done largely by the German side. The project objectives were mostly considered positive as to their clarity, feasibility and relevance. As had been found on earlier occasions, the establishment of joint objectives by both the local and the German side is of immense significance. Other important success factors have been (i) gearing the projects to the needs of the target groups; (ii) complementarity with other projects; and finally (iii) gearing project measures to market needs. 15. The evaluations have brought into the open also some structural problems in the drafting of project concepts. Thus, planning and implementation was mostly a matter for the German side to see to. However, a lot has changed meanwhile. Furthermore, two of the projects investigated have created separate structures to a considerable extent. These were impossible to maintain after the end of the project (nor should some of them have been continued). Although the transition from separate structures to integrated advice has brought about a change here, this aspects still merits attention in current projects. 16. Complaints by the partners were mostly related to two systematic aspects. Thus, the expenses for the consultancy services of German experts were often considered as disproportionate compared with the hardware supplied. Here, partner expectations obviously were not consistent with those of the German side. This reveals not so much

7 Summary Ex-Post Evaluation Regional Rural Development Programmes a problematic project concept but rather a problem of lack of proper information on the procedure (planning and implementation was considered as a matter for the German side to arrange), and in some cases also a lack of transparency regarding the use of the funds committed according to the respective government negotiations. 17. The termination of German support was often considered abrupt and incomprehensible. Phasing out was done according to German views and did not take sufficient account of the partners views and needs as seen by them. This certainly provides ground for a systemic conflict between the interests of the partners and those of the German side. 18. At the level of projects, the following conclusions can be drawn: none of the four projects show any system-oriented sustainability at the level of state executing organi sations. This is mostly due to the fact that insufficient consideration was given to the limited funds available on the part of state organisations, and the lack of political determination to continue the activities. Improvements of self-organisation capacities of the target groups were only partially successful. However, the living conditions of the target groups have improved in all project regions so that in this regard the evaluators can confirm the sustainability of the project measures on target group level. The best results were achieved through measures in the field of improvement of the infrastructure and promotion of the private sector. 19. The putting into place of new structures by means of development projects run a very high risk of not being sustainable. After all, in some cases a substantial difference existed between the expectations on the local partner side (hardware) and the concepts of the RRD projects (software). 20. Not all the findings and experience dating back to the 1970s and 1980s are still relevant today, since, firstly, the general climate has changed in the partner countries (e.g. economic reforms and decentralisation) and, secondly, development cooperation has undergone fundamental changes as a result. The former RRD project concept is no longer pursued, following the adoption of new general concepts by BMZ and GTZ. RRD projects today are conceived as multi-dimensional (different levels of action political, social, ecological and economic). Promoting rural development is a cross-sectional task. Poverty reduction and orientation towards the MDGs and national Poverty Reduction Strategies are basic requirements which such projects have to meet. 21. Overall costs for the evaluation series were rather high by comparison, due to the high number of staff on the part of the Consultants and the related expenses. On the other hand, account has to be taken of the fact that the evaluations carried out by far exceeded normal BMZ evaluations as to their depth and width of activities. 22. In organisational terms the establishment of local Advisory Groups for the selection of the Consultants and for the assessment of the expertises was of great significance. In terms of purely accompanying the investigations, the groups contributions have been rather marginal. 23. All in all, this serial evaluation has brought about a change in perspectives in parts through the deployment of local experts. However, the qualification profile of locally available Consultants varies considerably. The results of this serial evaluation do not suggest a general shift of evaluations to local organisations.

8 Summary Ex-Post Evaluation Regional Rural Development Programmes Comment of BMZ At a panel discussion at the BMZ in October 2005, the methods and results of the ex-post evaluation were discussed somewhat controversially among representatives from academic circles, implementing organisations and the BMZ. The panellists underlined the systematic ex-post approach as such to be a positive one, methodically speaking, as was the attempt to catch the local perspective by assigning local experts only although this does not always become recognisable. Critical remarks were voiced about the lack of hard data and facts in the synthesis report; the reference to the country studies which contain these data is not sufficient. All in all, however, the results seem plausible, given the range and depth of the investigation and the general developments in the countries. One would have wished for reliable cost-benefit analyses to have been made, bearing in mind that this poses a challenge for both traditional and modern types of development cooperation (projects, sector programmes, budget support). The target of accountability has been met, showing clearly positive effects; however, the relevance of the evaluation for future concepts and learning remains open to dispute. On the one hand, reference was made to the change in today s general environment and the concepts which have long been adjusted (e.g. liberalisation, donor harmonisation, BMZ concept on Rural Development). The role of development cooperation today consists in the flexible facilitation of processes rather than the implementation of measures. On the other hand, both the critical results of the evaluation (e.g. little ownership) and also the positive findings (poverty reduction at the target group level) offer content-oriented starting points for learning. Given a specific combination of circumstances, the elements of the former RRD-approach might still be considered as effective contributions to resolving problems (e.g. locally adjusted substitute structures in cases of bad governance). Integration into national policies and structures would be preferable but is not always realistic. The current trend towards state centralism (e.g. through budget support) is somewhat risky. What is needed in any case is to support a lobby for the poor and rural areas and, in doing so, make a contribution to feasible and effective pro-poor concepts. Ideological debates often hamper matters, on all sides and in particular where rural development is concerned. They must not be allowed to prevent learning from ex-post and other evaluations which therefore should be continued.

9 Summary Ex-Post Evaluation Regional Rural Development Programmes

10 Imprint Editor Federal Ministry for Economic Cooperation and Development Division of Development Education and Information Office Bonn Adenauerallee D Bonn Phone: + 49 (0) Fax: + 49 (0) Office Berlin Stresemannstraße 94 D Berlin Phone: + 49 (0) Fax: + 49 (0) poststelle@bmz.bund.de Editing Maria Tekülve Final editing Steffen Beitz Responsible Michaela Zintl As of July 2005 Orders please to: Publikationsversand der Bundesregierung Postfach Rostock Phone: + 49 (0) Fax: + 49 (0) publikationen@bundesregierung.de This summary is available in English on the internet under A printed copy of the summary and full text can be sent upon request. Contact: Karl-Peter.Schallenberg@bmz.bund.de