Investment Support under Rural Development Policy

Transcription

1 Investment Support under Rural Development Policy Contract 30-CE /00-41 EXECUTIVE SUMMARY 12 November 2014 Written by Metis/WIFO/aeidl November 2014

2 EUROPEAN COMMISSION Directorate-General for Agriculture and Rural Development Directorate E Economic analysis, perspectives and evaluation; communication Unit E.4 Evaluation and studies agri-evaluation@ec.europa.eu European Commission B-1049 Brussels

3 Investment Support under Rural Development Policy Contract 30-CE /00-41 EXECUTIVE SUMMARY 12 November Directorate-General for Agriculture and Rural Development Investment Support under Rural Development Policy

4 Europe Direct is a service to help you find answers to your questions about the European Union. Freephone number (*): (*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you). LEGAL NOTICE The information and views set out in this report are those of the authors and do not necessarily reflect the official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this study. Neither the Commission nor any person acting on the Commission s behalf may be held responsible for the use which may be made of the information contained therein. More information on the European Union is available on the Internet ( Luxembourg: Publications Office of the European Union, 2014 ISBN doi: /63445 European Union, 2014 Reproduction is authorised provided the source is acknowledged. Printed in Austria

5 Executive Summary Objective and scope of the study The objective of this evaluation is to review, analyse and test different methods for evaluating investment support measures in Rural Development Programmes (RDP).The selected methods were used to analyse investment support measures in eleven Member States. Results of this evaluation focused firstly, on assessing the strengths and weaknesses of different methodological approaches and identifying types of results to be obtained by six different methods. Secondly, the effectiveness, efficiency and impact of investment support measures were explored by applying these methods in the case study territories. Finally, the results of the case studies were used to assess to what extent the different approaches to targeting investment support have been effective in meeting the general objectives of rural development policies. Investment support measures in RDP aim at increasing productivity within the agricultural and forest sector and to diversify production in non-agricultural activities. Furthermore, related infrastructure investment and non-productive investments supporting agri-environmental schemes are supported. The target groups are economic entities (farm and forestry holdings; micro, small and medium sized enterprises in sectors processing agricultural and forest product) within the agricultural and forest sector and individuals attempting to diversify and establish businesses or services outside the agricultural sector, including tourism. These measures are part of Axes 1 and 3 (investment support to private beneficiaries and infrastructure investments) and partly, of Axis 2 (non-productive investments). Six methods were used to evaluate these investment support measures: IO (Input-Output model), a computational model using quantitative tools based on a specific theory of economic behaviour which applies widely available regional and national statistics on sectors and households as primary data input,; the IO method includes sectoral analysis, but is in fact a general equilibrium programming approach; PSM (propensity score matching) is an econometric method that uses micro-data (data on individuals and firms) or data on a large number of regions and programme information. The objective of PSM is to identify comparable observations of those who received and those who did not receive investment support and calculate the difference in the indicator of interest (e.g. Gross Value Added). MAPP is a method that combines qualitative and quantitative approaches by exploring the opinion of stakeholders on the outcomes of a programme. One of the results is a shared view of the participants on indicators of interest. Such results are derived in an interactive process. TBE (theory-based [impact] evaluation) is a method that defines procedures to derive evaluation results from desk research and expert interviews. SEA (strategic environmental assessment) is an approach rather than a method and is based on the SEA Directive 2001/42/EC. Member States have to carry out a SEA as part of the ex-ante evaluation to determine whether the RDP has significant environmental effects; CEA (cost-effectiveness analysis) is a set of different methods, that combine economic and environmental data in order to evaluate the effects of measures taken to enhance environmental quality; in many cases - but not always - CEA results are based on combining quantitative tools used in economic and environmental sciences. page 5

6 Methodology Different approaches were applied to answer the three overarching evaluation questions (EQ) of the study: EQ1 1 focuses on the appropriateness and suitability of methods. Thus concise and clear descriptions including strengths and weaknesses were developed for each method. A factsheet with a short description of the measure, data requirements, the approach how the counterfactual situation is established, the scale of indicators and the method for measuring efficiency, effectiveness and impact is presented. A typical workflow highlights the activities necessary to conduct the specific method. Data requirements are described in detail. The appropriateness of each measure is judged using the criteria rigour, robustness and validity and practicability based on views of experts in the project team and literature. Answering EQ2 2 different methods had to be applied in selected territories: A total of eleven different RDP territories with a geographical balance and representation of different types of measures and reasonable data availability were chosen (AT, CZ, CY, DE/Hessen, DK, ES/Galicia, FR, GR, PL, SK, UK/Scotland). For each of the methods the core team of experts specified the data requirements and geographic experts assessed the data-availability during a first screening exercise. For the fieldwork the necessary data for applying the tests were used. A large variety but not every combination - of investment measures in the study regions was analysed by different methods (in total 126 cases of different combinations of study regions, methods and investment support measures). In order to derive comparable results only a small number of indicators were used (mainly focusing on employment, income, value added and environmental effects, if applicable). The third evaluation question 3 (EQ3) asked for the effectiveness of different approaches to targeting investment support in meeting objectives of rural development policies. Three criteria were identified, that reflect approaches to targeting: eligibility criteria (by territory, investment type, beneficiary characteristics etc); aid-intensity differentiation (more funding for targeted groups); and criteria used for ranking by administrative authorities. These criteria are combined in various ways in the RDPs leading to a high number of approaches to targeting. They were used to analyse the effectiveness of targeting to achieve the objectives of the RDP and also to divert funds to the targeted groups. The evaluation question was addressed by testing a specific hypothesis: Different approaches to targeting might differ in how efficient they are in achieving the objectives of the RDP. It was thus envisaged that it might be possible to directly link the results for EQ2 where the efficiency is estimated to the question of targeting. 1 To which extent are the different evaluation methods described and/or tested in this exercise appropriate for the assessment of the effectiveness, efficiency and impact of the different types of investment support considered? 2 What is the effectiveness, efficiency and impact of the investment support studied in the selected RDP territories? 3 To what extent have the different approaches to targeting investment support studied been effective in meeting the general objectives of rural development policy and/or specific objectives included in the relevant RDPs? page 6

7 Selected results of case studies The application of the selected methods is explained in detail for each method separately as applied to one case study territory, where the conditions were specifically favourable. For Austria all A and B-type measures were evaluated firstly by an Input-Output analysis, where the total economic effects of the RDP were assessed to increase output, income and employment. Secondly, an econometric counterfactual method was applied, to analyse the effects of an investment support measure (M121): Indicators of programme beneficiaries were compared to target values and to outcomes of a control-group of non-beneficiaries. Also the structure of the total increase as the result indicators was analysed. From the results it became clear that the conditional PSM method 4 appeared to be the most rigorous, transparent and reliable method. A naive before-after approach generated results which were highly biased by over-estimating result indicators to a magnitude of factor 2-25 (!). On this basis a detailed PSM assessment of effectiveness, efficiency and impact could be provided, also comparing farm-level based results with country-wide data. At the same time, considerable time and administrative efforts were necessary to receive from national authorities all required farm bookkeeping/fadn and Paying Agency data. The MAPP method is explained for Scotland, where six of the most important investment schemes were analysed with a focus group of stakeholders. In several steps the expert and the focus group established the overall context, analysed trends for performance and impact indicators, discussed an influence matrix for RDP schemes and external drivers and developed an impact profile. The discussions revealed that RDP investment support had had an important impact on generation of non-farm jobs, especially long-term jobs in the food sector. Also RDP investment support has helped at best to maintain farm jobs among beneficiaries while jobs have declined in nonparticipating businesses. Results are judged to be of good quality, as the tools have worked well. Problems were in finding the right participants and overcoming a certain group fatigue. The method needs careful preparation in order to achieve good outcomes. In the Czech Republic s case study the Programme-theory-based evaluation method was illustrated. Based on a conceptual model according to the RDP intervention logic, the main project outputs at operations level have been compiled. This leads from the projects/operations level to gross results for the target group, e.g. holdings, small enterprises. During the judging phase the evaluator drew evidencebased conclusions on the effectiveness of the interventions and identified gross direct effects. It transpired that the effectiveness is quite high ranging from medium, high to even very high. For the Czech RDP all building blocks to establish a TBE along the intervention logic could be established at good quality level. Accordingly, robust judgements were possible. The main problem encountered was to put the information together in a consistent way, direct access to the monitoring system was a prerequisite for this. A Strategic Environmental Assessment and a Cost-Effectiveness Analysis were illustrated for investment in public infrastructures in Greece. The SEA has highlighted the explicit and implicit intervention logic in relation to the environmental issues and has shown areas for improvement in the programme-specific result indicators and the options for assessment of impacts. 4 combination of a binary PSM and DID page 7

8 CEA is a method where the strength lies in the credibility of the numbers feeding both the numerator and denominator. Even when using ex ante values the results can be used to trigger the debate for the explanation of deviations among operations and regions. The lack of current result indicator values renders the discussion of the representativeness of results impossible; however the theory-based assumptions are sound and valid. For both methods the consideration of the environmental dimension has been plausible for a measure under Priority Axis 1; the formulation of the theory of change on environmental aspects is clear, but incomplete on secondary effects (soil, energy). Problems encountered relate to the lack of periodic reporting of result indicator values and the lack of tracking of context indicators. These are necessary for a better understanding of the effects of a measure in a given timeframe and spatial context. There is no real solution to this problem: application forms, environmental permits and other documents have been screened and discussed but no systematic solution exists. Findings and conclusions Comparing methods (EQ1) When comparing the different methods against the judgement criteria for EQ1 it becomes evident, that there is not one but rather a range of methods that is best suited to address different aspects of an evaluation. In order to assess the appropriateness of methods to analyse investment support measures, fundamental problems related to the rigour of any evaluation have to be considered: This relates for example to assessing the counterfactual, i.e. what would have happened if the programme was not in place. To identify and test causal relations between a policy intervention and an outcome is one of the biggest challenges of every evaluation. Only a small set of approaches (typically specific econometric methods or experiments) are suited to provide such results by empirically estimating the counterfactual situation. If such estimates are not available, assumptions about the causal effects need to be made. In the context of this study, PSM was applied to identify causal effects. For other methods either findings in the literature or monitoring reports, assumptions and statements of stakeholders or results of PSM were used. Furthermore it is worthwhile considering specific effects such as leverage or deadweight effect. PSM can be used to directly address these effects by quantifying them at farm or firm level. At sector or regional level other effects need to be considered, including the increase of demand for investment goods and capacity adjustment after an investment was made. IO can be used to directly address these effects by quantifying them. It is worth considering not only the generic characteristics of a method, but also the context in which it is applied. In this study the practicability of methods in the context of EU RDPs was also a topic of interest. All methods need data (among them programme documents, reports, evaluation studies, etc.). Such data are partly readily available (e.g. input-output tables for most Member States), some data are available but scattered on different places (programme details and data on implementation and outputs), other data are available but accessible only with restrictions (e.g. many microdata), or have to be collected (e.g. interviews, participatory activities). In the context of this study, expenditure for data acquisition has been surprisingly low. Much more resources were necessary to locate, collect and prepare data. page 8

9 Another finding is that even if data that are necessary for a given method are known to exist, they may not be available due to data protection concerns or lack of capacity to match different data sets and make them useable for an evaluation. Based on an overview of the resources used in this study, methods can be ranked: data collection and field work were most resource intensive for TBE, followed by PSM, MAPP, and IO; SEA and CEA required the least resources; applying PSM required the most resources in order to derive results and considerable fewer resources per case study were necessary for handling data and deriving results when applying MAPP, IO and TBE; the least resources were used for SEA and CEA. The resource intensity is only one small facet and it is necessary to link the resources to the quality and depth of an analysis. SEA and CEA needed the least resources, assuming that necessary data would be available in the course of standard monitoring and reporting; unfortunately this was not the case. The report suggests a decision tool that can facilitate the identification of a method which best fits to the preferences and capacities of the user. Because the need for resources is an eminent element of any evaluation, it is evident that some results may be attainable only at higher costs. The tool should help a decision maker to structure the problem of method selection in a transparent and concise manner. Effectiveness, efficiency and impact of the investment support studied in the selected RDP territories (EQ2) There are as many answers as investigated indicators (efficiency, effectiveness, and impact), cases and methods used to obtain results. Furthermore information on environmental outcomes of investment support measures (measured by SEA and/or CEA) is rarely available and if available, often not conclusive. Efficiency of programmes depends firstly, on the specific result of interest (e.g. value added or labour) and, secondly, on the method used to measure efficiency. In most cases efficiency was at levels expected and observed already in previous studies. Only two methods (IO and PSM) provide quantitative results on efficiency. TBE gives no results on efficiency and results of MAPP are very scant. There is a large variance of efficiency indicators across regions when measured by the same method (either IO or PSM). In several cases efficiency of the same investment measure was analysed with different methods in the same region. As expected, results deviated. Even contradictory results were obtained. When scrutinised these results showed that contradictions are either due to subtle differences of definitions of efficiency between the methods or due to the fact that one method was using exogenously given parameters to measure efficiency (IO) while the other measured efficiency directly (PSM). Results for efficiency obtained by PSM show in almost all cases that for each Euro of public funding spent, the GVA (Gross Value Added) and family farm income had increased by (sometimes substantially) less than one EUR by the end of the programme period. page 9

10 Results on effectiveness were provided by applying IO, PSM, and TBE methods. One general conclusion from the analyses is that in many cases targets are not reported and if they are reported there is not a good correspondence between achievements and target. Impacts are effects that go beyond beneficiaries of the programme. Results on impacts were provided by applying IO, PSM, and MAPP. A major finding - derived from the analysis with the IO method - is that support of investments creates additional demand in the regions where the beneficiaries are located and frequently triggered an expansion of outputs. The benefit of using more than one method is exemplified by showing the effects of regional output responses when they are based either on reports of Managing Authorities or derived from micro-data by PSM. Results based on the latter are consistently lower. Effectiveness of targeting investment support (EQ3) The study did not find a significant relationship between the efficiency of the policy intervention and approaches to targeting. Three reasons can explain the lack of significant relationships: First, and most likely, not enough observations were available. The noise through efficiency and targeting approach measurement combined with insufficient variation prohibited uncovering possible relationships between efficiency and approach to targeting. Second, the approach to targeting does not in fact have a substantial impact on efficiency as indicated by the insignificant results. Third, the approaches to targeting described in the programme documents did not actually divert the funds to the targeted groups. In order to further investigate these question, an in depth analysis was made in Austria on measure 121. Measure 121 in Austria includes eligibility criteria, aid intensity differentiation and selection criteria. To test if these criteria divert funds to the groups specified, FADN survey data are used. Using the survey design information (weights and strata) it is possible to estimate population statistics. Based on the analysis the study concluded that not all elements of the approach to targeting necessarily work. Eligibility criteria, as they were easy to check, worked. For aid intensity differentiation criteria a statistically significant effect on average investment support was not found. Interestingly, even though simple to apply and check, eligibility criteria were not used extensively. The major conclusion from this analysis is that targeting approaches are as complex as programmes are. In order to better understand the ways targeting-approaches contribute to the efficiency of programmes it seems to be necessary to work further on deepening the knowledge by analyses as carried out in this study. Further conclusions In addition to the responses to EQ1, EQ2 and EQ3 further conclusions have been derived from this study. Use of combinations of methods: The number of users of evaluation results is large and each user has specific expectations. While some users need quantitative results, others are more interested in subtleties that cannot be expressed in figures. To serve both prototypical groups calls for a combination of methods. page 10

11 On the positive side various methods are available but some contradictory results are likely to turn up. Further methodological improvements may help to overcome this situation. At the same time, using more than one approach to measure the outcomes of a RDP contributes to a more complete picture of the overall effects. Costs of applying methods: One important observation is that non-quantitative methods impose high variable costs each case study costs more or less the same and the costs are not negligible. Quantitative methods such as IO or PSM exhibit scale economies analysing a large number of cases implies that the costs per case drop as more cases are analysed. However, in order to use such methods, it is necessary to invest in specific infrastructure like micro-data sets or regional IO tables. These economic implications are important to consider when programmes and evaluation strategies are developed. Specific knowledge: In order to analyse measures of RDP, very specific knowledge is necessary. The programmes are very complex and the situations are heterogeneous, even within countries. This is not unexpected because the programmes are fine-tuned to the situations in a given region. Use of standardised data sets: The availability of standardised data sets (e.g. inputoutput tables for EU Member States, FADN data) is a big advantage for quantitative methods. There are significant economies of scale for methods using such data. Given data were available for all regions, it might even be possible to analyze a large number of measures for all regions with a small team of experts. Such an approach seems to be necessary even in cases where in one region sample sizes of beneficiaries or nonbeneficiaries are too small. Having more regions where the same measure is applied might increase the sample size. FADN data: FADN data are very useful; however their usefulness is contingent upon the link to other administrative data. For the purpose of evaluation, such a link should be the standard procedure in all regions and programmes. This would be a way to identify causal effects for a large number of cases. Using such an approach would improve the validity of the evaluation studies considerably. IO case studies: The case studies show that this method and those that use the same data may deliver a wide range of interesting results. The IO method is a standard approach in economics and because it is so standardised there is large expertise available. What is not available are sufficient data for all regions. One recommendation is to establish the necessary data that are used for IO or similar models already at the beginning of the programming period (in particular regional input output tables). Such an approach would allow consistently deriving targets, and carrying out mid-term and ex post analyses within the same methodological framework for a number of important indicators. Recommendations for planning evidence-based evaluations: If it is not possible to identify a method that can be used to evaluate the intended effects of a measure, it is strongly recommended to reconsider the implementation of the measure. This recommendation is based on the insight that "you cannot control what you cannot measure". When a programme is designed and when targets are set, it is advisable to keep already in mind which method to use to evaluate the outcomes. It is evident that quantitative input data are necessary when quantitative results are expected. Collecting this information not only for beneficiaries but also for non-beneficiaries is important when econometric methods are intended to be used. page 11

12 doi: /63445 KF EN-N