TOOL #57. ANALYTICAL METHODS TO COMPARE OPTIONS OR ASSESS

Transcription

1 TOOL #57. ANALYTICAL METHODS TO COMPARE OPTIONS OR ASSESS PERFORMANCE 1. INTRODUCTION A crucial part of any retrospective evaluation is the assessment of the performance of the existing policy intervention. Similarly, when a new initiative is being considered, the impact assessment should compare and rank the policy options. This tool presents the various ways/methods in which both can be done. In addition, the tool presents complementary procedural guides on how to perform a multi-criteria analysis and costbenefit analysis which draw on the content of the separate tools describing methods which can be used to assess costs and benefits and the study prepared by the Centre of European Policy Studies on the assessing the costs and benefits of regulation 698. Every evaluation and impact assessment is different and the degree to which quantified information is available will also be different. As such, impact assessments and evaluations will have to employ one or more of the approaches according to the particular circumstances. They will still have to assess the economic, social and environmental impacts in an integrated way and to quantify them as far as is possible. 2. POSSIBLE METHODS The key methods which are available to inform the assessments are presented below. Cost Benefit Analysis Cost-benefit analysis (CBA) entails the monetization of all (or the most important) costs and benefits related to existing public intervention or all viable alternatives at hand. A step by step guide to undertake a CBA is given in appendix. CBA is mostly used during the appraisal stage of a new intervention. In its most recurrent form, it disregards distributional impacts and only focuses on the selection of the regulatory alternative that exhibits the highest net benefit. Accordingly, the most common methodology in cost-benefit analysis is the net benefits calculation, which differs from the benefit/cost ratio method that is typically used in cost-effectiveness analysis (being benefit minus costs, rather than benefits divided by costs). There are pros and cons of choosing CBA as the method to be used in comparing policy proposals. The principal advantage in the ability of CBA to use an objective unit of measurement (monetised values) to compare alternative options and choose the one that maximizes the size of the pie, i.e. societal welfare as described in mainstream economics. The shortcomings, however, are often quite critical for CBA, and mostly refer to the assumption that income can be a proxy for happiness or satisfaction, the fact that it willingly ignores distributional effects (despite some attempts to adjust the methodology to reflect them), and its lack of objectivity when it comes to the selection of certain parameters (e.g. the inter-temporal discount rate), which can tilt the balance in favour of certain regulatory options over others

2 In the evaluation context, CBA can help to determine the overall impact of an intervention and whether it has been worth undertaking. It also provides evidence on the validity and appropriateness of the assumptions and projections used in the impact assessment for the examined intervention. A major advantage of using CBA in retrospective evaluations lies in ensuring that costs and benefits of an intervention are considered in a structured and explicit way. CBA promotes fiscal accountability and can be used to demonstrate added value of EU interventions. CBA can be, and has been applied, within the EU system. However, it is typically less applied, and more difficult to apply, than in other regulatory systems for various reasons. For example, putting a monetary value on non-monetary costs and benefits can be sometimes difficult and will rely on a number assumptions. As a result, there is a risk that intangible values or outcomes may be under/overestimated or even overlooked. Accordingly, multi-criteria analysis tends to be used more frequently. Multi criteria Analysis (MCA) Multi-criteria analysis is a technique to reach a judgement based on an explicit set of objectives and associated criteria 699. It is particularly useful in case of complex interventions with diverse quantified impacts measured in different units and/or qualitative impacts (in particular factors which cannot be expressed in monetary terms). Typically, MCA will be used to assess and rank alternative options in an impact assessment, or to assess the extent to which a variety of objectives have or not been met, in a retrospective evaluation or fitness check. For example, the criteria chosen could include the impact on SMEs, the degree of protection of fundamental rights, consumer protection, etc. Multi-Criteria Analysis is particularly useful when impact assessment has to be reconciled with specific policy objectives, and as such is used as an instrument of ensuring the simultaneous assessment of effectiveness, efficiency and coherence of policies. This method allows the capture of distributional impacts (e.g. in terms of stakeholder types, EU regions/countries or time) and trade-offs between dimensions (such as between some economic, social or environmental impacts, or between some families of criteria). A separate tool provides more detail on how to use multi-criteria analysis. Least Cost Analysis Least cost analysis is primarily used in the impact assessment context. It only looks at costs, in order to select the alternative option that entails the lowest net cost. You should choose this method whenever benefits are fixed, and you only need to choose how to achieve them. Costs do not need to be precisely monetised or even quantified but their relative magnitude across options should be determined. Cost-effectiveness Analysis Cost-effectiveness analysis (CEA) entails that you quantify (not monetise) the benefits that would be generated by one Euro of costs imposed on society. While CEA is closely 699 See Tool #63 on Multi-criteria analysis. 452

3 related to CBA, instead of monetised benefits it uses other measures such as increased life expectancy, educational attainment, emissions abated etc. In impact assessment, the typical method used to compare options is thus the so-called benefit-cost ratio, which means dividing the benefits by costs. This method is normally used for all expenditure programs, as it leads to identifying the value for money of various expenditure programs. A typical question that can be answered through costeffectiveness analysis is how many jobs will be created for every Euro invested?, or, how many lives are saved by every Euro spent? 700 CEA is less easily applicable to interventions with more than one main objective. If the intervention aims to achieve a number of objectives (e.g. job creation and environmental protection), or have indirect impacts, the results of CEA may be misleading or irrelevant. In the evaluation context, the cost-effectiveness analysis will very often be used to compare the evaluated intervention against best practice or other interventions that aim to achieve similar objectives. It can also be used to assess the effectiveness of the implementation process where different implementation approaches have been pursued Counterfactual Analysis Counterfactual analysis is a statistical method devoted to quantifying whether a given intervention produces the desired effects on some pre-established dimension of interest. The challenge for quantifying effect is finding a credible approximation to what would have occurred in the absence of the intervention, and to compare it with what actually happened. Different types of counterfactual analysis exist: Randomized Controlled Trials, Difference in Difference; Propensity Score Matching; Discontinuity Design and Instrumental Variables. More precisely, a counterfactual analysis consists in comparing the outcomes of interest of the beneficiaries of a policy (the treatment group ) with those of a group similar in all respects to the treatment group (the comparison group ), except that it has not been subject to the policy. The comparison group informs us on what would have happened to the members of the treated group, had they not been exposed to the policy under scrutiny. This is the counterfactual. Strengths: The observed differences (over time, across individuals) in the outcomes between the treated group and the counterfactual (over time, across individuals) provide estimates of the causal effect of the policy. Limitations: Counterfactual Analysis requires extensive data sets on policy outcomes, collected before and after the intervention. There is also a need of pre-intervention outcome data which can represent an insurmountable obstacle. Finally, the challenge facing the evaluator is to avoid giving a causal interpretation to differences that are due to other factors, not to the intervention. It is necessary to identify the possible sources of bias arising in each specific situation and indicate which methods can overcome these biases, under which assumptions 700 See Tool #10 on Financial programmes and instruments. 453

4 The European Commission Competence Centre on Microeconomic Evaluation has expertise in designing counterfactual evaluations and can provide policy DGs with advice on possibilities and limitations. SWOT Analysis A SWOT analysis is used to identify the Strengths, Weaknesses, Opportunities and Threats in relation to a project/organisation and how such an assessment will change over time. In the context of evaluation, this method can be used for e.g. when assessing the services provided by a project/programme. A group needs to be convened to discuss different options and categorise them into a SWOT matrix. SWOT can take past weaknesses and transform them into a constructive learning process. SWOT is not an analytical tool per se; instead it is a way to synthesize preceding analyses and use them for developing a strategy. 3. WHICH METHOD IS MOST APPROPRIATE? Cost-benefit analysis is one of many methodologies that can be used in evaluations and impact assessments. The other methods described above may prove more appropriate depending on the case at hand. Cost-benefit analysis as the method to formulate the judgment if: At least all direct benefits and direct costs can be monetised, covering where possible the economic, social and environmental impacts of the proposal at hand (if benefits can be quantified, but not monetised, consider cost-effectiveness analysis): this requires an assessment of data availability in order to understand whether CBA will be feasible within a reasonable time frame. The magnitude of impacts justifies the effort and time needed to perform CBA (as a full-fledged CBA is normally more time-consuming than other, more qualitative techniques). Distributional impacts are unlikely to be substantial (otherwise, consider multicriteria analysis, or break down CBA by affected stakeholder without aggregating costs and benefits into a net benefits analysis). Cost-benefit analysis has a significant potential to identify and inspire efficient regulatory choices, but is subject to several weaknesses, related to its relative ignorance of distributional impacts, its reliance on income as a proxy for utility and happiness, and a number of other underlying assumptions, which can prove detrimental for the accuracy of the whole exercise. Cost-benefit analysis is also more challenging when assessing initiatives at the EU level, for the following reasons: The Commission requires an integrated assessment of economic, social and environmental impacts. However, monetizing some of the impacts, such as respect for 454

5 fundamental rights, would be a meaningless exercise, and as such should not be undertaken. Rather, multi-criteria analysis should be used in order to provide policymakers with a basis for informed decisions. The multi-institution, multi-level nature of EU policymaking makes it difficult to reach a sufficient level of accuracy in the analysis of certain costs and benefits. In particular, predicting the mode of enforcement and the related costs for public administrations at the national level is almost impossible at the ex-ante stage, unless rather extreme assumptions are formulated. This also means that compliance costs will be more difficult to predict and measure, as they partly depend on enforcement patterns. A related problem is the greater difficulty to perform cost-benefit analysis due to problems of data availability. The need to collect data from all Member States or, alternatively, to extrapolate data collected for some Member States to the EU28 makes the performance of cost-benefit analysis much more difficult at the EU level. In the EU impact assessment system, CBA is also more challenging as impact assessments are carried out for a wide variety of legislative and non-legislative initiatives, including white papers and communications for which policy options might not be fully detailed and as such difficult to analyse in terms of costs and benefits. CBA should be used as the method to compare alternative policy options if both benefits and costs vary according to the chosen alternative (if not, consider least-cost analysis). In addition, cross-cutting legislative initiatives which feature significant distributional impacts may be better analysed through multi-criteria analysis. In the evaluation context, the choice of the most appropriate method will vary for every evaluation. This will for example depend on the type of intervention being evaluated (e.g. spending, non-spending), scope of the evaluation (e.g. EU wide, pilot project), evaluation timing (e.g. interim, ex-post evaluation) and the availability and measurability of outcomes (e.g. what data already exist, key indicators). 4. UNCERTAINTY/SENSITIVITY The uncertainty which is inherent in the various estimates of costs and benefits should be explicitly recognised and quantified as far as possible as it may have an important bearing on the judgment as regards both the performance of public intervention and ranking of policy options in impact assessments. The influence of the key variables should be investigated by a sensitivity analysis. These variables should be allowed to vary in order to test the robustness of the final and should be linked to the drivers of the problem identified in the problem definition. Possible ways to approach the problem of sensitivity analysis are: Worst/best case scenario analysis: this requires adopting all the most conservative and all the least conservative values for variables used in the calculation of the Net Present Values, costs and benefits, and cost-effectiveness etc. Partial sensitivity analysis (i.e. changing only some of the assumptions, but not others) should be selectively used, for those key risk factors and underlying assumptions that are expected to tilt the balance in favour of one policy option. This 455

6 is often the case of variables such as the compliance rate, the evolution of consumer demand, etc. Monte Carlo sensitivity analysis is a more sophisticated technique that entails the creation of a distribution of net benefits by drawing key assumptions or parameter values from a probability distribution. While this is a more robust approach to sensitivity analysis, care needs to be taken in adopting reasonable and justified assumptions about the probability distributions which have been assumed. This type of analysis normally takes the form of a random sampling process to approximate the expected values and the variability inherent in the assumptions which are expressed as probability distributions for the most sensitive and uncertain parameters (risk variables). It is a computer-aided methodology through which many possible project scenarios are generated through a random selection of input values from the specified probability distributions. An example of this technique is provided in the tool on the use of analytical models 701. If the robustness of the basic assumptions cannot be examined numerically, a qualitative discussion on the appropriateness of each assumption can help readers to gauge the reliability of the results. 5. FURTHER INFORMATION Study prepared by the Centre of European Policy Studies on the assessing the costs and benefits of regulation. OECD Regulatory Compliance Cost Assessment Guidance. Unit C2 of the Secretariat General can provide advice on the content of this tool via its functional mailbox SG-C-2@ec.europa.eu 701 See Tool #62 on The use of analytical models 456

7 Appendix 10 steps to complete a Cost Benefit Analysis The following steps should be followed when completing a cost-benefit analysis. More detail can be found in chapter 3 of the CEPS study 702 and the Tool #59 on Methods to assess costs and benefits. Step Description 1 Decide whether CBA is the most appropriate approach to formulate a judgment. The advantages and disadvantages described in the main body of this tool should guide this decision. 2 Identify the full range of Costs and Benefits to be measured. Failure to identify significant impacts may skew the final judgment. 3 Partial or general equilibrium analysis. The choice will depend on the extent of the impacts and is important to help prevent the use of excessively costly and time-consuming methods (e.g. stated preference methods, or ex novo CGE modelling) for narrowly defined initiatives or for certain policy initiatives with non-binding effects. In this respect, you have to answer the following questions. Does the problem at hand affect several markets and present significant cascading and cumulative effects? Are there very significant impacts on the economy? If the answer is yes to both questions then you should opt for a general equilibrium approach. In this case, if you have no specific expertise in how to use general equilibrium models, you should refer to the impact assessment or evaluation unit of your DG and possibly seek the help of expert staff or external consultants. In all other cases, i.e. if the problem: Affects a limited number of markets/economic sectors, and/or Produces mostly direct effects on stakeholders, and/or Generates limited indirect, macroeconomic effects, Then you can address the problem and the related assessment of impacts through a partial equilibrium analysis. 4 Monetise direct costs for the public intervention in question or for all policy alternatives and calculate total direct costs. Are direct charges imposed on particular stakeholders/societal group? Are compliance costs increased including administrative burdens? What are the enforcement costs? 5 Monetise direct benefits. The following issues are relevant: 702 See Chapter 3 pp156 of the 2013 CEPS Study on Assessing the Costs and benefits of Regulation.

8 Are there cost reductions in regulatory charges, compliance costs and enforcement costs? Improvements in market efficiency should be monetised as far as possible (consumer surplus, producer surplus, and deadweight loss). Monetization of non-market benefits (health, safety, environment etc.) 6 Assess indirect impacts. Are there significant indirect costs? Are there significant indirect benefits? Are there other non-monetisable benefits (protection of fundamental rights, legal certainty, reduced infringement of legal rules etc.) 7 Determine when costs and benefits occur in the life of the initiative and apply social discounting to determine net present values Present impacts and formulate the judgement on the performance of existing public intervention or the comparison of the policy options. Present the different types of costs and benefits which have been monetised Present qualitative information on non-monetised costs and benefits Comparison should be performed in terms of the various cost/benefit categories, net benefits and net present value, distributional impacts on stakeholders. 9 Check the robustness of the results Sensitivity cases to assess influence of key variable/assumptions on uncertainty and on conclusions Check methods (no double counting, baseline versus policy option, consistent base currency used, spurious accuracy in results) Recognise any behavioural biases Assess interaction/interdependency between the categories of costs and benefits (e.g. enforcement costs rising as compliance costs decline) 10 Consider distributional and cumulative impacts On Member States if proportionate On future generations Richer and poorer sections of society SMEs 703 See Tool #58 on the Typology of costs and benefits. 458

9 459