Topic 2. Programme Evaluation: key Qs, methodologies & guidelines

Transcription

1 Topic 2. Programme Evaluation: key Qs, methodologies & guidelines Dr Micheál Collins 1. Why Evaluate? 2. The Public Spending Code 3. Appraisal/Evaluation Glossary 4. Life-Cycle of a Programme/Project 5. Guidelines & Requirements 6. Appraising Current Expenditure 7. Value for Money Studies 8. Other Notes 9. Discussion 10. Reading for Topic 2 1. Why Evaluate? Limited resources; unlimited uses Choices bases on systematic procedures Aim for efficient use of resources min inputs for a given output where output is clearly defined Answer the questions under five headings/purposes: 1

2 1) Planning Is the programme justified? Input into programme design Efficient resource allocation 2) Implementation Is the programme working and/or how can the programme be improved? 3) Accountability What was achieved? 4) Knowledge & Learning What interventions work and in what circumstances? Does the logic of the programme and its assumptions need to be questioned? Lessons for other applications 5) Development Institutional performance and strengthening Service quality But evaluation sometimes used for other, covert purposes Justify decisions already made Postpone decisions Public relations Compliance ideal? A rational exercise often undertaken for non-rational reasons (Weiss) To learn through systematic enquiry how to better design, implement and deliver public programmes and policies (EU EVALSED Guide) 2

3 2. The Public Spending Code Irish context: recent notable change Previously: range of various documents, guidelines and circulars problematic and inaccessible reform Publication of The Public Spending Code by DPER Housed under the CEEU Central Expenditure Evaluation Unit Input into this week and next Underpins methods & guidelines Plans to continue to update An input into decision making An introduction to the PSC: ( general points ) Building on Good Practice brings all the past documents together allows easier access, application and update a responsive and evolving code Aid to good decision making formalised the input into this process but (see later) Proportionality scale of appraisal or evaluation should be proportionate to the scale of what is being reviewed some points/guidelines later Appraisal never to be case-making role to be objective rather than campaigning Avoiding premature commitments When undertaking one, care not to represent it as a commitment to do / retain something it is a commitment to consider not implement / keep be prepare to abandon proposal/programme is appropriate: EU Funding Ireland moving to be a net contributor a national resource used in the same way as own revenue The availability of EU aid for a project is not a justification for investment in that project often parallel EU evaluation requirements (see EVALSED) 3

4 Code stresses how evaluation = an input into decision making rather than decision making: 3. Appraisal/Evaluation Glossary Lingo useful to define now Appraisal/Evaluation interchangeable before/after Sponsoring Agency Government Department, other state body Sanctioning Authority Usually the provide of the Government Minister or Department Cabinet above 100m (was 250m until Sept 2015) 4. Life-Cycle of a Programme/Project Irish guide: 4 stages UK guide: 6 stages 1. Appraisal before expenditure incurred preliminary appraisal if 5m+ statement of needs (why do it?) objective and degree these would be met alternative options indication of costs review and decide if more detailed appraisal needed if appropriate undertake detailed appraisal involves: clarification of objectives, exploration of options, quantification of costs and a method of selecting the best solution from competing options 4

5 2. Planning/Design Appraisal approval costing and planning more on method/steps coming up diagram 3. Implementation internal or contracts difference between time-defined and ongoing programmes 4. Post Implementation Review May be once established or once completed or both judge versus objectives lessons to learn (where express these?) 5

6 5. Guidelines & Requirements Scale of Appraisal Picking up on the proportionate point earlier have moved around over time current ones: Range Appraisal < 0.5m a simple assessment discuss 0.5m- 5m a single appraisal involving elements of preliminary and detailed appraisal 5m- 20m Multi-Criteria Analysis (MCA) 20m Annual 5m+ Annual 30m 5yrs+ Current: economic appraisal Capital: CBA or CEA Prior and mid-term evaluation 6

7 6. Appraising Current Expenditure Programme evaluation: 5 key questions Reflects DoF Value for Money manual (2008) Originally from CSF Evaluation Unit (1996) 1. Rationale 2. Relevance 3. Effectiveness 4. Efficiency 5. Impact Looking at each of these Relevant to reviews of existing programmes and considerations of new ones. Rationale What is the justification or rationale for the policies underpinning the programme? What is the underlying market failure justification for Government intervention? A necessary but not sufficient for government intervention to improve economic efficiency is that there is some form of market failure (HM Treasury) Could the problem be addressed through more direct means? Danger of second-best solutions Relevance What are the implications for the programme of changes in the wider socio-economic environment and in the context of overall Government policy? What societal needs or problems does programme address? (needs assessment) Is the programme relevant given current priorities? Is a current programme relevant give current situation and changes since it was introduced (fit-for purpose)? Implications of external changes for programme (continued relevance) 7

8 Effectiveness Is the programme meeting its financial and physical objectives? Generally addressed at level of: Inputs: Is the money being spent? Outputs: Results or immediate benefits Are they in line with expectations? If not, why not? Similar to a monitoring question Correlation to assessments of impact (later) Dependence on how the objectives have been framed Measuring effectiveness, challenges: Effectiveness in the private sector objectives normally clear: improve profitability (benchmark measure) gain mkt share increase share price / company value Effectiveness in the public sector objectives are difficult to specify why Difficulties in specifying effectiveness in the public sector People are unsure if asked Broad objectives difficult to pin down Can be complex improve health, provide educ. Often multiple objectives Easy versus difficult Conflicting objectives economic v social Objectives can change Different conceptions education Unplanned effects 8

9 Efficiency Could more be achieved for the resources invested? Considered a core question the best return for a given amount of inputs Issues such as: is it possible to get reduced inputs for same level of service is it possible to get additional outputs for same level of inputs is it possible to get improved unit cost ratio could you change the mix of activities/outputs to better deliver a given objective for same input level could you use alternative delivery approaches, e.g., outsourcing to private sector Impact What socio-economic changes can be attributed to the programme? to beneficiaries to society generally Need to take account of: Deadweight / displacement any unintended side-effects Objectives of programme (if property defined) will be of help. Most projects will be considered in the context of a sponsoring agency s business plan or a multi-annual investment programme Timing impacts on ability to answer these questions Nice summary, the evaluation cycle (David Hegarty, DOF) 9

10 7. Value for Money Studies 2007 VFM manual (website) Details in the DPER documents (website) Value for Money Policy Reviews (VFMPRs) Really focussed on assessing the earlier questions economy efficiency effectiveness the three Es DPER/CEEU attempting to establish a standardised approach Structure/Table of contents recommended by DOF: 6.2. Content of a VFM Review Report (from DOF guidelines) Title or Cover Page; Table of Contents; Glossary and Abbreviations/ Acronyms; Executive Summary; Introductory Chapter; Chapters reporting the findings of the VFM review, for example: objectives of Programme X and the Rationale for the programme; inputs, outputs and the trend in efficiency achieved; effectiveness; impact; continued relevance and alternative means of achieving Programme X objectives; and potential future performance indicators; and a Final Chapter with Conclusions and recommendations Annexes, to include: TOR Summary Sheet; the programme logic model template; members of the Steering Committee and evaluation team; details of surveys undertaken; review of international practice; major points raised by Quality Assessor and response by Steering Committee; and the final two Annexes should separately list the persons interviewed and the documents reviewed for the review. 10

11 Programme Logic Model: maps out the shape and logical linkages of a programme. provides a systematic and visual way to present and share understanding of the cause-effect relationships between inputs, activities, outputs and outcomes (results and impacts) 11

12 Answering the 5 key questions: 1. Rationale 2. Relevance 3. Effectiveness 4. Efficiency 5. Impact See notes attached to slides from VFM manual (p32-36) 8. Other Notes Links to Project evaluation next week Role, as mentioned earlier, for both: Post-project evaluations Periodic evaluations Missing: more on financial analysis over next two weeks 9. Discussion 12

13 10. Reading for Topic 2 Weiss article see website Department of Finance (2007) VFM manual see website DPER The Public Spending Code European Union (2012) EVALSED :The resource for the evaluation of Socio-Economic Development - Evaluation guide. s/guidance_en.cfm Scottish Government Guide: pfm/appraisal 13

14 VALUE FOR MONEY AND POLICY REVIEW INITIATIVE GUIDANCE MANUAL 3.4. Evaluation criteria Evaluation criteria for VFM Reviews The evaluation criteria are reflected in the specific questions that the evaluation will have to answer. In the case of a VFM review, the specific questions are set out in the template Terms of Reference and cover the evaluation criteria: rationale, efficiency, effectiveness, impact and continued relevance. The criteria encompass the established Value for Money criteria (economy, efficiency and effectiveness note that VFM reviews address economy under the efficiency heading). Other public policy evaluation criteria may also be relevant depending on the nature of the review (See Section 3.5). Each of the five evaluation criteria (rationale, efficiency, effectiveness, impact and continued relevance) is discussed in the following paragraphs Rationale Rationale is concerned with establishing why a public policy intervention is necessary in a given area. It requires consideration of the public policy objectives of a programme and the reasons for public sector provision or involvement. It is closely linked to the economic concept of market failure. Market failure is considered in more detail in Annex 1 but in simple terms, it exists where private individuals or firms do not produce the optimal level of a good or service from a societal perspective. A practical example of market failure is the need for subsidised bus services on socially desirable yet uneconomic routes. The rationale criterion usually also extends to whether the design of the programme or project is the most appropriate means of achieving the identified public policy objectives. In a VFM review, the study of rationale covers the following issues: Rationale Issues The validity of programme objectives Performance questions that can arise Why is a public policy intervention necessary? What market failure is being addressed? Indicators Reasons for public policy intervention Evidence of market failure Are the programme objectives valid? Whether the programme objectives were valid to start with and whether they are kept up to date Are the programme objectives compatible with objectives in the statement of strategy? Documented links to statement of strategy How does the programme fit-in with other programmes of the Department or programmes of other Departments/Agencies that target the same category of beneficiary or economic/social issue? Linkages and dependencies between programmes at the expected results level Is the programme rationale kept under review? The frequency of review 32

15 VALUE FOR MONEY AND POLICY REVIEW INITIATIVE GUIDANCE MANUAL Efficiency and economy Efficiency and economy look at performance in terms of the production of outputs and the acquisition and use of inputs (See table below). It is common for economy and efficiency to be studied together in an evaluation report although, in many cases, it is also worthwhile to study them separately. Economy Definition Securing the appropriate quality of inputs at the best price Programme Logic Model Inputs Efficiency Optimising the ratios of inputs to outputs Inputs, Activities, and Outputs Efficiency is a comparison of outputs to inputs and can be viewed in two different ways: whether the same level of output and result could be achieved with fewer inputs, i.e. at a lower cost; or whether a higher quality or quantity of outputs might be delivered from a fixed amount of input. Evaluations of economy are in-depth considerations of inputs. The template VFM review Terms of Reference (see Annex 2 to Chapter 2) include a separate criterion for a study of the trend of resource inputs. A study of efficiency and economy will cover some or all of the following issues. Efficiency Issues The throughput of outputs The timeliness of production of outputs Performance questions that can arise What are the outputs? How many are produced? Are output targets met? How long does it take to produce an output? Are time targets met? What is the backlog of production in the system? Is it getting better or worse? Indicators Actual number of outputs produced Outputs produced per hour, day, week or month Average time to produce a unit of output Movement in backlogs The Quality of the Output The Cost of an Output Management and resource allocation Economy Issues The cost of the resource inputs How complete and accurate are outputs? How many complaints are there about outputs? How much did the output cost per item and altogether? Distinguish between programme and administration cost. Are the management structures and allocated resources appropriate for the achievement of programme objectives? Is the measurement framework (indicators) adequate? Was the best price obtained? Could a cheaper price be obtained? E.g. performing work during normal hours instead of on overtime Accuracy level (e.g. of a report) Number of valid complaints about outputs Unit cost of output Management arrangements Budget allocations Proposed monitoring indicators and targets Actual cost of input Unit price or cost of input 33

16 VALUE FOR MONEY AND POLICY REVIEW INITIATIVE GUIDANCE MANUAL The quality or mix of the inputs Identification of wasted expenditure inputs paid for but unused How do the inputs compare with the inputs of equivalent programmes elsewhere? Was a higher and more expensive grade of staff used? E.g. an AP doing the work of a HEO Was the quality of inputs maintained? E.g. through staff training Cost of unused inputs Relative cost of inputs compared Actual number of staff days input Ratio of grades of staff used Percentage of payroll spent on training Effectiveness Effectiveness is defined in terms of the extent to which the objectives have been achieved and the planned benefits delivered. (Achievement of impact objectives is considered under the impact heading.) It should be kept in mind that the results can be positive or negative, and intended or unintended. In terms of the programme logic model, the study of effectiveness is a study of the relationship between outputs and results. Effectiveness will consider the immediate to short term effects that the availability of outputs had on the targeted beneficiaries. In a VFM review, the study of effectiveness should include the following issues: Effectiveness Issues The extent that the objectives were achieved? The way that the achievement of objectives is assessed Cost effectiveness Performance questions that can arise What did the programme achieve? Can the performance gap be measured or otherwise defined? What are the contributing factors to the performance gap? Is there satisfaction with the level of achievement of results? What methods are used to assess effectiveness? Is the achievement measured against a valid baseline? Is the feedback on effectiveness used in setting new objectives? Could the same results be achieved in a better or cheaper way? Indicators Quantitative and qualitative measures of actual and expected performance Input/output indicators that reflect contributing factors Appropriateness of indicators used Suitability and accuracy of baseline indicators Evidence of assessments of effectiveness informing policy development and objective setting Benchmark comparisons to other approaches to the same problem nationally and internationally 34

17 VALUE FOR MONEY AND POLICY REVIEW INITIATIVE GUIDANCE MANUAL Impact Impact is concerned with the wider effects of the programme for a larger group of persons (e.g. a sector) or for society as a whole (it includes the medium to long term effects on targeted beneficiaries); the successes and failures in achieving the wider objectives; and the main causes of this. In the study of impact, the same basic performance questions used for effectiveness are applied to the medium and long term sectoral objectives of policy interventions and programmes. The impact of a programme is the difference it has made. As with effectiveness, it should be kept in mind that impacts can be positive or negative, and intended or unintended The consideration of impact extends the evaluation of effectiveness into the wider sectoral or national level. The issues that should be included in impact assessment are: Impact Issues Socioeconomic objectives Performance questions that can arise What are the longer-term impacts on targeted beneficiaries? Indicators Indicators directly related to the targeted long-term impacts on direct beneficiaries What are the wider socio-economic effects of the programme? Relevant socio-economic indicators Causality Other public policy evaluation criteria Is it possible to isolate the programme contribution to wider impact? Were there unintended side effects? What are the other relevant policy impacts? Direct contribution of programme to strategic objectives Measures of additionality, deadweight and displacement Poverty impact assessment Equality impact assessment Regulatory impact analysis Rural proofing The definitions of economy, efficiency, effectiveness and impact always need to be customised to the inputs, activities, outputs, results and impact of the Department or programme being studied. This is why it is important to produce a programme logic model as part of the evaluation planning papers for all VFM reviews. (See Chapter 4) Continued relevance In a VFM review, the evaluation of the continued relevance of the programme covers the following issues: Continued Relevance Issues Performance questions that can arise Indicators Continued relevance What is the justification for the continued allocation of public funds to the programme? How does the programme fit in with other programmes of (i) the Department, (ii) other Departments, and/or (iii) other public bodies in delivering on the Government s overall policy objectives in the area in which the programme operates Estimate of programme benefits expressed in quantitative, qualitative and timing terms and taking account of what would have been achieved anyway in the absence of the programme. Changes in programme context/environment. 35

18 VALUE FOR MONEY AND POLICY REVIEW INITIATIVE GUIDANCE MANUAL Has the examination of effectiveness shown that the programme is achieving its objectives? The five evaluation criteria (in green) can be mapped to the programme logic model (in red) in the following way. Strategic Objectives Continued relevance Rationale Inputs Impacts Impact Efficiency Activity Effectiveness Outputs Results 3.5. Additional public policy evaluation criteria In addition to the five main evaluation criteria outlined above, VFM reviews will need to take into account a range of additional public policy criteria that are relevant to the design and implementation of Government programmes. These are sometimes referred to as horizontal principles. The most important are addressed briefly below, with references for further reading given in each case. Depending on the nature of the programme being reviewed, it may be appropriate to address the additional public policy criteria under one or more of the five evaluation criteria outlined in section above. For example, for a programme run by the Department of Social and Family Affairs, poverty impact assessment might be handled under Effectiveness and Rationale. Alternatively, they may be handled as separate criteria in their own right. Further guidance on the development of evaluation criteria in the planning of VFM reviews is set out in Chapter Poverty Impact Assessment Poverty Impact Assessment is defined as the process by which Government Departments, local authorities and State agencies assess policies and programmes at design, implementation and review stages in relation to the likely impact that they will have or have had on poverty and on inequalities which are likely to lead to poverty, with a view to poverty reduction. Under the Poverty Impact Assessment guidelines issued by the Office for Social Inclusion (OSI), Poverty Impact Assessment should be conducted prior to finalising any Memorandum for Government and in the preparation of SMI Statements of Strategy, Annual Business Plans, Estimates and Annual Budget proposals, National 36