Ex ante Evaluation in the EU:

Transcription

1 Ex ante Evaluation in the EU: Ex ante Evaluation Operational Programme of youth employment Ignacio Moral-Arce 6th july

2 What theory says What European Commision says What we did!! 2

3 Ex ante evaluation of public intervention Introduction. Theoretical approach The ex-ante evaluation is intended to help ensure that what is proposed in the programme makes sense, is logical and justified, and that the proposed priorities, objectives, measures and allocations of resources are appropriate in order to respond to the needs identified through the needs assessment. It should act as a check to see if the needs assessment is comprehensive and balanced, whether the objectives are in line with the needs identified, and whether the strategy, activities and resource allocation proposed in the programme are likely to achieve its objectives and targets. Where appropriate it should make recommendations to improve the draft programme. 3

4 Ex ante evaluation of public intervention Introduction. Theoretical approach the evaluation should answer the following questions: Does the programme represent an appropriate strategy to meet the challenges confronting the region or sector? Is the strategy well defined with clear objectives and priorities and can those objectives be realistically achieved with the financial resources allocated to the different priorities? Is the strategy coherent with policies at regional, national (including the National Strategic Reference Framework) and Community level? Lisbon objectives? Are appropriate indicators identified for the objectives and can these indicators and their targets form the basis for future monitoring and evaluation of performance? What will be the impact of the strategy in quantified terms? Are implementation systems appropriate to deliver the objectives of the programme? 4

5 What theory says What European Commision says What we did!! 5

6 EU Ex ante evaluation of Operational Programmes Period:

7 EU Ex ante evaluation of Operational Programmes Period: Exante evaluation TITLE V:MONITORING AND EVALUATION CHAPTER I: Monitoring Section I:Monitoring of programmes Article 48 Composition of the monitoring committee Article 49 Implementation reports Article 50 Functions of the monitoring committee Article 51 Annual review meeting Section II: Strategic progress Article 52 Progress report Article 53 Reporting by the Commission and debate on the ESI Funds 7

8 EU Ex ante evaluation of Operational Programmes Period: Exante evaluation TITLE V:MONITORING AND EVALUATION CHAPTER II: Evaluation Article 54 General Provisions Article 55 Ex ante evaluation 1. Member States shall carry out ex ante evaluations to improve the quality of the design of each programme. 2. Ex ante evaluations shall be carried out under the responsibility of the authority responsible for the preparation of the programmes. They shall be submitted to the Commission at the same time as the programme, together with an executive summary. The Fund-specific rules may establish thresholds below which the ex ante evaluation may be combined with the evaluation for another programme 8

9 EU Ex ante evaluation of Operational Programmes Period: Exante evaluation 3. Ex ante evaluations shall appraise: (a) the contribution to the Union strategy for smart, sustainable and inclusive growth, having regard to the selected thematic objectives and priorities, taking into account national and regional needs and potential for development as well as lessons drawn from previous programming periods; b) the internal coherence of the proposed programme or activity and its relationship with other relevant instruments; (c) the consistency of the allocation of budgetary resources with the objectives of the programme; (d) the consistency of the selected thematic objectives, the priorities and corresponding objectives of the programmes with the CSF, the Partnership Agreement and the relevant country specific recommendations and where appropriate at national level, the National Reform Programme (e) the relevance and clarity of the proposed programme indicators; (f) how the expected outputs will contribute to results; 9

10 EU Ex ante evaluation of Operational Programmes Period: Exante evaluation 3. Ex ante evaluations shall appraise: (g) whether the quantified target values for indicators are realistic, having regard to the support envisaged from the ESI Funds; (h) the rationale for the form of support proposed; (i) the adequacy of human resources and administrative capacity for management of the programme; (j) the suitability of the procedures for monitoring the programme and for collecting the data necessary to carry out evaluations; (k) the suitability of the milestones selected for the performance framework; (l) the adequacy of planned measures to promote equal opportunities between men and women and to prevent any discrimination, in particular as regards accessibility for persons with disabilities; (m) the adequacy of planned measures to promote sustainable development; (n) measures planned to reduce the administrative burden on beneficiaries. 10

11 EU Ex ante evaluation of Operational Programmes Period: Evaluators should propose a mix of methods which they will use to address the different components of the evaluation and to answer the key evaluation questions. It should act as a check to see if the needs assessment is comprehensive and balanced One of the main goals of the evaluation is to assess the programme strategy and theory of change underpinning the programme, and to participate in the programme design in an iterative and interactive processs 11

12 What theory says What European Commision says What we did!! 12

13 13

14 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 14

15 Background - SWOT Analysis APPRAISAL OF THE SOCIO-ECONOMIC ANALYSIS AND THE RELEVANCE OF THE STRATEGY TO THE NEEDS IDENTIFIED SWOT analysis and the needs assessment The Managing Authority in collaboration with other Ministry departments, agencies, institutes, etc. prepares the SWOT analysis and the needs assessment for the OP. The ex-ante evaluator gives feedback on these analyses. The evaluator should look at the baseline values of context and result indicators At this stage the ex-ante evaluator gives feedback on the programme's expected contribution to the thematic objectives outlined in the Common Provisions Regulation, the intervention logic proposed in relation to the needs identified, internal and external coherence, the coherence between expected outputs and results 15

16 Background - SWOT Analysis The analysis of the situation in terms of Strengths, Weaknesses, Opportunities and Threats (SWOT) is a prerequisite for the needs assessment and setting of relevant objectives to meet the challenges and needs. The SWOT analysis provides the background against which the interventions proposed can be checked to ensure that they are justified, relevant and adequate in terms of the optimal use of public funds. Identification of most important needs and major trends on the labour market (adaptability of the labour market, access to employment for everyone, social inclusion, enhancement of human capital, ), Identification of the underlying (assumed) causes of disparities 16

17 Background - SWOT Analysis Identify and appraise the medium and long term needs. In carrying out this task, ex ante evaluation has to: - assess the completeness of the SWOT analysis; - analyse causes of disparities identified; - identify and assess the driving forces towards sustainable rural development; - contribute to the quantification of context and objective related baseline indicators (common and programme-related) by verifying and, where appropriate, suggesting modifications of the indicators and figures; - assess and, where appropriate, suggest revisions of the ranking of disparities and priorities assigned to identified needs and their translation into 17 objectives and concrete priorities for action.

18 Background - SWOT Analysis the evaluators should review and incorporate lessons learnt from previous evaluations and consider also the potential to maximise Community added value Evaluation evidence is available from the ex ante evaluation, mid term evaluation and the update of the mid term evaluation for the period as well as the ex post evaluationss Additional work may need to be undertaken by the evaluator to complement the existing analysis, which may lead to recommendations for improvements to the analysis contained in the programme The evaluator should then assess the relevance of the strategy to the identified needs. 18

19 Background - SWOT Analysis The ex-ante evaluator should ensure that the SWOT analysis and the needs assessment: are comprehensive, based on full and appropriate data (context indicators), and provide a holistic picture of the sectors and communities; are logically interlinked with the identified needs and are properly justified by the SWOT; are consistent and complementary to that for other CSF programmes, as shown in the Partnership Agreement; 19

20 Background - SWOT Analysis In first versions, the evaluation team believes that, despite the solid context analysis presented by the programmer team, it might be appropriate to deepen this analysis, carrying out a more detailed study of the current situation, the evolution and main characteristics stand out. This in-depth study would have a complete picture of the needs and existing strengths. The programming team has taken into account most of the proposals of the evaluation team for conducting the SWOT analysis. Evaluation team made proposals in previous versions of the appraisal, and in order to have a more thorough view of the initial situation, In final version: the evaluation team considers that the SWOT analysis presented is complete and accurately reflects the main features where it locks the POEJ 20

21 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherency: internal and external (Contribution to Europe 2020 strategy) Consistence: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 21

22 Lessons from the past Ex ante evaluation of OP for youth employment Background Learning lessons "Don't repeat old mistakes, make new ones!" Most programme proposals follow on from existing programmes, and many EU initiatives will have similarities to other programmes carried out by the Commission or other organisations. It is critically important to extract the relevant lessons from earlier experiences. The following important knowledge can usually be gathered from evaluation reports and related studies: Relevance of the existing strategy and possible needs for its amendment Effectiveness of existing delivery instruments Critical factors affecting the implementation of a programme Types of problems encountered while monitoring and evaluations were carried out 22

23 Background Learning lessons Learning from ex post evaluation reports The ex ante evaluation that prepares the adoption of a future programme should first look at the mid-term evaluation of the period drawing to a close. This mid-term evaluation usually has produced conclusions on the first years of activity of a programme. It can thus already say something about the relevance and effectiveness of chosen strategies The ex ante evaluation should also look at the ex-post evaluation report of the period preceding the current one and possible thematic evaluations in the same policy area 23

24 Background Learning lessons In the programming period The economic context was quite favorable for the Spanish economy with high annual rates of job creation. However, the current economic crisis emerged in 2008, at the beginning of the programming period, it has generated effects devastating for employment. it would be appropriate to take into account in preparing the current period concerning: - Simplification and streamlining of procedures. - Avoid duplication of actions. - Improve management and control systems. - Encourage the use of financial instruments. the assessment team may conclude that the programming team has taken into account what happened in the previous period. is a starting point useful to assess progress carried out by the ESF, 24

25 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 25

26 Strategy : Relevance Relevance in the context of programming means that the programme specific objectives are aligned with the identified challenges and needs in relation to the Europe 2020 strategy and that they are given an appropriate weight in the programme. The ex-ante evaluators should thus analyse: - analyse the degree to which the measures identified are adequate to meet the objectives - whether the identified national or regional challenges and needs are in line with the Europe 2020 objectives and targets, the Council recommendations and the National Reform Programmes; - whether the investment priorities and their specific objectives consistently reflect these challenges and needs 26

27 - Strategy : Relevance Identify weaknesses (strengths) most pressing within the sector/region Prioritize the weaknesses (cooperation programmer - evaluator) Calculate the potential impact of the strategy Using a statistical measure that relates the ability to "pull" strategy (potential impact) on the importance of the weaknesses (strengths) = degree of partial relevance

28 - Strategy : Relevance 1 st Step: Prioritizing the weaknesses (same for strenghts) W1 W2 W1 W5 HIGH = 9 W3 W7 W8 W4 W5 W6 W2 W6 W3 MEDIUM = 6 W7 W8 W4 LOW = 3

29 - Strategy : Relevance Step 3. Calculation of the degree of partial relevance (combination of the two previous stages) The degree of partial relevance of the investment strategy refers to the ability of the package of each general objective to mobilize the weaknesses (enhance strengths). only takes into account the second and last row of the above table, so for every weakness relates the potential impact of the investment strategy on that weakness (strength) with the importance of it W1 W2 W3 Weak Power (Importance) Objective - measures Specf. Objective Specific Objective Specific Objective Specific Objective Total impact

30 - Strategy : Relevance Step 4: a plot where the X axis is the importance of weakness or strength is performed, and the Y-axis value of the potential impact on either weakness or strength Option 1: OLS E(potential impact import. Weak) Option 2: Correlation coefficient between potential impact and import. Weak 30

31 Strategy : Relevance Weaks strengths W1 - The unemployment rate (in 2013, years is 74.3%, in years: 51.9%) is high and has reached these levels despite the fall in the rate of activity. In addition, the youth unemployment rate is double that of the general population. W2 - Youth employment destroyed in the case of men comes mainly from construction (506,000) and manufacturing (291,000). In the case of women, it comes mainly from services (682,000) sector, W3 - The destruction of youth employment has focused on workers with lower levels of formal education, which can hinder their professional reorientation W4 - The working conditions are very precarious in youth employment W5 - Important values of long (and very long) unemployment rate W6 - Low knowledge of foreign languages W7 - Scarce relative weight of middle grade Vocational Training W8 - Important number of young people neither study nor work (NEET) W9 - In the NEET collective there are mainly young people from low levels of education, but the problem affects young people with all levels of education. For example, college students have a significant presence, especially in the case of women. W10 - Young people with lower levels of education are the least form. Need very proactive policies W11 - High rate of school dropout, doubling the values of the EU-27 W12 - Scarce entrepreneurship and low levels of youth selfemployment % S1 - The recovery of domestic consumption and the rise of tourism favor job creation in some of the sectors that have destroyed female youth employment (trade and catering) S2 - Recognition by young people of the importance of education and training. The scarcity of employment opportunities for the younger population has led to a return to training and decreased dropout S3 - Importance of family as a social network and support youth unemployment S4 - Best transversal skills levels in the rest of the active population (use of technologies, adaptation to change, teamwork, etc.) and increased susceptibility to both national and international labor mobility. S5 - Gender differences in relation to the labor market are shrinking steadily although the position of women remains unfavorable in almost all indicators % degree of import SO to improve mediaton SO to improve employability SO to promote entrepreneurship SO To promote hiring SO to improve responsiveness of labor market institutions % % % % % % % % % % potential impact

32 Strategy : Relevance Correlation coef: 0.56 positive correspondence between the main strengths and weaknesses identified in the SWOT analysis and potential impacts to be expected from the specific objectives and investment priorities Therefore, the strategy adequately address the main weaknesses of the area such as youth unemployment (W1) that job destruction occurs primarily in workers of low educational level (W3), and the existence of a significant number of young people who neither study nor work (W8) the evaluation team concludes that the final objectives of the strategy set out in the PO are relevant to the needs identified in the diagnosis of the situation. 32

33 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 33

34 Strategy : Internal coherence The (internal) coherence of the strategy should be evaluated. This will require an analysis of the relationships and complementarities between the different priorities(spec.objetives), including the contribution of each priority (SO) to the programme objectives Any conflict between priorities should be highlighted. The evaluators should analyse the relationship between the specific objectives of each priority axis, and between the specific objectives of the different priority axes. The evaluators should verify that complementarities and potential synergies are identified. analyse how the various measures complement each other in promoting the achievement of said objectives 34

35 Strategy : Internal coherence To examine the ability of each of the objectives is to influence the achievement of other objectives, either - by influence (achieving this objective can increase the probabilities of achieving other objectives of the program) - or by sensitivity (the achievement of other objectives can achieve the target in question). The methodology to analyze these effects: building a matrix in which the interrelationships between the different objectives of the strategy are scored. We assigned a value to each crossing (0, 3 and 5) depending on the intensity of interrelation between the objectives This matrix allows to obtain, by rows, intensity and functional link between the objectives, that is, the degree of influence of a SO on the rest. Similarly, but by columns, to determine to what extent a target is influenced by the other, establishing therefore the sensitivity of this objective (how can be affected for the other SO) 35

36 Strategy : Internal coherence SO1 SO2 SO3 SO4 How Affect SO1 X SO2 3 X SO3 3 3 X 3 9 SO X 15 How can beaffected /4 = 11,25 Graphic representation: put the value of each target in this chart where each point is given by the column value in the x axis, while the value of the column in the y-axis. Calculate the average and use it as a reference value 36

37 Strategy : Internal coherence Types of objectives: Strategic objectives: are those with a higher values than average, both in influence and sensitivity rating. they are key objectives of the strategy for its high potential. influential Objectives: have a high capacity of influence (above average), but a lesser degree of sensitivity (below average). Objectives with high degree of influence. sensitive Objectives: greater sensitivity than average, but less influence than average. its achivement depends on the achievement of other objectives degree of dependence on the rest. Independent objetives: have a greater degree of incorrelation with the rest Obj (both low influence and sensitivity to the mean) 37

38 es Ex ante evaluation of OP for youth employment Strategy : Internal coherence SO to improve mediaton SO to improve employability SO to promote entrepreneurship SO To promote hiring SO to improve mediaton SO to improve employability SO to promote entrepreneurship SO To promote hiring SO to improve responsiveness of labor market institutions How affect X X X X 1 10 SO to improve responsiveness of labor market institutions X 16 How is affected The evaluation team notes that almost all SO maintain some kind of relationship among them, either weak, moderate or intense. These synergies favor the functioning of the Operational Programme and the achievement of the overall objective of the intervention 38

39 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 39

40 Strategy : External coherence Given increased emphasis on the strategic nature of cohesion policy and the need for it to contribute to the achievement of the Lisbon objectives, an important component of the ex ante evaluation is an appraisal of the coherence of the strategy proposed with national policies and the Community policies outlined in the Community Strategic Guidelines. Therefore it is important that the ex ante evaluation verifies those synergies and the potential to deliver relevant elements of the Lisbon Strategy. The evaluator should examine the interaction between the different priorities and the coherence of the intervention logic of the Operational Programme and its priorities with the National Strategic Reference Framework and the Community Strategic Guidelines 40

41 Strategy : External coherence Relation with other relevant instruments The evaluators should assess the coherence of the programme with other relevant instruments at regional, national and EU level. On the one hand they should review the analyses of the programme contribution to other strategies and policiess method: to develop a matrix that cross the information between measures of our strategy (rows) with the objectives of other policies (columns), possibly with an ordinal scale as follows: The EU2020 strategy Europe 2020 is the framework for the coordination of economic and employment policies of the Member States of the EU. It establishes a cycle that extends to 2020, reference to design national reform programs. The Union Strategy for smart, sustainable and inclusive growth sets goals and shared objectives that should govern the actions of the Member States and the EU 41

42 Strategy : External coherence EU 2020 strategy has five objectives: - Objective 1: 75% of the population between 20 and 64 should be employed. - Objective 2: 3% of EU GDP should be invested in R & D. - Objective 3: The "20/20/20" should be achieved on climate and energy. - Objective 4: the percentage of early school leavers should be under 10% and at least 40% of the younger generation should have a tertiary degree. - Objective 5: Decrease of 20 million people risk of poverty Objectives EU2020 Specific objectives Youth employment OP SO to improve mediaton SO to improve employability SO to promote entrepreneurship Objective 1: 75% of the population between 20 and 64 should be employed. - Objective 2: 3% of EU GDP should be invested in R & D. - Objective 3: The "20/20/20" should be achieved on climate and energy. - Objective 4: the percentage of early school leavers should be under 10% and at least 40% of - Objective 5: Decrease of 20 million people risk of poverty Degree of cohorence among objectives % % % SO To promote hiring % SO to improve responsiveness of labor market institutions % the objectives of the EU 2020 strategy are covered by the specific objectives of the Operational Programme concerning employment and combating poverty and social exclusion, with more than 90%, while the other objectives of the strategy is not They are covered by Youth employment OP. degree of coherence among Objectives 100% 24% 24% 64% 92% 42

43 Strategy : External coherence European interventions: - EU2020 strategy - Position Paper European Commision National interventions: - Partnership Agreement National Reform Programme 2013 PNR Spain. - Strategy Entrepreneurship and Youth Employment. - National Implementation Plan Youth Guarantee, Youth Employment Initiative. 43

44 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 44

45 Strategy : Consistency Linkage between supported actions, expected outputs and results The ex-ante evaluation should assess the intervention logic of the programme and of each priority axis. This intervention logic should start with the change that the programme intends to bring in the Member State or region. This change (or intended result) should be achieved through operations delivering outputs. It is good practice to use a logical framework to clarify the intervention logic under each investment priority or priority axis. Such a stylised representation demonstrates the causal links between the different actions, the planned outputs and the intended results. It also helps to build a shared vision of the programme objectives and type of interventions to achieve them Examining the rationale of the strategy will require an in-depth analysis of the objectives and priorities of the programme. The ex ante evaluator should examine the theory underlying the strategy and assess its validitys 45

46 Strategy : Consistency This analysis is developed through two elements: The design of the hierarchy of objectives that clearly show the interrelationships between the general and strategic objective of public intervention, and objectives of other levels The design of the intervention logic is established for each of the activities, which presents the structure: resources, activities, outputs and outcomes 46

47 Strategy : Consistency Evaluating consistence of objectives Evaluating logic intervention 47

48 Strategy : Consistency Evaluation of the hierarchy (structure) of objectives of OP youth employment Through the analysis conducted by the evaluation team it can say that the OP is consistent in terms of relations between short-, medium- and longobjectives In addition, evaluators notes that there are synergies (SO8.7.1) 48

49 Strategy : Consistency Assesment intervention logic (for each activity) of OP youth employment SO To improve mediation. (outcome) Actions involving guidance, information assistance in job seeking Actions with placement agencies: Mobility programmes Logical framework for each activity of the Specific Objective: 49

50 Strategy : Consistency Assesment intervention logic (for each activity) of OP youth employment SE actions: there is a well marked logic in all activities. The evaluation team believes that under certain assumptions (and risk assumpions) the specific objective can be achieved with these actions. 50

51 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 51

52 Horizontal principles Horizontal principles the ex ante evaluator to assess "the adequacy of planned measures to promote equal opportunities between men and women, to prevent discrimination and to promote sustainable development". The ex ante evaluation should appraise how the aim of promoting equality has been taken into account in the preparation of the programme. For example: was the gender perspective taken into account in the socio-economic analysis? Is there any evidence that gender issues were explicitly taken into account in the programme design stage? Have equality bodies/ organizations and other relevant stakeholders been consulted? For the promotion of equal opportunities and the prevention of discrimination, the ex ante evaluation shall appraise the description of specific actions during the preparation, design and implementation of the programme 52

53 Horizontal principles equal opportunities between men and women Strategy Governance Indicators Budget Intensity comments context analysis is performed taking into account the breakdown by gender and age groups of both groups. In the investment priorities as well as specific objectives in the inclusion of this principle is observed. It appears to be in coordination with the Government, autonomous regions, social partners and other stakeholders in the development and implementation of POEJ agents. Although it should explicitly guarantee the participation of women across the board. All indicators related to individuals (participants or unemployed, inactive) are broken down by gender issue. Eligibility for funding will be based on the collective beneficiary, ie the age of 15 to 24 regardless of their nature (employment, training, education or social inclusion). However, the financial allocation does not appear broken down by sex. to prevent discrimination Strategy Governance Indicators Budget The measures take into account this principle such as people in situations of multiple discrimination and vulnerable groups. analysis have by age and autonomous regions in investment priorities as well as the specific objectives inclusion of this principle is observed. It appears to be in coordination with the Government, autonomous regions, social partners and other stakeholders in the development and implementation of POEJ agents. Although it should explicitly guarantee the participation of women across the board. The indicators are designed to address the status of unemployed and inactive rather than specific groups, but is implicit in the application of this principle Eligibility for funding will be based on the collective beneficiary, ie the age of 15 to 24 regardless of their nature (employment, training, education or social inclusion). Although the financial allocation does not appear explicitly broken down by groups. 53

54 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 54

55 Indicators Indicators, monitoring and evaluation Article 27(3) CPR on the content of the operational programmes requires that each priority axis shall "set out indicators to assess progress of programme implementation towards achievement of objectives as the basis for monitoring, evaluation and review of performances The purpose of this part of ex ante evaluation is to translate high-level policy goals into more tangible quantified or otherwise measurable objectives, and to define on what basis achievement will be measured Setting concrete measurable, or at least verifiable, objectives is fundamental to the success of your programme Evaluation of: Theoretical indicators and..values expected 55

56 Indicators The following types of indicators can be differentiated: Context indicators, used to describe the situation in the sectors and communities covered by the programme to conduct the SWOT analysis Financial indicators are measuring the financial input which should lead to certain outputs, results and impacts (e.g. budget allocated). Output indicators are directly linked to measures and operations Results indicators which capture the direct effects of interventions and are linked to focus areas Common indicators are output indicators and, in the case of the ESF, also result indicators. Their use is compulsory and they are annexed to the Funds regulations (ERDF, ESF, Cohesion Fund, ETC). Specific indicators: if common indicators don t capture the activity (and its effects) 56

57 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 57

58 Indicators What must be covered in the ex-ante evaluation? The evaluation team must cover the following tasks: First, determine that the achievements of all Union priorities included in the programme will be adequately assessed. The evaluator should check that: all necessary common indicators are included, identify gaps and support the Managing Authority in completing them programme-specific indicators have been proposed wherever necessary. If gaps in programme-specific indicators are found, the ex-ante evaluator should propose how to complete these gaps with appropriate indicators. programme-specific indicators proposed are relevant to specificities of the OP. The evaluator should also check their consistency with the common indicators; 58

59 Indicators And 35 more outcome indicators (by actions) 59

60 Indicators 60

61 Indicators Using and choosing indicators Indicators should reflect the central aspects of the results or outcomes that are being sought. However, defining the relevant indicators is not enough - mechanisms for collecting data on their values Criteria for indicators By definition, indicators only give a partial picture of the progress achieved. evaluators should assess whether the proposed programme-specific indicators are SMART: specific, measurable, available/achievable in a cost effective way, relevant for the programme and available in a timely manner 61

62 Indicators The theoretical properties that the evaluation team has reviewed for this battery of indicators (taking into account the regulatory requirements) were: - Relevance: Indicators should reflect the degree of compliance of activities and objectives to which the indicator belongs, catching therefore the direct impact on it. Its value must be influenced directly by actions. - Consistency: indicators are suitable with respect to the expected result for which have been proposed, capturing their essence. the evaluation team will assess whether a given indicator fully meets the property, meets moderately or not met, - Clarity: indicators have a clear name and unambiguous definition and easy to understand. That is, they should be easily understandable, free of complexities. - Feasibility: indicators should be built, as far as possible, based on available statistics. Its measurement should not impose an undue burden on fund managers. the evaluation team will assess whether a given indicator fully fulfills the property, moderately or not fullfil, 62

63 Indicators 100% of Product Indicators fulfill the properties of relevance, coherence and feasibility. only the indicator "including longterm unemployed" compliance with the clarity of it occurs moderately. In this sense, the evaluation team believes that the indicator could completely fulfill this charasteristic if referring to young participants The evaluation team verifies that the total outcome indicators meet the properties of relevance, coherence, clarity and feasibility 63

64 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 64

65 Indicators: Target values Suitability of milestones "Milestones are intermediate targets for the achievement of the specific objective of a priority, where appropriate, expressing the intended progress towards the targets set for the end of the period The evaluators should assess "the suitability of the milestones selected for the performance framework" the ex ante evaluators should first analyse whether milestones are relevant, i.e. "capturing essential information on the progress of a priority". They should also analyse whether the milestones can realistically be achieved For this analysis, they may consider the rhythm of implementation of the programme in the period and available resources. 65

66 Indicators: Target values The practical property that the evaluation team has reviewed the level of realism (feasibility) of the predicted values indicators 66

67 Indicators: Target values the evaluation team has valued positively for both milestones 2018 to 2023 goals, the application of the methodology of average costs used by programmers to calculate the values in those moments. the pace of financial executions is consistent with the evolution of the PO. the development thereof occurs linearly over time, except in the case of the Youth Employment Initiative where execution of resources occurs in whole, in the first two years indicators differentiated by gender, forecasts have been made using proportions of the previous period do not have to stay in this new period 67

68 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 68

69 Governance Governance: Analysis of good practices This section is intended to assess the changes proposed by the essential can be highlighted: - The concentration of resources and simplification, so that Spain simplify the current management model by reducing the number of operational programs and intermediate bodies, - Decentralization, in order to get a more distributed management model, greater role of regions, - Management efficiency through the implementation of e-government solutions, - Reducing burdens and improve administrative capacity, - The possibility of incorporating private actors in providing the national portion - Increased use of financial instruments to facilitate investment and offset the budgetary adjustment of Administrations. it is concluded that the principle of proportionality, reduction of administrative burdens, the participation of private agents and the use of financial instruments are present in the elaboration of PO 69

70 Governance Governance: human Resources In organizational terms, the main tasks to be developed in the ESF and thus in the development of POEJ are programming, management, certification and payments, evaluation, audit and dissemination. It detects evaluation team recommends increasing workers as possible in order to improve the efficiency and quality of all obligations have established Governance: Communication and coordination. The application of the principle of partnership For this, the evaluation team answers the questions: who participated? (Description of participants), when? (Through the meetings that have been held) and how?, identifying the connections between the agents involved 70

71 Data collections Tasks: assess the monitoring of the program, as well as assess their development to analyze its evolution in terms of efficiency and achieving the objectives set Evaluation team asses positively the development of information system for the implementation of the Youth Guarantee, Evaluation team analize very positive the generation of a database with the whole indicator system. It is also remarkable the positive effect it has on the side of governance. 71

72 Ex ante evaluation index of topics 1. Programme strategy 1.1. Starting point; Background Learning lessons 1.2. Evolution of Strategy:Logic in the intervention Relevance of programme objectives Coherence: internal and external (Contribution to Europe 2020 strategy) Consistency: Linkage between supported actions, expected outputs and results Horizontal principles 2. Indicators, monitoring and evaluation 2.1. Relevance and clarity of proposed programme indicators 2.2. Quantified baseline and target values. Suitability of milestones 3. Administrative capacity, data collection procedures and evaluation 4. Consistency of financial allocations 72

73 Financial allocations: Budget financial allocations the evaluation team analyzes mainly the distribution of EIE Funds must be consistent with the strategy proposed in which priority investments identified from diagnosis. we studied whether the axes that are best suited to meet the main objectives in the strategy, happen to be receiving more resources. graphic: the vertical axis of the figure measured for each priority axis, its strategic importance in the SWOT analysis of PO. the volume of each sphere represents the percentage of funding it receives from the total budget. The desirable in terms of consistency in financial allocations, is that the larger circles are located on the top of the chart, thus indicating that has given more to the strategic OE Evaluation: good consistency between financial allocations and the strategic nature of each priority axis of PO 73

74 Financial allocations: Budget temporal distribution of financial flows according to the information contained in the financing plan of the Operational Programme. The program provides that for all measures YEI and YEI-ESF will take place in 2014 and 2015, the investment priority 8.2 (with its corresponding N + 3), with direct action on the program's target population (NEET ). recommendation by the evaluation team, noting the possible emergence of risks of financial absorption with respect to OP, it is necessary to reduce as far as possible all restrictions, administrative or otherwise, to ensure standards of financial performance proposed by programming team. 74