ESF Ex-post Evaluation Synthesis

Transcription

1 Ref. Ares(2015) /06/2015 ESF Ex-post Evaluation Synthesis Contract No. VC/2015/0098

2 EUROPEAN COMMISSION Directorate-General for Employment, Social Affairs and Inclusion Unit A3 Analysis, Evaluation, External Relations Contact: Andris Skrastins European Commission B-1049 Brussels

3 EUROPEAN COMMISSION ESF Ex-post Evaluation Synthesis Contract No. VC/2015/0098 Directorate-General for Employment, Social Affairs and Inclusion [Ex Post Evaluation Synthesis] 2014 EUR [number]en

4 TABLE OF CONTENTS Introduction OVERALL APPROACH to the synthesis The purpose of the ESF ex post evaluation synthesis The scope of the ESF ex post evaluation synthesis and use of sources Evaluation criteria Our approach to the synthesis Providing missing information Drawing the data together Allocating Priority Axes to one single Priority (and sub priorities) Aggregating output and result indicators METHODOLOGY AND TOOLS Main tasks and deliverables Evaluation task 1: Follow up of ESF thematic ex-post evaluations Evaluation task 2: National level analysis National inventories Country synthesis reports Evaluation task 3: EU level analysis Thematic synthesis reports EU synthesis report REPORTING The Inventory of Evaluations The Country Reports The Thematic Synthesis Reports The Final Synthesis Report WORK ORGANISATION Roles and responsibilities of the team Functions, responsibilities and work flows Allocation of time and resources to the project and to the tasks Timetable Annex 1: The allocation of Priority Axes to ESF Priorities page 4 May 2015

5 TABLES, FIGURES AND BOXES Table 1. Evaluation criteria, sources and additional data gathering Table 2. Overview of tasks, data sources and outputs Table 3. Database structure for the thematic ex post evaluations and EU synthesis report Table 4. Overview of steps for the accomplishment of Task Table 5. Overview of steps for the accomplishment of Task 2A Table 6. Table of contents for country synthesis reports Table 7. Overview of steps for the accomplishment of Task 2B Table 8. EU Synthesis Report structure Table 9. Overview of steps for the accomplishment of Task 3A Table 10. Structure of the EU synthesis report Table 11. Context indicators for each ESF priority (as included in the EU27 templates) Table 12. Overview of steps for the accomplishment of Task 3B Table 13. Content of the inventory Table 14. Working days allocated to the partners in the Consortium Table 15. Allocation of working days to the tasks Table 16. Working days allocated to each member in the team (including tasks) Table 17. Working days per expert Table 18. Timetable Table 19. The allocation of pas in the thematic ex post evaluations and proposed allocation to promoting partnerships and increasing administrative capacity Figure 1. Tasks and deliverables of the ex post evaluation synthesis Figure 2. From evidence to findings to lessons learned Figure 3. Overview of the team structure and work division Figure 4. Generalized work flow Box 1. An example: Programme specific output and result indicators in A2E May 2015 page 5

6 Box 2. Discussion points /guidelines on the construction and handing over of the the database Box 3. Updating financial and physical performance data with AIR Box 4. Approach to efficiency in the three ex post evaluations Box 5. Template for the Phase 2 Country Reports Box 6. Draft Thematic Synthesis Report Structure Box 7. Final Report Structure page 6 May 2015

7 LIST OF ABBREVIATIONS A2E AIR ALMP CAV CE CIE CoE CSR EC EEN EES ESF EU LFS LMP MA MS NGO OP QA PES PL PR SFC SI TA TBE Access to employment and sustainable integration into employment Annual Implementation Report Active labour market policy Community Added Value Country Experts Counterfactual Impact Evaluation Code of expenditure Country synthesis report European Commission Expert Evaluation Network European Employment Strategy European Social Fund European Union Labour force survey Labour market policy Managing Authority Member State Non-governmental organisation Operational Programme Quality assurance Public Employment Service Project leader Public relations Structural Funds Common Database Social Inclusion Technical Assistance Theory Based Evaluation May 2015 page 7

8 INTRODUCTION This revised Inception Report is for the ESF ex post evaluation synthesis (Contract No. VC/2015/0098). The contractor is a consortium led by Metis GmbH with Fondazione G. Brodolini (FGB) and Panteia as partners. All partners will contribute to the study and each has one key expert in the core team. Herta Tödtling-Schönhofer (Metis) will manage the evaluation with operative support from Isabel Naylon (also Metis). Building on previous cooperation in the framework of the three thematic ex post evaluations 1, Isabel will manage Task 1. She will also manage Task 2 and ensure the delivery of the updated inventories and the country synthesis reports. Metis will supervise the drafting of the two thematic synthesis reports and the overall EU synthesis reports (Task 3) which will be in the hands of the Consortium partners Fondazione G. Brodolini (FGB) and Panteia respectively. All tasks are led by very experienced experts who have been analysing the current ESF programmes exhaustively in the recent past. Benno Savioli will support the core team in the updating of the inventories for which he produced the database in the ESF Expert Evaluation Network. The overview team will support the core team by preparing and structuring the available material and by supporting the core team members in drafting the reports. The present Inception Report includes a chapter on our overall approach to the synthesis and a description of our proposed methodology and tools for the management of the main tasks and outputs. The main task of the evaluation synthesis will be bringing together all the information and filling in the gaps as certain Priorities will have been thoroughly covered in some countries but not in others and the degree of depth of the information will differ. One of the main challenges in the evaluation synthesis as in the three thematic ex post evaluations will be the demarcation between the five 2 ESF Priorities (according to Article 3 of the ESF Regulation) which are the common thread throughout all the ex post evaluation exercise. This will largely have been solved by the three thematic ex post evaluations covering four Priorities (Adaptability and Human Capital, Social Inclusion and Access to Employment). A further issue will be linking the Annex XXIII and programme specific indicators to ESF Priorities and aggregating these on country level and subsequently on European level. We discuss these issues and our approach to them in Chapter 1 and the overall methodology in Chapter 2. The structure of the reports is given in chapter 3 and the overall management of the contract is described in Chapter 4. 1 Isabel is managing the Ex Post Evaluation of Access and Sustainable Integration into Employment. 2 Adaptability and Human Capital are considered one Priority as they have been treated as such in the ex post evaluation. page 8 May 2015

9 1. OVERALL APPROACH TO THE SYNTHESIS 1.1 The purpose of the ESF ex post evaluation synthesis In the course of 2014, three ESF ex post evaluations were launched by DG EMPL. These covered the ESF Priorities Adaptability and Human Capital (grouped under the ex post evaluation on Human Capital), the Integration of Disadvantaged Groups and Access and Sustainable Integration into Employment. The purpose of Ex Post Evaluation Synthesis is to synthesise the results of these thematic ex post evaluations and to provide supplementary evaluation syntheses of the ESF Priorities Promoting Partnership and Strengthening Institutional Capacity which were not covered by separate contracts. The ex post evaluation synthesis also includes 28 country reports summarising the outputs and results of the ESF investments across the ESF Priorities and presenting findings and conclusions as well as lessons learned and good practice. The synthesis should draw lessons for economic and social cohesion policy at Member State and EU level with reference to the European Employment Strategy as incorporated into the Integrated Guidelines for Growth and Jobs, within the framework of the Lisbon Strategy and the Europe 2020 Strategy, and the accompanying (country-specific) recommendations and national reform programmes. As well as serving the purpose of the ex post evaluation as defined in the General Regulation (Art. 49.3), the ex post synthesis evaluation will also have the function of supporting the communication of the Commission with the Council and the European Parliament of the main results of the ESF as well as to non-technical audiences. 1.2 The scope of the ESF ex post evaluation synthesis and use of sources The synthesis will cover the interventions of the European Social Fund during the programming period in all 28 Member States. As Croatia only joined the EU in 2013, at the end of the programming period observed by the ex post synthesis, the analysis will only be able to cover this short period of implementation. It will draw on the Operational Programme document, Annual Implementation Reports 2013 and 2014 and any available evaluations. The synthesis will cover all HR Operational Programmes in the 28 Member States in both Convergence and Regions. The ex post evaluation synthesis will rely mainly on existing evaluation material and information. However, to fill in the gap for the remaining two ESF Priorities not covered in the three thematic ex post evaluations (Promoting Partnerships and Strengthening Institutional Capacity), the contractor will undertake additional research on the basis of existing material and a limited number of interviews. May 2015 page 9

10 The main sources of information will be: ESF ex post evaluation: Access and sustainable integration into Employment (2014) ESF ex post evaluation: Investing in Human Capital (2014) ESF ex post evaluation: Supporting the integration of disadvantaged groups into the labour market and society (2014) Ex post evaluation of cohesion policy programmes financed by the European Development Fund (ERDF) and Cohesion Fund (CF) work package 12: delivery system DG Regio evaluation on effects of structural funds on GDP and macroindicators Evaluation of the reaction of the ESF to the economic and financial crisis, Final Report by Metis GmbH in cooperation with wiiw SFC2007 Annual Implementation Reports 2014 and previous AIRs when needed The deliverables of the ESF Expert Evaluation Network ( ) 3 The deliverables of the Preparatory Study (2013) Operational Programmes and amendments/implementing documents Relevant ESF Evaluations Relevant EU-wide evaluations 1.3 Evaluation criteria The synthesis of the ex post evaluation should bring all information together gathered during the programming period on each of the five ESF Priorities as defined the ESF Regulation (Art.3). The synthesis will present the lessons learned in terms of policy choices, target groups (with a special focus on specific target groups such as young people and older workers), programming, implementation, monitoring and evaluation of future programmes. Particular attention will be paid to: the extent to which resources were used effectiveness (quantified results, factors contributing to success and failure, identification of good practice) efficiency measured in terms of cost-effectiveness socio-economic impact In addition, reflecting the focus of the thematic ex post evaluations, findings will also be presented in terms of Community Added Value Gender sensitivity and 3 Including country reports, synthesis reports and inventories page 10 May 2015

11 Sustainability The table below provides an overview of what evaluation issues the ex post evaluation synthesis should report on, on Member State and EU level, and the use of existing sources already produced as well as needs for additional data gathering. Table 1. Evaluation criteria, sources and additional data gathering Evaluati on criteria Country synthes is reports (per priority ) Themati c EU synthes is reports (per priority ) Existing Sources Additional data gathering Contextu ali-sing ESF The extent to which sources were used Effective ness Efficienc y Socio economi c impact Other criteria: Information about the content and objectives of the OPs and the types of interventions funded (by ESF priority). Provide the context of the national socioeconomic situation and the national policies and other EU level strategies. Overview ESF spending per ESF priority (SFC2007 and AIR2014) Quantified results (the outputs of the interventions and the results achieved). Success factors Good practice Lessons learned Cost effectiveness Analysis of socioeconomic indicators (ESF big part of funding in social fields), qualitative evaluation (other). Community Added Value Gender sensitivity 27 filled-in overview country templates of three ESF Ex post thematic evaluation studies Country in-depth assessment reports from the relevant ESF Ex post thematic evaluation studies Available country level deliverables of the 'Ex post' evaluation of cohesion policy programmes financed by the ERDF and CF work package 12: delivery system Annual Implementation Reports 2014 Operational Programme documents Evaluation reports (till 30 June 2015 All Expert Evaluation Network country reports Relevant country-level deliverables of the Preparatory study for the ex post evaluation of ESF EU wide data (Annex XXIII and CoE) Commission reports (1) Employment and Social Update evaluations produced by MA after December 2013 till 30 June 2015 Information from Croatia (all evaluation reports, AIR , OP document) Promoting partnerships and Strengthenin g Institutional Capacity (OP documents; AIR ; Evaluations available up until 2015). May 2015 page 11

12 Evaluati on criteria Conclusi ons Lessons learnt Country synthes is reports (per priority ) Themati c EU synthes is reports (per priority ) Sustainability at the level of individuals and systems/structures/prod ucts and in terms of the sustainability of the interventions themselves in the absence of ESF funding For the EU level policy on economic and social cohesion policy choices, target groups, programming, and implementation monitoring and evaluations (with regard to the new regulatory requirements). Existing Sources Developments in Europe; (2) Fourth report on Economic and Social Cohesion; (3)Fifth Report on Economic, Social and Territorial Cohesion; (4) Sixth Report on Economic, Social and Territorial Cohesion DG Regio evaluation on effects of structural funds on GDP and macroindicators Evaluation of the reaction of the ESF to the economic and financial crisis Additional data gathering 1.4 Our approach to the synthesis The Ex Post Evaluation Synthesis is based mainly on previous work in the framework of the three thematic ex post evaluations and the approach to the synthesis will be aligned with this work, e.g. with regard to definitions of expenditure and structure of the presentation etc. It will also draw on the ESF Expert Evaluation Network and Preparatory Study where appropriate, e.g. in the updating of the Inventories of evaluations and the completion of the thematic reports on Partnerships and Institutional Capacity. The Metis report on the Evaluation of the reaction of the ESF to the economic and financial crisis for DG EMPL will also be drawn on. The synthesis will also integrate findings of the DG REGIO evaluation of the delivery systems as results will be useful for the Institutional Capacity report. Other DG REGIO reports that may be relevant are: a report on the effects of the crisis on the (ERDF) programmes (July) and an econometric report in the framework of Article 16 on the effects of structural funds on GDP and macro-indicators. The analysis will select the best information available and formulate synthetic messages on policy choices, target groups (with a special focus on young people), programming, implementation, monitoring and evaluation of future programmes. page 12 May 2015

13 The challenge of this synthesis report, but also the three thematic ex post evaluations is to bring together what has been gathered in other evaluation exercises and (1) providing missing information, (2) drawing the data together, (3) assuring a clear distinction between data related to each Priority, and (4) aggregating output and result indicators. These challenges should already have been dealt with in the individual thematic evaluations but will nevertheless require further thought in the ex post synthesis Providing missing information As mentioned above, the synthesis is based on existing information. Nevertheless there are still some blank spots to be filled: I. Cumulated data from the 2014 AIRs (including financial figures and programme specific output and result data) II. III. IV. Evaluations by Managing Authorities (MA) implemented between December 2013 and 30 June 2015 Information on the Priority Themes Promoting Partnerships and Strengthening Institutional Capacity. The synthesis should include activities for drawing up country fiches finally feeding thematic EU synthesis reports on these themes Data on Croatia from the Operational Programme document, Annual Implementation Reports 2013 and 2014 and available evaluations Drawing the data together As indicated above there are a wide number of data sources on which the thematic EU synthesis reports and country synthesis reports can draw. However, one of the main challenges working with these data is that their definitions and scope are overlapping and not always clearly demarcated (the most striking example is the level at which financial data is stored compared to data on physical progress). The challenge of the ex post evaluation is how to link these different sources: I. The purpose of the thematic ex post synthesis reports is to report on the ESF Priorities as included in Article 3 of ESF Regulation 1081/2006. These do not always correspond to the Priority Axes of the OPs. II. The Categories of Expenditure as defined in the Funds Regulation 1083/2006 and Implementation Regulation 1828/2006. The MS submit to the Commission an indicative breakdown by these categories. Categories relate to the ESF. Each category can be clearly linked to one of the ESF Priorities within the ESF Regulation (Article 3) but not always very obviously to a Priority Axis or the programme specific indicators. Nevertheless the Preparatory Study for this ex post evaluation teaches us that around 70 percent of the sub priorities / actions are linked to one specific category of expenditure indicating that in most cases a clear link can be made. May 2015 page 13

14 III. IV. Moreover, Member State Managing Authorities themselves have allocated budgets to specific categories, making the reconstruction of budgets by the evaluation team redundant. Annex XXIII data on participants are registered for every OP and reported on Priority Axis level. Nevertheless, Priority Axes are not always clearly linked to the Priorities in Article 3. The 117 OPs and their respective AIRs, include Priority Axes (first digit level), sub priorities / actions (second digit level), interventions (third digit level) that are differently operationalised across programmes. All programmes defined programme specific indicators in relation to the Priority Axes and sometimes actions, and also store information on the ESF budget. Priority Axes do not always show a clear relation with Priorities or Categories of Expenditure, and sometimes more Priorities are addressed at the same time or, vice versa, one Priority is addressed by more than one Priority Axis 4. Working with Priority Axis data allows us to combine financial spending figures with physical progress indicators on OP level 5. Besides, there are two crucial issues: The titles (face value) of Priority Axes sometimes suggest a different allocation to ESF Priorities than a careful reading of the actual actions and interventions behind them suggests (some headings do not accurately represent the content or are politically driven). Even if a Priority Axis is predominantly focused on one Priority, if this Priority Axis is financially significant (over a billion Euro), the other Priority that it also affects may also be highly significant in financial terms. V. Evaluations are implemented on different levels, sometimes dealing with a Priority Axis, sometimes a specific objective, sometimes a specific action, sometimes a concrete intervention or bundling of projects. Evaluations can also be transversal addressing several actions / interventions addressed by more than one Priority Axis. All sources will be used in the synthesis while being aware of the problems. Decisions will already have been made on how to deal with most of them in the thematic ex post evaluations but the synthesis will have to report on these decisions and their caveats. 4 There is a cluster of Priority Axes that have a direct link with a Priority (100% coverage); There is a cluster of Priority Axes that is dominantly addressing one of the ESF Priorities, but at the same time includes some actions / interventions related to another Priority ( % coverage); and there is a last cluster of Priority Axes that has an equal contribution to one or more ESF Priorities (20-80% coverage). 5 See outcomes of the Preparatory Study, based on a sample of 43 OPs, spending around 80 percent of ESF budget across Europe. page 14 May 2015

15 1.4.3 Allocating Priority Axes to one single Priority (and sub priorities) The thematic ex post evaluations have allocated Priority Axes to ESF Priorities as a basis for the evaluation synthesis. In order to avoid overlaps in the EU synthesis report between ESF Priorities (including the same Priority Axis) and possible double counting of budget and output/result indicators, a clear methodology feeding the decision how to allocate Priority Axes to ESF Priorities has been developed: I. Step 1: Identify the Priority Axes that show an overlap (address one or more of the thematic ex post evaluations). II. III. Step 2: Further explore those Priority Axes that show overlap on digit 2 level (and if necessary digit 3 level) to assess the dominant Priority in the Priority Axis (based on budget). Guiding criteria are how actions and interventions within this Priority Axis relate to the demarcation of clusters of interventions as identified in each thematic evaluation, the labelling of MA themselves, the labelling of relevant categories of expenditures, and expert opinion assessing the goals, target groups, and instruments of each action and intervention. Step 3: Allocate the Priority Axis to one of the ESF Priorities. In case it appears that Priority Axes substantially contribute to more than one Priority a solution should be found. In order to do justice to the actual nature of Priority Axes, the preferred option is to allocate distinct actions (at digit 2 level) to each Priority in case financial information and output and result data is available on this level. The three thematic evaluations will have addressed this in such a manner that the ex post evaluation synthesis can start with clearly demarcated overview reports on each theme. Ideally, the evaluation synthesis team will be able to start synthesising immediately without reallocating Priority Axes (and actions) to the respective ESF Priorities Aggregating output and result indicators Programme specific indicators are defined on OP level and in most cases these are related to PA or sub priorities. OPs have defined a wide variety of indicators measuring the progress of ESF also including different types of measurements (such as absolute numbers, percentages, totals or additional numbers / increasing percentage, etc). For some indicators targets are set for, others not. The challenge is to link these indicators to ESF Priorities (subject to the thematic ex post evaluations) and to aggregate these on country level and subsequently on European level for each ESF priority (but also across ESF priorities feeding the EU synthesis). For example, in total 2,387 indicators were found for A2E, of which 897 are labelled as output indicators, and 1,490 as result indicators. For the Human Capital theme, 955 result indicators were found of which 820 indicators with values. May 2015 page 15

16 For the ex post synthesis evaluation, two types of analysis are foreseen on these programme specific indicators: 1) Aggregating achievement values within and between OPs (such as summing up total number of participants, entities supported, or products developed) 2) Calculating (average) achievement rates within and between OPs (total achievement in 2013 divided by the target set). Both types of analysis pose some challenges that should be overcome in each of the thematic evaluations. In order to ensure a common approach across thematic evaluations, some guidelines and rules need to be drawn up for aggregating data. 3) Aggregating achievements In the context of the thematic evaluations as well as for the synthesis report it would be valuable to sum up some key performance figures of the ESF across programmes, ESF priorities, and countries. Monitoring information that allows aggregations across OPs and countries are: financial spending figures, Annex XXIII participant data, and (some) programme specific output and result indicators. I. Financial data: Allocated and certified expenditure budgets per PA and CoE can be summed up across PA, programmes, countries and EU level, since they have the same unit of measurement, namely Euro 6. II. Annex XXIII participants data: For aggregation on outputs of participants across PAs, OPs and countries the Annex XXIII participant data provides a more solid basis for aggregation than the programme specific output indicators related to individuals, since Annex XXIII data is recorded in a systematic manner across countries (total number of participants and for sub groups like gender, employment situation, age, migrants/minorities, disabled, other, and educational level 7 ). III. Programme specific output and result indicators: Aggregating achievement values of programme specific indicators can only be done if indicators have a common basis (definition, scope and unit of measurement). A first analysis of the programme specific indicators shows that there is a wide variety of programme specific indicators (having different units of measurement and different scope, 6 Regarding financial data we suggest to distinguish: (1) Allocations (OP, PA, CoE) of ESF-funding in terms of what has been approved by the COM as EU-contribution up-front; (2) Commitments made by member states when granting funds to beneficiaries (to be distinguished ESF money and national public); (3) Expenditure declared (again: ESF and national public to be distinguished) should be expenditure that is the base for re-imbursement from the COM (while assessing the certified expenditure one should be aware of the frontloading of ESF introduced with anti-crisis-measures). Committed budget is not recorded in the data collection template for task 1 (only allocated budget and certified expenditure). 7 Programmes are recording the number of entries and departures of individuals participating in ESF. In the thematic evaluation information is collected on the number of entries (to get a more realistic picture on programmes performance since the programme is not finalised yet, and to assure we are not missing a high number of participants in the data set). One individual could however participate in more than one ESF project and is therefore counted more than once. As a result Annex XXIII is reporting about the number of participations and not the number of persons benefitting from ESF. page 16 May 2015

17 addressing different objectives, types of interventions and target groups). Nevertheless, there are still a number of (common) indicators that frequently appear across and within each ESF priority. Some of these also allow cross ESF priority aggregation (which is interesting for the synthesis). In each of the thematic evaluations we therefore need to carefully assess each programme specific indicator whether these belong to common groups of indicators, allowing aggregations of achievement values in the end. In order to group indicators we propose to follow a deductive and inductive approach. a. Pre-define groups of indicators that have similar definitions, scope and unit of measurement and allocate individual indicators to these groups. i. Starting point is to define whether an indicator relates to the performance of an individual participant, entity (such as an organisation, enterprise, network, partnership supported / established), product (such as a programme, project, tools, curriculum, approach developed), or system (such as a reform implemented). ii. Secondly, where possible, indicators could be classified in terms of type of intervention (does it concern an intervention related to training, job placement, counselling?). The definition of indicators, however, does not often give an indication of the types of intervention addressed. In many cases, different types of activities referring to the same objective are integrated in the same output indicator. In these cases the evaluator is not able to construct groups. iii. Subsequently, one should assess whether an indicator can be considered as output or result indicator. Output indicators relate to operations supported and measures an output which is directly produced/supplied through the implementation of an ESF operation, measured in physical or monetary units (such as supported people, supported entities, provided goods or services delivered). Result indicators capture the expected effects on participants or entities brought about by an operation. Result indicators go beyond output indicators in so far as they capture a change in the situation, in most cases related to supported entities or participants, e.g. in their employment situation. Programmes label the indicators themselves as output or result indicators. The reality is that sometimes the same indicator is labelled as output indicator in one OP while in another OP this indicator is labelled as result indicator (depending on the action or the objectives of the May 2015 page 17

18 action / programme). An example is that for Human Capital the number of participants gaining a qualification or certificate is often considered as result indicator, while for Access to Employment this indicator can be considered as an output indicator, finally contributing to employment. While grouping indicators we attempt to aggregate similar indicators in terms of definition and level of measurement despite whether these are labelled as output or results. Moreover, sometimes result indicators are actually macro-impact indicators (such as national or regional employment rates of specific groups) and not related to single participants. These indicators should be considered as context indicators and will not be further assessed in the analysis. iv. Finally, one should assess the unit of measurement of indicators. This mainly relates to (1) number or (2) percentages. Where possible percentages will be recalculated to absolute numbers (based on information available on the total population if available). b. While allocating programme specific indicators to groups of indicators, where necessary, new groups will be formed or combined. In the first case, this means that in case we see more indicators that have a similar definition and unit of measurement within the pre-defined group, we decide to define a sub group. In other cases, we bring indicators together - being different - but having similar unit of measurement and scope. For example, in case sustainable employment is measured using different time periods (like 3, 6, or 12 months after the intervention), these can be grouped together as one indicator called sustainable employment. Also different types of entities supported (like SME, PES, NGOs) can be regrouped in total number of organisations supported. A suggestion is to split up this group in final beneficiaries and intermediary organisations like PES and NGO s. c. Per group of indicators, where possible, values will be summed up. In case the indicator is measuring numbers, we are able to sum up numbers in case they concern the same object (like individuals, entities, products). When indicators are measuring the achievement in percentages we face some specific challenges. Percentages are mostly used for measuring the results of an intervention (like the percentage of participants that successfully completed a training course, gained a qualification or found a sustainable job). In these cases it would be recommendable to calculate the absolute number. Therefore, information is needed on the page 18 May 2015

19 total number of participants 8 that will be multiplied with the percentage (like 30% of 1000 participants = 300). A precondition is that there is clear information on the total population the indicator is reporting on (preferably a clear relation between output total number of participants and result). Only in this way are we able to calculate absolute numbers. If this is not possible, another option is to calculate average percentages between indicators (like the percentage of participants that have a sustainable job). However, this is not the preferred option since it does not provide robust outcomes. 8 The term participants actually refers to participations meaning number of entries into measures rather than number of people who participated in measures. The fact that one person may well have participated in more than one measure is taken into account. May 2015 page 19

20 Box 1. An example: Programme specific output and result indicators in A2E A close look at the output indicators reveals that most output indicators relate to individuals (total 617), followed by products (176) and entities (107). Concerning the number of output indicators across countries one see that Belgium, Germany, Italy, Spain, and the UK already represent 69% of the total number of indicators. As these countries also gave the largest number of OPs and the indicators are sometimes very similar, particularly in Italy and Spain, this means a reduced overall number of indicators. The indicators in the German, Belgium and the UK programmes are more distinct. A first inventory of the volume and types of indicators per PA, OP and Member State shows a quite diverse picture. Many of the indicators are unique in their own context and difficult to aggregate. They can nevertheless be broadly grouped around some common categories. Concerning individuals, these would broadly correspond to the Annex XXIII data whereby Annex XXIII data is probably the more robust being collected systematically by PA and OP and covering all participants in ESF interventions. Concerning entities, the categories discerned were: (1) number of enterprises created; (2) number of enterprises supported; (3) number of networks supported; number of centres developed; (4) number of PES supported; (5) number of organisations supported. As indicated most indicators relate to the number of supported organisations, allowing some form of aggregations on this very basic level. The problem, however, is that the indicators do not have a complete coverage across PAs, OPs, and countries, making it difficult to do comparative analysis. Concerning products, the categories distinguished were: (1) number of projects; (2) number of campaigns; (3) number of programmes; (4) number of trajectories; (5) number of tools / databases; number of studies / evaluations; (5) number of agreements / conventions. These groups allow doing some aggregations at very basic level, however it should be noticed that most indicators (except of the indicator on number of projects) do not frequently appear across OPs. In addition to the content or definition issue, there is also an issue concerning indicator values. Most output indicators are measured in absolute numbers; however 6.7% of all output indicators (60 in total for A2E) use percentages as unit of measurement. Sometimes the indicator measures the relative share of a particular target group of the total population, and sometimes the increase in percentage compared to a certain baseline or target. In the second interim phase of the A2E ex post, the programme specific output indicators will be further analysed, grouped, and assessed on their unit of measurement. A large number of programme specific result indicators can also be found (in total 1,490 indicators) for the selected PA relevant for A2E. Of these 1,490 indicators, the majority relate to individuals (1,337), a smaller number relate to entities (total 86) and products (64) and very few to systems (3). Assessing the number of indicators across the different MS, one sees that also here Italy, Spain, and the UK represent most of the indicators page 20 May 2015

21 (68% of total indicators), mainly linked to the large number of OPs. Italy represents in total 42% of the indicators. Again, in Italy, Spain and the UK, the indicators show little variation between OPs. The large amount of indicators is in these cases also explained by the fact that the same indicators are used to measure the results for all target groups (men, women, young people, older people, migrants etc.). Another conclusion to be drawn is that most OPs report on the number of job entries and do not provide evidence on job sustainability. These would be the two most interesting indicators to observe in this field in the synthesis and further research will be done in the next interim phase of the A2E ex post. (2) Calculating achievement rates across PA, programmes, and countries For those indicators that have a target, achievement ratios can be calculated. This can be done in two ways: a. If targets are set for absolute numbers, these can be compared with the achievement values (target / achievement) b. If targets are set for percentages, the achievement rate can be calculated in two ways. The first option is to calculate the relative share of the achievement compared to target (when the target is 20% and the achievement is 10% we calculate the achievement rate 10/20 = 50%). The second option is to calculate the difference between target and achievement in absolute terms (when the target is 20% and the achievement is 10%, we can calculate the distance to target, by extracting the target from achievement: 10-20=-10%). Since the first option allows a better analysis between indicators, this is the preferred type of analysis. Subsequently, we can calculate the average achievement across indicators for each PA, OP, and groups of programme specific indicators on EU level. This provides input for making statements such as: 40% of indicators across programmes achieved their target, or 60% of output indicators related to individuals achieved their target. However, it is important to mention that in some cases it is questionable whether targets set are realistic. Indicators that show a high over- or underperformance therefore need special treatment It is being discussed with the teams responsible for the three thematic evaluations, whether indicators with very high or low achievement values have been excluded from the analyses. It is advisable to leave out those indicators showing an over performance of 300% or those that have an underperformance of 10%, while calculating average achievement percentages across indicators. The parameters for excluding indicators is being discussed and decided between the responsible evaluation teams. May 2015 page 21

22 A critical note related to the target values set by programmes is that many programmes have adjusted their targets over the programming period in order to better align them to the actual progress made in the programme, while other programmes kept the targets set from the start of the programme. As a result, one could question whether measuring the achievement rate is meaningful. With proper adjustments, the moving target can be reached in the end and a potential failure to meet targets be disguised. Problem of exclusion of single indicators that do not have a value or cannot be grouped Some indicators cannot be included in a further analysis on country and EU level due to the following reasons: Indicators that do not report an achievement value for the period Indicators that are unique for certain programmes and not appear frequently or not at all in other OPs and cannot be grouped (such as the indicators measuring the yearly reduction of 5% of the cost of an intervention, as included in the ESF programme for the German Speaking Community in Belgium). Each thematic evaluation should report for which indicators one of the above described situations apply. Concluding remarks Based on the experience with the A2E ex post and the anticipated experience of the other ex post evaluations, it is expected that a grouping of common or similar programme specific indicators will be possible with some caveats and compromises. Nevertheless, given the variety described above, full aggregation and coverage will not be possible since unique indicators not belong to a group should be excluded from the analysis. Reviewing the indicators, one can conclude that the financial data and Annex XXIII data comprise the best information base for providing an EU wide overview of the output of ESF. The programme specific indicators only represent a part of ESF spending. The output and result indicators for individuals show the most commonalities, followed by the indicators related to entities. Those formulated for products are very diverse and difficult to aggregate. As a result, we also need to indicate that the programme specific indicators are reporting on the minimum contribution of ESF. We do not propose to extrapolate achievement values of programme specific result indicators to the whole of ESF. For example sustainable employment indicators for A2E are only measured in 16 countries (and most cases are related to a specific PA or sub priority). As a result we propose to only report on these indicators for the 16 countries (and will not extrapolate these figures to the other 11 countries). The evaluators should always be 9 There are indicators that only have a value in 2014 (or even later) because the value should be produced by a separate evaluation / study at the end of the programme. page 22 May 2015

23 clear about the coverage of the indicators they are reporting on, avoiding false interpretation by the reader of the final evaluation reports. May 2015 page 23

24 2. METHODOLOGY AND TOOLS 2.1 Main tasks and deliverables The ex post synthesis evaluation comprises three main tasks and a number of sub-tasks. The deliverables and their interlinkage are shown in the figure below. Figure 1. Tasks and deliverables of the ex post evaluation synthesis Task 2. National level analysis Task 1: Follow-up of ex-post evalu-ations 27+1 updated Inventories 27+1 Country Reports Task 3. EU level analysis 2 Thematic reports (partnership/cap) EU Synthesis Report As discussed above, the ex post will pull together the material that has been produced so far and complement the existing information with additional research. The main methods that will be used will be desk research, comparative research, completing the database with statistical information coming from Eurostat, the SFC2007 and national sources, some complementary interviews and a focus group with high level experts and Commission officials to validate the findings. The main tools will be existing material coming from the three thematic ESF ex post evaluations and the Ex Post Evaluation of Cohesion Policy Programme Work Package 12, the ESF Expert Evaluation Network, the Preparatory Study, EU-wide evaluations (e.g. on gender or on specific target groups, in particular young people and older people), newer evaluations carried out by the Managing Authorities, OPs and implementing documents, especially in Croatia, AIRs 2014, SFC2007, Eurostat, national databases etc. 0Table 2 below summarises the main tasks, data sources and outputs required for the fulfilment of the contract. The following chapter describes how we intend to fulfil the individual tasks. page 24 May 2015

25 Table 2. Overview of tasks, data sources and outputs Tasks Data sources Output Task 1: Follow-up of the ESF thematic ex post evaluation studies 1: Follow-up of the ESF thematic ex post evaluation studies Task 2: National level analysis 2 A: National inventories of evaluation 2 B1: Country synthesis reports Three thematic ex post evaluations Ex post evaluation ERDF: Work package 12 Minutes steering groups / reports Previous inventory by the EEN as a basis ESF evaluation reports produced by MA after December 2013 till 30 June 2015 All evaluation reports produced in Croatia 27 filled-in overview country templates of three ESF Ex post thematic evaluation studies Country in-depth assessment reports from the relevant ESF Ex post thematic evaluation studies Available country level deliverables of the 'Ex post' evaluation of cohesion policy programmes financed by the ERDF and CF work package 12: delivery systems Annual Implementation Reports 2013 and SFC2007, in particular Annex XXIII and programme-specific indicators Operational Programme documents Attending steering group meetings of the four Ex Post Evaluations (including Work Package 12 of the DG REGIO Ex Post Evaluation) providing feedback and advice Minutes 28 National Inventories Overview Inventory of all Member States 28 Country synthesis reports structured by ESF priority including: information on OP objectives and types of interventions contextualisation of the above vis-à-vis national situation and strategies and EU-level strategies overview of expenditure, outputs and results achieved, findings on all evaluation criteria with a particular focus on effectiveness, based on evaluation evidence conclusions for the policy on economic and social cohesion at Member State level outline lessons learned in relation to policy choices, target groups, programming, implementation, monitoring and evaluation at least one good practice per Member State according to robust evidence-based criteria, from the list of good practices already selected in the ex-post evaluations, or new good practices 10 For the themes covered by the three thematic ex post evaluations, only the AIRs 2014 will need to be consulted to update information. For the themes PP and SIC, the earlier AIRs, 2013 in particular, will need to be used. May 2015 page 25

26 Tasks Data sources Output Task 3: EU level analysis 3A: Thematic EU synthesis reports on promoting partnerships and strengthening institutional capacity 3B: EU Synthesis report Evaluation reports (till 30 June 2015) All Expert Evaluation Network country reports Relevant country-level deliverables of the Preparatory study for the ex post evaluation of ESF Evaluations available up until 2015 Country synthesis reports EU wide data (Annex XXIII and CoE) Context data (Eurostat) Employment and Social Developments in Europe Fourth report on Economic and Social Cohesion Fifth Report on Economic, Social and Territorial Cohesion Sixth Report on Economic, Social and Territorial Cohesion Country synthesis reports Overview assessment for 27 Member States from the relevant ESF Ex post thematic evaluation studies Thematic EU synthesis reports on promoting partnerships and strengthening institutional capacity In-depth horizontal assessment of interventions in Member States selected for the relevant ESF researched for the other priorities The country synthesis reports will also include additional data on Promoting Partnerships and Strengthening Institutional Capacity. The country synthesis report for Croatia will also be included based on the OP, AIR 2014 and any available implementation information, contextual data (e.g. from Eurostat and national statistics office) and evaluations. country synthesis reports will be a minimum of 15 and an absolute maximum of 30 pages long, including tables and a one-page executive summary. One thematic synthesis report on Promoting Partnerships One thematic synthesis report on Strengthening Institutional Capacity Each of maximum 50 pages One synthesis report structured along ESF Priorities which: provides information on objectives established in the Operational Programmes and types of interventions funded, together with a contextualisation of this information vis-à-vis national situations and strategies and EU-level strategies provides an overview in terms of expenditure, and as regards the outputs achieved, based on the most important data collected through Annex XXIII and programme-specific indicators presents findings on all evaluation criteria, with a particular focus page 26 May 2015

27 Tasks Data sources Output ex post thematic evaluation studies Available deliverables of the 'Ex post' evaluation of cohesion policy programmes financed by the ERDF and CF work package 12: delivery system Expert Evaluation Network synthesis reports Preparatory study for the ex post evaluation of ESF Other relevant evaluations of the ESF carried out by the Commission on effectiveness, based on evaluation evidence presents conclusions for the policy on economic and social cohesion at EU level outlines lessons learned in relation to policy choices, target groups, programming, implementation, monitoring and evaluation The EU synthesis report will be 50 pages long, including tables and a three page executive summary. May 2015 page 27

28 2.2 Evaluation task 1: Follow up of ESF thematic ex-post evaluations Given the request to base the Ex Post Evaluation of Access and Sustainable Integration into Employment (being carried out by Metis) on the other two thematic ex post evaluations (being carried out by ICF), the Consortium has a good overview of what is being done in all three thematic ESF evaluations and is in good contact with the other two evaluation teams. The existing good cooperation base will be used to accomplish Task 1 of the ex post synthesis as efficiently and effectively as possible. A maximum effort is being made to align the three ex post evaluations and this effort will be maintained throughout the existing contracts in order to develop a common methodological approach and to deliver comparable final reports for the ex post synthesis evaluation. In the thematic ex post evaluations, a common approach was used in the selection of OPs in Member States with a large number of OPs for the indepth assessments, selection of interventions for in-depth study, presentation of findings (report templates, tables, maps, definitions). For the overview (Task 1), a common approach was also developed in order to have comparable data from the three ex posts with a clear demarcation between ESF Priorities. This will be extremely useful when it comes to writing the synthesis report as important decisions on how to allocate OP Priority Axes (and if necessary sub-priorities or actions) to ESF Priorities have already been taken by the evaluation teams and agreed by the Commission. A huge effort was made to avoid overlap in Task 1 but in a limited number of cases it proved impossible to allocate the whole PA to one Priority (e.g. in FI, LT, LU, SE). In these few cases, a special mention will made in the Synthesis that the PAs have been allocated to one PA to avoid double counting but that a significant amount of funding according to CoE has gone to another Priority (in most cases from A2E to SI). It is also expected that the presentation of the conclusions and lessons learnt in relation to policy choices, target groups (in particular young people), programming, implementation, monitoring and evaluation will be aligned in all three thematic ex posts. The contractors will familiarise themselves with the ex post evaluation of Cohesion Policy Programmes financed by the European Development Fund (ERDF) and Cohesion Fund (CF) work package 12: delivery system" (DG REGIO) published in June This ex-post evaluation is concerned with the effectiveness and efficiency of the delivery system of cohesion policy for the period and will identify the main success factors that support implementation of all funds. This evaluation will provide useful information for the thematic report on institutional capacity. The approach to Task 1 of the ex post synthesis evaluation is based on the existing good cooperation between the evaluation teams. Building on the existing cooperation the project manager will: page 28 May 2015

29 (Continue to) familiarise herself with all the reports published under the four ex post evaluations (including the DG REGIO ex post evaluation, work package 12) (Remain in) contact with the evaluation teams concerning methodology and deliverables with the aim of making these as complementary and comparable as possible Ensure that overlaps and double counting have been dealt with satisfactorily and agreed by the Commission, including for the Priorities Promoting Partnerships and Strengthening Administrative Capacity. Ensure that definitions, terms, funding categories used etc are harmonised Participate in steering committee meetings as requested, e.g. on 21 st April 2015 Cooperate in the development of comparable methodologies/deliverables Provide advice when appropriate and requested Encourage other evaluation teams to build a database in which information is merged on Priority Axis of every programme, including information on the ESF Priority it addresses, and basic information on budget spent and output and result indicators in order to be able to combine the databases for the ex post synthesis evaluation. At the very least, an excel sheet should be produced collating all the financial, output and result data for the EU27 not only single country fiches. Preparation, setting up a database In order to produce EU wide information we are dependent on the consistency in the way data is collected in the separate thematic ex post evaluations (and the thematic EU synthesis reports on Promoting Partnerships, and Strengthening Institutional Capacity). In order to be able to start from a structured information base, it is recommended that every thematic evaluation produce a database (in Excel) with basic information per relevant Priority Axis that is labelled to a single ESF Priority, including information on financial allocations and expenditures, participant data, and programme specific indicators. In order to streamline the approach between the thematic evaluations two notes were produced in the inception phase and shared with the three evaluation teams and the European Commission. One note focused on issues surrounding the aggregation of programme specific indicators (which has been discussed in the previous section). Another note focused on the construction and handing over of the database with information on financial data and programme specific indicators. Several discussions have taken place with the evaluation teams responsible for (1) the human capital evaluation and (2) Access to Employment evaluation during the inception phase while the discussion with the team responsible for the thematic evaluation on Social Inclusion still needs to be had. The box below includes May 2015 page 29

30 some important points that should be taken into account while producing and handling over the database for each thematic evaluation. Box 2. Discussion points /guidelines on the construction and handing over of the the database Discussion note on the construction and handling over of the databases Since the synthesis of the ex post further elaborates (and update) some of the products delivered in each of the thematic evaluations, some guidelines should be formulated how each thematic evaluation handles over the data(bases) produced and analysed. This is important since the evaluation team responsible for the synthesis report needs to update the financial, Annex XIII data and programme specific indicators with 2014 values (AIR2014). Moreover, based on the AIR2014, the analysis should be updated and reconstructed 11. This has some consequences for: 1. Set up of the database with information on financial and physical data. The databases of every thematic evaluation should have a similar structure (including similar variables) so easy to merge into one database. 2. Transparency on the analysis done aggregating programme specific indicators by means of log books / syntaxes (including the link of individual programme specific indicators to groups of indicators), Annex XXIII data and financial data. This information is needed to reproduce the analysis done, including the most recent AIR2014 data. Evaluation teams should share their final databases and inform the synthesis team how they dealt with the construction of (1) the database (and relevant variables recorded) and (2) analysis done aggregating data. Each aspect is further discussed below. (1) Database structure: In total every thematic evaluation produce four databases that can be linked with a common identifier (CCI and PA). Most of the information captured in the database is extracted from SFC2007 (information on PA level). Especially in case information is reported on sub priority level (not available in SFC2007) it should be completely transparent where financial and physical information is stored in the AIR and whether additional calculations were carried out by the thematic evaluators to construct a value on financial and physical performance (in case information on sub priority level is not available). The evaluation teams should provide log books when they allocate budget and number of participants to a sub 11 We assume that the programme specific indicators for each OP have not changed dramatically since the AIR2013. page 30 May 2015

31 priority level based on some assumptions. As a general rule it was decided that the EU27 templates only zoom in on the sub priority level when financial information, Annex XXIII data, and programme specific indicators are available in the AIR on this level. If this was not the case, the whole PA should be allocated to one ESF priority. At least the following information should be available in each database that should be carefully checked within each evaluation team (during / after the meeting) 12. A. Database on programme specific indicators Each record is an indicator Country name CCI OP name OP objective (convergence / competitiveness/multiple objective) PA name and number ESF priority (A2E, HC, ADAPT, SI) Indicator name original Indicator name translated in English Unit of measurement (number / percentage) Initial baseline Final target 2007, 2008, Achievement Total achievement (already available in 2013 achievement number or reproduced by evaluation team) Achievement rate (target / final achievement 2013) Allocation to grouping of indicators (in case of aggregating programme specific indicators) B. Database on Annex XXIII data Each record is a PA Country name CCI OP name OP objective (convergence / competitiveness/multiple objective) PA name and number ESF priority (A2E, HC, ADAPT, SI) Total number of participants Male Female Employed 12 The assumption is that the databases show a lot of commonalities since these are based on the same EU27 template. May 2015 page 31

32 of which self-employed Unemployed of which LTU Inactive of which in education/training Young people (15-24 years) Older people (55-64 years) Migrants Minorities Disabled Others Primary or lower secondary education (ISDEC 1 and 2) Upper secondary education (ISCED 3) Post-secondary non tertiary education (ISCED 4) Tertiary education (ISCED 5 and 6) Database with financial data per PA Each record is a PA Country name CCI OP name OP objective (convergence / competitiveness/multiple objective) PA name and number ESF priority (A2E, HC, ADAPT, SI) Total funding of the OP (Union and national) Total amount of certified eligible expenditure paid by beneficiaries Corresponding public contribution Implementation rate (%) C. Database with financial data on CoE per OP Each record is an OP Country name CCI OP name OP objective (convergence / competitiveness/multiple objective) Total funding of the OP (Union and national) per CoE (allocated budget) (2) Transparency analysis performed aggregating financial and physical data In order to repeat the analysis done in each of the thematic evaluations, each thematic evaluation should make transparent how they aggregated page 32 May 2015

33 financial, Annex XXIII and programme specific indicators, by producing: A log book on how this analysis is done and assumptions behind o Which indicators are included / excluded in the analysis o Special treatment of percentage values o How to deal with indicators that have no target value o How to deal with indicators that have no value for 2013, or no value at all o How to deal with calculating achievement rates (target / achievement value) o Additional calculations done on total achievement (already included in 2013 achievement value or not) o On what level aggregations are done (OP, country, EU level) Syntaxes for the analysis performed in Social Statistical Software Formulas in Excel for calculating totals How individual indicators are allocated to groups of indicators Each evaluation team should share this information with the Commission and the synthesis team. Answers on the above mentioned issues allows the evaluator - responsible for the EU synthesis report - to merge databases, and to further work with one database for further analysis feeding the EU synthesis report as well as the country synthesis reports (see Table 3 below). This database will be constructed during Task 1 but will be completed with updated information from the AIR 2014 during the country synthesis work (see Task 2). May 2015 page 33

34 Table 3. Database structure for the thematic ex post evaluations and EU synthesis report MS CCI Programme name Priority Axis Link PA to ESF Priority # # # # # # # # Background information Budget AIR version Years covered Allocated budget Certified expenditure # # # # # # # # Total number of participants # # Male # # Female # # Employed # # of which self employed # # Unemployed # # of which LTU # # Inactive # # of which in education/training # # Young people (15 24 years) # # Older people (55 64 years) # # Migrants # # Minorities # # Disabled # # Annex XXIII data Others # # page 34 May 2015

35 Primary or lower secondary education (ISCED 1 and 2) # # Upper secondary education (ISCED 3) # # Post secondary non tertiary education (ISCED 4) # # Tertiary education (ISCED 5 and 6) # # Indicator title and definition Indicator title and definition English Type I (output / result) Type II (individual, entity, product, system) Type III (type of group) Unit of measurement Priority Initial Baseline Final Target Achievement 2007 Achievement 2008 Achievement 2009 Achievement 2010 Achievement 2011 Achievement 2012 Achievement 2013 Achievement 2014 Achievement Achievement rate % # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # This database will be used for updating some key figures from the EU27 analysis based on the AIR2014, mainly related to certified expenditures, participant data (Annex XXIII), and 2014 values of programme specific indicators. Based on this updated data some calculations are reproduced within task 2 (country level) and task 3 (on EU level). The difference is that May 2015 page 35

36 task 2 intends to analyse and aggregate data on national level across OP and ESF priorities (and find common indicators on priority themes and national level), while task 3 tries to aggregate data on EU level (and find common indicators on priority themes and EU level). Table 4. Overview of steps for the accomplishment of Task 1 Task Who Task 1: General overview Step 1: Familiarisation with all the reports published under Project manager the four ex post evaluations (including the DG REGIO ex post evaluation, work package 12) Step 2: Contact with and advice to the evaluation teams Project manager + concerning methodology and deliverables Bert-Jan Buiskool Step 3: Participate in steering committee meetings as Project manager requested 2.3 Evaluation task 2: National level analysis National inventories In the framework of the ESF EEN, 27 national inventories of evaluations were created as Excel files. The aim of the inventories was to record in very succinct form the contents of the ESF evaluations that had been produced in the period, in particular with a view to filtering out the findings of the evaluations according to ESF Priority. The inventories consist of a number of sheets which present the evaluation plans, the evaluations, the evaluation documents, the evaluation approach, evaluation method, the main findings, and the main findings according to intervention types and target groups (see below). All the inputs can be filtered according to the subject of interest, e.g. counterfactual evaluations or evaluations concerning women or young people, older workers etc. Objectives The main objective of this task (2A) is to update the existing inventories with new evaluations bearing in mind that the last update was in October A new inventory will be created for Croatia. The overall aim is to have a complete overview of the evaluation activity and the main findings of the evaluations in relation to interventions and target groups in the programming period. The tool is useful as a basis for the Evaluation Synthesis in that the evaluation findings will provide additional and specific evaluation evidence under the priorities which may not have been included in the thematic ex post evaluations. The findings are categorised according to Priority (Policy Field) and therefore easily allocated to the relevant chapter of the country reports. The tool is also useful for policy makers and researchers and any interested stakeholders who would like to know more about what the ESF achieved and how specific instruments worked. page 36 May 2015

37 Data and information sources For the update of the 27 existing inventories and the new inventory for Croatia, the main sources of information will be the internet (the web pages of the main stakeholders in the ESF: Managing Authorities, Public Employment Service, national and regional authorities, EU and international (e.g. ILO, OECD) sources). In addition, it may be necessary to contact the ESF Managing Authorities to ask specifically about recently published evaluations or even those in planning (as the Inventory also allows for nonfinalised evaluations). The geographic units of DG EMPL also be asked to provide any evaluations they have been provided with or are aware of. The cut-off date for the evaluations to be included will be June Tools The basic structure of the inventories is shown in the Chapter 3, Table 13. The structure of the Inventory is a series of sheets covering: the evaluation plans, the evaluations, the evaluation documents, the evaluation approach, evaluation method, the main findings, and the main findings according to intervention types and target groups. Each sheet has a number of columns with information pertaining to the evaluation. Activities and team members The updating of the inventory will be coordinated by the project manager with the support of Benno Savioli (research team). He originally created the inventory in the framework of the ESF Expert Evaluation Network and will be responsible for ensuring its smooth functioning and for aggregations of findings for the country and final synthesis reports. The country experts will be responsible for the actual filling in of the inventory, i.e. for the identification and screening of the evaluations and for the categorisation of the technical information and findings of the evaluations. A guidance note will be provided to the experts for doing this. The quality of the findings and the correct and complete categorisation will be checked by the overview team. Table 5. Overview of steps for the accomplishment of Task 2A Task Task 2: National level analysis (Lead: Benno Savioli) A: National inventories of evaluations Step 1: Preparation of Guidance for country experts Step 2: Identification and reading of the evaluations by country experts Step 3: Categorisation of the technical information and findings of the evaluations Step 4: Quality check of the findings and correct categorisation Who Benno Savioli Country experts Country experts Benno Savioli (with support by Metis) May 2015 page 37

38 2.3.2 Country synthesis reports Objectives The country synthesis reports requested under Task 2B have the aim of summarising the most recent and most robust evaluation evidence available and providing a succinct overview of the outputs, results and impacts of the ESF in each of the 28 Member States in relation to target groups and types of intervention and the evaluation findings identified in the available material. The most reliable information will be used not a summary of all evaluations available. Older sources such as the EEN will only be used when information gaps need to be filled. The country reports include sections on the evaluation criteria in which the achievements per ESF Priority are analysed according to: Extent to which resources were used Effectiveness Efficiency/cost-effectiveness Socio-economic impact CAV Gender sensitivity Sustainability Particular attention will be paid to the issue of effectiveness and information will be synthesised from the in-depth studies (in the framework of the thematic ex post evaluations) which have the aim of providing insights into what worked and how. Also important is the issue of gender sensitivity. This theme is explicitly covered in the three thematic ex post evaluations and will also feature in the country synthesis reports as one of the evaluation criteria covered in all the packages of the ex post synthesis. Our understanding of gender sensitivity is that it covers both the aspects of gender mainstreaming (equal access and participation in interventions) and specific measures for women and also gender budgeting, i.e. ensuring that equal amounts of funding benefit men and women. Furthermore, the relative effectiveness of ESF supported interventions for men and women will be looked at. This means that gender balance will be looked at in terms of participation, budgets and effectiveness across all of the ESF supported interventions in the country synthesis reports. An issue that will be of importance in the country and thematic studies on Institutional Capacity in particular will be that of sustainability. Sustainability will be investigated at the level of individuals and systems/structures/products and in terms of the sustainability of the interventions themselves in the absence of ESF funding. The conclusions of the country synthesis reports will cover all the evaluation criteria and the lessons learned will focus on the policy choices, target groups, programming, implementation, monitoring and evaluation to draw page 38 May 2015

39 lessons for the future. The focus will be particularly on specific target groups such as young people, older workers, women and types of interventions that have proven effective for these specific target groups. An example of good practice will be included in each of the country synthesis reports. It will make sense to use those examples that have already been given in the thematic ex post evaluation in-depth country studies as these will have been well-researched and can be relied upon to be based on sound evidence. Care will be taken to take a diverse range of Good Practice across ESF Priorities. In order to select good practices we will isolate the most robust examples of practices identified in the previous thematic ex-post evaluations which will also mean researching additional good practice in the fields of Promoting Partnership and Strengthening Institutional Capacity. Data and information sources The main challenge in this task will be to synthesise the information coming from at least six country reports of one kind or another per country that will have been produced in the framework of the various evaluation contracts tendered out by the Commission. There is at least one per country emerging from one of the three thematic ex post evaluations plus one per country at EU27 level. Then there are at least one per country from the DG REGIO study and at least three per country that are of relevance from the ESF EEN (Access to Employment, Social Inclusion and the Final Country Report as well as the country reports on women and young people). In addition to the country reports, there are other sources of information that it will be essential to draw on: the AIRs for 2014 and the newer evaluations as captured in the inventories (Task 2A) as well as OPs and implementing documents in cases where information is missing on activities (e.g. for partnership and institutional capacity and for Croatia) (see also Table 2). The geographic desks will also be consulted in the case of missing information or to check if information is to the best of their knowledge correct. The information gathered will be summarised into a maximum of 30 pages which will involve an extremely economical use of tables and a careful visualisation in order to ensure that all the information is succinctly and comprehensively reported. Table of contents The table of contents will broadly reflect the structure of the both the indepth country reports and the EU 27 overview templates which will be used in the three thematic ex post evaluations The information will already be structured as it is needed for the country reports in most cases (with the exception of Croatia and for the two themes that have so far not been treated). This will be particularly useful in those countries with many OPs as country level aggregation will have taken place for the ESF Priorities already covered in those countries where it is reasonably easy (Italy and Spain) and solutions proposed in those countries where it is not (Germany). Given the May 2015 page 39

40 shortness of the country synthesis reports, the information will have to be condensed in those countries with many OPs although these will no doubt be the longer country reports in the end. It is important that the detail is not lost in the aggregations and this may lead to longer explanations in e.g. Germany or countries in which the demarcation is particularly difficult on account of the coverage of the Priority Axes such as Finland. Table 6. Table of contents for country synthesis reports Chapter/secti on Executive summary Sub-chapter Source of information Introduction 1 1. Context, general policy background and ESF investment in all ESF Priorities 2 Financial volumes, outputs, results and findings by ESF Priority 1.1 Economic and social challenges in the Member State assessing Europe 2020 indicators (such as GDP development, employment and unemployment figures, education, lifelong learning, poverty index etc.) and strategic response 1.2 Investment priorities in the MS (in relation to challenges and EU level strategies) Summary of main types of interventions funded across ESF Priorities and main target groups 2.1 Increasing adaptability and Human Capital - Financial absorption (SFC2007 and AIR 2014) - Outputs (SFC2007 and AIR 2014) - Results (Programme specific indicators EU 27 country templates from the 3 thematic ex posts General overviews from the 3 thematic ex posts ESF HC ex post (EU27 template and in-depth reports and horizontal analyses for specific countries) AIR 2014 ESF EEN final country report Relevant evaluations Pages page 40 May 2015

41 Chapter/secti on Sub-chapter Source of information and other results gathered in the framework of the thematic ex post evaluation) - Extent to which resources were used - Effectiveness - Efficiency/costeffectiveness - Socio-economic impact - CAV - Gender sensitivity - Sustainability 2.2 Enhancing access and sustainable integration into employment - See Reinforcing social inclusion - See Promoting partnerships - See Strengthening institutional capacity - See 2.1 ESF A2E ex post (EU27 template and in-depth reports and horizontal analyses for specific countries) AIR 2014, SFC2007 A2E database ESF EEN final country report Relevant evaluations ESF Disadvantaged Groups ex post (EU27 template and in-depth reports and horizontal analyses for specific countries) AIR 2014, SFC2007 ESF EEN final country report Relevant evaluations OPs, AIRs, SFC2007 ESF EEN final country report Relevant evaluations OPs, AIRs, SFC2007 ESF EEN final country report Relevant evaluations 3 Conclusions In relation to resources EU 27 country 3 Pages May 2015 page 41

42 Chapter/secti on 4 Lessons learned 5 Good practice Sub-chapter Source of information used, effectiveness, efficiency, CAV, sustainability and gender sensitivity and socio economic impact In terms of: policy choices, target groups, programming, implementation, monitoring and evaluation One evidence based example templates from the 3 ex posts In-depth reports from the 3 ex posts EU 27 country templates from the 3 ex posts In-depth reports and horizontal analyses from the 3 ex posts In-depth and reports and horizontal analyses from the 3 ex posts Pages 2 2 Tools The main tools to be used in the country synthesis reports will be the reports listed in the ToR and above (under data sources) and the database produced for Task 1 (at least under the ESF Access and Sustainable Integration into Employment Ex Post Evaluation 13 ). Activities and team members The country reports will be drafted on the basis of a thorough reading of the existing material and drafting according to the above template. The basic information for the country reports will be pulled together and updated (with AIR 2014) by the overview team. This is exacting work and needs to be done very carefully, especially in the case of constructed indicators. The process is described in Box 3 below. 13 It is recommended by the Access to Employment evaluation team that similar databases be set up in the other two ex post evaluations. page 42 May 2015

43 Box 3. Updating financial and physical performance data with AIR 2014 The 3 thematic evaluations include information on the financial and physical progress for each OP and relevant PA from the AIR2013. This includes information on the: Allocated budget and certified expenditure per PA on Annex XXIII data per PA on (number of entries) Programme Specific Indicators per PA (target and 2013 realisation) This information will be updated in task 2 with the information included in the AIR of 2014 (dated on ). This information is generally available in the SFC2007. Only in those cases where PA are split between different ESF Priorities is information gathered on sub-priority level. Since this information is not available in SFC2007, the AIR should be used to find more detailed financial and physical information on the level. For every country, an Excel sheet will be developed that follows the structure of the database as presented in task 3B (including three worksheets, one on financial data, one for Annex XXIII data, and one on programme specific indicators per OP/PA). These worksheets are already filled in with information collected during the thematic evaluations as much as possible with an extra column in which the values for 31 December 2014 will be reported. These country sheets will be integrated in one database with all indicators per ESF priority, providing input for the EU synthesis report. For the country reports, only the basic information will be reported on the financial and physical progress of the programme. This includes updated information of Section 4.4 of the EU27 template providing answers to the question on which have been the key positive achievements and underachievements in the country in terms of the outputs, results and impacts for ESF investment. The tables with programme specific indicators will be included in the Annex. For updating and aggregating the programme specific indicators with 2014 data some necessary steps need to be undertaken in each of the thematic evaluations. In case individual indicators are aggregated on country level (by for example summing up similar indicators between OPs) it should be clear and transparent how this was done. Each thematic evaluation should provide details on which indicators are grouped / grouped together and explain the origin of every indicator, e.g. providing source and page number if using the AIR If this is not done, the evaluator responsible for the synthesis is not going to be able to recalculate the output and result of the programme on the basis of the 2014 data. Therefore, each programme specific indicator should be labelled and attributed to a group and the formulas calculating aggregated values should be transparent and reproducible. May 2015 page 43

44 This is something that will have to be very carefully followed up in Task 1 of the ex post synthesis to ensure that the data are transparent in all three thematic ex posts. The recent information included in the 27 country reports of the thematic ex post evaluations will be a major source of information for the country synthesis reports. The level of depth of information differs per country since for every thematic ex post evaluation an in-depth assessment is made in only a limited number of countries (8-12 countries) on that particular theme. However, each country is covered by at least one in-depth study and by all five thematic synthesis reports. Information will be taken from the three thematic ex post evaluations and complemented by additional information from the AIRS 2014, the final ESF Expert Evaluation Network reports and recent domestic evaluations, in particular for the ESF Priorities not covered by the three thematic ex post evaluations. With the aim of ensuring a similar approach and quality across the 28 country reports, a maximum amount of compilation will be done in-house (by Metis) with the aim of producing reasonably complete basic country reports. The national experts will be consulted for those sections of the reports which cannot be done by the overview team. They will be asked to contribute new findings (identified during the updating of the inventories which will be one of the first tasks to be undertaken during the ex post synthesis), provide analyses of the information compiled, draw conclusions and identify lessons learned. They will also be expected to verify the good practice or suggest other examples for the three ex-post evaluations. The country research will be preceded by a pilot study (Pilot country synthesis report) on the basis of AIR 2013 as this will be available at the time of commencement of the contract. If all ESF Priorities are to be covered in the pilot study as suggested in the ToR, the Member States which come into question for the pilot study will be those which cover all Priorities set out in Article 3 of the ESF Regulation, meaning that only countries with Convergence OPs come into question due to the Priority Strengthening Institutional Capacity which is only in Convergence regions. Additional criteria for the pilot study could be: Limited number of ESF OPs Simple or relatively simple allocation of Priority Axis to ESF Priorities Ample evaluation material (i.e. evaluations completed) If we comply with the above criteria (including that of a country with a limited number of OPs for manageability), we would actually only have the choice between Austria, Bulgaria and Poland. Other Convergence OPs in countries with only a few or one ESF OP do not have all the ESF Priorities covered by their Priority Axis. Partnership often seems not to enjoy its own Priority Axis although it may be included as a transversal theme of course. While the three countries mentioned: Austria, Bulgaria and Poland seem page 44 May 2015

45 good choices for pilot studies also on account of available evaluation material. Of the three, it is proposed to take Bulgaria as Bulgaria has all ESF Priorities and presents a number of challenges as a pilot as there is overlap between the PAs. It will also be a test of the in-house overview team cooperating with an external expert. The remaining country synthesis reports are to be delivered in batches. The criteria for deciding on the batches could be either to choose the countries included in each of the thematic ex post evaluations (as these are almost mutually exclusive with the exception of France and Italy) or those with accepted AIRs for The logic of the first would be that the in-depth country reports would cover at least one ESF Priority in depth in the countries which could be an interesting point of departure for the remaining Priorities in that country. It would make sense here to start with the countries from the ex post that is the furthest advanced or finished which is presumably the Integration of Disadvantaged Groups, then Human Capital, and finally Access to Employment. A further criterion could be the availability of the AIR 2014 as it is absolutely necessary to have the final AIRs for completing the country reports. The batches of country synthesis reports would be managed by one core team expert. The first batch would be managed by Isabel Naylon, the second by Costanza Pagnini, the third by Bert-Jan Buiskool. The core team experts would ensure the high quality of the reports delivered and a quick turnaround in the delivery of revised reports. Given the logic that the reports are produced as far as possible in-house, it is hoped that quality issues will not be an issue. Still, a comparative review of all reports in terms of coherence check will be done by Isabel Naylon, and for editorial issues, by Haris Martinos. Table 7. Overview of steps for the accomplishment of Task 2B Task Task 2: National level analysis B: Country synthesis reports (Lead Isabel Naylon) Step 1: Pilot country study Step 2: Screening and pulling together of the existing material by the overview team Step 3: Filling in of the blank parts of the country report templates by the country experts Step 4: Finalisation of reports by overview team Step 5: Drafting Step 6: Delivery of country reports in batches Who Core team and country expert Overview team Country experts Overview team Core team Project manager with responsible sub-task manager (for each of the 3 batches) May 2015 page 45

46 2.4 Evaluation task 3: EU level analysis Thematic synthesis reports Objectives of the thematic EU synthesis reports and overall approach The two thematic synthesis reports will provide an overview of the implementation of the ESF Priorities Promoting Partnerships and Strengthening Institutional Capacity at EU level in terms of participants, outputs and results, of typologies of interventions and on the basis of available evaluations findings - effects and socio-economic impacts. As these two ESF Priorities were not covered under the three thematic ESF ex post evaluations, the additional two EU thematic synthesis reports on Promoting Partnerships and Strengthening Institutional Capacity will provide the key information needed in order to compile the EU synthesis report covering all the ESF Priorities. Promoting Partnerships and Strengthening Institutional Capacity Priorities cover a marginal proportion of ESF resources and they differ from the other Priorities addressed by the ESF as they are not directly focused on education/training or employment-related objectives for individuals, but rather focus on strengthening structures and entities directly or indirectly involved in implementing such objectives. For this reason available data on outputs and results might not be representative of the actual relevance and effectiveness of related interventions and additional qualitative information will have to be collected. The tender specifications mention that these reports shall follow in their structure and analytical approach the overview assessment of 27 MS of the thematic ex post evaluations. Consequently our approach to the analysis of thematic priorities is centred on the analysis of a basic set of information at the MS level in line with what has been collected during Task 1 of the expost thematic evaluations for the preparation of the Country Templates. This information encompasses quantitative and qualitative data. Quantitative data refers to the collection of financial and physical indicators (outputs and results) for the relevant Priority Axes (or, where clearly identifiable, sub-pas). Qualitative information will be collected in order to give a more complete overview of how these Priorities have been dealt with in the different MS, to describe the objectives established in the OPs as regards each of the two ESF Priorities and to provide information on the types of interventions supported and the target groups. This information will be analysed also having regard to the national situation and strategies and EU-level strategies under the two ESF Priorities even though, necessarily, they will not provide the same level of detail and in-depth analysis as the other three thematic ex post evaluations. page 46 May 2015

47 The analysis of this information will provide a first picture of the extent to which the resources were used and of the effectiveness of implemented interventions (in terms of results) and efficiency measured in terms of financial resources spent in order to achieve them. Information collected shall be consistent with the methodology elaborated in Task 1 of the thematic ex post evaluations attributing specific Priority Axes to ESF Priorities and ensuring that no overlap between ESF Priorities happens. In addition to information collected through the analysis of dedicated PAs in the OPs, the EU synthesis thematic reports will also collect and analyse key information on typologies of interventions that can be clearly traced back to the relevant category of expenditure (cat. 80 for Promoting Partnerships and 81 for Strengthening Institutional Capacity) outside of dedicated Priority Axes. This information will be treated separately in order to ensure that overlapping with other ex post thematic evaluations is clearly identified. The combination of these two criteria (analysis of data concerning both dedicated PAs as well as CoE) thus provides a clear demarcation of the scope of the evaluations, although they will not necessarily and exhaustively cover all the interventions which can be traced back to these Priorities. Partnership building in particular is sometimes supported as a transversal theme across all Priority Axes and is more difficult to detect. Likewise, capacity building objectives as we will explain in the following paragraphs are pursued across Priority Axes outside of those explicitly devoted to this Priority. In addition to the above some specific and additional analysis will be carried out. It will focus mainly on issues of programming strategies, effectiveness and sustainability which represent an important element of analysis for both Partnership and Institutional Capacity themes. The scope and methodology for this additional research will be identified on the basis of information collected through the Country synthesis reports and will necessarily focus on a limited number of MS and OPs. In the following we provide a more detailed description of the approach we will adopt for the delivery of the two synthesis reports starting from the specificities of each ESF Priority to be investigated, tacking stock of accumulated knowledge. EU Synthesis Report on Promoting Partnerships Art. 3.1(e) of ESF Regulation 1081/2006, describes the policy field as follows: Promoting partnerships, pacts and initiatives through networking of relevant stakeholders, such as the social partners and non-governmental organisations, at the transnational, national, regional and local levels in order to mobilize for reforms in the field of employment ad labour market inclusiveness. This description helps us to define the demarcation and the main focus of the evaluation task, i.e. interventions formally included, under each OP in all the MS, under the Promoting Partnership specific May 2015 page 47

48 Priority when the Priority is explicitly and clearly identified within an OP Priority Axis (as it is, e.g., in the case of Belgium, Bulgaria, Germany, and France). When the adoption of such a criterion is not feasible, interventions under Reg. 1828/2006 Priority Theme 80 (dedicated to Partnerships) will be considered under each OP as falling under the ESF Priority. Based upon a quick assessment of the main reference documents (Metis- Eureval, 2011 and Metis-EEN 2014 and related country reports, and the Preparatory Study) we can already trace a first outline of the most relevant characteristics of Promoting Partnership ESF Priority which can be used as the basis for formulating key evaluation questions or drivers: The policy field, not particularly relevant for its financial weight, is very differentiated in terms of interventions and it has been locally implemented in highly contextualised ways; Partnership and networking have been (correctly) generally assumed as a means for implementing activities/interventions (therefore focusing on systems and structures) and not as an end in themselves; There are some key variables that may be used for understanding (and probably grouping) the policy field implementation and results. These are: a) the subjects involved in partnership/networking; b) the territorial (or policy) level covered by partnership and/or networking (national, regional, local); c) the types of intervention/s implemented through partnership; d) the issue/s (such as items, themes) covered by partnership and/or networking; Three main types of interventions can be identified around which the grouping could be further detailed, these are: i) Bringing together a range of actors from across public, private and non-governmental sectors so as to promote, establish and sustain partnerships; ii) Strengthening local support structures and policies, where the focus is on the local dimension as a way of enforcing intervention effectiveness; and iii) Promoting Transnational Partnerships, for sharing knowledge and collaborate on areas of common interests ; According to preliminary research carried out we can anticipate that in some MS the PP priority is not part of the ESF programming or its implementation cannot be clearly identifiable (such as Cyprus, Estonia, Latvia, Luxembourg, Netherlands, Slovakia, Sweden); while in some others the priority has been implemented transversally (we could say it has been mainstreamed), also in these cases it is difficult to isolate and evaluate its implementation, outputs and results (France, Greece, Hungary, Ireland, Poland, United Kingdom, Finland, Portugal); The Priority-related evaluation reports are (at least until 2013) very few (for 2011, Metis et. al., p.69, the evaluations so far hardly deal with the promotion of partnership and networking, only 2% of the findings address this policy field ) and, together with other policy fields, very poor in presenting sound evidence about effectiveness (2014, Metis et al., p. 91). page 48 May 2015

49 From the same data sources we also learn that participants (read entries) in these interventions were more than 136,000 and they are mainly concentrated in five MS (CZ, DE, HU, IT, RO). We also have information on their occupational status and gender. No information is available about entities involved, which are otherwise relevant for such a policy field which is focused on structures and systems rather than on services delivery. The most important results (considering 2012 result indicators and 2013 evaluation reports) are strongly linked to the thematic focus of the established partnerships (such as knowledge sharing, new target groups involvement, good practices exchange, delivery of effective interventions) and consequently not easily comparable. Very few robust evaluations concerning effectiveness are available (up until 2013) while a number of evaluation reports (very few in general for this policy field) identify facilitating factors for partnerships development and operations (such as regular networking, financial incentives) as well as ways in which partnerships contribute to a more effective evaluation of the ESF in general. As a whole, a very limited number of good practices has been identified for this policy field and they are concentrated in 4 MS (AT, HU, IT, SI). The above indicated key-variables (intervention logic, composition of supported partnerships, types of interventions, issues covered by partnerships), together with physical and financial information and duration collected through AIRs and SFC are the most relevant elements that can be collected within the scope of the present study and on which the analysis will be focused, at MS level, in order to provide an overview of the PP priority programming strategies and implementation. Within this framework, the main research questions that will be tackled at the MS level in the report are the following: - How has the PP priority been integrated at MS level in ESF programming (analysis of intervention models and logics and programming set-ups) - How has the PP priority been implemented in terms of typologies of interventions (i.e. what type of networking activities: partnerships, pacts and initiatives and at what level of intervention: transnational, national, regional or local) and beneficiaries (such as social partners, NGOs), resources spent - How much has been achieved in terms of outputs and results and, according to available evaluations, impacts (in terms of mobilisation for reforms in the field of employment and labour market inclusiveness) - How do achievements compare with programming expectations (as represented by the specific OP indicators aggregated by MS) May 2015 page 49

50 - What can we say about the future sustainability, gender sensitivity and CAV of the interventions promoted under the PP priority? - What is the level of consistency between the PP priority and the relevant Country Specific Recommendations 14 ; The extent of the completeness with which these research questions will be answered will depend to a great extent on PP programming set-ups adopted by each MS. We expect that these questions will receive a more complete answer for those MS in which the PP priority is totally or partially implemented under dedicated priority axes/measures. According to preliminary research carried out so far this is the case for Austria, Belgium, Bulgaria, Czech Republic, Denmark, Italy, Lithuania, Malta, Romania, Slovenia, Germany, and Spain). According to the ESF Expert Evaluation Network (2014) in these MS training, employment, local development and geographical mobility are the issues mostly covered by partnerships and networking activities; VET providers, social partners, NGOs and local institutions are the most frequently involved actors; in general, both system actions and actions targeting people are equally supported within the PP. Answering the above indicated questions will allow to draw an overall picture of both quantitative and qualitative) evidence focusing on all the programming and implementation steps of Promoting Partnerships Policy Field; in addition, such evidence will allow to define MS clusters according to PP programming and implementation strategies, on the one hand, and by actual funding, output and results, on the other. Furthermore, the PP evaluation questions match the thematic synthesis report structure described in the Table of contents section below which will be produced using evaluation questions answers as inputs. From a methodological point of view, Promoting Partnerships evaluation questions will be answered in the following ways: - The consistency between the PF programming strategies and their implementation pathways (interventions and actions; implementing subjects; target-groups, etc.) will be equally assessed through a desk analysis of existing information reinforced by the evidence deriving from the in-depth research on a limited number of MS (case-studies); - The correspondence between expected and actual outputs and results indicators (mainly at MS level) is going to be evaluated through a desk comparison between the two groups of indicators quantifications; explanations of positive and/or negative differences will be based upon country experts contribution as well as on official implementation document analysis (mainly 2013 and 2014 AIRs); - The coherence between the expected and actual PF expenditure will be assessed in the same way as output/result indicators expected versus actual values; 14 Covering the period page 50 May 2015

51 - The evaluation of the relevance of PF implementation impacts is going to be mainly based on available PF specific impact evaluations (very few at the moment) eventually integrated, for the involved MS, by evidence coming from the case-studies carried on by country experts; - Finally, the CAV, gender sensitivity and future sustainability assessment of PF implementation pathways and (if possible) results will be based on country experts analysis using dedicated grids and/or check-lists. Such methodological design includes, as far as possible, the establishment of groups of indicators derived, at MS level, both from Annex XXIII Reg. 1828/2006 and Programme indicators. However, due to the peculiarity of PP with respect to the other ESF priorities as well to the differences in approaching the issue in each MS, it will be reasonably more feasible to identify aggregated (that is at EU level) output indicators than result indicators (which are likely to be quite different among MS and in some cases among OPs within the same MS). EU Synthesis Report on Strengthening Institutional Capacity In line with the 2006 Community Strategic Guidelines for Cohesion, which identify good governance and capacity building as key issues to be addressed, Article 3.2(b) Regulation EC 1081/2006 mentions the strengthening of institutional capacity and the efficiency of public administrations and public services at national, regional and local level and, where relevant, of the social partners and nongovernmental organisations, as one of the main ESF Priorities for the period. This Priority has been introduced for Convergence regions and Cohesion MS as a key element for promoting structural adjustments, growth and jobs, as well as economic development. It is seen as a tool which will contribute to reforms, better regulation and good governance, especially in the economic, employment, education, social, environmental and judicial fields. Although the share of overall funding volumes is relatively small, institutional capacity building plays an important role in improving the delivery of ESF services, their efficiency and their effectiveness. In general, the actions under the capacity building priority theme should support the reform of public administrations and public services, including their modernisation, both in attaining sustainable socio-economic growth and reducing disparities and in ensuring good governance. The support provided under the priority should enable public administrations and public services to become a factor of competitiveness, development and growth for the Member States and regions. Clearly, these actions are strongly dependent on the specific national and regional context of each MS, making it somehow more complicated to classify and compare them (with respect for instance to other ESF Priorities). Each eligible MS has outlined the strategy for reinforcing the institutional capacity in the National Strategic Reference Framework, indicating whether May 2015 page 51

52 strengthening institutional capacity is implemented through a separate Operational Programme (with one or more Priority Axis), or through one or more Priority Axis in other programmes, especially regional programmes or a combination of both. In this context, four Member States (Bulgaria, Romania, Hungary and Greece) set up a dedicated administrative capacity building Operational Programme, while nine others (the Czech Republic, the three Baltic States, Poland, Slovenia, Slovakia, Malta and the UK in Wales) included it as a Priority Axis in one of their programmes, mainly in regional programmes. Others, like Italy, combined the two approaches with a dedicated national programme and Priority Axes in regional ones. As a starting point the EU synthesis report on Strengthening Institutional Capacity will analyse and compare quantitative data (financial allocations, outputs and results) concerning the interventions linked to the Priority in line with the demarcation of axes across priorities. Following this, it will examine ESF Operational Programmes or Priority Axes specifically dedicated to Capacity Building across all Member States (with reference to the Convergence objective). In addition, in order to expand the qualitative assessment of the Priority (for which overlapping is not an issue) and in consideration of its horizontal nature, the report will also explore Priority Axes which are not specifically targeted at Strengthening Institutional Capacity but which nonetheless foresee a relevant component of capacity building activities. This can be done by looking at the categories of expenditures. ESF Priority Strengthening Institutional Capacity is in fact thematically linked with category of expenditure n. 81 Mechanisms for improving good policy and programme design, monitoring and evaluation at national regional and local level, capacity building in the delivery of policies and programmes ; however it can be related to other important categories, such as category n. 80 (Promoting partnerships, pacts and initiatives through the networking of relevant stakeholders for the mobilisation of reforms), which is the object of the other thematic synthesis report; and categories n. 65 (Modernisation of labour market institutions) and n. 72 (Reforming education and training systems), which are covered under the three ex-post thematic evaluations. Finally, an additional point to be considered by the report is the relationship between the Strengthening Institutional Capacity Priority with actions and objectives of the technical assistance interventions. The ESF finances actions aiming at supporting the structures for the management and implementation of the Fund under Technical Assistance and should therefore be clearly distinguished from the activities financed under the Institutional Capacity Priority. However, the Institutional Capacity Priority aims at supporting the reform of the administrations irrespectively of their role in Structural Funds management and implementation and some overlap between these two typologies of interventions occur. For this reason it will be necessary to verify the use of resources destined to capacity building and if the actions analysed are really interventions of capacity building or their specific objectives cover mainly the needs of the management of Structural Funds. page 52 May 2015

53 Strengthening institutional capacity relates to two main types of intervention: - improvement of mechanisms of good policies and sound programming and implementation (such as through studies and expert advice), including support for dialogue between the public administration and its citizens and private bodies - capacity for the delivery of policies and programmes (through training and specific support to key services). Clearly, while public administration capacity building targets primarily institutions (systems and structures), the capacity building of individuals (e.g. staff of institutions) can be equally important to improve the ability of these institutions to perform in a more effective and efficient way. That is why ESF promotes an integrated approach to such priority: assistance to individuals (for personalised support such as training of staff) and assistance to entities (single institutions or a system of such). Furthermore any assessment of strengthening institutional capacity should also look whether relevant programmes and measures follow a horizontal or sectoral approach, i.e. whether they pursue: - overall modernization and administrative reform (including in the judiciary) horizontal approach or - improvement of mechanisms and strengthening of institutions (always according to the integrated approach targeting both systems and individuals) working in specific areas of cohesion policy (such as employment, education, health and social policies) - sectoral approach Another task of this analysis concerns the extent to which the SIC priority responds to, or is coherent with, the development needs of MS as formulated in the CSR, with a particular focus on the sub-categories like Improving the effectiveness and efficiency of the public administration, the Judiciary reform, the Improvement of the business environment and the Anti-corruption Public procurement. The above indicated key-variables provide us with a general analytical framework for assessing the implementation of the SIC priority. These elements, together with physical and financial information and duration extracted from AIRs and SFC are the most relevant information that can be collected within the scope of the present study and on which the analysis will be focused, at MS level, in order to provide an overview of the SIC priority programming strategies and implementation. Within this framework, the main research questions that will be tackled at the MS level in the report are the following: - How has the SIC priority been integrated at MS level in ESF programming (analysis of intervention models and logics and programming set-ups) May 2015 page 53

54 - How has the SIC priority been implemented in terms of typologies of interventions and beneficiaries, resources spent? Concerning interventions the assessment will explore whether for example supported interventions are aimed at: (i) modernization and/or administrative reform (including interventions targeted at judiciary reform). Interventions will be grouped according to a process dimension (e.g. modernizing and optimizing internal processes, development of quality management systems, adoption of IT systems, improving the interaction between institutions and with stakeholders, e-government / e-justice...) or to a structure and organizational dimension (developing appropriate administrative structures, incl. through reallocation of functions, decentralization, improving management structures). (ii) Skills enhancement of relevant staff? In this case interventions will be grouped according to the specific actions envisaged: e.g. traineeship programme, learning network, training. Sectoral interventions will be grouped according the relevant policy field One more element of analysis of the interventions will relate the level of intervention (national, regional or local) Concerning beneficiaries this question will explore in particular which bodies and services have benefitted from ESF support for SIC (public administrations and public services but also social partners and non-governmental organizations) at the - How much has been achieved in terms of outputs and results and, according to available evaluations, impacts (with a view to reforms, better regulation and good governance especially in the economic, employment, education, social, environmental and judicial fields). - How do achievements compare with programming expectations (as represented by the specific OPs indicators aggregated by MS) - What can we say about the future sustainability, gender sensitivity and CAV of the interventions promoted under the SIC priority? - What is the level of consistency between the SIC priority and the relevant Country Specific Recommendations 15. Such methodological design includes, as far as possible, the establishment of groups of indicators derived, at MS level, both from Annex XXIII Reg. 1828/2006 and Programme indicators. Given the peculiarity of the SIC 15 Covering the period page 54 May 2015

55 priority, it will be reasonably more feasible to identify aggregated (that is at EU level) output indicators than results indicators (which will be probably quite different among MS and in some cases among OPs within the same MS). Data and information sources As already mentioned above, the data and information sources feeding the thematic EU synthesis reports refer in the first instance to the material which is already available and which has been described in detail in the Tender Specifications and in the previous chapters of the present Inception Report. These refer in particular to: Operational Programmes and Annual Implementation Reports (with particular reference to the 2013 and 2014 AIRs which were not reviewed by the EEN Country Reports); AIRs are a key source of information in order to update the quantitative assessment and double check information from the SFC2007, to provide more detailed information on typology of interventions implemented under the dedicated Priority Axes, and to identify Priority-relevant objectives and actions outside of dedicated Priority Axes. The analysis of the AIR will consider the following parts of the document related to the Programmes or Priority Axes for Promoting Partnerships and Strengthening Institutional Capacity: 1. Section 2 overview of the implementation of the Operational Programme in particular for the general information about the Programme e for the financial allocation referred to the relevant categories of expenditures; 2. Section 3: implementation by priority including the description of the actions activated and their physical implementation. SFC2007 database; for compiling the quantitative assessment The updated inventory of evaluations up to 30 June 2015 that will be analysed in detail in order to identify evaluations (on-going, final but above all ex post; thematic and ESF priority-wide; results and impacts-focused, also those produced in the framework of ex post evaluation of Cohesion Policy Programmes;) that have been carried out at the EU or MS level and that might provide important elements concerning effects, impacts, lessons learned and policy recommendations relevant to the two Priorities (the ESF assessment on administrative and institutional capacity and the forthcoming ERDF, ESF and CF ex post study on delivery system will be of particular importance for the Institutional Capacity report) DG REGIO ex post evaluation of delivery systems, WP12 which is expected to provide interesting insights and complementary information on the delivery systems of Cohesion Policy Preparatory Study; this report will be particularly useful in providing information for identifying the main clusters of interventions, but also the programme architecture of 43 OPs (Priority axes, actions, and May 2015 page 55

56 interventions) including Promoting Partnerships and Strengthening Institutional Capacity EEN synthesis and country reports last round; these provide information on outputs, results and short descriptions of the ESF Priorities (up to 2013) and will be useful for completing the quantitative assessment as well as for providing key information for filling-in the dedicated sections of the Country synthesis reports Context data, such as Employment and Social Developments in Europe; Fourth report on Economic and Social Cohesion; Fifth and Sixth Report on Economic, Social and Territorial Cohesion, Quality of public administration report 16, 2015 EU Justice Scoreboard 17. It should be noted that most of the relevant information derived from the above data sources will be included and analysed in the Country synthesis reports (sections on Promoting Partnerships and Strengthening Institutional Capacity); consequently these reports will represent a key information source for the drafting of the EU synthesis reports. In addition, as further explained in Tools and activities section/step 2 below, the PF implementation in some MS (max 5, to be agreed with the Commission) will be more deeply analyzed through specific case studies aimed at more deeply exploring the scope and effectiveness of a sample of PF implementing interventions. Table of contents Each thematic EU synthesis report will be 50 pages long and its table of contents broadly reflect the structure of the other thematic ex post evaluations. A tentative structure is outlined below: Table 8. EU Synthesis Report structure Introduction Executive summary Report background and objectives (3 pp) Key-data concerning ESF Priority delivery: a general overview (10 pp.) ESF Priority operations and Background and report objectives and scopes Overall progress in ESF Priority implementation Analysis of clusters of interventions Evaluation results and evidences Conclusions Report objectives ESF and Promoting Partnerships/Capacity building Analysis of outputs (EU and MS level) Analysis of results (EU and MS level) Types of interventions and interventions COM(2015)116 page 56 May 2015

57 interventions (10 pp) Evaluation data, approaches, methods and results (15 pp.) Conclusions: implications for future policies (3 pp) clustering: numbers and facts (EU and MS level) Types of interventions: numbers and facts (EU and MS level) Evaluation sources, approaches and methods Evaluation evidences (EU and MS level) Main factors influencing interventions results and impacts Tools and activities Step 1. Desk analysis of existing information During this first step all relevant information and documents for the drafting of the EU synthesis reports on Promoting Partnerships and Strengthening Institutional Capacity will be collected and analysed, in particular concerning implementation of the relevant ESF priorities in OPs/MS, identification of dedicated PAs consistent with the other three ex-post thematic evaluations, implementation processes, results and evidence from specific evaluations (see the data sources indicated above). This activity will entail a review of available material and will result in a preliminary draft of the overview assessment of the 28 MS; it will also serve to identify the information that is needed from Country experts and to pre-compile the relevant sections of the Country synthesis reports (see Task 2). During this first step the core team will also build a preliminary clustering of typologies of interventions implemented in response to each of the two priorities. In addition the following tools will be implemented: MS/OPs table with Priority Axes dedicated to Promoting Partnerships and Strengthening Institutional Capacity. A preliminary table, compiled according to information in EEN country and synthesis reports, is provided in Annex I. The table will be revised to make sure that no overlapping exists among the five ESF priorities. On the basis of this table it will be possible to identify for each dedicated Priority Axis the relevant financial allocation, outputs and results indicators, targets and beneficiaries. This forms the basis for the qualiquantitative assessment. MS/OPs table with financial allocations per category of expenditure which have a direct relevance with each of the two Priorities (n. 80 for PP and n. 81, 65 and 72 for SIC). This information will be matched with the MS/OPs table with dedicated Priority Axes so as to map MS according to their programming decisions on how to deal with the two priorities (i.e. whether they have dedicated specific OPs/PAs or have adopted a transversal approach whereby relevant interventions are distributed across OPs/PAs or a combination of both); MS positioning May 2015 page 57

58 might be used as an additional element for the selection of a number of MS/OPs for a more in depth analysis. Step 2. Identification of information gaps and collection of information at country level On the basis of the analysis undertaken during Step 1, the sections of the country synthesis reports dedicated to Promoting Partnerships and Strengthening Institutional Capacity will be precompiled and circulated to the country experts for double-checking and updating on the basis of the analysis of the AIRs and relevant evaluations carried out in MS. Country experts will also be asked to fill in information gaps concerning dedicated OPs/PAs objectives. During this step, a limited amount of additional (mostly field) research will be envisaged aiming at studying in more detail the implementation and results of the two ESF priorities in a restricted number of MS. This information will help shed light on the scope and effectiveness of specific typologies of interventions or ESF set-ups for the implementation the priorities. Field research will be carried out by Country experts and will mainly involve the collection of additional information/data through telephone interviews (with MAs and/or relevant stakeholders). The selection of case studies will be agreed with the European Commission once Country synthesis reports have been completed and will be based as far as possible on the criteria used for the ex post thematic evaluations (e.g. relative importance of dedicated Priority Axes/OPs; socio-economic contexts). In addition representativity of different ESF set-ups for implementing the priorities will be taken in consideration (i.e. MS that have dedicated specific OPs/PAs to the priorities vs MS that have dealt with the priorities horizontally i.e. outside of dedicated Priority Axes). Step 3. EU-level overview of ESF Priority and drafting of the Thematic EU synthesis report This step is dedicated to synthesising the information collected from Country experts (Country synthesis reports and additional information from case studies) and other EU-level relevant documents. The final result shall be an overview of the implementation, outputs, results and effects of ESF Priorities Promoting Partnerships and Strengthening Institutional Capacity. The overview will adopt a comparative approach analysing available information across MS, objectives, financial allocations, actions, outputs and results. MS/OPs will be classified according to the importance that they attribute to each of the ESF priorities Promoting Partnerships and Strengthening Institutional Capacity, and whether they do so through dedicated OPs/Priority Axes or through a clear horizontal approach. This will allow making comparisons in terms of results, impacts, and MS socioeconomic contexts. page 58 May 2015

59 As previously mentioned, the overview will include both quantitative as well as qualitative analyses. Quantitative analysis will provide comparable and standardised information, consistent with that of the three ex-post ESF thematic evaluations covering financial information, outputs and results. Qualitative information will provide information on objectives established in the Operational Programmes and types of interventions funded, together with a contextualisation of this information vis-à-vis national situations and strategies and EU-level strategies. Particular attention will be given to synthesise any existing evaluative evidence on impacts and effects of implemented strategies with a view to identifying strengths and weaknesses as well as facilitating factors or obstacles. Team members One core team member (Costanza Pagnini) will be in charge of coordinating the activities and drafting the Thematic EU synthesis reports. She will be supported by a member from the research team (Diego Teloni), who will discuss and review the reports. Country experts will draft a separate small chapter during their complementary information provision and provide country-specific insight concerning the need for further in depth assessment. Table 9. Overview of steps for the accomplishment of Task 3A Task Task 3: EU level analysis A: Thematic EU synthesis reports (Lead: Costanza Pagnini) Step 1. Desk analysis of existing information Step 2. Identification of information gaps and collection of information at country level Step 3. Comparative analysis and drafting of the reports Who Core team Core team and Country experts Core team EU synthesis report Objectives The second part of the EU level analysis concerns the main product of this project: the production of the EU synthesis report. This report will synthesize the analysis, findings, conclusions and lessons learned from the ex post evaluation of the ESF This report will be structured around the ESF priorities and will both be used as a basis for the communication to the Council and the EU Parliament and be the main communication to other relevant audiences. From this perspective the language will have to be non-technical and the report very concise (maximum 50 pages). May 2015 page 59

60 The template for the EU synthesis report The ToR indicates which information the EU synthesis report should include. Based on these requirements the following structure of the synthesis report is proposed. Table 10. Structure of the EU synthesis report Executive Summary Introduction Adaptability of workers Access and Sustainable Integration into Employment Social Inclusion Human Capital Promoting Partnerships Strengthening Institutional Capacity Conclusions and lesson learned on the EU level policy on economic and social cohesion Policy context Aims Methods Contextualising ESF The extent to which resources were used Effectiveness Efficiency Socio economic impact Community Added Value Gender sensitivity Sustainability Conclusion and lessons learned See above See above See above See above See above policy choices target groups programming implementation monitoring evaluation of future programmes Data and information sources The EU synthesis report will be based on several sources as already identified in the ToR. The main secondary sources for the EU synthesis report are: The country synthesis reports (all 28 MS) Overview, comparative analysis, conclusions, lessons learnt from the four thematic ex post evaluation reports and the two thematic EU synthesis reports, Other possible sources are: Additional background material: such as Employment and Social Developments in Europe, the reports on Economic, Social (and page 60 May 2015

61 Territorial) Cohesion, relevant reports from Eurofound and Cedefop, national studies with an EU perspective The new Regulations for the Cohesions Funds and the ESF in particular and the Guidance documents related to these regulations (in particular in connection to the objective to provide lessons learned which could be useful for the new programming period). The level of depth of information differs per country since for every thematic ex post evaluation an in-depth assessment is made in only a limited number of countries (8-12 countries) on that particular theme. However, each country is covered by at least one in-depth study and by all five thematic synthesis reports. Despite, or rather because of, the fact that the EU synthesis report is mainly based on existing data sources, we suggest organising a peer group seminar in Brussels near the end of the process to reflect on the main conclusions and lessons learnt. For this seminar we would like to invite senior officials from the EC, the main evaluators for each of the thematic ex post evaluations, and a limited number of senior experts. The aim would be to validate the main findings and conclusions and lessons learnt in a peer setting. In the following sub sections we explore our approach for every sub section in the EU synthesis report (see Table 10 on the structure of the report above). Activities and tools Step 1: Aggregating information in the database Based on the work done in Tasks 1 (setting up of the database) and Task 2 (filling in of the database at country level), Task 3 tries to aggregate data on EU level (and find common indicators between priority themes and EU level). Step 2: Contextualising ESF (Section 1 in the synthesis report) First of all, the EU synthesis report should provide information on objectives established in the Operational Programmes and types of interventions funded, together with a contextualisation of this information vis-à-vis national situations and strategies and EU-level strategies. This analysis will be implemented by plotting all relevant contextual information (Europe 2020 indicators relating to economic development, employment, education, lifelong learning and social inclusion), general trends in Country Specific Recommendation over the programming period, and focus of ESF programmes in countries. Here we would like to build further on the scoreboards that were developed in each of the thematic ex post evaluations where the key indicator trends are presented for each Member State and the EU-27 average. Each thematic evaluation also May 2015 page 61

62 provided an overview of relevant Priority Axes per ESF Priority. Having this information, the following outputs can be created: I. Sub-step 1: Typology of different country contexts per ESF Priority (3-4 clusters) assessing Europe 2020 indicators (such as GDP development, employment and unemployment figures, education, lifelong learning, poverty index etc.) and other. Here we take the development over the period, also to identify whether contexts are positive. page 62 May 2015

63 Table 11. Context indicators for each ESF priority (as included in the EU27 templates) A2E HC ADAPT SI GDP expenditure capita PPS_EU27) Bienual (%) Real per (in growth Employment rates by age and sex Employment rates by highest level of education attained Unemployment rates from 15 to 64 years Part-time employment Temporary employees as percentage of the total number of employees Self-employed (total number: thousands) Young people not in employment and not in any education and training Job vacancy rate (2008 onwards) Initial education: Extent of ESL (Eurostat), Unemployment of young people (Eurostat), PISA scores (2006, 2009, 2013, OECD) (disaggregated by gender and age where available) Higher education: HE attainment (Eurostat) (disaggregated by gender and age where available) Continuing adult education: Employment of low skilled (Eurostat), Participation in LLL (Eurostat) (disaggregated by gender and age where available) Funding: Expenditure on E&T (Eurostat), Expenditure on R&D (Eurostat). Percentage of enterprises providing CVET Percentage of companies systematically checking of training needs % of establishment giving employees time for training Employment rate Unemployment rate Long term unemployment rate Inactivity rates among the low skilled (ISCED 1-2) Inactivity rates among individuals of foreign nationality Share of young people (15-24) not in education not in training Share (%) of population aged 18 and above at risk of poverty or social exclusion Share (%) of individuals (aged 18 and above) born in a foreign country at risk of poverty or social exclusion Percentage of expenditure in different social protection areas in relation with the GDP Funding: spending participants LMP and May 2015 page 63

64 II. III. IV. Sub-step 2: Trends of development challenges based on the analysis of the Country Specific Recommendations during the programming period. Types of CSR will be clustered and countries will be mapped along clusters of CSR, showing different challenges identified across the EU. Sub-step 3: Typology of programmes assessing type of focus of programmes per ESF Priority (type of interventions supported and relative financial contribution ESF). It will be assessed whether countries concentred their budget on a limited number ESF priorities (concentration principles) or not. You could identify programmes that (1) focus on a wide variety of ESF priorities, (2) medium number of interventions, or (3) low number of interventions, having a more focused approach. Moreover, we will identify the focus of each programme (by identifying the ESF priority / COE that receives the largest budget). Sub-step 4: countries will also be compared on how the way the ESF A2E investment contributed to the national policies. Overall assessment are made whether ESF is mainly used as additional funding to mainstream policies, or whether ESF was used to test new and innovative activities, or whether ESF was used to reach new target groups, or finally whether ESF was used to improve the delivery systems and methods. Based on the four steps above, countries can be clustered and illustrated with some examples coming out of the country synthesis reports (task 2) and the thematic evaluation reports on each of the 6 ESF Priorities. It would be interesting to see whether ESF is addressing the exact issues in a country that shows the greatest need (linking the different sub steps). Step 3: Assessment of evaluation criteria The extent to which resources are used: The first step in the implementation of ESF is the mobilisation of the financial inputs provided by the EU and the Member States. The utilisation of the financial resources is a condition sine qua non for achieving any kind of impact. Per ESF Priority we will assess: Budget allocated to every Priority Axis and finally spent (certified expenditure), calculating absorption rates (here we makes use of the latest AIR2014). These budgets will be aggregated on ESF Priority level. Budget allocated to the relevant Categories of Expenditure per ESF Priority. Information explaining factors for low absorption (qualitative analysis). page 64 May 2015

65 By comparing budgets spent on Priority Axis and the budgets allocated to categories of expenditure, we are able to identify deviations, possibly explained by the allocation of Priority Axis to one single ESF Priority (cross check). Effectiveness: For each ESF Priority an overview will be provided in terms of outputs and results achieved based on the most important data collected through Annex XXIII and programme specific indicators. Annex XXIII indicators are registered on Priority Axis level, and centrally captured by the European Commission. Within task 2, this database will be updated with participant data coming from AIR Programme specific indicators are stored on Priority Axis level, and OPs follow different approaches leading to a wide variety of indicators, also including different types of measurement like absolute numbers, percentages, total or additional, etc. Therefore, the ex post evaluation synthesis can only aggregate values of indicators that have a common ground (unit of measurement) and can therefore only provide a picture of the minimum contribution of ESF (see chapter on programme specific indicators). Moreover (average) achievement ratios are calculated for the programme specific indicators (achievement value / target value) per PA, OP, ESF priority, country (captured by task 1) and EU level (captured by task 3) 18. For each ESF Priority an overview will be provided of the main physical outcomes of ESF financed interventions. The results of the analysis in the thematic ex post evaluations and the thematic synthesis reports for the other two ESF priorities PP and SIC will be screened and used for evidence of effectiveness, success factors, good practice, and lessons learned. The aggregated findings from the evaluation inventories will also be drawn on for the assessment of the effectiveness of ESF interventions as well as relevant EU level evaluations and research projects. We will also identify some common indicators across ESF priorities and countries to provide a picture on the performance of ESF integrally (see Section 1.4.4). Efficiency is related to the question of whether the results have been achieved in a cost-efficient way. Since it is hardly possible to assess the efficiency of spending by linking financial spending per Priority Axis and output and result indicators (also given the broadness of interventions included in each Priority Axis), we will base the assessment of efficiency mainly on the assessment made in the thematic ex post evaluations and the 18 The problem, however, is that a relatively large amount of indicators do not have a target value, so the average achievement rate is not representative for all indicators. Another challenge is that many OPs have adjusted their targets over the programming period, in order to better align them to the actual progress made in the programme. As a result, one could question whether measuring the achievement rate is meaningful. With proper adjustments, the moving target can be reached at the end this way disguising a failure. May 2015 page 65

66 thematic synthesis reports for the other two ESF priorities (in-depth country reports) and summarise the outcomes. This is assessed in terms of the extent to which the desired effects were achieved at a reasonable cost, how economically the resources used were converted into outputs and results, and the relationship between the outputs and results of ESF interventions and the resources used to deliver them 19. The main components of the approach are listed in the box below. The Evaluation Synthesis will summarise the results of this investigation in the three ex post evaluations. Box 4. Approach to efficiency in the three ex post evaluations The analysis of the efficiency of interventions in the three ex post evaluations compares the scale and scope of interventions (cost per participant) and the relative difference the intervention makes. This requires: A clear understanding of how the costs of ESF interventions compare to their outputs and results achieved and at what costs the desired results have been delivered, including mapping the relationships between the cost vs outputs and results achieved. An understanding and analysis of the measured efficiency of ESF interventions compares relative to other similar interventions funded from different national/eu funding streams. An analysis of the relative efficiency of the different ESF interventions and the reasons behind these differences in efficiency. In order to answer these questions, the approach to the efficiency analysis of interventions in the three ex post evaluations incorporates the following key elements: Comparison of the levels of output/result per EUR invested per person / entity across different projects supported under the interventions. Ranking of the cost per participant / entity of different ESF interventions. Comparison of the results of different ESF interventions and other similar interventions (funded from different streams) to support similar themes (i.e. obtaining information on cost per unit of matched comparators). Analysis of the views of the ESF stakeholders and other independent experts about cost-effectiveness gathered in the interviews and the A2E e-survey. 19 See definition in the Commission s guidance on evaluation, European Commission (2001) and Evalsed Sourcebook: Methods and techniques: P page 66 May 2015

67 Socio economic impact: We will synthesise evidence gained on socio economic impact for every ESF priority as included in the thematic synthesis reports for the three thematic ex post evaluations and the thematic synthesis reports on Promoting Partnerships and Strengthening Institutional Capacity (in-depth country reports).the country synthesis reports will be checked for updates (AIR 2014). Community Added Value: We will synthesize evidence gained on Community Added Value for every ESF priority as included in the thematic synthesis reports for the three thematic ex post evaluations and the thematic synthesis reports on Promoting Partnerships and Strengthening Institutional Capacity (in-depth country reports). The country synthesis reports will be checked for updates (AIR 2014). Gender Sensitiveness: We will synthesize evidence gained on gender sensitiveness for every ESF priority as included in the thematic synthesis reports for the three thematic ex post evaluations and the thematic synthesis reports on Promoting Partnerships and Strengthening Institutional Capacity (in-depth country reports). The country synthesis reports will be checked for updates (AIR 2014). Conclusions and lessons learned: Conclusions and lessons learned should be concise, logical, evidence based, analytical and relevant. However, experience shows that in many evaluations, a logical fault line appears between findings, conclusions and lessons learned. We will attempt to produce a common thread between these different results. Step 4: Conclusions and lessons learned As a first sub-step the core team will cluster the conclusions for the EU level policy on economic and social cohesion and lessons from the different thematic evaluations and country synthesis reports along the following lines: Policy choices: an assessment will be provided whether the focus of programmes are in line with the emerging development needs over the programming period. Target groups: assessing whether and how the relevant target groups are addressed by the programmes giving the challenges countries face, in particular concerning young people. Programming, and implementation: assessing whether programming and implementation can be improved, also taking into account the new Regulations for the programming period (ESF Regulation and Common Provisions Regulation) that have introduced many changes and novelties, and reinforced obligations for the ESF. Monitoring and evaluations (with regard to the new regulatory requirements): improving monitoring systems, amongst else addressing the issue to further standardise the definitions, interpretations and consistent collection of monitoring data (outputs May 2015 page 67

68 and results), including linking better administrative systems Lessons will be drawn on the planning, implementation, and use of evaluations in policy making. Special focus is on the topic how to improve the evidence on the (net) effectiveness of interventions. In a second sub-step the core team will categorise these lessons based on categories like time-frame (short/mid/long term), addressee (MA, IB, beneficiaries etc.), type of action (e.g. termination of an activity, intensification etc.) In a third sub-step the evaluation team would like to discuss the outcomes during a peer group seminar in Brussels (Step 5) for which we would like to invite the following discussants (maximum 15): Senior officials from the EC (DG EMPL, DG EAC, DG REGIO), Responsible evaluators for each thematic ex post evaluation, Limited number of experts that were involved in the ex post evaluation exercise During this seminar we want to prioritize the conclusions and lessons learned for the future (also taking into account the political, institutional and practical feasibility to implement the lessons learned). The figure below provides a schematic overview how we are planning to translate the evidence collected into concrete findings, provide a judgment, and finally develop conclusions and lessons for the future. Figure 2. From evidence to findings to lessons learned Evidences from data collection are cross analyzed to obtain findings which are then used for judgement and to formulate Lessons learned Evidence #1 Finding#a Evidence #2 Finding#i Evidence #3 Finding#b Lessons learned Evidence #4 Finding#ii Finding#c Evidence #5 Evidence #6 Finding#d page 68 May 2015

69 Table 12. Overview of steps for the accomplishment of Task 3B Task Task 3: EU level analysis B: EU synthesis report (Lead: Bert-Jan Buiskool) Who Step 1: Preparation, setting up a database Core team with support of overview team Step 2: Contextualising ESF Step 3: Assessment of evaluation criteria Step 4: Conclusions and lessons learned Step 5: Validation of findings through Peer Review Seminar Step 6: Drafting Core team Core team Core team Core team + peer reviewers Core team May 2015 page 69

70 3. REPORTING This chapter presents the tables of contents of the various deliverables. 3.1 The Inventory of Evaluations The structure for the inventory was already developed in the framework of the ESF Expert Evaluation Network. The structure is robust and has proved useful in filtering for various elements of information, e.g. types of interventions or target groups evaluated, types of evaluations carried out, findings on various themes etc. The structure is given in Table 13 below. page 70 May 2015

71 Table 13. Content of the inventory Evaluation Plan (table I_EvanPlan) no update expected 0 Country (MS) admin. /polit. Unit No. Version number Version date Basic info on evaluation plans Level for which evaluation plan is available Please specify level No. of eval. to be carried out Link to monitoring Specification of arrangements Specifications I Content & Use Description of the scope of the evaluations Specification of scope(s) Main eval. Questions Specification of main evaluation questions Potential use of the evaluation Specification of the potential use Type of evaluation Eval Plan Specifications II Time, Money & Overall comment Time frame for evaluation Specification Information on Finanical resources foreseen Specification Overall Comments to Evaluation Plan Your notes and excerpts May 2015 page 71

72 Implementation of the evaluation (table II_EvanImpl) II.1 Basic info on a specific evaluation 17 Country (MS) admin. /polit. Unit No. Short title (english) Full title of evaluation contracted /announced (original language) Title of Evaluation contracted /announced (english translation) Authority/ Institution/ Organisation in charge II.2 Specification : Type, evaluator, periods & status Type of evaluation Evaluator Period covered: BEGIN [dd/mm/yyyy] Period covered: END [dd/mm/yyyy] Start of the evaluation (planned or actual) [dd/mm/yyyy] End of the evaluation (planned or actual) [dd/mm/yyyy] Status of evaluation II.3 Specification: Coverage (Programmatic & policy level, territorial level) and intended use Level of evaluation SFC Code of OP please specify evaluation level Territorial level If other, please specify Potential If 'mixed' or if otherwise 'unclear' (intended) use of please specify the evaluation page 72 May 2015

73 Evaluation documents (table III_EvanImpl_Doc) 18 Country (MS) admin. /polit. Unit No. Ref. Eval Type of evaluation document available III. 1 Evaluation documents Title, type & topics (and availabilty) ( characters!) Short title (english) Title of the document Title in english translation Details /main topics Comments Evaluation approaches (table III_EvalImpl_Appr) 20 Country (MS) admin. /polit. Unit Ref. Eval Evaluation approach If other, please specify III. 2 Evaluation approaches Specifications If Impact Evaluation which design? If 'Impact other', please specify (not more than 510 characters!) Evaluation approach general comments Data collection method (table III_EvalImpl_Meth) III. 3 Data collection methods 53 Country (MS) admin. /polit. Unit Ref. Eval Data collection methods If other, please specify General comments Findings of the evaluation (table III_EvalImpl_Find) III. 4 Findings reported Basic info on content and reference 200 Country (MS) admin. /polit. Unit ID Finding Ref. Eval Describe or specify finding Provide sources and references (short) Preferably the most relevant reference on TOP please! May 2015 page 73

74 III. 4 Findings reported Evaluation criteria and policy field(s) relating to Finding refers (1st) to Key Eval Criterion Finding refers (1st) to Policy Field Finding refers (2nd) to Key Eval Criterion Finding refers (2nd) to Policy Field Finding refers (3rd) to Key Eval Criterion Finding refers (3rd) to Policy Field General comments Findings by Intervention types (table IV_EvalFind_IntTpe) IV. 1 Findings & Interventiontypes Assignments and Sources 188 Country (MS) Ref. Eval Ref. Findg Types of interventions Additional sources used (if not already listed in this very inventory) and bibliographic information Comments Findings by target group (table IV_EvalFind_TrgtGrp) IV. 2 Findings & Target groups Assignments and Sources 250 Country (MS) Ref. Eval Ref. Findg Types of Target Addressees Sub type of Addressee Additional sources used (if not already listed in this very inventory) and bibliographic information Comments page 74 May 2015

75 3.2 The Country Reports Box 5. Template for the Phase 2 Country Reports Table of contents: Country Report for XXX all Priority Axes Executive summary Introduction 1. Context, general policy background and ESF investment in all ESF Priorities 1.1 Economic and social challenges in the Member State assessing Europe 2020 indicators (such as GDP development, employment and unemployment figures, education, lifelong learning, poverty index etc.) and strategic response 1.2 Investment priorities in the MS (in relation to challenges and EU level strategies) 1.3 Summary of main types of interventions funded across ESF Priorities and main target groups 2. Financial volumes, outputs, results and findings by ESF Priority 2.1 Increasing adaptability Financial absorption (SFC2007 and AIR 2014) Outputs (SFC2007 and AIR 2014) Results (Programme specific indicators and other results gathered in the framework of the thematic ex post evaluation) Extent to which resources were used Effectiveness Efficiency/cost-effectiveness Socio-economic impact CAV Gender sensitivity Sustainability 2.2 Enhancing access and sustainable integration into employment See Reinforcing social inclusion See Enhancing human capital See Promoting partnerships See Strengthening institutional capacity See May 2015 page 75

76 3. Conclusions 3.1 resources used 3.2 effectiveness 3.3 efficiency 3.4 CAV 3.5 sustainability 3.6 gender sensitivity 3.7 socio economic impact 4. Lessons learned 4.1 What are the key lessons in terms of policy choices - were the policy choices made appropriate? What actions were missing? 4.2 What are the key lessons in terms of target groups - were the appropriate target groups chosen? 4.3 What are the key lessons in terms of the appropriate programming? 4.4 What are the key lessons in terms of the effective implementation - were the right types of activities funded? Were the target groups chosen prioritised in the implementation? Has the response to the challenges posed by the crisis been adequate? Were the delivery mechanisms functioning appropriately? How could the effectiveness of activities funded by ESF been increased? 4.5 What are the key lessons drawn in terms of the robustness of the monitoring (such as setting of appropriate output and result indicators, any challenges in the regular monitoring and reporting on the indicators chosen)? 4.6 What are the key lessons drawn in terms of the robustness of the evaluation systems (such as availability of data measuring progress on output, result indicators, the choice of evaluations conducted and not conducted, the use of impact evaluation methods such as counterfactuals etc) 5. Good practice 5.1 Description of one evidence based good practice in Member State Annexes: Information sources page 76 May 2015

77 3.3 The Thematic Synthesis Reports Box 6. Draft Thematic Synthesis Report Structure Table of contents Executive Summary 1. Synthesis 1.1 Scope and aims of evaluation, links to other evaluations 2. Promoting partnerships/capacity building: context analysis 3. ESF Programming: Promoting partnerships/capacity building 3.1 Focus on PP/CB policy priorities at MS level 3.2 Target groups addressed 3.3 Objectives, types of interventions and clustering under PP/CB priority 3.4 Contribution of PP/CB priorities to Europe 2020 and CSRs 4. Financial and physical performance 4.1 Financial performance of PP/CB-related OPs/PAs 4.2 Analysis of outputs (EU and MS level) 4.3 Analysis of results (EU and MS level) 5. Evidence from evaluation data 5.1 Sources, approaches and methods 5.2 Evidences 5.3 Main factors influencing interventions, results and impacts 6. In-depth assessment of ESF investment in PP/CB (case studies) 7. Conclusions: implications for future policies May 2015 page 77

78 3.4 The Final Synthesis Report Box 7. Final Report Structure Table of contents Executive Summary Introduction Policy context Aims Methods 1. Adaptability of workers Contextualising ESF The extent to which resources were used Effectiveness Efficiency Socio economic impact Community Added Value Gender sensitivity Sustainability Conclusion and lessons learned 2. Access and Sustainable Integration into Employment The extent to which resources were used Effectiveness Efficiency Socio economic impact Community Added Value Gender sensitivity Sustainability Conclusion and lessons learned 3. Social Inclusion The extent to which resources were used Effectiveness Efficiency Socio economic impact Community Added Value Gender sensitivity Sustainability Conclusion and lessons learned 4. Human Capital The extent to which resources were used Effectiveness Efficiency page 78 May 2015

79 Socio economic impact Community Added Value Gender sensitivity Sustainability Conclusion and lessons learned 5. Promoting Partnerships The extent to which resources were used Effectiveness Efficiency Socio economic impact Community Added Value Gender sensitivity Sustainability Conclusion and lessons learned 6. Strengthening Institutional Capacity The extent to which resources were used Effectiveness Efficiency Socio economic impact Community Added Value Gender sensitivity Sustainability Conclusion and lessons learned 7. Conclusions and lessons learned on the EU level policy on economic and social cohesion policy choices target groups, in particular young people programming implementation monitoring evaluation of future programmes May 2015 page 79

80 4. WORK ORGANISATION This chapter presents the organization and management of the work with regard to the following five aspects: Roles and responsibilities of the team Allocation of time and resources to the project and to the tasks, including the allocation to each expert proposed The timetable and the overview of the workflow The quality assurance measures 4.1 Roles and responsibilities of the team This sub-chapter presents the roles and respective responsibilities of the proposed team, considering also the different roles and responsibilities of the partners in the Consortium. Figure 3 provides an overview of the structure of the work and the team, including sub-task coordinators. Figure 3. Overview of the team structure and work division Metis GmbH leads the Consortium and both Consortium partners, Fondazione Brodolini and Panteia, participate in the core team and take the role of sub-task coordinators. We have set up three teams: the core team, the overview team as support to the core team and the research group, i.e. the national experts. The distribution of work between the core team, the overview team and the research group among the partners is shown in Table 14. We have allocated page 80 May 2015