Data Fitness Initiative for Counterfactual Impact Evaluation (CIE)

Transcription

1 Data Fitness Initiative for Counterfactual Impact Evaluation (CIE) Dear All, As previously announced, DG Employment, Social Affairs and Inclusion (DG EMPL) and the Centre for Research on Impact Evaluation (CRIE) are launching the 2018 edition of the Data Fitness Initiative for Counterfactual Impact Evaluation (CIE) with the aim to select suitable ESF interventions for which CRIE will conduct CIE studies. CRIE is a project jointly established by DG EMPL and DG Joint Research Centre (DG JRC), to provide scientific expertise and methodological support to the Commission and Member States for the evaluation of the impact of ESF funded interventions managed by DG EMPL. The general objective of the Data Fitness Initiative for CIE (D4C) is to promote awareness among ESF Managing Authorities (MAs) on the requirements of datasets to carry out CIEs of ESF-funded interventions, and thereby to increase the use of CIEs 1 and the existing scientific evidence regarding the impact of ESF type interventions. As in 2017, the 2018 Data Fitness Initiative for CIE takes the form of a two-type support to ESF MAs: 1. accompany 'data-ready' MAs with CIE: D4C-R (ready); 2. accompany other prospective applicants in getting 'data-ready' for future CIE evaluations: D4C-F (future). With the former type, ESF MAs are invited to provide CRIE with the data that are needed for carrying out a well-designed CIE of a specific ESF funded intervention, while with the second one they are selected in 2018 according to the proposed interventions to be evaluated and they will be guided by CRIE through the year in the data collection and/or data linkage procedures to be ready for a CIE study in The CIE studies carried out by CRIE can be used by the MAs to feed into the evaluations foreseen in their Evaluation Plans so as to respect the regulatory requirements on the evaluation of impacts (Common Provisions Regulation 1303/2013 Art. 54 and Art 56.) and on the evaluation of the Youth Guarantee (ESF Regulation 1304/2013 Art. 19). The CIEs will be performed by CRIE at no cost for the Member States. D4C-R: CRIE will select the most suitable applicants on the basis of both quality of programme design, data availability and involvement in the previous rounds of Data Fitness Initiative. Priority will be given to the projects of high policy relevance (labour market interventions, especially related to the implementation of Youth Guarantee). Detailed information regarding the intervention will have to be provided to CRIE along with the dataset and its associated data documentation and 1 For more information about CIEs, please see Annex III.

2 codebook,. In particular, the data provided need to include information on the treated units of the intervention under scrutiny as well as data on a comparison group. D4C-F: CRIE will select the most suitable applicants according to programme design and description of the intervention as well as the description of the data sources potentially available for the future CRIE. Priority will be given to applicants who are in the condition to get 'data-ready' in 2019 for CIE evaluations to be carried out in Please find in appendix more details on the terms of participation, the modalities for selecting candidates and a checklist of required documents for both D4C-R and D4C-F. How to participate In order to establish collaboration with DG JRC in the framework of the project CRIE for a CIE study, MAs are invited to provide information regarding: - the ESF intervention to be evaluated (for both D4C-R and D4C-F); - the datasets related to the ESF/YEI intervention to be used for the evaluation (D4C-R only); The selection of MAs participating in this initiative is organized in two phases: Phase 1 ( December 9, 2017 February 4, 2018) Applicants are invited to: - fill in the online questionnaire: in order to provide information about the ESF-intervention to be evaluated and the data to be used for the evaluation; - provide the ESF intervention documentation listed in Annex I by uploading it to the questionnaire before submission. Phase 2 (February 5 March 2, 2018) Proposals will be ranked separately 2 for D4C-R and D4C-F according to the criteria listed in Annex II. Applicants of short-listed proposals of both types will be contacted in order to agree upon conditions on how to provide CRIE with the data (D4C-R) and how to manage the joint data collection activities during the year (D4C-F). In addition to this, they will have to provide the documents listed in the checklist in Annex I for Phase 2. For D4C-R, the projects received will be selected to establish a collaboration agreement with DG JRC in the framework of the project CRIE according to the criteria listed in Annex II. 2 Proposals not selected in D4C-R might be considered for D4C-F on behalf of CRIE selection committee.

3 The deadline to submit the final version of the data for DC4-R is 31 March Who can participate The 2018 Data Fitness Initiative for CIE is open to national or regional MAs in charge of ESF/YEI management and/ or ESF evaluation as lead applicants. MAs can participate in this initiative together with: beneficiary institutions implementing ESF/YEI interventions (e.g. institutions carrying out training programs); institutions, research centers, consultancies or agencies involved in data collection; as co-applicants. D4C-R applicants (lead and co-applicants) will be responsible for providing and preparing the datasets for the evaluation. What type of ESF intervention Interventions eligible for the evaluation should consist of any type of ESF/YEI interventions implemented in, or Programming Periods, in the context of national or regional Operational Programmes. What type of data Data used for the evaluation might originate from different data sources (e.g. administrative, survey or monitoring sources). A detailed description of their linkage procedure should be provided in the proposal. Proposals submitted for the CIE study may be based on data already used for previous intervention evaluations. How we will proceed D4C-R: CRIE will select the two most suitable datasets for the implementation of a CIE study in 2018 on the basis of: policy relevance (prioritizing labour market interventions, especially Youth

4 Guarantee) and design of the ESF intervention to be evaluated, data availability and quality, clarity of data documentation and applicability of CIE methods and involvement in the previous rounds of D4C Initiative according to the criteria for Phase 1 and Phase 2 (2.a) listed in Annex II. D4C-F: CRIE will select the most promising interventions in order to guide the MAs in the data collection and/or data linkage procedures being ready for a future CIE evaluation in 2019 according to the criteria for Phase 1 and Phase 2 (2.b) listed in Annex II. The conclusion of the D4C-R projects is planned by the mid November The D4C-F projects are planned to be prepared to be data ready by mid November Successful applicants of D4C-R will establish a Collaboration Agreement with DG JRC 3 that will govern relationships during the subsequent research activities. The collaboration projects should start after signature of agreements and provision of data. The indicative timetable for this initiative includes the following activities: carrying out of CIE (April October 2017) for D4C-R and presentation of the state of play of the projects in the 2018 Community of Practice meeting (June 2018) for both D4C-R and D4C-F. Successful applicants of D4C-R will be invited to participate in: a kick-off meeting to be held at the DG JRC in Ispra (VA), Italy or in the participant s Member State to agree upon terms and modalities of the CIE studies to be carried out. a meeting in the participant s Member State to visit the data holders (national statistical officers, tax authorities and public employment offices among others) to check together the data availability and hence plan the data preparation to carry out the CIE study. Successful applicants of both types will be invited to participate in: the 2018 Community of Practice, to present the D4C-R and D4C-F projects. For additional information, please contact CRIE at: jrc-crie-data@ec.europa.eu. 3 Activities conducted by CRIE will be handled by DG JRC in the context of the collaboration between DG EMPL and DG JRC.

5 Annex I: Checklist of required documents The following table includes the documents required for D4C-R and D4C-F in Phase 1 to provide information regarding the ESF intervention and the datasets at disposal of MAs, and the documents that must be provided in Phase 2 to complete the selection procedure. Applicants can use the table as a checklist in order to verify compliance with all requirements. Phase No. Type Document Specification and content 1 1 D4C-R ESF intervention documentation D4C-F Detailed description of the ESF intervention indicating the institutions involved in the planning and implementation phases, the duration, the aims and the target groups. 2 2.a D4C-R Data codebook Detailed description of the datasets indicating the format and the list of variables with labels. In case of survey data, the documentation needs to include a detailed description of the survey methodology. 2 2.b D4C-F Data sources Detailed description of the potential data sources available for the CIE evaluation. 2 3 D4C-R Declaration A declaration signed by the lead applicant stating the right to use data in accordance with data protection and privacy legal requirements.

6 Annex II: Modalities for selecting candidates In order to select candidates for the two types of support, CRIE will take into account mainly the policy relevance (labour market interventions, especially related to the implementation of Youth Guarantee) and design of the ESF/YEI intervention to be evaluated, data availability, readiness-towork, and quality (D4C-R), clarity of data documentation (D4C-R and D4C-F) and previous involvement in D4C Initiative. These elements will be evaluated in order to assess the potential of the proposals in terms of applicability of CIE methods. The proposals will be ranked separately for D4C- R and D4C-F by CRIE according to the potential of applicability of CIE methods. Phase 1 CRIE will assess the eligible proposals based on questionnaire responses according to the criteria listed below. The maximum rating for each criterion is indicated in parentheses. The maximum total points that a proposal may obtain is 50. Proposals that receive at least 30 points will be considered for Phase 2 and applicants will be contacted by CRIE to agree upon the terms and modalities of agreement. Applicants of D4C-R will be able to use CIRCABC platform to send datasets. ESF intervention (25 points) Relevance of ESF intervention to be evaluated [D4C-R; D4C-F]; Data (25 points) Cross-sectional dimension (population size) [D4C-R; D4C-F]; Quality of the datasets (availability of outcome, treatment and additional control variables, availability of observations belonging to an appropriate group of non-participants) [D4C-R]; Quality of the potential datasets available [D4C-F]; Phase 2 After provision of datasets and documentation required for Phase 2, CRIE will assess the proposals based on the submitted datasets according to the criteria listed below. The maximum total points that a proposal may obtain is 50. Ready-to-Work Data (30 points) [D4C-R] Quality of the dataset:

7 - number and type of outcome variables; - number and type of treatment variables; - variables describing the treatment (selection, duration, etc.); - variables describing characteristics of treated and untreated units (demographic info, socioeconomic background, etc.); - number of relevant variables; Time dimension: - availability of pre-intervention data; - time span of post-intervention period; Clarity of the codebook; Format; Representativeness of the sample; Quality of data (number of missing, etc.). Prospective Data (30 points) [D4C-F] Quality of the potentially available dataset: Clarity of the data description; Feasibility of data collection and data linkage procedures. CIE methods (20 points) Applicability of one or more appropriate CIE methods [D4C-R and D4C-F]. Proposals will be separately ranked considering the sum of points obtained in Phase 1 and Phase 2. The maximum total points that a proposal may obtain in the two phases is 100. Priority will be given to: - ESF/YEI interventions related to the implementation of Youth Guarantee; - Datasets which have not been used for evaluation purposes of the same ESF intervention; - Data from administrative data sources;

8 - Applicants who did not receive CRIE support under the D4C-R in the previous periods (if applying for D4C-R); - Applicants who did not receive CRIE support under D4C-F in the previous periods (if applying for D4C-F).

9 Annex III: What is CIE? And what is it for? CIEs may provide good quality evidence of the net effects (impacts) of interventions, including ESF funded interventions. They only do so however, if they are well planned and executed appropriately. The main methodological distinction in CIEs is between evaluation designs that are experimental and those that are quasi-experimental. The experimental approach is commonly referred to as the randomised control trial and sometimes also as social experimentation. In an experimental approach, the persons supported by the ESF funded intervention would be selected randomly from a potential population of eligible participants to the intervention. The random assignment assures that, on average, those exposed to the program (treatment group) and those who are not (control group) are similar and thus, the difference on a specific result between the two groups can be attributed to the intervention. Quasi-experimental methods such as: difference-in-differences (DiD); regression discontinuity design (RDD); instrumental variables (IV) and propensity score matching (PSM) are different approaches to mimic experimental methods, in such a way that participants and not participants in the intervention are, before benefiting from the ESF funded intervention, similar and the difference in the result between the two groups can be attributed to the intervention. The application of CIE methods requires the availability of data related to both units taking part in the intervention under analysis, defined as treated, and units not participating, defined as untreated or control units. Not participants may include both eligible and non eligible units. More detailed information about CIE methods and their implementation can be found in the publication Design and Commissioning of Counterfactual Impact Evaluations: A Practical Guidance for ESF Managing Authorities, at: on the CRIE web-site, at: and on CRIE YouTube channel, at: