MLP My career (for partners) A common system for monitoring and evaluation of PES and ESF MA

Transcription

1 MLP My career (for partners) A common system for monitoring and evaluation of PES and ESF MA

2 Practice reported in the e-survey of the study about the ESF monitoring and evaluation systems the Flemish case The Flemish partner will present their approach to gear the ESF operations towards better results through monitoring and evaluation. In particular the integrated system of monitoring ALMPs implemented by the Flemish PES and ESF will be presented.

3 ALMP There are three main categories of ALMP (active labour market policies): Public employment services, such as job centres and labour exchanges, help the unemployed improve their job search effort by disseminating information on vacancies and by providing assistance with interview skills and writing a curriculum vitae. => PES and ESF Training schemes, such as classes and apprenticeships, help the unemployed improve their vocational skills and hence increase their employability. => PES and ESF Employment subsidies, either in the public or private sector, directly create jobs for the unemployed. These are typically short-term measures which are designed to allow the unemployed to build up work experience and prevent skill atrophy. (not in PES nor in ESF).

4 Organogram Minister of Work Minister of Finance Social and Economic Council of Flanders (SERV) Policy council Inspection of Finance AA My career Department of Work and Social Economy ESF CA / ESF MA Flemish Employment and Vocational Training Service (VDAB) PES SYNTRA Flanders

5 My Career over the years VDAB PES digitalisation process 1981 Simona From filing cards to digital system / unemployedvacancies 1994 WIS (work and information system) Computer access vacancy kiosks ESF progr. periods VDAB projects only AMI (labour market info system) Still existing Oracle dbase / Growing need for analysis 2001 My VDAB Online registration, virtual space for resumes, vacancy posting, / First steps of automatic matching Push & Pull CVS (client following system) Evolution of services provided on the Labour market PES + Service providers (counsellors / trainers / coaches / ) Projects for unemployed and early school leaving Progressively also projects for workers 2013 My Career (for partners) All projects with direct participants Cofinancing VDAB - ESF (1 FTE) Cofinancing VDAB - ESF (1 FTE)

6 Participant Vacancies Activities Service provider Enriched data-connections Job seekers PES- Counsellors Employees Employers Ministry of Education Public social welfare offices Third party service providers MY CAREER FOR PARTNERS Unique number Characteristics Competences <output indicators> Employers Temporary agencies Visit Councelling sessions Coaching Internships Work experience Training courses Apprenticeship Address Name of the counsellor Employment status (Dimona) Diploma (LED) <results indicators> ESF monitoring and evaluation Historical data

7 My Career and the OP OP Pr TO (IP) Target group My Career PES competence P1 8 (1,2,3) Unempl/inact Yes Yes P2 10 (1,3) Students/worker Yes No P3 9 (1,2,5) Unempl/inact Yes Partial P4 8 (5,7) (workers) No* No P5 inno tn All No Yes P6 TA No No * Not a direct target group

8 Flow of the set up OP ? Progr. period Regulation Input in EGESIF Operational Programme Translation of regulation and guidance in MLP system, setting up the system (core team that can be extended with members of ESF and VDAB study services) Closure (staff service ESF MA and VDAB) Annual Per call Per project Annual report Programming the reporting requirements (core team) Call Writing the project fiche, containing registration obligations for the beneficiary (specific per call). Drawing up the programming requirements for the monthly monitoring tables and for the interim of final reporting table. (interaction btw head of service MA, call manager, Staff service ESF MA and the VDAB) Monthly verification of data before sending to beneficiary (call manager, Staff ESF and VDAB) AIR (staff ESF MA) Montab to beneficiary (enhanced with key indicators) Monthly verification by the beneficiary and adding the non MLP indicators (montab) (Beneficiary call manager - Staff ESF) Consolidation per call Process (or impact) evaluation (head of service, staff ESF MA, call manager) Saltab (Beneficiary call- and project manager staff ESF)

9 Examples Printscreen for beneficiaries Inputfile (direct or xml) Montab and saltab (key figures) Note : montab (monitoring table) monthly communication btw ESF MA and beneficiary Saltab (table with data agreed upon) final table providing the participant and indicator list agreed upon by both MA and beneficiary

10 How many now? Number of internal users ESF MA 52 Number of external users ESF MA (number of addressees of 519 (17/11/2017) mails sent on a monthly basis) Number of projects 260 (17/11/2017) # Policy budget supported by the system euro total cost Average number of participants a year

11 Advantages for ESF MA Advantage for MA when setting up the OP Historical data for Target setting when writing the OP Advantage for MA when evaluating the priorities Non ESF data available for evaluation purposes (control groups): Low cost No problems with privacy Advantage for MA when setting up the calls Setting up standard scales for unit costs, even for activities new to ESF Insight in the total pathway of the final target groups Advantage for MA in the follow up of the projects Monthly monitoring of the advancement of the projects => possibility to intervene Possibility to make agreements with PES to send participants to the activities Advantage for beneficiaries and ESF MA with respect to the indicators Most indicators are delivered automatically based on the national number and thus do not have to be asked to the beneficiary, nor to the participant

12 Advantages for PES Transparency for beneficiaries Same system to work in regardless of the instance financing the calls Transparency for participants All data in one place regardless of the instance financing the calls Transparency for PES counsellors All activities by ESF in the place which is familiar to them Advantage for PES management and policy makers Overall reporting at Flemish level of all activities with job seekers

13 Non ESF data available for evaluation purposes Advantages : amount of data available no privacy issues low cost no waste of time Having non-esf data is not the only thing that matters for impact evaluation: Non-ESF and ESF funded participants may be participating in exactly the same (type of) interventions => the impact would then obviously be zero Only when ESF funded actions are sufficiently different and/or less/more costly than what people would otherwise get is there an interesting evaluation question, Variables from MLP are not sufficient for conducting robust propensity score matching designs But they can produce a good RDD design And they can help figure out if there are pathway types linked to certain profiles

14 Added value of ESF common indicators Added value of common output and result indicators Mainly comparability at EU level When setting up the OP, the common indicators were the beacons for setting up the specific indicators, closer to the Flemish system Those specific indicators are being used in all calls as beacon for orienting the actions Why specific indicators? Necessity to set up realistic targets based on historical data For some priorities (mostly workerrelated), there are no common indicators available Direct comparability was higher in because System of EU was copying the Flemish system/policies Both target orientations were action based (contextual issue) E.g. FL target overrepresentation of target groups in actions (PES) E.g. FL target % of unemployed leaving unemployment 12 months after entering PES

15 Benchmarking performance using ESF common indicators Difficult, given the fact that, when issuing calls or tenders, both PES and ESF MA want to/have to allow for new organisations to participate. Doing so could easily be seen as a discriminating action. However output (and result) indicators are always used in the set-up of new (similar) calls -> the importance of learning based on process evaluation Sometimes when extending the duration/the scope of calls the performance of beneficiaries on both output (or results) is taken into acccount

16 Challenges with ESF common indicators Most common indicators were already available in the Flemish system It is good therefore that referral can be made to national definitions for migrants and disabled people Problems are that some indicators are not available, like household indicators (to be asked to the beneficiary) My career is not set up for audit purposes => ESF MA has to set up systems to freeze data at a given moment (saltabs) Problems also because PES is less concerned about the employment status at a given moment (vision of a transitional labour market): The list of types of participants is more diversified Doesn t always coincide with the three statusses that condition the system of the EU indicators

17 Questions? Also at Tine Stryckers (call manager ESF MA) Elien Vranckx (VDAB study services) Patrick De Mooter (staff ESF MA) Benedict Wauters (evaluation ESF MA) Veerle Moens (programme manager ESF MA)