Expert Workshop: Local statistics for decision-making on well-being and vulnerability Livorno, 17.6. 2015 What can a CIE tell us about the origins of negative treatment effects of a training programme Miroslav Štefánik miroslav.stefanik(at)savba.sk
Motivation Data availability (Official registers of unemployed and Social insurance data) Critique of the training programme Counterfactual impact evaluation studies come in a wide stream of literature
Description of the training programme Training activities are implemented (also subcontracted) by regional offices of the Centre of Labour, Social Affairs and Family (COLSAF) The content of the trainings provided is widely defined (increasing employability) All registered unemployed are eligible to participate in the training, capacities are very limited Evaluation period 2007-2013 Data allow us to follow the participants 24 months after the training From 1/2007-4/2008 trainings were designed and organised (also subcontracted) by regional PES offices From 5/2008 training providers are selected by a public procurement at the national level. Content of the trainings is decided based on the requests from regional offices (general skills). Bratislava remains out of this mechanism. In 7/2010 new national projects are introduced with a rapid decline in numbers of participants and the accessibility of the trainings
Periods of implementation Source: Database on registered unemployed provided by COLSAF
Outcome indicators Working income constructed from the assessed base of social insurance payments at the end of each month Employment constructed using the information about the registration for social insurance payments (for each month)
Propensity score matching Probit model to predict the propensity score variable (PSV) log Pr( I- Participation in the training(0,1) Ii 1 Xi) X- vector of observed characteristics (all information available from the database): Individual characteristics (gender, age, region, level and field of education,...) Previous participation in other ALMM Pre-treatment unemployment (date of entering, length and no. of previous unemployments,...) Previous working experiences (days of previous working experience, economic sector and occupation,...) Family background (kids, marital status,...) Declared skills (PC skills, languages,...) 0 2 X
PSM model applied: 1:1 matching of the nearest neighbour Replacement was allowed Exact matching/subgrouping based on regional offices Two matching variables PSV The date of entering unemployment
Sensitivity analysis: PSM using caliper radius (0.00075) Marginal improvement in balance 46,6% of participants were excluded, leaving us with 21 288 OLS estimation
Assumptions behind ex-post (control group selection) selection Unconfoundedness assumption After ensuring the balance on observable characteristics, non-participants outcomes have the same distribution that participants would have experienced if they had not participated. There are no unobservable characteristics influencing the outcome. Assumption of common support An area of common support exists=characteristics of participants and non-participants overlap. For each analysed participant, there is a non-participant which is sufficiently similar.
Distribution of the PSV before matching 0 1 0.2.4.6.8 1 Graphs by p46
PSV Balance achievement N 1 758 123 Log likelihood 181 862,2 Prob > chi2 0,0000 Pseudo R2 0,5574 Sensitivity 28,24% Specificity 99,75% Positive predictive value 68,54% Negative predictive value 98,65% Correctly classified 98,42% 0 1 0 1 2 3 0.5 1 0.5 1 Pr(p46) Graphs by p46
Mean Proportion in % Control group Participants Database Balance improvement N 32.651 32.651 2.354.850 mean (date of entry) 25.12.08 26.12.08 2.9.10 99,84% mean(length of previous u) 511,36 530,02 312,59 91,42% mean(age) 38,49674 38,32553 34,95795 94,92% mean(psvar) 0,4148981 0,4173019 0,0622178 99,32% Male 45,22 47,97 54,12 55,28% NP 9,79 12,48 36,42 88,76% Single 37,39 37,7 50,77 97,63% Previous occupation ISCO 1 15,57 17,59 30,70 84,59% ISCO 2 2,9 2,93 1,58 97,78% ISCO 3 4,8 4,73 3,18 95,48% ISCO 4 14,07 13,99 7,62 98,74% ISCO 5 8,04 7,6 4,7 84,83% ISCO 6 13,36 13,7 11,69 83,08% ISCO 7 0,58 0,62 0,93 87,10% ISCO 8 15,42 15,12 13,21 84,29% ISCO 9 15,16 14,37 17,23 72,38% Foreign language 75,85 76,02 66,19 98,27% Graduate 2,76 2,73 2,54 84,21% Level of highest education achieved No elementary 0,09 0,08 0,51 97,67% Elementary 18,53 19,15 24,16 87,62% Lower socondary 0,43 0,43 1,07 100,00% Vocational secondary 26,11 26,11 28,21 100,00% Upper socondary vocational 39,29 37,72 30,05 79,53% Upper secondary general 5,46 5,36 4,12 91,94% First stage university 0,44 0,44 0,99 100,00% Second stage university 21,09 20,3 17,24 74,18% Ph.D. 0,02 0,03 0,14 90,91% Field of highest education achieved Field of education 1 19,26 19,94 26,26 89,24% Field of education 2 0,34 0,53 0,64-72,73% Field of education 3 24,5 24,17 21,94 85,20% Field of education 4 17,3 17,15 15,68 89,80% Field of education 5 6,23 6,31 5,18 92,92% Field of education 6 0,79 1,04 1,51 46,81% Field of education 7 20,36 20,59 19,8 70,89% Field of education 8 9,14 8,58 7,57 44,55%
Date of entering unemployment Control Treatment 5.0e-04 0.001.0015 01jul2006 01jan2009 01jul2011 01jan2014 01jul2006 01jan2009 01jul2011 01jan2014 Graphs by p46
Imputing the date of end of treatment for the control group Participants Entering unemployment (Balanced) End of the treatment Control group Number of days until the end of training Entering unemployment (Balanced) Start of the reference period Imputed end of the treatment
Results ATT on earnings: Comparison of methods OLS PSM NN PSM Caliper Month Coef. S.E. p. N Coef. S.E. p N Coef. S.E. p N 6-101,5 2,87 0,000 1757898-20,25 2,36 0,000 60168-49,87 2,0666 0,000 41380 12-82,64 3,19 0,000 1757805-16,44 2,96 0,000 59889-38,80 2,4935 0,000 41380 18-29,02 2,66 0,000 1757296-3,03 3,19 0,3433 58907-25,71 2,9141 0,000 41380 24 30,77 3,63 0,000 1756729 13,2 3,54 0,0002 57802-7,07 6,97 0,311 41380
PSM estimations by period of implementation (Employment)
PSM estimations by period of implementation (Employment)
PSM estimations by period of implementation (Employment)
PSM estimations by period of implementation (Employment)
PSM estimations by period of implementation (Employment)
PSM estimations by period of implementation (Employment)
PSM estimations by period of implementation (Employment)
PSM estimations by period of implementation (Earnings)
PSM estimations by period of implementation (Earnings)
PSM estimations by period of implementation (Earnings)
PSM estimations by period of implementation (Earnings)
PSM estimations by period of implementation (Earnings)
PSM estimations by period of implementation (Earnings)
PSM estimations by period of implementation (Earnings)
Findings Evaluated training measure seems to have initial negative impact on participants chances to get employment and on their income The length of this initial (negative) impact varies between periods of implementation Positive impact of the measure is observed after 24 months (on average). In some periods of implementation positive impact is observable even earlier, in some periods there is none positive impact observable. Provided trainings seem to be less effective during and after the crisis. The way of implementation also plays a role in shaping the impact of the measure.
Thank you for your attention Miroslav Štefánik miroslav.stefanik(at)savba.sk