What can a CIE tell us about the origins of negative treatment effects of a training programme

Transcription

1 Expert Workshop: Local statistics for decision-making on well-being and vulnerability Livorno, What can a CIE tell us about the origins of negative treatment effects of a training programme Miroslav Štefánik miroslav.stefanik(at)savba.sk

2 Motivation Data availability (Official registers of unemployed and Social insurance data) Critique of the training programme Counterfactual impact evaluation studies come in a wide stream of literature

3 Description of the training programme Training activities are implemented (also subcontracted) by regional offices of the Centre of Labour, Social Affairs and Family (COLSAF) The content of the trainings provided is widely defined (increasing employability) All registered unemployed are eligible to participate in the training, capacities are very limited Evaluation period Data allow us to follow the participants 24 months after the training From 1/2007-4/2008 trainings were designed and organised (also subcontracted) by regional PES offices From 5/2008 training providers are selected by a public procurement at the national level. Content of the trainings is decided based on the requests from regional offices (general skills). Bratislava remains out of this mechanism. In 7/2010 new national projects are introduced with a rapid decline in numbers of participants and the accessibility of the trainings

4 Periods of implementation Source: Database on registered unemployed provided by COLSAF

5 Outcome indicators Working income constructed from the assessed base of social insurance payments at the end of each month Employment constructed using the information about the registration for social insurance payments (for each month)

6 Propensity score matching Probit model to predict the propensity score variable (PSV) log Pr( I- Participation in the training(0,1) Ii 1 Xi) X- vector of observed characteristics (all information available from the database): Individual characteristics (gender, age, region, level and field of education,...) Previous participation in other ALMM Pre-treatment unemployment (date of entering, length and no. of previous unemployments,...) Previous working experiences (days of previous working experience, economic sector and occupation,...) Family background (kids, marital status,...) Declared skills (PC skills, languages,...) 0 2 X

7 PSM model applied: 1:1 matching of the nearest neighbour Replacement was allowed Exact matching/subgrouping based on regional offices Two matching variables PSV The date of entering unemployment

8 Sensitivity analysis: PSM using caliper radius ( ) Marginal improvement in balance 46,6% of participants were excluded, leaving us with OLS estimation

9 Assumptions behind ex-post (control group selection) selection Unconfoundedness assumption After ensuring the balance on observable characteristics, non-participants outcomes have the same distribution that participants would have experienced if they had not participated. There are no unobservable characteristics influencing the outcome. Assumption of common support An area of common support exists=characteristics of participants and non-participants overlap. For each analysed participant, there is a non-participant which is sufficiently similar.

10 Distribution of the PSV before matching Graphs by p46

11 PSV Balance achievement N Log likelihood ,2 Prob > chi2 0,0000 Pseudo R2 0,5574 Sensitivity 28,24% Specificity 99,75% Positive predictive value 68,54% Negative predictive value 98,65% Correctly classified 98,42% Pr(p46) Graphs by p46

12 Mean Proportion in % Control group Participants Database Balance improvement N mean (date of entry) ,84% mean(length of previous u) 511,36 530,02 312,59 91,42% mean(age) 38, , , ,92% mean(psvar) 0, , , ,32% Male 45,22 47,97 54,12 55,28% NP 9,79 12,48 36,42 88,76% Single 37,39 37,7 50,77 97,63% Previous occupation ISCO 1 15,57 17,59 30,70 84,59% ISCO 2 2,9 2,93 1,58 97,78% ISCO 3 4,8 4,73 3,18 95,48% ISCO 4 14,07 13,99 7,62 98,74% ISCO 5 8,04 7,6 4,7 84,83% ISCO 6 13,36 13,7 11,69 83,08% ISCO 7 0,58 0,62 0,93 87,10% ISCO 8 15,42 15,12 13,21 84,29% ISCO 9 15,16 14,37 17,23 72,38% Foreign language 75,85 76,02 66,19 98,27% Graduate 2,76 2,73 2,54 84,21% Level of highest education achieved No elementary 0,09 0,08 0,51 97,67% Elementary 18,53 19,15 24,16 87,62% Lower socondary 0,43 0,43 1,07 100,00% Vocational secondary 26,11 26,11 28,21 100,00% Upper socondary vocational 39,29 37,72 30,05 79,53% Upper secondary general 5,46 5,36 4,12 91,94% First stage university 0,44 0,44 0,99 100,00% Second stage university 21,09 20,3 17,24 74,18% Ph.D. 0,02 0,03 0,14 90,91% Field of highest education achieved Field of education 1 19,26 19,94 26,26 89,24% Field of education 2 0,34 0,53 0,64-72,73% Field of education 3 24,5 24,17 21,94 85,20% Field of education 4 17,3 17,15 15,68 89,80% Field of education 5 6,23 6,31 5,18 92,92% Field of education 6 0,79 1,04 1,51 46,81% Field of education 7 20,36 20,59 19,8 70,89% Field of education 8 9,14 8,58 7,57 44,55%

13 Date of entering unemployment Control Treatment 5.0e jul jan jul jan jul jan jul jan2014 Graphs by p46

14 Imputing the date of end of treatment for the control group Participants Entering unemployment (Balanced) End of the treatment Control group Number of days until the end of training Entering unemployment (Balanced) Start of the reference period Imputed end of the treatment

15 Results ATT on earnings: Comparison of methods OLS PSM NN PSM Caliper Month Coef. S.E. p. N Coef. S.E. p N Coef. S.E. p N 6-101,5 2,87 0, ,25 2,36 0, ,87 2,0666 0, ,64 3,19 0, ,44 2,96 0, ,80 2,4935 0, ,02 2,66 0, ,03 3,19 0, ,71 2,9141 0, ,77 3,63 0, ,2 3,54 0, ,07 6,97 0,

16 PSM estimations by period of implementation (Employment)

17 PSM estimations by period of implementation (Employment)

18 PSM estimations by period of implementation (Employment)

19 PSM estimations by period of implementation (Employment)

20 PSM estimations by period of implementation (Employment)

21 PSM estimations by period of implementation (Employment)

22 PSM estimations by period of implementation (Employment)

23 PSM estimations by period of implementation (Earnings)

24 PSM estimations by period of implementation (Earnings)

25 PSM estimations by period of implementation (Earnings)

26 PSM estimations by period of implementation (Earnings)

27 PSM estimations by period of implementation (Earnings)

28 PSM estimations by period of implementation (Earnings)

29 PSM estimations by period of implementation (Earnings)

30 Findings Evaluated training measure seems to have initial negative impact on participants chances to get employment and on their income The length of this initial (negative) impact varies between periods of implementation Positive impact of the measure is observed after 24 months (on average). In some periods of implementation positive impact is observable even earlier, in some periods there is none positive impact observable. Provided trainings seem to be less effective during and after the crisis. The way of implementation also plays a role in shaping the impact of the measure.

31 Thank you for your attention Miroslav Štefánik miroslav.stefanik(at)savba.sk