THE EVALUATION OF VOCATIONAL TRAINING: A COMPARATIVE ANALYSIS..

Transcription

1 Ref.# 100 THE EVALUATION OF VOCATIONAL TRAINING: A COMPARATIVE ANALYSIS.. Eduardo Luís Campos Soares Tomé Centro de Investigação em Economia Europeia e Internacional (CEDIN) Instituto Superior de Economia e Gestão (ISEG) Universidade Técnica de Lisboa (UTL). Rua Miguel Lupi, Lisboa. Telephone: Fax: E mail: eduardo.campos.tome@clix.pt

2 ABSTRACT In this paper I will try to compare the evaluation methods of vocational training public programs. I will study the American case, the other national cases (mostly OECD countries), and the European Union case. I will split the exposition in 8 items: 1. Theory and Methods; 2. The American Case; 3 The Other National Governments; 4 The European Union Case; 5 Explications to a strange situation; 6 Implications; 7 Conclusions; 8 Note on the Portuguese Case. As a conclusion, I will suggest that OECD countries and specially the EU should follow the American trend, and make impact microeconometric studies, in order to assess the training effects of those programs; doing that, it will be possible to conduct better policies, and to spend public money in better ways. Additionally I also will try to explain why is it that Americans are much more advanced in those studies than the EU and other countries; the reasons will be linked to scientific history and to the administrative actions. Finally, I will write some ideas about the Portuguese Case (my country), which is a fine example of the problem I would like to enlighten. 1

3 1.Theories and Methods. A. Theories. In Economics, Vocational Training is studied as a chapter of Human Resources economics. Manuals on the subject are few ( see CHAPMAN (1993), ASHENFELTER and LALONDE (1996), HECKMAN LALONDE and SMITH (1999) ), although many papers are written on it every year (about papers and books are nowadays listed on the subject at ECONLIT database). Great economists (Adam Smith (PAUL (1989, pp.58-9)), Marshall (PAUL (1989 pp.59-60)), Stuart Mill (SHACKLETON (1995 pp.10)), Pigou (STEVENS (1996 pp.21)), Keynes (KEYNES (1990 pp.191)) stressed the importance of Vocational Training and qualifications in economic life. But nowadays, the most important theory that is used to study Vocational Training is Human Capital Theory (HCT), which was originated by studies of American scholars from the late fifties to the early seventies (Jacob Mincer (MINCER (1958, 1962, 1974)), Walter Oi (OI (1962)), Ben Porath (BEN PORATH (1967)), and Gary Becker (that received the Nobel Prize on Economics in 1992) (BECKER (1962, 1976, 1980))). Synthetically, those studies stress the importance of the non homogeneity of labour, and the positive impact of some human characteristics (school levels, on job training, ability, health) in the success of each individual (and the organization where he works) in the labour market, mainly in what concerns wages, and employment. Also, this studies emphasize the role of firms as providers of training, bearing in mind the direct link between wages and productivity, and the possibility of biding contracts between the worker and the firm; in this subject it is very important to analyse the way the training funding is shared between workers and firms. Some other complementary theories help to understand the general findings of the HCT, by analysing the market failures and equity problems that may arise from the normal operations of the labour markets: - the discrimination theory (MCCONNEL and BRUE (1989 pp.104-6), PAUL (1989 pp.72-80), SPENCE (1975)), points to fact that people give signals that enable firms to screen them; 2

4 consequently, some people (the social excluded minorities) will be usually wrongly judged, (because they give negative signs) and receive less training opportunities, and less rewards than the ordinary worker; - the dual markets theory (PAUL (1989 pp.80-90), CHAPMAN (1993 pp.55-8), REYNOLDS MASTERS MOZER (1986 pp.165-6)), analyse the way the labour market is organized and define the existence of two segments, one in which training is valued, the other in which it is not; the first segment is composed of big and oligopolystic firms, the second of small firms in perfect competition; - the Marxist theory (MCCONNEL and BRUE (1989 pp.106-8), BOWLES and GINTIS (1975)) calls for a new society in which training should not be a way of supporting the ruling class. In recent years, some interesting ideas have been developed, like those by Margaret Stevens about transferable training.(stevens (1996, 1999)); in this case, in an imperfect competition environment, there is a market failure, because a poaching firm can reap the benefits of the shared investment made by the firm and the worker. The economic analysis of training addresses the questions of demand and supply, and also the problem of government actions. In what concerns the State, three major ideas are market failures, government failures and equity problems. In short, the State should produce training if a social benefit emerges from that production, and should act in order to solve market failures that reduce training from its due level: this may mean to fund private training operations through subsidies. Additionally, we know that Vocational Training is a complex field, that can also be studied by other sciences: sociology (remember Frederick Taylor, Elton Mayo, the Sociotechnic School and the Post- Taylorism studies about the new technologies ) psychology, management. etc. B. Methods. The economic evaluation of training may be done in very different ways. The easiest way is to study results, that is to answer the question What happened?. Results indicators are the number of supported people and the money spent (the costs incurred) in the public 3

5 programs. These indicators have big advantages, (they are easy to collect and to understand, they can support policies easily), but also a serious problem (they do not tell nothing precise about the economic consequences (and benefits) of those programs). The difficult way to evaluate training is to do impact studies, answering the question: - What difference does the program make to people involved in it?, and thus dealing with the economic consequences of the program. Those impact methods analyse training mainly in a within without way. Experts on the field include James Heckman, the Nobel Prize of Economics in 2000 (See for example: BURTLESS and ORR (1986), LALONDE (1986), BURTLESS (1995), HECKMAN and SMITH (1995), ASHENFELTER and LALONDE (1996), HECKMAN SMITH and LALONDE (1999)). Studies using those methods are very difficult to make, for some very interesting and simple operative reasons: - they involve the comparison of two groups of economic agents (individuals, firms, etc): those that participate and those that do not participate; - participants and non participants should have the same observable (qualifications, age, capital levels, etc.) and non observable (ambition, motivation, social climate, ) characteristics: as this is very difficult to happen, most impact studies face a crucial selection bias problem that threatens to bias the estimates made (for example, very motivated people would obtain more rewards than the average individual from the same course); - in experimental methods, the two samples are drawn from people that wanted to participate, but people are randomly assigned to the two groups like that selection bias should be very limited, but big ethical problems arise; - in non experimental methods participants are compared to similar people of the labour force selection problem arise because, for example, firms tend to give much more training to successful workers, and more ambitious people tend to attend more courses; 4

6 - in general it is better to compare the trajectories of those two samples, before, during and after the program, using panel longitudinal data, with the help of complex microeconometric methods of estimation; - cross sections methods may be used also, but there are some types of calculations that can t be made using only cross sections data; so it is more appropriate to use panel data; - in experimental methods one equation is used, which is named the impact equation; - in non experimental methods, to deal with the selection bias, a second equation is used, named selection rule. - in the left hand side of the impact equation we should introduce the variables that show the program impact (for example, wages, employment, productivity, exports); if we have access to panel data we may use lagged differences between those variables; in cross section studies we do not have this chance; - in the right hand side, of the impact equation we should introduce the variables that account for the program participation; the easiest way of doing this is to use a binary variable which assumes the value of 1 when the economic agent in question participates, and which is zero otherwise; we may also introduce other control observable variables that may effect the evolution of the left hand side variables; these variables may be lagged in panel data, and have to be current variables in cross sections studies; - in the selection rule, the left hand side represents the participation in the program, and the right hand includes control observable variables that may explain the participation in the program; this second equation is used to endogenize the participation: this means that the binary variables included in the impact equation may be adjusted, reducing the selection bias that they usually involve. The most interesting feature of impact methods, is that using simple impact equations, that in a general way are only adjustments of Mincer wages functions, we can obtain important estimates of policy outcomes. And, even more significantly, whit these methods, that compare individuals that were and that were not subject to a stimulus, we can, whit more or less use of mathematics, study many alternative problems in different fields of society: medicine (is a new drug more efficient than the 5

7 previously used), psychology or sociology (is a new method of social or psychological help really performing well), etc. After the estimation of benefits (through those impact methods), we can compare them with costs, obtaining the rates of return of the programs. Reading the best texts on the matter, it can be concluded that there are also many other problems with those methods: - ethical problems people may disagree preventing others from receiving benefits for scientific purposes, as it may happen in experimental studies; - financial, political legal and administrative costs those studies have to be implemented with the public support; a program is only evaluated if the administration as doubts about it; if the preevaluation idea is very firmly good or bad, the administration tends not to evaluate; in addition, in experimental studies the fact that some people are prevented from taking part also has the implication that the local administration often refuses to participate; - technical problems: the program may introduce substitution (participants substitute non participants), dislocation (public support substitutes current private support) and defensive effects (public support generates private spending) in the labour market ; if the program is sufficiently large general equilibrium approaches should be used, because the whole environment may have changed due to the program; correct sampling of participants and non participants should be made; contamination bias should be accounted for, that is it should be checked if non participants were allowed to receive a stimulus similar to the program one; - statistical due to the variety of estimates that may be made with different impact equations, it is important to compare those estimates and to choose between them. Finally, impact methods should not be confused with two other ways of looking at training programs: - the method of comparison before after only answers to the impact problem if the situation is homogeneous or stationary, that is if during the whole period (before, during and after training), the situation of non participants and participants was basically the same; but, nothing guarantees us that this occurs; 6

8 - the input output method, where the investment in training is viewed in a macro economic way, and is studied using multipliers that compute direct, indirect and induced effects of investments. 2. The American Case. Public sponsored training programs made in the USA by the Federal Government, since the 60s, were evaluated at impact levels. The first of all was the Manpower Development Training Act (MDTA) from The first major paper on training impact evaluations, using non experimental methods (ASHENFELTER 1978), was about this program. This study had several characteristics that it is important to point out: - a very short period of time of participation was analysed (the year of 1964) ; - two samples, of participants (MDTA 1964 trainees) and non participants (elements from the Continuous Work History Sample of the American Social Security Administration), were used; so, two data bases were previously made, by the Federal Government; - longitudinal data (from 1959 to 1969) about the members of those two groups were collected; - scientific economic and econometric knowledge was used in a microeconomic sense, by high rated experts. This study was a very important complement to the official reports of execution made about the MDTA operations; in 1969 the MDTA helped persons (namely young dropouts and disadvantaged adults), at a cost of around 1.3 billion 1994 dollars (LALONDE (1995 pp.150)). Ashenfelter found that the program had had some positive impact in the participants wages (from 150 to 500 dollars in the male case, from 300 to 600 in the female case (ASHENFELTER (1978 pp.56)) but he was also sceptical about is own estimates, because he found sample selection bias that undermined them, specially in the female case (ASHENFELTER (1978 pp.57)). Other studies that followed on the MDTA arrived to similar conclusions. 7

9 In 1973, MDTA was followed by the Comprehensive Manpower Training Act (CETA). This was also a huge program (supporting individuals in June 1976, mostly with training but also with employment services, with a cost of dollars per participant (CARD and SULLIVAN 1988 pp.500 and 526)). This program was widely studied with non experimental methods; ten studies were made about two years (1976 and 1977), using two important data sets on the non participants from the Current Population Survey and on the participants from the Continuous Longitudinal Manpower Survey (LALONDE (1995 pp.156-7)). That massive effort give raise to a discussion summarized in another paper (BARNOW (1987)), because different impact equations and selection rules, applied to different sub-samples made by different matching techniques between participants and non participants produced very different impact estimates. Even if the general idea that emerged was that CETA had positive effects in adult women wages, the range of results obtained brought serious doubts about non experimental methods; and in the men and the young cases, some studies obtained positive estimates, others negative ones (LALONDE (1995 pp.157)). At approximately the same time, three important authors made non experimental studies about another program (the National Supported Work Demonstration (NSW); see LALONDE (1986) and FRAKER and MAYNARD (1985)); that program had been studied by a experimental method; the problem appeared because those authors found estimates that were different between themselves and were also different from the experimental ones. As a consequence, the non experimental methodology that was to be used in the Job Training Partnership Act (JTPA), that in 1982 had replaced CETA, was dramatically changed; in the beginning only a non experimental study was expected, using a database on participants (JTLS Job Training Longitudinal Survey) and non participants (SHOW Survey of History of Work). Finally both experimental and non experimental methods were used because it was thought that experimental methods were essential, although not sufficient (URBAN INSTITUTE (1996 pp.17)). 8

10 The JTPA was also a very huge program and in 1995 supported nearly economically disadvantaged persons at a cost of 2.6 billion dollars, and nearly dislocated workers at a cost of ; JTPA provided employment and training services (LALONDE (1995 pp. 152)). The JTPA evaluation implementation faced very big administrative problems because only 16 (in 649) local administration services decided to participate (URBAN INSTITUTE (1996 p.3)). The experience took place between November 1987 and September 1989, involving about individuals, one third of which where non participants. The impact estimates obtained were positive for adults but negative or null for young people (LALONDE (1995 pp.160-1)). In addition, other experimental studies were made in the USA; the most important were the NSW ( ), that was already mentioned, and the Work Welfare Experiments (1982-8), but experimental studies were also made in San Diego, Florida, and California (HECKMAN LALONDE SMITH (1999 pp )). As a result, studies on wages (see for example COUCH (1992), HECKMAN ICHIMURA SMITH TODD (1998)), and employment (see also CARD and SULLIVAN (1988), HAM and LALONDE (1996) were published in highly rated scientific reviews, creating the basis to a new field of knowledge. Some papers were also published about the impact of training on firms (see for example HOLZER e al (1993), BISHOP (1994), BARTEL (1995)). Surveys and comments on these studies were also made by the some authors: LALONDE (1995), MAYNARD (1993), OECD (1998), HECKAMN SMITH LALONDE (1999). In the end, even if sometimes their conclusions were contradictory, these studies created a very important and solid scientific industry : the knowledge, the papers, the methods, and ultimately even the Nobel Prizes, were produced, exist and can be mentioned. A mix of government willingness (in special collecting the data that scientists needed) and economic and econometric tradition (analysing those data), coincided in the same objective; as a result, the USA are on the top of the world in this field of knowledge. 9

11 3. The Other National Governments. Mainly in OECD countries, some sporadic impact studies on training programs began to be made in the 80s, and were incremented in the 90s. Data about those studies can be collected in several surveys: BJORKLUND (1994), BRADLEY (1995), BJORKLUND and REGNER (1996), OECD (1998), HECKMAN LALONDE and SMITH ( 1999). Only three experimental studies were made on the labour market, in Sweden (in 1975 to the employment service of Eskilstuna), Norway (in 1991 to training courses to unemployed people), and the UK (to the Restart program in 1987) (BJORKLUND REGNER (1996 pp )), generally originating positive impact estimates. Non experimental studies were made on wages (in Denmark, Sweden, United Kingdom, Germany, Ireland, Canada, Colombia), employment (in Canada, Norway, Denmark, Sweden, Finland, Austria, Belgium, Ireland, Spain, Italy, United Kingdom, France, Germany, Netherlands), and on the impact of training in firms (in the Netherlands, Canada, Denmark, Sweden, France, United Kingdom, Ireland, Japan). Estimates where also generally positive, specially in what concerns employment and the firm outcomes; on wages the evidence was mixed. In the whole, those national countries are following the USA: results studies are looked at as the art infancy ; those Governments also fell concerned with the effect of the use of public money; but investments made are not as big as the American ones, and, even if it is possible to learn with others experiences, those countries are in a worse position because they do not have the tradition of the Americans. 4. The European Union Case. Founded in 1957, by the Treaty of Rome, the European Social Fund began operating in While preparing my PhD thesis I studied the ESF evolution exhaustively; very shortly, four main administrative phases can be easily defined; basic figures are indicated in Table 1. 10

12 In what concerns evaluations, till 1990, even 1994, almost nothing was made about the evaluation of ESF impacts; for many years, the institution was evaluated by what we called the results (see 1.B); reports of operations results can be found mainly in the General Report on the Commission Activities ( ), the Annual Report on the ESF ( ), the Annual Reports on the Structural Funds ( ). Basically those data respect to the number of supported individuals, and value of support given by the ESF to each Member State; some statistics were also made about the number of employed people after the operations (specially after 1984). Some authors confirm the general idea that evaluation was the minor field in the ESF (GIZARD (1991 pp.504)), that the evaluation tradition of ESF has only just being developed in the early nineties (PLANAS CAZAL (1994 pp.34)); accordingly a generic strategy of evaluation to the ESF was proposed at the time (LINDLEY (1996)), and in 1997, the EU was still establishing indicators to survey the labour market (SCHMID (1997 pp. 425)). And although the evaluation was officially acknowledged as a major field of work since 1990 (see Regulations on the Structural Funds published in 1988, 1993, and 1999) it was only in 1999 that the European Commission published a study in evaluation methods (EU (1999)) in which the problem of impact methodologies was addressed. The main exceptions we know were the studies to the Youth Training Scheme (YTS) in the UK (BRADLEY (1995)) and to young people and long term unemployed in Spain (BALLART (1998)). The YTS studies were dubious in wages terms, but not in employment ones; the Spanish study discovered positive effects in the employment situation of trainees. In the last decade other studies were made that in some way were other exceptions to that general rule: - macroeconomic studies were made to assess the impact of all EU structural actions, specially in Objective 1 regions (whose GDP per capita was below 75% of EU average). The estimates obtained were good (see for example EU 1995 pp.10): EU structural actions were responsible for 0.5 % to 1% of annual growth. But, we should notice that macroeconomic studies on the ESF face the problem of 11

13 collateral effects, meaning that it is not sure if the ESF is additional: it may just represent expenses that would have been made any how; - some studies on the macroeconomic impact of ESF in the employment creation were also made (see for example EU 1999b pp.122), but in this case it was pointed that it was difficult, although essential, to distinguish between the net effect and the gross effect of those investments; - related to the period of , and beginning in 1997, a set of interim evaluations were performed (see EU (1998)); the reports on these evaluations show that mainly evaluators studied administrative and implementation related problems, the access to the ESF, and were at pains dealing with the socio economic consequences of those investments. Finally we should remember that since the seventies doubts about misappropriation and waste of funds in the ESF were common following the Annual Report of the European Cour des Comptes on the Comission activities. It was clear by then (GIZARD (1991 pp.504) that the ESF should pass from a stage in which it surveys the amount correctness and the legality of funds applied, and the number of supported people, to a stage in which it would inquiry the impact of the operations. That is, for many years the EU made a confusion between auditing and evaluation, and between expenditures (that is results) and effects (that is impacts and cost benefits analysis). If we only look to the program legality, we are not able to assess its impacts, because, firstly we only deal with costs, secondly we do not account for the benefits, and we do not search them comparing the program with its environment. In this context, it is very interesting to make three final remarks: - that the ESF appeared basically at the same time of the MDTA, and that it has been contemporary to the other Federal Government programs; - that the ESF appeared at the same time of the HCT; - contrary to other programs mentioned, the ESF was not evaluated at a microeconomic level, applying the HCT. 12

14 5. Explanations to a strange situation. There is a absorption logic that prevails in the ESF. It is guided by broad concerns of international and regional politics, like social cohesion, and board views of market failures and equity. In addition, this logic satisfies itself with results analysis, because ESF is seen as a gift, or a compensation (for the financing of the EU budget) to the recipients: the ESF share in costs is sometimes 75%, sometimes 70%, generally more than 50% of the program costs, so the direct burden the national Governments in its direct funding is very much reduced, and the burden of individuals and firms is almost zero, giving the idea of gratuity. So, in the ESF, apparently there is no cost, because ESF is almost free to those who receive it. As the program costs are effectively seen as benefits, the real benefits, that is, consequent differences in future outcomes, are not really perceived, nor studied, not even checked. The HCT, even with all its question marks, brings training to a cost benefit logic, accounting for the use of public money (but the same logic can be used to firm investments), in a very different way. Subsidies are seen as scarce resources, and costs; future benefits have to be inquired, rates of return should be calculated. This HCT logic is much more national, but it makes sense that it should be used in the ESF by the European administration because European subsidies are scarce European funds. From the start, the scope of the analysis is totally different in the two logics. The position of the national administration is different in the two logics: in the normal case the administration may be easily made responsible (at all levels) for national investments, so it is natural to make impact studies; in the ESF case, the national administration also tend to catch ESF money (and to be accounted for it), and so tend to favour results studies. 6. Implications. Without microeconomic impact evaluations it is much more difficult to direct funds usefully, and it is much more easy to waste funds, even in the training case, where there is a substantial Economic 13

15 Theory to support investments: the ideas mentioned in 1, show that there is theoretic support to make private and public expenditures in training, but also to evaluate them. A fine microeconomic evaluation is essential to define the program effects, and to guide future policies more wisely. 7. Conclusions. The EU should copy, at least in some way, the US federal government actions to evaluate training; in comparison with the other countries, the USA have more tradition, and knowledge; the main problem is that the absorption logic, still dominates the ESF. This means that the ESF participation in program costs should not be seen as gifts to countries (like in this logic), but as money spent in active labour market policies, that should be evaluated, with cost benefits impact analysis, as it happens with the programs referred in 2 and 3. Some reports and information mentioned suggest that although some effort began to be made recently progress in that particular field was limited (see 5, EU ( 1995, 1998, 1999, 1999b) and the Annual Reports on the Structural Funds), and probably will continue to be made slowly (see 5 and Regulations on the Structural Funds for from July 1999 on Evaluation): impact evaluations will continue to be made totally ex-post, a very big period of time will be studied, only after 2006, making it difficult to achieve timed conclusions this is a pity because in the seven years of between 2000 and 2006 it was probably possible to make some studies like those that were described for the MDTA in the USA. It may be surprising or even shocking to find out, but in this field the EU has been lagging behind the USA and other national countries for many years, is still trailing and will probably continue to be in the near future. Academic knowledge and public actions made of the USA the world leaders in this field of knowledge. 8. Note on The Portuguese Case. Due to a complex economic and politic history in the 20 th century, in my country training was virtually non existent before 1986; then Portugal became a member of the EEC and began to receive large sums of ESF money. Even official statistics on training only began to be regularly published in

16 The FSE results on Portugal ( ) were enormous : - the number of helped people as immense (around 9% of the labour force each year on average); - a very important amount of money was spent (ESF total costs were of about 0.8% of the Portuguese GDP, and 2% of the Budget ); almost 70% of that money was absorption : only 30% were funds coming directly from national budgets. - in relative terms, Portugal has been only beaten by Ireland in the absorption of ESF funds. All this means that the ESF impact in my country should be carefully evaluated; the large dimension of the investment means that general equilibrium approaches should be used. But the absorption logic dominates the situation, and Portugal followed the EU trend. During the period a set of 17 interim evaluations was made in I checked these evaluations and concluded that their basic characteristics were very different from the impact evaluations mentioned in this paper (see 1.A, 1.B and 2): - all previous program years where analysed (in a very strictly defined ex-post analysis); - all the structural funds (regional, agricultural, social) in a program were analysed (so evaluators might had to deal with roads, agriculture, education and training in the same study); - although external evaluators made the studies, their scope was internal, and in only one case I found mentioned the control group strategy defined in IB ; - the studies focused mainly on results (money spent, individuals helped), the coherence of the program, and the definition of indicators to be used in the future; - longitudinal data bases of participants and non participants were not mentioned; - we may even find out some doubts about the theoretical models that should be used. During the preparation of my PhD thesis I made several tries on the impact evaluation field in Portugal that I may summarize: - the ESF had a positive impact of the access of training by the workers of small firms, and by small firms in general (the analysis was made with data from the official statistics on Portuguese vocational training programs); this happened although big firms and state agencies were crucial to ESF absorption; 15

17 - the ESF had a positive impact on firm productivity; I found this making a small microeconometric impact study to the period of ; - training has a positive impact on exports, and the ESF helped an increase in exports in the textile sector (I found this using official data on exports and on vocational training operations); - in what concerns wages and employment the findings were much more complex; in general I found positive impacts on wages (when I used officially published sector data) and no effect on employment (probably because some firms substitute skilled workers by unskilled ones); but when I studied the youth case in I found that the employment effect was huge, but not the wages effect; that meant that probably firms used ESF to employ people but only rewarded them in wages terms after several years of practice. 16

18 Bibliography ASHENFELTER, Orley "Estimating the Effect of Training Programs on Earnings" - Review of Economics and Statistics - LX - pp ASHENFELTER, Orley; LaLONDE, Robert Introduction - in Orley Ashenfelter and Robert Lalonde editors: The Economics of Training - volume I - Theory and Measurement - pp ix - xvi - Edward Elgar Publishing Limited - Great Britain. BALLART. Xavier Spanish Evaluation Practice versus Program Evaluation Theory - Evaluation pp BARNOW, Burt The impact of CETA programs on earnings: a review of the literature - Journal of Human Resources - n 2 volume XXII pp BARTEL, Ann Training wages growth and Job performance - Evidence from a company Database - Journal of Labour Economics July - pp BECKER, Gary Investment in Human Capital: A Theoretical Analysis - The Journal of Political Economy volume LXX part 2 - Supplement - October pp BECKER, Gary The Economic Approach to Human Behaviour - The University of Chicago Press - Chicago. BECKER, Gary Investment in Human Capital: Effects on Earnings - in Human Capital : A theoretical and empirical analysis whit special reference to education - Chicago - University of Chicago Press - Chapter II - pp BEN PORATH, Yoram The production of human capital and the life cycle of earnings - Journal of Political Economy August - pp BISHOP, John On Job Training of New Hires - in D. Stern, M. Ritzen (editors) - Market Failures in Training - Berlin - Springer - Verlag - - pp BJORKLUND, Andres Evaluations of Labour Market Policies in Sweden - Industrial Journal of Manpower pp

19 BJORKLUND, Andres; REGNER, Hakan Experimental Evaluations of European Labour Market Policies - -in Gunther Schmid, Jaqueline O'Rilley, Klaus Schomann (editors) - International Handbook of labour market policy and evaluation - Edward Elgar Publishing Company Ltd - pp BOWLES, Samuel; GINTIS, Herbert The problem with Human Capital Theory: a Marxian critique - American Economic Review - May pp BRADLEY, Steve The Youth Training Scheme: a critical review of the evaluation literature - International Journal of Manpower pp BURTLESS, Gary The case for randomized field trials in economic and policy research - Journal of Economic Perspectives Spring pp BURTLESS Gary, e ORR, Larry Are classical experiments needed for manpower policy? - Journal of Human Resources - volume 21 - n 4 pp CARD, David; SULLIVAN, Daniel Measuring the Effect of Subsidized Training Programs on Movements in and Out of Employment - Econometrica pp CHAPMAN, Paul The Economics of Training - Harvester Manchester - UK. COUCH, Kenneth New evidence on the long-term effects of unemployment training programs - Journal of Labour Economics October - pp EU The execution of the Structural Funds Reform in 1993 Luxemburg. EU Conclusions of ESF mid term evaluations - Luxembourg EU Socio economic programs evaluation - Collection MEANS - Luxembourg. EU 1999b - 9th Annual Report of the Structural Funds 1997 Luxemburg. FRAKER, Thomas; MAYNARD, Rebecca The Adequacy of Comparison Groups Designs for Evaluations of Employment Related Programs - Journal of Human Resources - XXII (2) - Spring - pp

20 GIZARD, Xavier La mise en oeuvre de la réforme du Fonds Social Européen - in Le Chômage de longue durée: comprendre, agir, évaluer - Préface Martine Aubry.- Syros Alternatives - Paris pp HAM, J; LALONDE R The effect of sample selection and initial conditions in duration models: evidence from experimental data - Econometrica pp HECKMAN, James; ICHIMURA H; SMITH, Jeffrey; TODD, P Characterizing the selection bias using experimental data - Econometrica pp HECKMAN, James; SMITH, Jeffrey Assessing the case of social experiments - Journal of Economic Perspectives Spring - pp HECKMAN, James; LALONDE, Robert; SMITH, Jeffrey The Economics and Econometrics of Active Labour Market Programs - in - Orley Ashenfelter e Robert Lalonde (Editors) - Handbook of Labour Economics - volume 3A - Chapter 31 - North Holland pp HOLZER Harry; BLOCK, Richard; CHEATHAM, Marcus; KNOTT, Jack Are Training Subsidies for Firms Effective? The Michigan Experience - Industrial and Labour Relations Review volume July pp KEYNES, John Maynard Teoria Geral do Emprego, do Juro e da Moeda - São Paulo - Editora Atlas - Translated from the 1973 edition of the McMillan Press. LALONDE, Robert Evaluating the Econometric Evaluations of Training Programs with Experimental Data - American Economic Review September - pp LALONDE, Robert The promise of Public Sponsored Training Programs -Journal of Economic Perspectives - Volume Spring - pp LINDLEY, Robert The European Social Fund: a Strategy for Generic Evaluation - -in Gunther Schmid, Jaqueline O'Rilley, Klaus Schomann (editors) - International Handbook of labour market policy and evaluation - Edward Elgar Publishing Company Ltd - pp MAYNARD, Rebecca Evaluating Employment and Training Programs: Lessons from the USA - International Journal of Manpower - Volume 14-3/4 - pp

21 Mc CONNELL, Campbell; BRUE, Stanley Contemporary Labour Economics - Mc Graw Hill - International Edition - second edition - Singapore. MINCER, Jacob Investment in On Job Training and Personal Incomes Distribution - Journal of Political Economy pp MINCER, Jacob On-the-job training: costs, returns, and implications. - The Journal of Political Economy - volume LXX part 2 - Supplement - October - pp MINCER, Jacob Schooling, Experience and Earnings NBER Columbia University Press New York. OCDE Human Capital Investment - An International Comparison Paris. OI, Walter Labour as a quasi-fixed factor The Journal of Political Economy - 70 pp PAUL J La relation formation - emploi: un défi pour l' économie - Economica Paris. PLANAS, Jordi; CASAL, Joaquim Avaliar os resultados do FSE: uma tarefa complexa. Alguns problemas específicos da acção avaliadora. - Formação Profissional CEDEFOP - Berlim - pp REYNOLDS L., MASTERS S., MOSER C Labour Economics and Labour Relations - Prentice-Hall Inc - New Jersey. SCHMID, Gunther The Evaluation of Labour Market Policy: Notes on the state of the art - Evaluation volume 3-4 pp SHACKLETON, J Training and Unemployment in Western Europe and the United States - Edward Elgar -- Aldershot - England. SPENCE, M Job Market Signalling - Quarterly Journal of Economics - 87 pp

22 STEVENS, Margaret Transferable training and poaching externalities - in Alison Booth and Denis Snower editors - Acquiring skills: market failures, their symptoms, and policies responses - CEPR - Cambridge University Press - pp STEVENS, Margaret Human Capital Theory and UK Vocational Training Policy Oxford Review Economic Policy Vol 15 1 Spring pp URBAN INSTITUTE 1006 Does Training for the disadvantaged work? evidence from the National JTPA study Edited by Larry Orr Ashgate Publishing Ltd. 21

23 Table 1 Phase Supported People (millions) Gross Budget (Million ECUs) Share in EEC Budget (%) Main gross Recipients (%) (1) (2) (3) (4) (5) I Germany 44 Italy 36 II Italy. 32 UK 27 III Italy 22 UK 21 IV Spain 20 Germany 14 UK 13 Main relative recipients Ireland Greece Ireland Portugal Greece Ireland Portugal Greece Notes: 1 In ESF history, phase I is , phase II is , phase III is , phase IV is since Here, phase II is , phase III is , and phase IV is In phases II, and III (1984-8) data refer to annual authorizations; 4 and 5 relate to data of 3. Source: EU official documents and reports. 22