How to govern and evaluate labour market policies Statements and Comments Erling Barth Institute for social research This paper discusses some topics related to the government and evaluation of labour market policies in Sweden as discussed in the Discussion Paper by Åsa Sohlman. The Norwegian context, which is quite similar to that of Sweden in many respects, will be outlined first. Then follows a discussion of transferability issues, with a focus on some of the dilemmas present with respect to the government and evaluation of labour market policies. The next section provides a brief account of some of the issues presently under debate in Norway. Because the Swedish paper is mainly concerned with outcome evaluations, and the relationship between such evaluations and policy formation, I will focus on this types of issues as well, and only briefly consider issues related to internal efficiency, auditing etc. a couple of places below. 1. Policy Context in Norway In an international comparison by OECD 2003, Norway spent 0.79 percent of GDP on active labour market programmes in 2001, while Sweden spent 1.39 percent. Relative to the unemployment rate the figures were 0.22 and 0.29 respectively. Both countries rank high in terms of resources used on active labour market polity measures relative to total spending on labour market programmes, 63.9 (N) and 59.2 (S) percent (OECD, 2003). This type of policy measures represent examples of what is often called activation policies within the areas of both employment and social welfare. In Norway, the new government that took over in 2005 has declared its intention to increase the use of active labour market programmes. 1.1 Aims The main aim of Norwegian labour market policies is to facilitate for a well functioning labour market. The present Government states in their Soria Moria declaration of their joint political platform that The struggle against unemployment and for an including work life is a main goal for our policy. They list increased number of individuals on active labour market programmes, both for regular unemployed and for disabled persons specifically, as well as a youth guarantee ensuring individual follow-up of youth with respect to work, education or training. They also want to increase the possibility of combining social support with work. In connection with poverty reduction, they introduce a concept of welfare contracts, which is yet to be developed in detail. 1.2 Institutional structure The Ministry of Labour and Social Inclusion develops and enacts labour market policies in Norway. The Norwegian Public Employment Service is the government agency responsible for implementing labour market programmes. It has 18 local directors and 163 local offices that 81
provide services for job-seekers, administer training programs and manage unemployment benefits. In 2005 the Government presented a white paper proposing a reorganisation of employment and welfare administration in Norway. The objectives are: More people at work, fewer on benefits, a user friendly, user oriented system, and a coordinated efficient welfare and employment administration. In order to reach these objectives a new central government employment and welfare agency will be established to absorb the tasks that are presently carried out by the Labour Market Administration and the National Insurance Service. These two agencies will then be closed down. At the local level the government is establishing a joint front-line service with a municipal-central government employment and welfare office in each municipality. This office will be a familiar gateway to the employment and welfare services and will be perceived by the users as a single entity. The office will offer a range of services for the unemployed and enterprises, people on sick leave, disability pensioners, and people who receive financial social assistance, pensions and family benefits. The employment and welfare offices will have accounting responsibility and they will be able to activate the measures that are required to meet individual needs. (See http://odin.dep.no/filarkiv/243554/new_welfare.pdf) 1.3. Evaluation The Ministry of Labour and Social Inclusion is responsible for bringing about independent evaluations of the effects of labour market programmes. They normally commission this work to independent research institutions. This is done partly directly and partly through the Norwegian Research Council. As in Sweden, the existence and availability of large scale, high quality register data ensures high quality data for evaluation purposes in Norway (see eg. Raaum and Røed, 2003). A summary and discussion of evaluation studies is given in Raaum, Røed and Torp (2002). Recent examples of evaluation efforts based on register data are Hardoy et al (2006) who evaluates the effect of the extended youth guarantee on employment and educational outcomes for young people, and Røed and Raaum (2006) who evaluates the effect of labour market programmes on unemployment duration. The office of the Auditor General of Norway generally audits the accounts as well as in general performing performance audits and corporate control of the central government and government agencies. It also puts requirements on the agencies to evaluate performance of public programmes. 2. Transferability The governance and evaluation of labour market policies and programmes are highly relevant topics in the Norwegian context, in particular because of the two developments outlined above: The present Government s renewed emphasis on labour market programmes has implications in terms of governance and evaluation as well. 82
The huge organisational reform related to the merging of welfare and employment agencies requires new knowledge in both areas, as well as new knowledge on the interaction between the two. The Swedish experience is also of high relevance for Norway, as it is in many policy areas, simply because the two countries are quite similar in many respects. This holds both regarding the substantial field, the labour market, as well as for modes of governance and public policy. In my view, there are also some common problems related to governance and evaluations of this type, in particular for small countries. This has to do with scarcity of resources and some tradeoffs related to long-term commitments, the development of specialised skills, independent evaluations etc., points that will be discussed in the following. 2.1. Quality Evaluation of labour market programmes requires the use of advanced scientific methods. Methodological challenges arise in particular related to issues of selection, self-selection and heterogeneity of effects. Proper evaluation requires methods that satisfy internal validity, i.e. that the measured results represent unbiased estimates of the actual effect of the programme on the population under investigation, rather than just reflecting some empirical correlation. Furthermore, the methods need to provide answers to issues of external validity, i.e. to what extent the estimated effects may be generalized beyond the investigated group of participants or programmes. For programme evaluations, the main problem is related to the fact that we cannot observe the counterfactual of non-participation for each individual. Methods to deal with this type of situations are discussed in Blundell and Costas-Dias (2000) and Heckman et al (1999). There are also serious problems related to measurement issues, both in terms outcome variables, costs and of other factors. As mentioned above, both Sweden and Norway have good register based data available for research and evaluation purposes. IFAU in Sweden has a reputation for high quality evaluations. Partly this is due to the quality of the manpower at IFAU and partly it is due to the active and close link with university researchers, in particular the Economics department of University of Uppsala, but also with top experts from abroad. The Discussion paper brings up the notion of a critical mass needed to produce highclass empirical results. Building and maintaining top expertise in this field require a long term commitment as well as sufficient resources. It would be of interest to see a more elaborate discussion of conditions for high quality work, and of what type of institutional arrangements that fosters high-quality evaluations. 2.2. Independence Clearly, a certain level of independence is a prerequisite for reliable evaluations. Policy evaluations conducted by those formulating or implementing the policy do not have the necessary credibility. Self-evaluations may be of high importance, in particular for obtaining internal efficiency, but evaluations of output, and outcomes relative to policy goals require involvement of independent agents. 83
The Swedish IFAU also has a reputation of being independent and reliable. Since IFAU is partly a government agency and partly a research institution, this is of particular importance. The Discussion paper links this topic to the division of labour among the main actors. It would be of interest to have a discussion in more detail about the institutional framework that is set up specifically to ensure independence and a clear cut division of tasks. 2.3. Competition For small countries, there is a limit as to how many expert units for governance and evaluation purposes one may reasonably sustain in any given field. At the same time, both for quality reasons and for reasons of independence, some competition among independent agents may be warranted. For a small country there appears to be a trade off between ensuring competition and building and sustaining high-quality research units. Relations with the international research community is one way of obtaining quality and independence, but again such relations require high-quality research on own part. This type of dilemma may be present to different degrees in different countries, also depending on size, of course. The Discussion paper raises this issue in relation to diversity. The Swedish approach has been to focus the effort by establishing IFAU. A discussion of several issues related to the trade-off between competition and the concentration of effort could in my view be fruitful. 2.4. Output and outcome efficiency The efficiency of government agencies depend on efficiency in two dimensions. IFAU appears to focus on issues related to the efficiency of different outcome measures. There are also important issues related to internal efficiency, both in terms of cost effectiveness and in terms of output. Modes of governance in public organisations are debated and under continuous change. There has also been some research attempting to assess the productivity of local labour market agencies, see eg. Torp et al (2000), but the internal efficiency of the public sector is for the most part left to own assessments and evaluations. It would have been of interest to hear if this type of internal efficiency is also considered by the IFAU or if it is left to the agencies themselves. 3. Evaluations and utilisation A summary of Norwegian research on the effects of labour market programmes is given in Torp et al (2002). They summarise as follows: Research has not so far reached the final goal of uncovering the casual effects of programmes, but it is still possible to draw some conclusions: There is little doubt that some form of mandatory activity tied to the provision of benefits reduces moral hazard problems and stimulate job search. Programme participation normally has a positive effect on job-chances and income. An optimal volume of programmes is not identified There is an international consensus that wage subsidies work, but there are caveats related to this result The volumes of the programmes should be adjusted according to the business cycle. 84
In terms of knowledge gaps, the Discussion paper claims that there are not enough evaluations to permit meta-evaluations. Presumably this is related to Swedish studies. It is worth pointing out that there recently have been some attempts to use international sets of evaluation results to conduct meta studies (see eg. references in OECD 2003). Kluve (2006) is a very recent attempt at using a meta-analytical framework. Based on a host of studies from all over Europe, he concludes that: It is the program type that matters for program effectiveness, rather than labour market institutions or the business cycle. While direct employment programs in the public sector appear detrimental, wage subsidies and services and sanctions can be effective in increasing participant s employment probability. The issue of utilisation is very interesting. I don t know of any study trying to assess this issue for Norway. To me it seems sensible that New policy initiatives and reforms are less often motivated by evaluations while continues improvements and adjustments of policy measures are more often motivated by evaluations. as referred to in the Discussion paper. I would thus not necessarily view this as a critical point against the evaluation efforts, rather the opposite. 4. The Current Debate in Norway The role and scope of active labour market programmes is one of the issues that divide the present Government from the previous one. The effectiveness of labour market programmes is thus a debated topic in Norway. There is also, as in Sweden, a debate related to the efficiency of the public employment services versus private temporary work agencies. In light of the huge reform in the management of employment and welfare policies in Norway, organisational issues related to the new organisation are of course also at top of the agenda. There is a general discussion of the organisation and the division of tasks in the public sector. Topics are: the role of markets incentive provision in the public sector issues of decentralisation versus control general issues related to governance in the public sector division of tasks and responsibilities the role of competition and interactions with the private sector. the role of and interaction with private organisations (both interest groups like unions and non-profit organisations) Several of these issues go in the general direction that is described in the ToR paper for Sweden: greater flexibility to decide on how they (government agencies) should be organised and how the tasks should be carried out. followed by increased focus on accountability and the measurement of outcomes. All of this debate is also present within the field of labour market policy. The 85
governance of evaluation and related issues do not, however, appear at the front of public debate, but constitute important interrelated aspects of the above listed topics. 5. Closing remarks The Swedish approach to governance of monitoring and evaluation of labour market policy, as outlined in the Discussion paper, has provided impressive results. In particular IFAU has a reputation of producing reliable high-quality analysis, and has a high standing in the international research community. It also appears to me that their results are very useful for policy evaluation and for policy making in Sweden. The results are of high interest for other countries, both in terms of research and in terms of policy formation. It seems evident that a lot of resources have been utilised in the development of this tool for policy evaluation, and in my view the experiences from Sweden shows that such efforts are worth their costs. In my view, the pivotal points for useful evaluations are high quality and independent researchers. Included in this list is close association with the international research community. It is crucial that the governance of monitoring and evaluation of labour market policy is designed such as to ensure this high level of quality and independence. The advantage of the Swedish approach presented here is the focussing of resource utilisation. The challenges lie in maintaining the highquality focus, and on keeping the distance to the policy makers. A model of commissioning projects to independent research organisations and agencies implies more competition and may provide a more direct way of obtaining credibility and independent results. Within such a model, however, the challenge may lie in the development and maintenance of skills and a critical mass of qualified personnel. In my view, a move from the Swedish approach in the other direction, a shifting of evaluation efforts towards the agencies that formulate policy or actually implement the policy measures, would not be wise. Such a shift would be detrimental to both quality and credibility. In Norway the renewed interest in active labour marked policies and the upcoming reform in employment and welfare management puts strong requirements on monitoring and evaluation efforts. In this work, we may have a lot to learn from the Swedish experience. 86
References Blundell, R. and M Costas-Dias (2000), Evaluation Methods for Non-experimental Data Fiscal Studies 21: 427-68. Hardoy, Ines, Knut Røed, Hege Torp og Tao Zhang (2006) Ungdomsgarantien for 20-24 åringer, Har den satt spor? ISF rapport 2006:4. Heckman, J., R.J. LaLonde, and J.A. Smith (1999) The economics and econometrics of active labor market programs in O. Ashenfelter and D. Card (eds.) Handbook of Labor Economics 3, Elsevier, Amsterdam. Kluve, Jochen (2006) The effectiveness of European Active Labor Market Policy IZA Discussion paper # 2018. IZA, Bonn. OECD (2003), Employment Outlook. Raaum, Oddbjørn, Knut Røed og Hege Torp (2002) Riktig satsing i arbeidsmarkedspolitikken? Norsk økonomisk tidsskrift vol 116:167-84. Røed, Knut and Raaum, Oddbjørn, (2006): Do Labour Market Programmes Speed up the Return to Work?, Oxford Bulletin of Economics & Statistics, forthcoming Røed, Knut and Raaum, Oddbjørn, (2003): Administrative Registers - Unexplored Reservoirs of Scientific Knowledge?, Economic Journal, vol 113: f258-f281 Torp, Hege, Dag Fjeld Edvardsen og Sverre A. C. Kittelsen, (2000): Evaluering av formidling. En effektivitetsanalyse av arbeidskontorenes samlete virksomhet basert på DEA, ISF-Rapport 2000:8. 87
88