Summary Table of Peer Country Comments

Transcription

1 Austria Most evaluation research is contracted out from public authorities, while monitoring is done internally and on a high standard. The Ministry of Labour and the Public Employment service hold the main responsibility for labour market policy evaluation in Austria. Evaluation projects are occasionally also commissioned by social partner organisations, Territorial Employment Pacts or provincial governments. Scientific funds may act as additional source for financing labour market policy research and offer more mid-term perspective. Availability of administrative individual data for labour market policy research is favourable. The Austrian social insurance database covers individuals labour market history since PES data cover all relevant business processes of the PES. Operative data are documented and organised in the PES data warehouse together with the social insurance database. Evaluation studies are carried out based on administrative data. The favourable databases increase the potential of (quantitative) identification of impacts. In some research projects administrative date are linked to surveys and combined with more qualitative research approaches. Some success factors in the Flemish example also hold for Austria: e.g. the importance of an initiative coming from within the public authorities. In both countries a relatively small group pushed the development of the data warehouse projects. The Austrian databases seem to have advantages with respect to comprehensive coverage, precise measurement of spells on a daily basis, the long available employment histories. A clear data-privacy agreement allows researchers a privileged access to administrative data. Limitations of the Austrian database concern missing or unreliable characteristics as well as unexploited potential of linking additional databases. Are medium-term contracts, such as those awarded to the Policy Research Centre WES or calls for single evaluation project more suitable to maximise the impact of evaluation research on policy? What are the strategies to overcome the limitations of administrative data, e.g. the only monthly information in the Flemish case? Which examples may serve as good practice for data protection solutions? Is the situation in Austria too liberal? Availability of administrative data and register data is improving in many EU-countries. What would we need to make first steps in exploiting administrative data on individuals for European research projects? Bulgaria In Bulgaria evaluation of labour market policies is carried out by the Government and its internal services and by external contractors. The EA through its regional structures collects, maintains and analyses the most important (and extensive) administrative about the labour market in Bulgaria. The experience of Belgium may be applied in Bulgaria, because there are no legislative restrictions or specificities that require fundamental reform. The work on the establishment of warehouse What is the future for VIONA and may it be expected that it will be fully replaced by Policy Research Centre me? Where information could be 1

2 Methodologies used for current internal policy evaluations are rudimental and consist of straightforward calculations of statistical indicators and comparisons with previous and current activities. Most often, determination of research and evaluation priorities is based on the priorities of the labour administration, but in independent research projects they could be selected by the research team. The importance of policy evaluations is recognized in the process of the preparation of new policies and measures and at the end when results are reported, but less in the other stages of the policy cycle. An update of is needed to reflect recently introduced requirements towards job-seekers such as gained key skills, potential for professional and geographical mobility and other personal characteristics. The experience of Belgium in this field may be used in Bulgaria, because it would provide a comprehensive and up-to-date approach towards these socioeconomic characteristics of the registered unemployed. According to the legislation, the exchange of data between institutions must be undertaken based on signed agreements. Current practice in Bulgaria does not allow best use of available data in evaluation, improving the effectiveness of a range of processes related to maintaining and accessing data would improve, if the warehouse practice is introduced. institution in Bulgaria will not be difficult due to the steps already undertaken. Utilisation of both administrative data sources alongside survey data is important to facilitate comprehensive labour market policy evaluations, it should be recognised that this is an important aspect for successful evaluation In the medium term, accumulation of sufficient capacity to establish proposed Policy Research Centres is expected. The Belgium experience illustrates that work to establish such capacity is a long term process, and, in the Bulgaria case, the VIONA me can function successfully and in parallel with the Policy Research Centre me. found on the methodology of the research on mandatory activation of older unemployed, and on the case study on unemployed transition to employment in Belgium? Do plans exist in Belgium on research of the effectiveness of the labour market policy evaluations? Attempts have not been made to use administrative statistics for outlining the longitudinal status of the labour market positions that persons have occupied. However, availability of such data on the background of unemployed persons in labour offices 2

3 and the legal requirements regarding protection of personal data makes this exercise possible in Bulgaria. Denmark Evaluations are conceived as an important tool for actors at all levels of the system in improving the design and implementation of active labour market policy. The National Labour Market Authority (NLMA) has in recent years put increasing emphasis on evidence-based policy formation Denmark has a long tradition for the use of administrative registers for quantitative research and evaluation. A major driving factor has been the introduction of the personal identification number (CPR-number) in The use of and access to administrative data is supervised by a special public agency the Danish Data Protection Agency. When it comes to combining the data sets under the auspices of the NLMA with other administrative registers Statistics Denmark plays a key role in giving access to combined data sets in a protected environment. A number of aggregated databases have been developed over the years, which combines information from several administrative registers into well documented variables at the individual level and over several time periods (longitudinal data). The key success factors identified in the Host Country Discussion Paper are, to a large degree, also relevant in the Danish context. In contrast to the Flemish case of the Crossroads Bank project, little coordination at the central level of the development of combined registers have taken place What approaches are taken when it comes to accessing the data in the registers? Who can get access? What are the conditions including the costs to be covered? Are there special facilities to support the combined use of register data and surveys or does this take place on an ad hoc basis? What is the assessment of evaluations based on administrative registers compared to evaluations based on other sources? How high are they ranked in the hierarchy, when it comes to the credibility of the results? 3

4 In a number of cases register-based information has been combined with survey data in order to conduct both basic research and evaluations Estonia The Ministry of Social Affairs (MoSA) and Unemployment Insurance Fund (UIF) are responsible for evaluating labor market policy in Estonia. There is no systematic approach to evaluation and the division of tasks between MoSA and UIF regarding evaluation is vague. Impact evaluation is limited, the majority of studies are process evaluations. Funding is given for specific research project with limited duration (4-7months), the calls for tenders are issued on an ad hoc base and only limited resources are available for external research projects. Strong evaluation culture and clear vision of the balance of internal and external research projects in Flanders/Belgium. Long-term research projects combined with ad hoc calls enable to develop coherent research s on the one hand and to respond flexibly to unforeseen needs. Good quality data and reasonable privacy protection rules are essential for linking administrative for labour market research. In theory it is possible to link different administrative via personal identification code in Estonia. In practice the procedure for preparing tailor-made, which contain information from different registers is agreed separately for each research project. The creation of linked containing anonymous individual level data is hindered by the strict privacy protection rules. There is no data warehouse comparable to the Flanders/Belgium example operational in Estonia: the Labour policy information system, which could be the Estonian counterpart, is still being developed. Administrative do not contain all the variables required for policy evaluation: the are built for operational purposes rather than for research objectives. How is it decided what to evaluate internally and what should be contracted out? Are there any guiding principles? What is the balance between quantitative and qualitative evaluations carried out in the framework of Viona/Policy Research Centers Programme? Actual policy is delivered by four autonomous agencies in Flanders/Belgium. Has this implementation structure been evaluated and has it proved to be effective from the point of view of the unemployed? 4

5 Germany The IAB (Institute for Employment Research) is a special organizational unit within the PES that is by law primarily responsible for conducting labour market and evaluation research. The IAB s Research Data Centre (FDZ) has the primary task of making administrative data available to internal and external researchers. A key FDZ product is the Integrated Employment Biographies dataset, which links diverse data in a consolidated data set for evaluation purposes. It includes PES records on receipt of benefits, participation, employment histories and biographical information for all registered jobseekers in a longitudinal data set. FDZ activities include not only updating and quality monitoring but also documentation of the data sets and advising users on the analytical potential and limitations of the data sets it provides It is not sufficient to link administrative data sets and to make them available to outside researchers. There is an additional need for a research-oriented intermediate organization or staff that maintains and documents data sets, continually monitors data quality and advises researchers. Using administrative data to evaluate labour market policies is not an either/or proposition. Combining administrative data with tailor-made survey data makes a lot of sense in offsetting the typical shortcomings of administrative data. Is there an overall framework for ALMP evaluation research in Belgium in a given period? What is the impact of data protection regulations? What type of infrastructure, staff resources and expertise are available in the Data Warehouse? How difficult is it to link regional PES data to the national Data Warehouse through the national identification number? Can this process be automated? The FDZ data products are intensively used in German evaluation research. External researchers can access data from the FDZ in three ways: On-Site Use in Nurnberg, Remote Data Access and Scientific Use Files. Major public evaluations in Germany have used this mixed-methods approach, supplementing it with indepth qualitative implementation studies. 5

6 Greece Given the unfavourable labour market developments and the fiscal constraints, emphasis is given, in the short term, to the implementation of Active Labour Market Policies (ALMPs), in a way to improve labour market outcomes The most common type of labour market policies evaluation studies relate to the evaluation of the Operational Programmes (OPs) and ALMPs interventions implemented under the CSF and the NSRF The Ministry of Economy and Finance is responsible for the determination of research and evaluation priorities, for the conducting of evaluations of the relevant OPs. The methodological approaches used in the evaluations of labour market policies cofounded by ESF follow closely EU methodological guidelines In common with the VDAB in Belgium, the Greek Manpower Organisation (OAED) holds an Unemployment Register consisting of individual records, so as to allow the issuance of the Unemployment Card and the successful job matching. Modernization and upgrading of services provided by the Greek PES (OAED), encouraging the use of internet for the e-completion of various procedures (submitting of applications, payments, etc.). A new Information System at the Ministry of Labour has been established, permitting the interconnection of the information systems and relevant data bases of OAED, Social Security Fund (IKA), Ministry of Interior and General Inspectorate of Labour. The new system should further enrich data relating to the unemployed with relevant information on additional skills, previous professional experience and other important elements, such as disability or other social vulnerability status, household Some of the success factors of the developments presented in the host country discussion paper could readily be applied and transferred to Greece while some others require significant changes The development of the common data warehouse within the new established Information System of the Ministry of Labor requires strong political commitment and cooperation willingness amongst the decision makers of different ministries and other institutional bodies, which is rather uncertain under the current circumstances in Greece. The content of the databases included in the new established Information System should be reformulated and enriched in order to be adapted to the needs of data driven evaluation analysis. The privacy legislation should be modified following the Scandinavian model in order to avoid restrictions and methodological complications on evaluative research. Clarification of the content of data bases used for data driven evaluation of labour market policies concerning specific individuals characteristics. Further details on how the access of researchers to administrative data is regularly arranged in the case of tailormade data demand. Regular production and publication of administrative data for the evaluation of labour market policies. Possibility to make changes in the law for the individual data protection in order to facilitate the use of individualized administrative data for social research purposes. Provision of additional quantitative and qualitative knowledge on both the global results and side effects of the labour market interventions. 6

7 Evaluation studies are regularly commissioned by the Management Authorities of relevant ministries and notably as a rule, these are conducted by external contractors (independent experts or private consulting firms and research institutes). An operational system for the on-going internal evaluation of ALMPs schemes, as a core service offered by the PES, is not well developed. position and income/living conditions Evaluation studies reveal that both administrative data driven analysis and survey based analysis, utilized in a mixed way, are suitable for both monitoring and evaluation of labour market policy measures. Administrative data driven analysis is most suitable for the monitoring of the implementation of the interventions, while the survey based analysis is most suitable for the evaluation of the impact of various measures, providing additional quantitative and qualitative knowledge on both their global results and side effects. The reconstruction on individual level unemployment histories, based on administrative data provident by the Greek PES, is still not possible. The evaluative capacity of OAED will be strengthened if all contacts that took place between local PES and the individual unemployed, as well as the results of all the Individualized PES Interventions (Profiling, Individual Action Plan etc) were included in the Management Information System of the organisation. Italy A decentralized institutional system characterizes the country, thus generating a multi-level governance system in charge for different phases of policy cycle in the labour market domain, from ming to implementation and consequently for monitoring and evaluation of impacts. Monitoring and evaluation tasks There is a large set of statistical information, both from surveys and from administrative records, used for monitoring and for designing of labour market policies and analysis of their effects: large pieces of the mosaic information on employment/unemployment are available, but chipped, of variable quality, with redundancies and together with conspicuous flaws, and especially widely disconnected. At the present, the administrative data are By recognizing the opportunity represented by the on-line mandatory communication system of labour contracts, which has created a standard input of the information flow, the Italian authorities have taken the decision to strengthen the process aimed at the production of reliable and comparable statistics on the labour market, according to a common standard of specific administrative How to organize the use of interrelated sectoral databases to evaluate and steer targeted labour policies? How is organized the set up and the transfer of information related to vacancies and future trends in employment positions from the employers to the PES? Which leverages would be useful to push the different players in adopting coordination and 7

8 are therefore also delegated to the Regional Authorities, which have organized the evaluation plans with different methodologies and degrees of integration among data sources (national statistics, PES databases, ESF databases, etc.) and with different partnership agreements with other local players (research institutes, etc.). Evaluation questions, so far as utilized methodological approaches (quantitative and qualitative) are various and the central level is still demonstrating a lack of coordination capacity so far. The recent reform of the labour market policies (Article n. 1 of law 92/2012) states that the interoperable information system which shall be set up in short term will be both produced by and accessible to a number of accredited players, belonging to public administration so far as to universities, research institutes recorded in at least seven different databases, although none of them are fully interconnected. A very recent reform of the labour market (Law n. 92/ June 2012) has strengthen the role of the national level (ministry of labour) in its functions of strategic coordination and evaluation of policies and introduces a clear system for monitoring and evaluation by entrusting to the ministry the tasks related to managing a special task force (inter institutional technical working group) recently established to implement this new deal in evaluation of LM policies. After the activation of the on-line communication system for the labour contracts (in Italian, the so-called CO, comunicazione obbligatoria ) to be sent mandatory to the PES by the employers, a great opportunity has been offered to better understanding the dynamics of the labour market, because PESs do now have individual administrative files originated by the same communication model as defined at national level, timely comparable, and subject to the same source used. A good example is the report recently published by the national pension and social security institute in cooperation with two national labour agencies (ISFOL and Italia Lavoro) aiming at evaluating the effectiveness of the combined active and passive policy measures adopted in years 2009 to 2011 for unemployed who are beneficiaries of wage compensation fund. A bottom up experience to be mentioned is the recent foundation of a multi-regional working group for analysis of labour market data, which is also publishing a bulletin on regular basis. The working group is composed by experts of ten out of twenty of the Italian Regions, and of the two Autonomous Provinces of Bolzano and Trento. The institutional structure of the governance system of data-driven evaluations in Belgium is an important benchmark for those countries, as it is Italy, that has yet a multi-level governance approach to ALMP, and that fore might very integration of database as the right approach? 8

9 or regional in-house labour agencies. rules. much appreciate a well-done balance between centralized and federalist division among evaluation tasks. Ireland Labour market policy and its evaluation is fragmented across a number of departments. Central Expenditure Evaluation Unit (CEEU) evaluates policies as well, which in the current economic climate is focussing more on the effectiveness of departments current expenditure mes, which includes labour market mes. The government has established guidelines for conducting evaluations. Very little independence in the conducting of evaluations from start (Terms of Reference established by departments) to finish (evaluations mainly undertaken internally). Lack of an evaluation culture, and also an understanding among labour market policy Only one rigorous evaluation has been conducted using administrative data: this evaluation was carried out on Ireland s activation strategy, the National Employment Action Plan. It would appear that the findings from this evaluation have had an impact on policy making as the new integrated employment and support service that has just been launched by the government, Intreo, has been designed to counter some of the failings in the old activation system that was identified in the NEAP evaluation. Follow-up survey data has been used mainly to evaluate the effectiveness of labour market mes, particularly mes designed to help the unemployed find employment. Descriptive output based methodology applied in most evaluations instead of measuring the counterfactual using econometric Ireland has commenced the creation of a National Data Infrastructure, similar to Belgium s Datawarehouse, but the country is still a good bit behind the situation in Belgium, in terms of the existence of a Datawarehouse that contains administrative data that can be used for labour market policy evaluation. Ireland is also lagging behind Belgium in relation to the existence of a culture of evaluation, and an understanding and openness among policy-makers on the importance of conducting evaluations using appropriate data and methodologies to establish the effectiveness of labour market mes. What led to the creation of a research unit within the Department WSE? What has given rise to the culture of research and evaluations at the national-level, and the importance of explicitly funding labour market policy research/evaluations? Are there any labour market experts on the Policy Research Centre WSE SC? What is the Belgium federal administration Science Policy? Who manages the CBSS? Has the Belgium Datawarehouse ever considered linking household-level administrative data? 9

10 makers on the importance of using appropriate data and methodologies to conduct evaluations. techniques. Netherlands The main objectives of the Dutch Ministry of Social Affairs and Employment (SZW) are policy development and enforcement of laws and regulations. SZW is not involved in policy implementation and policy administration. The available budget for external evaluations has been decreasing over the last years and so is the spending. Policy information it is not always usable and used. Available information is not sufficient to judge policy outcomes and is not incorporated in policy planning completely and in time. In the Netherlands, the most elaborate databank is the so-called microdata set of CBS Statistics Netherlands. Other interesting applications of administrative data in the field of labour and social security are the administrative data of UWV concerning social security arrangements and the labour market information system of the public employment service section of UWV. Contrary to survey data, administrative data are not collected at the request of the researcher but for purposes outside the reach of researchers and their commissioners. The better/more can be linked with each other the more records and/or variables and/or data over a longer period of time can be expected to be found in the constructed combined dataset. The value of using administrative data is primarily depending on datarelated and institutional preconditions. The application of administrative cannot fully replace other quantitative or qualitative research methods. For which type of policy evaluation administrative data are of use? Is the usability of quantitative data based on administrative data not overestimated? It is always retrospective use of old data. What will be the influence of the growing use of digital media on the availability of different kinds of administrative data? 10

11 The role of problem exploring and phenomenon describing research is decreasing. Monitoring and evaluation is increasingly organised by linking databases of different authorities and organisations. The application of administrative data presupposes (the striving for) a high standard of policy evaluation and policy knowledge. Without this ambition there will be a lack of public, political and finally institutional and financial support for making administrative data accessible for research and evaluation. Norway Extensive use of administrative register data has a relatively long history in Norway. Research and evaluation is conducted both by external research institutes, evaluators, and by the policy implementing unit NAV Use of administrative register data is a cost effective, high quality source of data for research Administrative data allows researchers to have better control of research design The use of administrative register data requires matching routines and data sharing Statistics Norway collects and matches data registers from various sources and makes them ready for research purposes Use of administrative register data may alleviate several methodological challenges associated with evaluation studies and the analysis of economic behaviour in general Do you have any suggestions as to how to improve on the timing of data delivery to researchers and analysts in order to improve the quality and relevance of evaluations? Have you considered the combination of randomized experiments and the use of administrative register data? What sources would be best for international comparisons, surveys or registers? The success factors seemed to rely partly on good will on part of some individuals. Could you think of institutional factors that would promote the ability to use 11

12 such data for these purposes in countries where administrative data are currently less available for research and evaluation purposes? Poland Ministry of Labour and Social Policy is responsible for labour market policy on the national level and the local government and poviat authorities on the regional level In Poland an integrated model of relevant data for evaluation of labour market policy has not been developed yet The internal evaluation is limited to monitoring gross employment effectiveness of active mes External evaluations are based on more rigorous methodologies but are seldom conducted. These studies are conducted by public and private scientific and research The best source of information on the unemployed is the register of the local employment office The data warehouse, that collects information from all employment offices does not function comprehensively A new Central System for Analyses and Reporting, which will enable integration of data from social welfare and social insurance is intended to be implemented Despite the lack of central database two pieces of evaluation research based partly on administrative resources have been conducted recently The project of methodology for evaluating active labour market policies is established by external Completing the work on the data warehouse is essential to implement a model of policy evaluation which makes effective use of administrative data. The evaluation requirements place on Poland in order to use the new period of ming the structural funds may be an accelerating factor in the requirement to develop evaluation data sources and methods in Poland. The existing law on the protection of personal information will be a hindrance in developing the system for evaluating labour market policy, particularly when it comes to nongovernmental researchers gaining access to data. Are there any special legal authorisations that govern an issue of access for potential evaluators to individual information on the unemployed and employed? What criteria are used for the selection of external evaluators? Are the existing databases facing the missing data problem? 12

13 institutes as well as by research teams that are created at universities researchers Slovenia Slovenia is a small country with a centralized public employment system. Evaluation of labour market policies is commissioned by the Slovenian MLFSA and AEP measures are implemented by the PES at the local levels. The researchers and academic community plays a minor, ad hoc, role in policy analysis and evaluations. The PES database with outside links to some other administrative databases allows limited traceability of unemployed. Traceability is limited due to a lack of integration of relevant administrative databases and missing in-depth evidence on competencies of unemployed to better target labour market mismatches between labour supply and labour demand from employers. Evaluation is mainly focussed on short-term practical activity to undertake resource and output audits, with some ad hoc analysis and evaluations of impact. Administrative data allows some longitudinal analysis for specific cohorts, which is undertaken by the PES and partly by researchers have been used in the case of the PES AEP database. Simplification and harmonization of administrative is an ongoing activity, which it is recognised could provide far greater understanding of the effectiveness and efficiency of labour market policy. The LFS database and other survey data have been used in analysis and evaluations for the research purposes to supplement administrative data. However, it is recognised that it would be beneficial to simplify and harmonise the economic and social security Who should be the crucial player(s) for enrichment, simplification and harmonization of administrative databases? What should be the role of the researchers and academic community in harmonization of administrative databases, analysis and evaluations of measures? How to analyze efficiency in the employability of unemployed? How to analyze labour market flows inside and outside the country? 13

14 databases and to start to collect evidence on the competences of unemployed or competences of different job seekers. In order to implement the lessons from the host country it would be important to understand the budgetary requirements and additional administrative burdens placed on contributing organisations. Sweden The Institute for Labour Market Policy Evaluation (IFAU) is the Ministry of Labour s agency for labour market policy evaluation. The Riksdag, through the Swedish National Audit Office, conducts evaluations on labour market policy. Some evaluations is on the borderline between research and evaluation and can be financed by research funding bodies. Building databases for research, monitoring and evaluation has a long tradition in Sweden. Overall good access to rich data for academics, but less quality for private actors. On national level it is mainly data driven impact evaluations. Systems similar to the Crossroads Bank project exist in Sweden. Privacy concerns can imply compromises and there is a risk that data availability. Policy research centres could mean improvements from methodological points of view. Especially since they facilitate methodological variety. How about ex-ante evaluations? They can be important for constructing for ex-post evaluation purposes. Is there a possibility to combine survey and register data (to attach register information to conducted surveys)? Is it financially possible for small projects to use administrative data? 14