EXAMINATION OF THE METHODS USED TO EVALUATE THE PERFORMANCE OF REGIONAL EMPLOYMENT OFFICES

Transcription

1 SUMMARY EXAMINATION OF THE METHODS USED TO EVALUATE THE PERFORMANCE OF REGIONAL EMPLOYMENT OFFICES (REOs) NATIONAL RESEARCH PROJECT 45 OCTOBER 2003 PRINCIPAL AUTHORS: DR. GIOVANNI FERRO-LUZZI PROF. YVES FLÜCKIGER ASSISTANT AUTHORS: DR. JOSÉ RAMIREZ DR. ANATOLE VASSILIEV

2 RAPPORT DE SYNTHÈSE 2 INTRODUCTION The revision of the Unemployment Benefits Act in 1995 led to major changes in the way the authorities deal with the problem of unemployment in Switzerland. Official policy with regard to unemployment is no longer based on passive management but active measures such as help in finding a new job and matching registered job-seekers with vacant positions. These active measures are implemented by Regional Employment Offices (REO) whose main aim is to reintegrate the unemployed. The size of these REOs is relative to the pool of unemployed people whose cases they deal with, while the number of REOs per canton is proportional to the size of the canton in question. Apart from setting up and operating the REOs, the cantonal authorities have also created logistical services for labour market measures (LLMM). As a result of the REO/LLMM/Cantonal Authorities 2000 agreement signed by the Federal Department of Economic Affairs and the cantons, a system for evaluating the performance of the REOs has been set up. As part of the active policy with regard to the labour market this evaluation is aimed at encouraging the REOs to increase their economic efficiency. Depending on the mean classification of their REOs, cantonal accounts will be debited or credited according to a previously defined bonus/malus system. Evaluating the performance of the REOs was the main focus of the present study. According to the terms of the REO/LLMM/Cantonal Authorities 2000 agreement the performance of the REOs is evaluated on the basis of Atag, Ernst and Young s 1999 model, which consists of a simple ratio analysis. Our research proposes alternative methods for evaluating the economic efficiency of the REOs. In the present report the results of these methods are resumed and compared with those obtained using the Atag, Ernst and Young method (referred to below as the Atag method). It goes without saying that, from the point of view of how the REOs function, our purely economic approach is necessarily limited in various ways. We are convinced, however, that an examination of the performance of the REOs is one aspect of the evaluation of the public employment service which deserves attention, even beyond the agreement mentioned above. There is no doubt that the criterion of efficiency does not replace but complement other criteria by which the activities of REOs or other public offices can be evaluated. THE METHODS USED TO EVALUATE THE PERFORMANCE OF THE REOS Like any other model, there is room for various improvements in the Atag model. For example, it does not provide any solutions as to how the REOs can improve the way they function such as nominating one or several REOs as benchmarks of efficiency with which other REOs can be compared in terms of input-output efficiency. In addition, the Atag model does not enable the productivity of the REOs nor the trend in productivity over time to be measured. Using the Atag model one cannot determine the type of economies of scale which characterise the activity of REOs. It does, however, allow the relative performance of REOs to be evaluated, taking into account

3 RAPPORT DE SYNTHÈSE 3 exogenous factors such as the professional and socio-demographic characteristics of the pool of job-seekers who are registered with them. In our study the objective was to calculate a function of the production possibilities (FPP) of REOs using different approaches. Output was defined solely as the number of successfully placed job-seekers or, in addition, output results as defined by the seco 1. As input we took the number of job-seekers and the number of counsellors per REO, or a variant of the latter based on the available information and the model calculated. The three calculation methods taken into consideration can be combined in two types of models: parametric and non-parametric. In the first type, no specific form is given to the FPP of the REOs and more than one output factor can easily be taken into account. In the second type of model, however, it is easier to assess the efficiency of the REOs together with the productivity of the input used and to demonstrate how the productivity and the efficiency of the REOs evolve over time, while at the same time taking exogenous factors into consideration. This final report devotes a separate chapter to each of the three models calculated. The first two papers 2 present two so-called non-parametric models: the data envelopment analysis (DEA) model and the model based on directional analysis (the DIR model). In both cases, calculations were carried out on the basis of the same data as those used by Atag in the evolution of the performance of REOs over the period April 1998 to March 1999 (termed period 1, cf. Atag, Ernst & Young, 1999, p. 12). The calculation method associated with the DIR model includes many new aspects. For example, various benchmarks can be attributed to each REO depending on the type of output considered, which is not possible with methods such as DEA. The third paper 3 concerns a parametric model which was calculated using observations made in This consists of a stochastic frontier analysis model (SFA model) of the FPPs and of technical efficiency based on the data obtained from a sample group 4. 1 Over a given period these output results are: 1) the mean duration of unemployment; 2) the number of people going into long-term unemployment, i.e. more than 1 year; 3) the number of unemployed people who are no longer entitled to claim unemployment benefit; 4) the number of people who become unemployed (again) less than 4 months after having found a job. The seco has weighted each of these criteria according to its importance. 2 Performance measurement and determinants of inefficiency of regional employment offices: A nonparametric frontier analysis for Switzerland and Economies of scale in job placement and efficient benchmarks for employment offices. 3 An efficiency comparison of regional employment offices operating under different environmental conditions. 4 For more details concerning the models we used in our research see the non-technical summary included in our final report.

4 RAPPORT DE SYNTHÈSE 4 COMPARISON OF THE SCORES OBTAINED FOR EACH REO AND CANTON USING THE DIFFERENT METHODS The literature on the subject compares mainly the parametric and non-parametric approaches on the basis of the DEA and SFA models respectively 5. As a general rule, these studies reveal a strong correlation in the ranking between the results obtained using these models, although it must be said that there are substantial differences with regard to the two first bases for allotting calculated scores for efficiency (i.e. mean and variability). As far as concerns our study, for each of the periods mentioned above we compared the results obtained using the Atag model with those obtained using the three other models, i.e. on the one hand the DEA and DIR models for the period April 1998 to March 1999, and on the other the DEA and SFA models for the year We simply adjusted the efficiency scores in each of the three models in relation to the respective mean in order to establish classifications which are comparable with those obtained using the Atag model. Tables 1 and 2 below provide a brief summary of the results for each model according to the period taken into consideration. From Table 1 it can be seen that the mean efficiency scores obtained using the DEA and DIR models are very close. The results obtained using the DEA model for example imply that, given the input observed, the REOs could have increased their output by 15.4% on average (= ). TABLE 1: Frequency and statistics of efficiency scores obtained using different calculation methods (period: ) DIR DEA ATAG Mean (0.862)* Standard deviation (0.109) Minimum (0.503) Maximum (1.000) (0.846) (0.092) (0.573) (1.000) < 95 33** > Note: *The figures in brackets are calculated efficiency scores. **Number of REOs per category. Total number of REOs included: 129. In terms of classification, the lower part of Table 1 indicates the number of REOs in each of the four categories of results associated with the bonus/malus rule set out in the 5 See for example Linna and Käkkinen (2003), Cummins and Zi (1998) and Sharma, Leung and Zaleski (1997). In their study (1998) Cummins and Zi observed considerable differences between the two methods, while Sharma, Leung and Zaleski (1997) found only minor differences.

5 RAPPORT DE SYNTHÈSE 5 REO/LLMM/Cantonal Authorities 2000 agreement (cf. A. E. & Y., 1999, p. 10). As can be seen, the differences are quite substantial seen from this aspect. The numbr of REOs which exceed the mean by less than 5% varies between 26 for the DEA model and 45 for the DIR model, while the Atag model gives a result of 28. If we now look at the second period analysed, the results of which are summarised in Table 2, we see that there are marked differences between the two models generally compared in the literature (DEA and SFA). On average, the efficiency scores obtained using the SFA model are significantly higher than those obtained using the DEA model, while the rate of variability is lower a result which is generally corroborated in the literature on this subject 6. In terms of classification, the lower part of Table 2 indicates certain differences between the three models, as for the first period. TABLE 2: Frequency and statistics of efficiency scores obtained using different calculation methods (period: ) SFA DEA ATAG Mean (0.825) Standard deviation (0.075) Minimum (0.468) Maximum (0.922) (0.689) (0.120) (0.092) (0.882) < > Note: *The figures in brackets are calculated efficiency scores. **Number of REOs per category. Total number of REOs included: 129. Although these rates are informative, they do not tell us for example if a majority of the REOs at the bottom of the scale according to the DEA model occupy the same position if a different model is used. Table 3 provides a more precise picture in the form of matrices of the correlation coefficients with regard to ranking for the various models used. For the first period there is a strong correlation in the ranking between the two models we calculated, while this correlation is less marked if one compares the results obtained using these two models with those produced by the Atag model. It should be noted, however, that for this period all the calculated correlations in the ranking are highly significant from a statistical point of view. In other words, a not insignificant number of REOs are ranked the same whichever model is used. As far as regards the scores calculated for the year 2001, the correlation in the ranking between the Atag model and 6 See Cummins and Zi (1998) and Sharma, Leung and Zaleski (1997).

6 RAPPORT DE SYNTHÈSE 6 the two others is non-existant, while it is relatively marked and significant for the DEA and SFA models. TABLE 3: Matrices of the correlation coefficients for ranking (Spearman) for the classification obtained using the different methods Period: Total number of REOs included: 129 Period: Total number of REOs included: 72 DIR DEA ATAG SFA DEA ATAG DIR SFA DEA 0.573** DEA 0.594** ATAG 0.328** 0.325** ATAG Note: ** Significant at a level of Again for the purposes of comparing the models we used, we then combined the efficiency scores obtained for each REO per canton. Table 4 below gives the correlation coefficients for ranking at a cantonal level. As can be seen, in every case the correlation is positive and some are even more marked than at a more individual level. It is important, however, to take into consideration the fact that for 2001 the number of REOs per canton included in our own calculations (i.e. for the SFA and DEA models) is lower than the number used by Atag, which is not the case for the first period 7. TABLE 4 : Matrices of the correlation coefficients (Spearman) between the classifications by canton resulting from the various methods used Period: Total number of cantons included: 24 Period: Total number of cantons included: 20 DIR DEA ATAG SFA DEA ATAG DIR SFA DEA 0.552** DEA 0.677** ATAG * ATAG 0.483* Note: **Significant at a level of 0.01; *significant at a level of Appenzell Inner- and Ausserrhoden counted as one canton. In order to better visualise the differences observed at a cantonal level we have drawn up a graph of the differences in ranking by comparing the classifications associated with the various models. Figure 1 shows these differences using an abscissa axis where the 7 For the first period, the numbers of REOs per canton included are: Aargau: 11; Appenzell Inner- and Ausserrhoden: 2; Basle-Land: 6; Basle-Stadt: 3; Berne: 16;Fribourg: 5; Geneva: 1; Glarus: 1; Grisons: 6; Jura: 3; Lucerne: 5; Neuchâtel: 2; St. Gallen: 6; Schaffhausen: 1; Solothurn: 9; Schwyz; 2; Tessin: 5; Thurgau: 3; Unterwalden: 1; Uri: 1; Valais: 5; Vaud: 16; Zug: 1; Zurich: 16.

7 RAPPORT DE SYNTHÈSE 7 cantons are arranged in the order shown by the Atag model. It can be seen that for the canton of Geneva (one single REO, aggregated), which leads the field, there is no difference in ranking between the three models, while in the case of the Valais, there is a significant difference between the Atag and DIR models for example since this canton s ranking varies from 4 th to 24 th position (= ). It should also be noted that for a canton such as Zurich, where a high number of REOs were included, the differences in ranking between the three models are negligible FIGURE 1: Differences in ranking between the cantons resulting from the various methods used (period: , total number of REOs included: 129) dir-dea dir-atag dea-atag GE ZG LU VS NE NO BL GR FR TG AG SH SZ SO VD BS UR ZH TI AI- AR BE JU SG GL PRODUCTIVITY OF INPUT AND OUTPUT OF SCALE OF THE REOS The question of output of scale is of a certain importance from an economic point of view. In this connection we sought to determine whether the size of the REO has a significant influence on the way it functions. In effect, if output of scale is not constant substantial benefits can be gained by combining or decentralising certain REOs. The Atag model does not provide an answer to this question. In contrast, two of the three models we used enabled us to determine the type of output of scale which is characteristic of how an REO functions. These are the DIR model, calculated on the basis of data for the first period, and the SFA model, where data from the year 2001 were used. A comparison of the results obtained using these two models is even more relevant since they are based on quite distinct mathematical approaches. The output factors taken into consideration for calculating the DIR model were the number of unemployed people who found a job and the number of people who became unemployed (again) less than 4 months after having found a job, while the input factor was the number of people seeking work and the number of counsellors per REO. As far

8 RAPPORT DE SYNTHÈSE 8 as concerns the SFA model, we took the same input factor and only the number of people who found jobs as the output factor. Of the 132 REOs used for calculating the DIR model for the first period, 124 were seen to have a decreasing output of scale. The other REOs had a constant or increasing output of scale. In the SFA model it was not possible, however, to calculate directly the phase of output of scale for each of the REOs included, but the conclusion reached is similar, despite the fact that the number of REOs and the period used differ from those applied in the DIR model. The output of scale throughout the calculated FPP was also seen to decrease: a rise of 10% in the two input factors taken into consideration resulted in an increase in the number of people who found jobs of 8.8%. Using the SFA model we were also able to deduce an increase of 10% in the number of counsellors; the other input factor being fixed, the result was a rise of 0.8% in the number of people who found jobs. Although the latter result concerning productivity among job counsellors is highly informative we also thought it would be interesting to analyse the results of a questionnaire sent to the REOs 8 in order to test certain hypotheses concerning how the counsellors use their time or the level of human resources capital represented by the pool of counsellors. This was also done using the SFA model. The counsellors work was divided into three types of tasks: counselling, administrative tasks and others (e.g. management, coordination). The results obtained using the SFA model clearly indicate that if the time devoted to administrative tasks is increased the REO s productivity falls. More precisely, by fixing the time devoted to other tasks, an increase of 10% in the time given up to such tasks reduces the rate of finding jobs for the unemployed by 0.8%, while a similar change in the amount of time counsellors spend counselling job-seekers or carrying out other tasks raises output by 1.2% and 1.9% respectively. We were also able to ascertain that a rise in the number of counsellors who have a federal diploma had a positive effect on the mean productivity of the REOs. This was also the case with regard to the mean length of service of the counsellors. The variation in output in relation to the proportion of counsellors who have been employed at the REO for over 3 years is 0.17, while it is 0 in relation to the proportion of counsellors who have been in their present jobs for less than 1 year. Finally, the most variable output-input ratio naturally involves the number of jobseekers. In effect, our different calculations showed that on average and all things being equal, a rise of 10% in the number of people entitled to unemployment benefit translates into an 8% increase in the number of people who find jobs. CONCLUSIONS The diversity of alternatives to the Atag model which we examined as part of our study is of enormous value in calculating the economic efficiency of REOs. The differences between these various models and the Atag model in terms of calculated efficiency scores are not so important at a cantonal level as at the level of the individual REOs. One reason which could explain such differences may be linked to the fact that the 8 See Appendix 2 of the report entitled Analysis of the results of the questionnaire sent to the REOs.

9 RAPPORT DE SYNTHÈSE 9 exogenous influencing factors taken into consideration in the models we tested differed somewhat from those used in the Atag model. One of the strengths of the non-parametric models that we studied lies in the possibility of attributing to each REO one or several reference REOs (depending on whether one or several output factors are taken into account) which, with the same input rate, achieve a higher rate of output. From this point of view, there is no doubt that our method is superior to that used by Atag, which results in only one relative classification of the REOs without providing any indication at all of possible ways in which they might improve their performance. By providing each REO judged to be inefficient with at least one model office, the former can devise ways of improving its performance in relation to the aim it wants to prioritise by comparing the way it functions with that of its model(s). Secondly, whatever the method chosen (i.e. parametric or non-parametric) or the period considered, our results show very clearly that the activity of the REOs is characterised by decreasing or constant output of scale. This original result is of interest because it contradicts the accepted idea that grouping employment offices improves productivity in relation to input. Our conclusion may at first appear to be surprising, but it can be explained by the fact that the way in which REOs function does not allow for savings in fixed costs associated with the tangible assets of the REOs (furnishings, premises, IT equipment). On the contrary, our results show that by reducing the size of the REOs the fixed costs per unit of output could even be reduced. Finally, by processing the results of the questionnaire sent to the REOs we were able to considerably improve our analysis of the performance of the REOs in relation to the models which do not take this into account, such as the Atag model. The fact of integrating variables relating to staff qualifications and the types of tasks carried out (administrative, job counselling or others) reveals the necessity of taking into account the productive organisation of the REOs, not only to evaluate their performance but also to provide them with ways to improve. This method is sufficiently promising for further studies to be carried out in this area. REFERENCES Atag, Ernst and Young (1999). "Calcul des résultats des ORP compte tenu des facteurs d'influence exogènes", Rapport final, version à l'intention du seco et des directeurs des offices cantonaux. Cummins J. D. and H. Zi (1998). "Comparison of frontier efficiency methods: An application to the U.S. life insurance industry", Journal of Productivity Analysis 10, Linna M. and U. Häkkinen (2003). "Determinants of cost efficiencies of Finnish hospitals: A comparison of DEA and SFA", draft, National Centre for Welfare and Health, Helsinki, Finland.

10 RAPPORT DE SYNTHÈSE 10 Sharma K.R., P. Leung and H.M. Zaleski (1997). "Productive efficiency of the swine industry in Hawaii: Stochastic frontier vs. Data Envelopment Analysis", Journal of Productivity Analysis 8,