FACULTEIT ECONOMIE EN BEDRIJFSKUNDE. HOVENIERSBERG 24 B-9000 GENT Tel. : 32 - (0) Fax. : 32 - (0)

Transcription

1 FACULTEIT ECONOMIE EN BEDRIJFSKUNDE HOVENIERSBERG 24 B-9000 GENT Tel. : 32 - (0) Fax. : 32 - (0) WORKING PAPER The Ooghe-Joos-De Vos Failure Prediction Models : A Cross-Industry Validation Hubert Ooghe (*), Jan Camerlynck (**) and Sofie Balcaen (**) July /107 (*) Professor, Ernst & Young Chair of Growth Management and Institute of Credit Management (Gerling-Graydon-IFB), Vlerick Leuven Gent Management School and Department of Corporate Finance, Faculty of Economics and Business Administration, Ghent University, Belgium. (**) research assistant, Department of Corporate Finance, Ghent University, Belgium. Mailing address: Department of Corporate Finance, Ghent University, Kuiperskaai 55E, B-9000 Ghent, Belgium; tel: ; fax: , Hubert.Ooghe@rug.ac.be, Sofie.Balcaen@rug.ac.be D/2001/7012/08

2 ABSTRACT Faced with the question whether the Belgian failure prediction models by Ooghe, Joos and De Vos (1991) can be easily applied in all industries and for all sizeclasses, this study compares the performance of the OJD models across 18 different industries and different sizeclasses. After a brief theoretical review of the logistic regression modelling technique, which was used to design the OJD 1991 models, and the performance measures that are used to evaluate these models, we report type I and type II error rates corresponding with the original cut-off points of the models. Furthermore, we calculate new optimal cut-off points, as well as Ginicoefficients. Finally we report the reductions in unweighted error rates when using the new cut-off points instead of the original ones, and the graphs of the trade-off functions. As can be concluded from the performance results and the trade-off functions, there s a wide range of performances for the different industries. However, we notice that the OJD 1991 models perform best for classical manufacturing industries - such as chemicals, paper and printing, textiles and apparel, paper and printing and metal - and financial services., while the models show the worst performance for service industries - such as real estate, hotel, restaurant and catering, wholesale and personal services - and the no-industry category. Furthermore, when using the new cut-off points for the industries, the unweighted error rates of both models (i.e. the model 1 year prior to failure and the model 3 years pior to failure) are significantly reduced. A second remarkable finding is that the OJD model 1 year prior to failure has the best predictive ability for large companies and companies with complete form annual accounts, while it performs worst for companies without employees and companies with an abbreviated form of annual accounts. On the other hand, when we compare the predictive ability of the OJD model 3 years prior to failure for the different sizeclasses and subsamples with repect to form of annual accounts, we find no significant differences. Furthermore, the performance differences between the various subgroups with respect to industry, sizeclass and form of annual account of the 3 ypf model the are much smaller than those of the 1 ypf model.

3 1. INTRODUCTION Failure prediction or financial distress models are much-discussed in accounting and credit management literature. A lot of studies have been dedicated to the search for the most effective empirical method for failure prediction. Besides the comparison of different scoring techniques on the same data set, much attention has been paid to the prediction of failure in different industries (Platt, 1989 and Platt & Platt, 1991). In Belgium 1, the first financial distress models were estimated in 1982 by Ooghe and Verbaere. These models predicted financial distress on the basis of multiple discriminant analysis. Ever since, extensive research has been done into the prediction of failure of Belgian companies. Inspired by the results of these research efforts, Ooghe, Joos and De Vos estimated a second generation of models in In these models, the authors used the technique of logistic regression in order to predict financial distress. In the years to follow, research efforts have been concentrated on the validation of these models. In this paper, we validate the Ooghe-Joos-De Vos 1991 failure prediction models on one data set of Belgian company accounts, in order to find out whether these failure prediction models can be easily used in different industries and for different sizeclasses. As our goal is not the re-estimation, but a large and global validation of these models, we work with populations and samples as large as possible. It is also our objective to suggest possible explanations for differences in performance of the models for the different industries. This paper is organised as follows. It starts off with a short explanation of the logistic regression modelling technique, which is used in the OJD 1991 models. Section 3 contains a theoretical elaboration of the different performance measures that are used to evaluate the performances of the models for the different industries. In particular, we discuss type I and type II error rates based on old and new cut-off points and we compare the performances of the models in a more global way with Gini-coefficients. Section 4 discusses and investigates the Ooghe-Joos-De Vos failure prediction models, which are analysed in this paper. Section 5 describes the population and the sampling methodology and section 6 presents the results of 1 For an overview of financial distress studies in Belgium between 1982 and 1991, with an emphasis on those conducted at the Department of Corporate Finance of Ghent University, see OOGHE, JOOS and DE BOURDEAUDHUIJ (1995) 3

4 our empirical research. The final section concludes with an overview of the most important findings. 2. MODELLING TECHNIQUES Modelling techniques for two-group classification in general and failure prediction in particular can roughly be classified in four different groups: classical statistical techniques, recursive partitioning analysis (or tree classification), neural networks and genetic algorithms 2. This section will elaborate on the classical statistical modelling technique of logistic regression, which is the technique used in the OJD failure prediction models under investigation. In logit analysis, conditional probabilities or logit scores lying between 0 and 1 (on a sigmoidal curve) are determined with the next formula (Hosmer & Lemeshow, 1989): 1 P( y = 1 X ) = P1 ( X) = + ( k k ) (1) 1 e b b X b X The exponent in formula (1) expresses the so-called 'logit'. The coefficients are estimated with the maximum likelihood method. The likelihood function in formula (2) is maximised: n L( b) = P (2) i= 1 yi 1 yi 1 (X i ) [1 P1 (X i )] with P 1 (X i ) = probability of failure of ith firm, b = vector with k estimable parameters b 1, b 2,,b k, X i Y i = vector with characteristics of ith firm, = 1 if ith firm fails, 0 if it doesn't fail. On the basis of their logit score and a certain cut-off point, firms are classified into the failing or the non-failing group. Logit analysis is often used in classification studies because this 2 For a comprehensive summary of methodological issues on estimation and evalution of credit scoring models, see JOOS, OOGHE and SIERENS (1998) 4

5 method has some favourable qualities. For example, it is not necessary to adapt the method for disproportional samples 3 for only the constant term b 0 is distorted (Maddala, 1992). 3. PERFORMANCE CRITERIA The performance of a classification model indicates how well the model performs. In econometric literature it is called 'goodness-of-fit'. Evaluation of performance is possible in two different contexts: one can use the original dataset that was used to estimate the model or a new validation dataset. As it is not our intention to present an exhaustive overview of the various performance measures, only two different performance measures will be discussed in this paper: measures based on a classification rule and measures based on the inequality principle (Joos, Ooghe and Sierens, 1998) Measures based on a classification rule Since classification is the principal goal of the failure prediction models, it is obvious that measures based on a classification rule are frequently applied. A firm is categorised as failing or non-failing, on the basis of the following classification rules. For a continuous score model, the classification rule can be formulated as follows: y 1 if the logit or discriminant score yˆ i of firm i > y 0 if the logit or discriminant score yˆ iof firm i y i = (3) with y i = estimated class of firm i, y = threshold or cut-off point. 3 In classification research, state-based samples (the probability of being selected depends on the state of the firm i.e. non-failing or failing) are often used instead of pure random samples. Since the number of failing units is smaller than the number of non-failing units in most databases, random sampling would lead to very small samples of failing firms, and to inaccurate models. 5

6 The classification rule divides the scores into two subdivisions, which causes two types of misclassification costs: 1. Type I error: represents the credit risk : a failing firm is classified as a non-failing one. 2. Type II error: represents the commercial risk : a non-failing firm is classified as a failing one. In this respect, the optimal threshold or cut-off point can be determined as the score for which the average of both types of errors is minimised. In addition, following Koh (1992) the population proportions and misclassification costs can be involved in the identification of the threshold as well. Here, the global cost function (4) must be minimised: with expected cost = π failing, π non failing C Type I, C Type II Type I, Type II EC π C TypeI + π C TypeII (4) = failing TypeI non failing Type II = population proportion of failing and non-failing firms, = misclassification cost of type I and type II error, = type I and type II misclassification percentage resulting from resp. type I and type II errors. The population proportions show the frequency of failing and non-failing firms in the population. The misclassification costs can be very different for both type of errors in the context of credit granting, because the classification of a failing company as a non-failing one can have more severe consequences than the classification of a non-failing as a failing one. If these cost factors are integrated, it is obvious that the classification process is dependent of the risk behaviour of the decision-maker and his attitude towards the proportion of cost factors. Besides minimising a cost function, which is only one way to evaluate the performance of a classification model, it is also possible to evaluate the performance statistically; without taking the population proportions and the misclassification costs into account. In this respect, significance can be tested by using the Kolmogorov-Smirnov test (Siegel & Castellan, 1988). This test is based on the cumulative distribution functions of the scores of the nonfailing (F non-failing ) and failing (F failing ) firms. The score for which the greatest difference (D non- 6

7 failing, failing) between the cumulative distribution function of non-failing and failing firms exists, is the optimal cut-off point with minimal classification errors. In this context, abstraction is made of population proportions and misclassification costs. D non failing, failing = max[ Fnon failing ( y) Ffailing ( y) ] (5) with D non failing, failing = maximum difference between the cumulative distributions F non failing ( y) of the scores of non-failing and failing firms, = cumulative distribution of the scores of non-failing firms, F failing ( y ) = cumulative distribution of the scores of failing firms, y = discriminant or logit score Measures based on the inequality principle The performance of a model for a certain industry can be demonstrated graphically with the construction of a trade-off function. Here, the cumulative frequency distributions for nonfailing and failing firms, are located in a co-ordinate system with the type I error (= F failing ( y) ) on the X-axis and the type II error (=1- F non-failing (y) ) on the Y-axis (Steele, 1995). It is clear that the best performing (i.e. most discriminating) model has a trade-off function that coincides with the axes. After all, a perfect model categorises each failing firm as a failing one (the type I error is always 0) and each non-failing firm as a non-failing one (the type II error is also 0 for every value). On the other hand, the worst model (i.e. a model that can not make a difference between non-failing and failing firms) has a linear descending trade-off function from 100% type II error to 100% type I error. In this case, F failing ( y) F non failing ( y) coincide (for each score, there are just as much non-failing as failing firms), which results in complementary type I and type II errors for each score. In this respect, a model is considered to be better performing for a certain industry as the curve is situated closer to the axes. and 7

8 Figure 1: Trade-off function: best, worst and estimated classification models Type I error Best Model Estimated Model (example) Worst Model Type II error Each element of this trade-off function represents an optimal threshold or cut-off point for given classification costs (C Type I and C Type II ) and population proportions ( π π ). failing and non failing The difference between the estimated model (trade-off function) and the worst model (linear descending function) is an aggregated performance measure and is presented by the Ginicoefficient. In a normal situation, this coefficient lies between 0 and 1 and is equal to the proportion of the area between the estimated model and the worst model (i.e. the grey area in figure 1), and the area between the worst and the best model (i.e. the triangle with the axes as sides). As a consequence, a higher Gini-coefficient corresponds with a curve that is situated closer to the axes and thus with a better performing model. A negative Gini-coefficient implies that a model classifies more companies falsely than it classifies them correctly. An empirical appropriation of the Gini-coefficient is presented in the formula below (Joos, Ooghe and Sierens, 1998): GINI ˆ 2 = = 1 n x i= 1 max ( x i y max x i 1 n i= 1 x )( y ( x max i 1 i 2 y x max i i 1 + y ) ) y i 1 + y 2 i (6) 8

9 with x i, y = type I and type II error with threshold i, i x max, y = maximum type I and type II, i.e. each 100%. max 3.3. Other measures Two performance criteria that are not used in this study are R 2 -type measures and measures based on entropy (Joos, Ooghe and Sierens, 1998). Several R²-type measures, which indicate the percentage of the variance that is explained by the model, are possible. The count R² measure, which reports the number of correctly and the number of falsely classified firms, is the most suitable measure. However, since the number of correctly and falsely classified firms is already measured by the measures based on the inequality principle, especially by the Gini-coefficient, we decided not to use these measures. The concept of entropy originates from the information theory of Shannon (1948) and was originally introduced in econometrics by Theil (1971). Measures based on entropy were used as performance measures in failure prediction research by Zavgren (1985) and Keasy and McGuinness (1990). It has to be noted, though, that entropy measures only evaluate the discriminating ability of the model and do not allow to take misclassification costs and population proportions into account a posteriori. As a consequence, we did not use entropy meaures in this study. 4. THE OOGHE-JOOS-DE VOS FAILURE PREDICTION MODELS As a consequence of the rise in failures in the Belgian economy and the fact that both the Royal Decree of Octobre 8, 1976 and the Balanscentrale of the National Bank of Belgium led to an increased availability of financial data, a lot of attention was paid to failure prediction models. In 1991, Ooghe-Joos-De Vos built new failure prediction models based on the technique of multiple logistic regression. In their selection procedure of the variables, the practical ease of use of the models for the outsider analysts was beared in mind. Also Ooghe, Joos and De Vos focused on profitability 9

10 and liquidity measures, since they considered these to be the most important ones. Moreover, variables with reference to solvency and intragroup relationships and ratios concerning added value were included. Table 1 :Failure prediction models of Ooghe-Joos-De Vos, 1991 X1 Variables Codes complete form Codes abbreviated form 1 year prior to failure Direction of the Financial Leverage (1 if > 0, 0 if <0) X2 Accumulated Profits + Reserves/Total Liabilities less deferrals and accruals {( 70/66-66/ <65> ) / 20/58 } - {(-<65> ) / ( /48 )} ( ) / ( 10/49-492/3 ) id. {( 70/66-66/ <65> ) / 20/58 } - {(-<65> <656>) /( /48 )} X3 Cash / Total Assets ( 51/ /58 ) / 20/58 ( 50/ / ) / 20/58 X4 Overdue Short-Term Priority ( ) 1 if >0, else 0 id. Debts (1 if >0, 0 else) X5 Operational Net Working Capital ( / ) / id. / Total Assets ( 20/58 ) X6 Net Operating Result / Working ( 70/64-64/ ) / ( id. Assets + 22/ /41 ) X7 Financial Debts (Short ( 430/8 ) / ( 42/48 ) id. Term)/Short-term Liabilities X8 Guaranteed Portion of Amounts ( ) / ( /48 ) id. Payable by the Firm CO Cut-off Point : 0,3117 X1 Accumulated Profits + Reserves/Total Liabilities X2 Publication lag X3 Overdue Short-Term Priority Debts (1 if >0, 0 else) X4 Operational Cash Flow before Taxes Capital Investments/Total Assets 3 years prior to failure ( ) / ( 10/49-492/3 ) ( ) 1 if >0, else 0 {( 70/66-66/70 - <65> <631/4> + <635/7> ) - ( <854> )} / ( 20/58 ) X5 Relationships with Affiliated Enterprises X6 Debt/Total Liabilities ( /48 ) / ( 10/49-492/3 ) id. CO Cut-off Point: 0,2137 {( 70/66-66/70 - <65> <631/4> - <635/7> ) - ( <8545>)} / (20/58) ( ) / ( 20/58 ) ( ) / ( 20/58 ) Table 1 illustrates the composition of the model under consideration, reporting the included variables and the standard codes from Belgian annual accounts. The coefficients of the logistic regression can not be indicated, because of an exclusive licence contract with a supplier of financial data. 10

11 5. POPULATION AND SAMPLES 5.1. Definitions of failing and non-failing firms Before describing the population and the sampling method, some definitions frequently used in this study (faling firm, non-failing firm, accounts 1 year and 3 years prior to failure), are given. Failing firm: a firm in the situation of bankruptcy or with a request for a judicial composition or with an official approval of a judicial composition. Non- failing firm: firms characterised by the following juridical situations are included in the group of non-failing firms: - Termination of activity - Early dissolution - liquidation - Liquidation followed by a merger with another company - Liquidation followed by an absorption by another company - Closing of a liquidation - Without any particular legal status In other words, not only normal firms without any particular legal status, but also firms that cause doubt about the economic reason for their juridical situation are considered to be nonfailing firms and are included in the non-failing group. As it is our aim to validate failure prediction models, it is necessary to do this in a realistic situation and hence consider these firms as non-failing ones. Accounts 1 year before failure: accounts of a failed firm of which the closing date falls within the period [date of failure, date of failure - 365days] Accounts 3 years before failure: accounts of a failed firm of which the closing date falls within the period [date of failure - (2 * 365days), date of failure - (3 * 365days)] 11

12 5.2. Population For the validation Belgian accounting data from the period are used. It concerns published annual accounts of non-financial companies subject to the Royal Decree of October 8, 1976 on the annual accounts of companies. These data are obtained from the CD-ROMs of the National Bank of Belgium. In Belgium, companies are bound to deposit their annual accounts in a prescribed form dependent on their size. A distinction is to be made between large companies that have to prepare their annual accounts in a complete form and smaller companies that are allowed to prepare their annual accounts in an abbreviated form. The group of large companies consists of the companies of which the number of employees exceeds 100 and the companies that meet at least two of the following three criteria : - Number of employees (yearly average): more than 50; - Turnover (V.A.T. excluded) (yearly average): more than 200 million Belgian francs; - Total assets: more than 100 million Belgian francs. Smaller companies that don t meet these criteria, are allowed to prepare their annual accounts in an abbreviated form. The population from which the failing and the non-failing samples is taken, consists of all firms having annual accounts for at least one year in the period It concerns companies identified by their V.A.T number. According to the classification of juridical situations as described above, this population is divided into a population of failing companies ( companies) and a population of non-failing companies ( companies). In both populations, the following type of companies are not included because of their special situation: - Management activities of holding companies and co-ordination centres; - Public administration; - Education; - Health and social work. Since the aim of the study is to validate the OJD 1991 model across different industries and sizeclasses, the next step in the sampling procedure is to divide the populations of failing and non-failing companies into different subpopulations. 12

13 The industry classification that is used in this study, is primarily based on the industry classification in the yearly study The financial situation of the Belgian companies (Ooghe and Balcaen, 2000) carried out by the Department of Corporate Finance of Ghent University. It should be noted, though, that two types of industries in this study do not correspond with the former industry classification: the financial services industry (17) is added and a large number of firms that can not be ranged under any of the specific indusries, since their industry codes are not reported, are grouped into a separate category no industry specified (18). In table 2, the industry distribution of the failing and the non-failing population is shown. The industries with the largest number of companies are: business services (15), wholesale (10), retail (11) and construction (9). Table 2 : Industry distribution of the population of failing and non-failing companies Industry N non-failing N failing % non-failing % failing % non-failing % failing companies companies per industry per industry industry / total industry / total 1 Agriculture ,04% 3,96% 1,69% 1,00% 2 Utilities ,24% 1,76% 0,09% 0,02% 3 Metal ,79% 8,21% 3,01% 3,85% 4 Food ,61% 6,39% 1,41% 1,37% 5 Chemicals ,12% 4,88% 0,75% 0,55% 6 Textiles and apparel ,26% 11,74% 0,94% 1,78% 7 Timber ,77% 9,23% 0,83% 1,20% 8 Paper and printing ,03% 6,97% 1,58% 1,69% 9 Construction ,63% 8,37% 10,02% 13,08% 10 Wholesale ,90% 8,10% 15,54% 19,59% 11 Retail ,08% 7,92% 15,35% 18,88% 12 Hotel, restaurant & catering ,19% 12,81% 5,28% 11,09% 13 Transportation ,49% 8,51% 3,81% 5,07% 14 Real estate ,08% 4,92% 5,00% 3,69% 15 Business services ,64% 4,36% 17,64% 11,50% 16 Personal services ,26% 5,74% 3,18% 2,76% 17 Financial services ,45% 2,55% 0,51% 0,19% 18 No industry specified ,62% 1,38% 13,38% 2,68% Total ,46% 6,54% 100% 100% N: Number of companies Besides the industry classification, we make a classification according to the size of the companies. First of all, we identify the form of the annual accounts (i.e. complete or abbreviated) of each company, since this form is a proxy for company size. Furthermore, we check whether the companies report personnel expenses in their annual accounts. As a result, 13

14 companies are classified into one of three sizeclasses: large companies, small and medium sized companies and companies without employees. Companies that do not report their personnel expenses, are classified as companies without employees (CWEs). Companies that do report personnel expenses and have a complete form of annual accounts, are large companies (LCs) and companies with personnel expenses and an abbreviated form of annual accounts are small or medium sized companies (SMCs). Finally, a classification is made according to the form of annual accounts of the companies. We make a distinction between companies with a complete form of annual accounts, which are the larger companies, and companies with an abbreviated form of annual accounts, which are the smaller companies. In the population of failing companies a division is made according to the year of failure (i.e. the year in which a firm is classified as failing on the basis of its juridical situation). In this respect, we identify 7 groups within the population of failing companies having at least one annual account between 1992 and 1998: 1. failing in the period failing in failing in failing in failing in failing in failing in 1998 When looking at these groups, it appears that some companies that were failing in the period have deposited their annual accounts in the period , since the overall population includes all companies having at least one annual account in the period For example, a company that requested for a legal composition but did not fail, is still able to deposit its annual accounts. In this study, only firms having annual accounts up to three years prior to the year of their failure (1992 for firms failing in 1995, 1993 for firms failing in 1996, 1994 for firms failing in 1997 and 1995 for firms failing in 1998) and of which at least one annual account is available 14

15 on the CD-ROM of the National Bank of Belgium, are included in the population of failing companies. As a result, companies failing before 1995 are excluded and the population of failing companies is reduced from to The population from which the failing sample is taken, consists of all firms that failed in 1995, 1996, 1997 and For each company that failed in 1995, 1996, 1997 or 1998, the annual accounts 1 year and 3 years prior to failure (i.e. 1 ypf and 3 ypf), if available and if not concerning an extended financial year, are used and compared with annual accounts of nonfailing companies in the same period. The population of non-failing companies consists of all firms that where non- failing on January 1, Only firms with annual accounts in 1992 or later and of which at least one annual account is available on the CD-ROM of the National Bank of Belgium, are included. The non-failing sample contains companies Sample construction Because of the large number of companies in the original non-failing population, the population of non-failing companies is reduced, using a systematic selection technique. Firstly, the total population of non-failing companies is divided into 18 subpopulations concerning industry, 3 subpopulations concerning sizeclass and 2 subpopulations concerning form of annual accounts. Secondly, according to the systematic selection technique, each fifth element within each of these subpopulations is withheld. This procedure results in several reduced subpopulations. Hence, the original population of non-failing companies is reduced to a population of non-failing companies. The validation of the OJD failure prediction model is conducted for 3 types of subsamples of failing and non-failing firms, according to (1) industry, (2) sizeclass and (3) form of annual accounts. In table 3, the sample distribution of failing and non-failing companies with respect to their industry, their sizeclass and their form of annual accounts are reported. 15

16 Table 3 : Distribution of samples of failing and non-failing companies with respect to their industry, sizeclass and form of annual accounts Non-failing Failing Industry 1 Agriculture Utilities Metal Food Chemicals Textiles and apparel Timber Paper and printing Construction Wholesale Retail Hotel, restaurant & catering Transportation Real estate Business services Personal services Financial services No industry specified Size Class 1 Large Companies (LC's) Small and Medium Sized Companies (SMC's) Companies without employees (CWE's) Form annual accounts 1 Complete Form Abbreviated Form Total The further sample procedure is conducted as follows: in each of the 3 types of subsamples, the non-failing companies are divided into 6 equal groups. For each group, the annual accounts of one specific year in the period , if available and not concerning an extended financial year, are selected. Consequently, for the non-failing companies, the following annual accounts are withheld: Non-failing firms in group 1: annual account of 1992 Non-failing firms in group 2: annual account of 1993 Non-failing firms in group 3: annual account of 1994 Non-failing firms in group 4: annual account of 1995 Non-failing firms in group 5: annual account of 1996 Non-failing firms in group 6: annual account of

17 With a view to comparing these non-failing companies with failing companies, accounts of the two relevant years are taken together. The sample procedure is explained in table 4. Table 5 gives an overview of the number of companies in both subsamples with respect to the sizeclass of the failing and for the non-failing companies, while table 6 illustrates the number of companies in both subsamples with respect to the form of annual accounts. Table 4: Sampling procedure Year of 1ypf 3 ypf Failing firms Non-failing firms failure annual accounts annual accounts failing in 1995 group 3 & group failing in 1996 group 4 & group failing in 1997 group 5 & group failing in 1998 group 6 & group 4 Table 5 : Subsamples of failing and non-failing firms with respect to the sizeclass: large companies (LCs), small and medium sized companies (SMCs) and companies without employees (CWEs) Category Number of companies Number of companies Number of companies LCs SMCs CWEs Failing in Failing in Failing in Failing in Failing in Non-failing in Table 6 : Subsamples of failing and non-failing firms with respect to the form of annual accounts: complete form and abbreviated form Category Number of companies Number of companies complete form annual accounts abbreviated form annual accounts Failing in Failing in Failing in Failing in Failing in Non-failing in

18 6. RESULTS AND INTERPRETATION This section discusses the results of our cross-industry validation of the Ooghe-Joos-De Vos 1991 failure prediction models on our data set of Belgian company accounts. Firstly, we report the performance results of the original publication of the OJD 1991 models as well as previous validation results (section 6.1.). Secondly, we discuss the general results of the validation of the OJD 1991 models. We report type I, type II and unweighted error rates (UER) corresponding with the original cut-off points as well as new error rates corresponding with newly calculated cut-off points for each subsample with respect to industry, sizeclass and form of annual accounts. In our study, we define new optimal cut-off points for the various subsamples as the failure prediction score for which the unweighted average of type I and type II error rates reaches the minimum. This is the most objective way of working, because the allocation of weights to the different types of errors is subjective and depends on the risk aversion of the risk analyst. Furthermore, we don t take population proportions into account because of the unbalanced proportion of sample sizes (see table 3). The overrepresentation of non-failing companies and the corresponding focus on the minimisation of type II error rates (see formula 4) would lead to cut-off points that are too high and would cause a decision process that is too tolerant. Besides the unweighted error rate, the trade-off function, as measured by the Gini-coefficient, can also be used to evaluate the fit of the model for a certain subsample (industry, sizeclass or form of annual accounts). Contrary to the discussion of the type I and type II errors separately, this measure gives a more global judgement and is suited for the comparison of performances of the OJD 1991 models for the different subsamples. In the discussion of the performance results of the OJD models, we will focus on the unweighted error rates (UER) and not on the Gini-coefficients. However, if we consider the Gini-coefficients, almost the same conclusions can be drawn. Only, small differences can be noticed in the ranking order of performance results for the different industries and sizeclasses. Furthermore, we do not investigate the differences between the original and the newly calculated cut-off points for the different subsamples, but we discuss the reduction in the 18

19 unweighted error rates using the newly calculated cut-off points for each subsample instead of the original cut-off points. We discuss these validation results for all companies and for the different subsamples. Firstly, in section 6.2. the results of the cross-industry validation are reported. Secondly, in section 6.3. the results of the validation on the sizeclass subsamples are discussed. Thirdly, section 6.4 reports the validation results for the subsamples with respect to the form of annual accounts. Finally, section 6.5 gives a graphical presentation of the research results with the construction of trade-off functions for all subsamples Performance results of the Ooghe-Joos-De Vos 1991 failure prediction models in the original study and previous performance results Table 7 contains the classification results obtained by the authors themselves in their original study on their validation sample. Table 7: Classification results of the OJD 1991 models in the original study (Ooghe & Van Wymeersch, 1997) Cut-off Type I error Type II error Unweighted error rate Validation sample Validation sample Validation sample 1 year prior to failure 0, ,7% 22,4% 18,5% 3 years prior to failure 0, ,3% 34,1% 26,2% In table 8, the results of a previous validation (Ooghe and Camerlynck, 1999) on a sample of complete form annual accounts are shown. Firstly, the type I, the type II and the unweighted error rates corresponding with the original cut-off points, are presented. Secondly, new cut-off points and the corresponding error rates are illustrated. Finally, in the last column the Ginicoefficients, which are independent of changing cut-off points, are reported. In table 9, the results of a previous validation on a sample of companies with complete and abbreviated annual accounts, are shown. 19

20 Table 8 : Performance results of a previous validation of the OJD 1991 models for a sample of large companies with complete form of annual accounts (Ooghe & Camerlynck, 1999) 1 YPF Original cutt-off point Type I- error, original 24.5% Type IIerror, original 34.5% Unweighted error rate, Original 29.5% New cutt-off point Type I- error, new 26.4% Type IIerror, new 29.9% Unweighted error rate, new 28.1% Gini, new 53.6% 3YPF % 24.8% 35.4% % 32.3% 34.2% 37.3% Table 9: Performance results of a previous validation of the OJD 1991 models for a sample of companies with complete and abbreviated form of annual accounts (Ooghe and Camerlynck, 1999) 1 YPF Original cutt-off point Type I- error, original 17.6% Type IIerror, original 36.0% Unweighted error rate, original 26.8% New cutt-off point Type I- error, new 23.9% Type IIerror, new 28.0% Unweighted error rate, New 25.9% Gini, new 61.3% 3YPF % 42.9% 33.2% % 37.4% 32.8% 44.9% 6.2. Performance results of the Ooghe-Joos-De Vos 1991 failure prediction models with respect to industry In this section, the results of the validation of the OJD 1991 models for the 18 industries (see table 10), are discussed. If we take a closer look at the validation results of the OJD 1991 model 1 year prior to failure based on the original cut-off point, we see that this model has the lowest unweighted error rate (21,71%) and hence the best performance in the chemicals industry, while it performs worst in the real estate industry (37,26%). Other industries for which the model has relatively low unweighted error rates and hence a high performance are: paper and printing (22,63%), metal (23,86%) and textiles and apparel (24,10%). Other industries for which the predictive ability of the model is rather poor are: the hotel, restaurant and catering industry (36,63%), the no-industry category (34,80%) and financial services (33,04%). The validation results of the OJD 1991 model 1 year prior to failure based on the new cutoff points for the different industries indicate more or less the same industries as the ones for which the 1 ypf model has the lowest and the highest unweighted error rates. This model performs best for the chemicals industry (20,15%) and in the paper and printing industry 20

21 (22,30%), while it performs worst for the no-industry category (33,67%), real estate (33,38%) and the hotel, retaurant and catering industry (32,39%). Table 10 also demonstrates that, using the new cut-off points instead of the original cut-off point, the unweighted error rates of the model 1 year prior to failure are reduced for all industries. In absolute terms, the unweighted error rate of the 1 ypf model is reduced by 1,6%, which means a relative reduction 5,7%. The absolute and relative reduction in the unweighted error rates is most significant for the following industries: financial services, hotel, restaurant and catering, real estate and agriculture. It appears that the first three industries for which the reduction in unweighted error rates is most significant, are industries for which the OJD model 1 ypf performs relatively bad (i.e. industries for which the model has very high original unweighted error rates). When looking at the validation results of the OJD 1991 model 3 years prior to failure based on the original cut-off point, we notice that the OJD model performs best for the industry of financial services (29,08%). Besides, textiles and apparel (31,08%), paper and printing (31,37%) and construction (31,50%) show low unweighted error rates. On the other hand, the hotel, restaurant and catering industry (40,58%) and personal services (37,53%) are the industries for which the model performs worst. The validation results of the OJD 1991 model 3 years prior to failure based on the new cutoff points again indicate more or less the same industries as the ones for which the model performs best and worst. The model performs best for financial services (24,00%), textiles and apparel (28,88%), chemicals (29,70%) and paper and printing (29,78%). On the other hand, the model shows the highest unweighted error rates for the hotel, restaurant and catering industry (36,91%), personal services (34,39%) and wholesale (34,25%). As can be concluded from table 10, the use of the new cut-off points instead of the original one allows for a reduction in the unweighted error rates of the OJD model 3 years prior to failure for all industries. Just like the validation results of the 1 ypf model, there is a significant absolute (1,6%) and relative (almost 5%) reduction in the unweighted error rate of the model. The reduction in the unweighted error rates is most significant for the following industries: financial services, the hotel, restaurant and catering industry, personal services and agriculture. Two of these industries, more in particular personal services and the hotel, 21

22 restaurant and catering industry, are also mentioned as the ones for which the 3 ypf model performs worst. 22

23 Table 10 : Performance results of the OJD 1991 models with respect to industry 4 Model 1 year prior to failure Industry Type I-error, original Type II-error, original UER, original New cut-off point Type I-error, new Type II-error, new UER, new Gini, new Absolute change in UER Relative change in UER Agriculture 24,39% 31,91% 28,15% 0, ,96% 15,58% 25,27% 57,22% -2,88% -10,23% Utilities n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. Metal 21,87% 25,84% 23,86% 0,309 21,23% 25,84% 23,54% 63,88% -0,32% -1,34% Food 26,44% 35,55% 31,00% 0, ,48% 23,70% 29,09% 49,01% -1,91% -6,15% Chemicals 19,40% 24,01% 21,71% 0, ,88% 16,41% 20,15% 68,62% -1,56% -7,19% Textiles and apparel 20,09% 28,10% 24,10% 0,409 25,23% 22,02% 23,63% 65,72% -0,47% -1,95% Timber 24,20% 26,50% 25,35% 0, ,11% 21,37% 23,74% 62,94% -1,61% -6,35% Paper and printing 17,31% 27,95% 22,63% 0, ,94% 30,66% 22,30% 67,05% -0,33% -1,46% Construction 31,70% 23,06% 27,38% 0, ,14% 33,28% 25,71% 61,02% -1,67% -6,10% Wholesale 25,59% 30,51% 28,05% 0, ,22% 31,59% 27,91% 54,95% -0,14% -0,52% Retail 24,55% 30,16% 27,36% 0, ,80% 25,28% 27,04% 58,31% -0,31% -1,15% Hotel, restaurant & 19,00% 54,25% 36,63% 0, ,81% 28,96% 32,39% 42,85% -4,24% -11,58% Transportation 25,57% 30,03% 27,80% 0, ,51% 25,37% 27,44% 55,83% -0,36% -1,29% Real estate 29,64% 44,87% 37,26% 0, ,53% 16,23% 33,38% 43,52% -3,88% -10,40% Business services 29,87% 32,56% 31,22% 0, ,29% 28,08% 30,69% 50,02% -0,53% -1,70% Personal services 17,32% 46,49% 31,91% 0, ,61% 30,42% 30,02% 51,38% -1,89% -5,92% Financial services 28,00% 38,07% 33,04% 0, ,87% 32,00% 28,44% 52,93% -4,60% -13,92% No industry specified 28,92% 40,68% 34,80% 0, ,60% 16,73% 33,67% 42,40% -1,14% -3,26% Average 24,34% 33,56% 28,95% 0, ,72% 24,91% 27,32% 55,74% -1,64% -5,66% Model 3 years prior to failure Industry Type I-error, original Type II-error, original UER, original New cut-off point Type I-error, new Type II-error, new UER, new Gini, new Absolute change in UER Relative change in UER Agriculture 11,22% 54,56% 32,89% 0,366 29,59% 30,73% 30,16% 48,22% -2,73% -8,30% Utilities n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. Metal 28,22% 37,01% 32,62% 0, ,67% 43,08% 31,88% 44,16% -0,74% -2,27% Food 23,31% 44,25% 33,78% 0, ,38% 38,90% 32,64% 40,85% -1,14% -3,37% Chemicals 28,81% 34,95% 31,88% 0, ,51% 28,88% 29,70% 49,05% -2,19% -6,85% Textiles and apparel 28,99% 33,16% 31,08% 0, ,29% 37,47% 28,88% 47,78% -2,20% -7,06% Timber 26,52% 39,03% 32,78% 0, ,64% 25,07% 31,86% 46,32% -0,92% -2,81% Paper and printing 18,69% 44,04% 31,37% 0, ,69% 40,87% 29,78% 47,98% -1,59% -5,05% Construction 24,00% 39,00% 31,50% 0, ,13% 38,57% 31,35% 45,44% -0,15% -0,48% Wholesale 20,74% 48,52% 34,63% 0,236 24,33% 44,16% 34,25% 38,96% -0,39% -1,11% Retail 15,91% 50,73% 33,32% 0, ,51% 33,68% 32,10% 45,32% -1,23% -3,68% Hotel, restaurant & 15,21% 65,95% 40,58% 0, ,37% 42,44% 36,91% 31,88% -3,68% -9,06% Transportation 19,04% 45,11% 32,08% 0, ,40% 48,50% 31,45% 46,89% -0,62% -1,95% Real estate 17,12% 51,63% 34,38% 0, ,50% 47,14% 33,82% 42,90% -0,56% -1,61% Business services 19,77% 47,44% 33,61% 0,234 23,54% 42,60% 33,07% 42,11% -0,53% -1,59% Personal services 16,50% 58,56% 37,53% 0, ,36% 38,42% 34,39% 36,45% -3,14% -8,37% Financial services 4,35% 53,81% 29,08% 0,2745 4,35% 43,65% 24,00% 61,24% -5,08% -17,47% No industry specified 22,76% 41,39% 32,08% 0, ,27% 44,04% 31,16% 46,95% -0,92% -2,87% Average 20,07% 46,42% 33,24% 0, ,91% 39,31% 31,61% 44,85% -1,63% -4,92% 23 4 For the utilities industry, performance results could not be calculated, since the occurence of failing firms in this industry is neglectable (see table 3).

24 In figures 2 and 3 the type I, type II and unweighted error rates for the OJD model 1 year and 3 years prior to failure, based on the newly calculated cut-off points for the 18 industries, are presented. This allows us to make a comparison between the performances of the OJD model 1 year prior to failure and the OJD model 3 years prior to failure. Figure 2 : Type I, type II and unweighted error rates of the OJD 1991 model 1 ypf for the 18 industries, based on the new cut-off points 60,00% 50,00% 40,00% T ype I error Type II error Unweighted error rates 30,00% 20,00% 10,00% 0,00% Agriculture Utilities Metal Food Chemicals Textiles and apparel Timber Paper and printing Construction Wholesale Retail Hotel, restaurant & catering Transportation Real estate Business services Personal services Financial services No industry specified Average Figure 3 : Type I, type II and unweighted error rates of the OJD 1991 model 3 ypf for the 18 industries, based on the new cut-off points 60,00% 50,00% 40,00% Type I error Type II error Unweighted error rates 30,00% 20,00% 10,00% 0,00% Agriculture Utilities Metal Food Chemicals Textiles and apparel Timber Paper and printing Construction Wholesale Retail Hotel, restaurant & catering Transportation Real estate Business services Personal services Financial services No industry specified Average Comparison of figures 2 and 3 reveals that, on average, the OJD model 3 ypf has higher unweighted error rates than the model 1 ypf. This means that the 1 ypf model generally has a better predictive ability. In other words, it is easier to predict which firms will fail in the near future than to make long term failure predictions. Furthermore, it is clear that, in comparison with the 1 ypf model, the 3 ypf model generates higher type II errors and lower type I errors. 24

25 This means that a large number of non-failing firms show signs of financial weakness on a 3 year prior to failure model. More in particular, we find these higher type II errors in the following industries: real estate, financial services, transportation, business services and the no-industry category Performance results of the Ooghe-Joos-De Vos 1991 failure prediction models with respect to sizeclass In this section, the results of the validation of the OJD 1991 models on the three different sizeclasses, that are presented in table 11, are discussed. Table 11 : Performance results of the OJD 1991 models with respect to sizeclass Sizeclass Type I- error, original Type IIerror, original Unweighted error rates, original Model 1 year prior to failure New cutoff Type I- Type IIerror, point error, Unweighted error rates, new Gini, new Absolute change in UER Relative change in UER new new LCs 22,08% 24,41% 23,25% 0, ,29% 24,07% 23,18% 65,29% -0,06% -0,28% SMCs 25,23% 28,87% 27,05% 0, ,39% 25,43% 26,91% 68,01% -0,14% -0,52% CWEs 27,56% 39,92% 33,74% 0, ,27% 21,51% 32,39% 48,87% -1,35% -4,00% Average 24,96% 31,07% 28,01% 31,32% 23,67% 27,49% 60,72% -0,52% -1,85% Sizeclass Type I- error, original Type IIerror, original Unweighted error rates, original Model 3 years prior to failure New cutoff Type I- Type IIerror, point error, Unweighted error rates, new Gini, New Absolute change in UER Relative change in UER new new LCs 42,57% 28,04% 35,31% 0, ,47% 29,64% 34,56% 37,11% -0,75% -2,12% SMCs 18,95% 46,09% 32,52% 0,264 26,62% 37,22% 31,92% 57,11% -0,60% -1,85% CWEs 17,85% 49,65% 33,75% 0, ,64% 34,76% 32,70% 47,11% -1,05% -3,11% Average 26,46% 41,26% 33,86% 32,24% 33,87% 33,06% 47,11% -0,80% -2,36% As can be seen from the validation results of the OJD 1991 model 1 year prior to failure, based on the original cut-off point and on the newly calculated cut-off points for the three sizeclasses, the unweighted error rates are the lowest for the group of large companies (respectively 23,25% and 23,18%), whereas they are the highest for the group of companies without employees (respectively 33,74% and 32,39%). Consequently, the OJD model 1 year prior to failure performs best in failure prediction of large companies and performs worst in failure prediction of companies without employees. When using the newly calculated cut-off points instead of the original ones, we notice a significant absolute (1,35%) and relative (4,00%) reduction in the unweighted error rate of the 1 ypf model for the companies without employees. This is the sizeclass for which the 1 ypf 25

26 model performs worst (i.e. for which the model has the highest original unweighted error rates). For the other sizeclasses, we do not find any significant reductions in the unweighted error rates. Contrary to the validation results of the 1 ypf model, the validation results of the OJD 1991 model 3 years prior to failure, based on both the original cut-off point and the newly calculated cut-off points, show that there are no significant differences in the performances of the model. The ranges of unweighted error rates of the 3 ypf model, both based on the original and the new cut-off points, are much smaller (respectively [32,52%;35,31%] and [31,92%;34,56%]) than the ranges of the 1 ypf model (respectively [23,25%;33,74%] and [23,18%;32,39%]). However, if a sizeclass has to be appointed as the one for which the 3 ypf model performs worst, it should be the group of large companies. Furthermore, table 11 reveals no significant differences between the sizeclasses with respect to the reduction of the unweighted error rates when using the new cut-off points instead of the original ones. Nevertheless, the reduction in the unweighted error rate is most significant for the group of companies without employees. In figure 4 the type I, type II and unweighted error rates for the OJD 1991 models 1 year and 3 years prior to failure, based on the newly calculated cut-off points for the three sizeclasses, are presented. Figure 4 : Type I, type II and unweighted error rates of the OJD 1991 models 1 ypf and 3 ypf with respect to sizeclass, based on the new cut-off points 1 ypf model 3 ypf model 45,00% 40,00% 35,00% 30,00% 25,00% 20,00% Type I error Type II error Unweighted error rates 45,00% 40,00% 35,00% 30,00% 25,00% 20,00% Type I error T ype II error Unweighted error rates 15,00% 10,00% 5,00% 15,00% 10,00% 5,00% 0,00% BCs SMCs CWEs Average 0,00% BCs SMCs CWEs Average 26