Model Validation Specific Insolvency Rating Model

Transcription

1 Pension Protection Fund 28 May 2014 Pension Protection Fund Model Validation Specific Insolvency Rating Model

2 Purpose and use of this report This report has been prepared only for the Pension Protection Fund ( PPF ) and solely for the purpose and on the terms agreed with the PPF in accordance with our engagement letter dated 12 September We accept no liability (including for negligence) to anyone else in connection with this report. It may not be provided to anyone else without our prior written consent. This report must be read in its entirety; reading individual sections in isolation could be misleading. This report may contain information obtained or derived from a variety of sources. We have not sought to establish the reliability of all of those sources of information or otherwise verify the information so provided. Accordingly no representation or warranty of any kind (whether express or implied) is given by to any person (except to the PPF under the relevant terms of the Engagement) as to the accuracy or completeness of the report. Contacts julia.dickson@uk.pwc.com Tel +44 (0) Mob +44 (0) craig.d.prior@uk.pwc.com Tel +44 (0) Mob +44 (0) PricewaterhouseCoopers LLP 7 More London Riverside London SE1 2RT Tel +44 (0) Facsimile +44 (0) PricewaterhouseCoopers LLP is a limited liability partnership registered in England with registered number OC The registered office of PricewaterhouseCoopers LLP is 1 Embankment Place, London WC2N 6RH. PricewaterhouseCoopers LLP is authorised and regulated by the Financial Services Authority for designated investment business.

3 Contents 1. Executive summary Introduction PPF Success Criteria 3

4 1. Executive summary 1.1. Introduction Experian have created a PPF-specific insolvency score for the sole purpose of providing the PPF with an objective evaluation of the likelihood of any given levy-paying employer becoming insolvent during the next 12- months. The purpose of this report is to provide an independent review for the PPF on the model proposed by Experian. This Executive Summary provides a summary of our findings in relation to the procedures we have performed on the PPF-Specific Insolvency Risk Model ( the model ). During the review have employed their model validation framework to conduct the review of the model, and evaluated the models performance against the success criteria stipulated by the PPF. Overall, we are of the view that the model is better suited to score employers of the PPF universe than Commercial Delphi (Experian s off-the-shelf scoring model) and is an improvement on the other off-the-shelf generic models we have examined. In particular, we note that the model: shows increased predictive power in comparison to Commercial Delphi; demonstrates statistically significant levels of performance; produces insolvency scores that exhibit acceptable levels of stability historically; demonstrates increased operational resilience due to the use of primarily published financial data in scoring algorithms. However, we note the following limitations in relation to the development of the model in comparison to industry standard: The segmentation approach undertaken by Experian has been primarily driven by the availability of data, with a risk-based criteria being the secondary consideration. Industry practice is to take a riskbased approach to segmentation. Whilst Experian have not taken an industry practice approach, we are comfortable that an element of risk concentration is incorporated into the segmentation. The score of a parent is systemically incorporated into the model algorithms to derive a rating of employers that receive wider support group, without considering the degree of parental support. Industry practice is to directly substitute the credit rating of the parent to a subsidiary where formal guarantees are in existence. Where implied support exists subsidiary ratings are typically prudently allocated on a standalone basis or in exceptional cases overridden to reflect parental support. However, we view Experian s approach to be acceptable, pragmatic and statistically relevant due to the lack of a centralised credit function in the PPF dedicated to performing these overrides. Experian has split financial variables into discrete buckets in evaluating the relationship between the financial metric and observed insolvency rates. These are then transformed into continuous variables, recognising that a model built on a continuous basis is likely to have increased robustness, increased stability and reduced susceptibility to large step changes in model outputs. Industry practice is to derive these relationships using continuous variables directly, reducing the potential of model over-fitting and increasing model stability. This is the preferred approach; however, we recognise that the deterioration in model performance arising from Experian s approach will not be material. Experian have proposed individual models during the model development exercise based on the most statistically efficient solutions. Industry practice would be to produce a suite of candidate models for the consideration of the PPF, allowing key stakeholders to select the final model based on a balance between predictive power, stability, robustness, and diversity of financial metrics. In Experian s validation testing on the model, the following validations were not documented: o Accuracy of predicted default rates for each of the individual scoreband segments;

5 o Calibration methodology and rating philosophy employed in the model development process, which would inform the principles that would be used to determine frequency at which the model will be re-calibrated; o Benchmarking the scorecard variables to those contained within the models of other service providers. It should be noted, that the use of the model outputs for levy banding purposes was not included in the scope of this review and therefore is not contained within this report PPF Success Criteria At the outset of the development of the revised insolvency model the PPF defined a series of success criteria upon which the model was to be evaluated. These are listed below with a summary of findings against each of these criteria Table 1 Summary of Results Criteria A. Improved predictive power B. Adherence to best practice Comments The revised model has improved predictive power in comparison to Commercial Delphi (Experian s off-the-shelf scoring model). Predictive power is primarily measured by the Gini coefficient (the Gini ) where for portfolios of this nature a Gini in excess of 50% is deemed appropriate based on industry standards. The model has an aggregate Gini of 70.7% with individual sub components of the model having Gini s ranging from 49.0% 75.5% and therefore is in alignment with industry standards. No documentation of the model calibration methodology has been provided. The calibration is based on the entire development population from and hence is a predominantly through-the-cycle calibration approach which would seem appropriate for the PPF portfolio. Ongoing monitoring and validation is required to ensure that this remains appropriate. The model has been built in alignment with industry practice and understood to be fitfor-purpose. It should be noted however that, in our view, deviations from leading industry practice do exist, as listed below. These have been highlighted to and discussed with the PPF and Experian during the course of the model development process in order to determine whether timing and budgetary constraints would allow modifications to the model in light of these deviations. Portfolio segmentation Industry best practice would be to take a risk-based approach to segmentation of the portfolio (e.g. industry type and size of entity) in order to produce the most robust and efficient rating. Instead, the portfolio has been segmented primarily based on available data (type of corporate) and secondarily based on risk concentrations (e.g. changing default rates by employer size). As such, whilst Experian have not taken an industry practice approach, we are comfortable that an element of risk concentration is incorporated into the segmentation. Scoring algorithms for individual factors in the model were not derived on a continuous basis: Factor scores were derived using a discrete bucketing approach, reducing the degree of granularity of model outputs and potentially over-fitting the model to the available data. To overcome this Experian made factor scores continuous by linearly interpolating between the discrete factor scores for a given variable to increase the granularity of model outputs. While this approach partially addresses the limitations posed by using discrete factors, the interpolated continuous relationship is still dependent on the buckets used in the discrete approach and as such, an element of over-fitting risk still remains. Development of alternative candidate models: When developing credit rating

6 models, institutions typically create a number of potential candidate models, which are presented to key stakeholders for discussion. One preferred solution is then selected for implementation based on a combination of factors including predictive power, stability, resilience, diversity of financial metrics etc. During the PPF model development exercise Experian have not presented alternative models for consideration by the PPF, but instead we understand Experian have identified and proposed the most statistically efficient model for implementation. As such, the finalised model may not be one which is the most appropriate for the PPF, given alternatives have not been discussed and reviewed by an expert panel within the PPF. Use of core credit risk metrics following on from the point above Experian have not performed a review of factors contained within the model against those contained within alternative models employed in the industry. In our view, constrained candidate models using a broader, more diverse set of financial metrics should have been developed and a performance comparison should have been made against the statistically derived model chosen for implementation. A more diverse model with only a minor reduced level in performance efficiency would have further reduced the potential for model over-fitting. Use of parent score in scoring algorithm The score of a parent is systemically incorporated into the model algorithms to derive a rating of employers that receive wider support group, without considering the degree of parental support. Industry practice is to directly substitute the credit rating of the parent to a subsidiary where formal guarantees are in existence. Where implied support exists subsidiary ratings are typically prudently allocated on a standalone basis or in exceptional cases overridden to reflect parental support. However, we view Experian s approach to be acceptable, pragmatic and statistically relevant due to the lack of a centralised credit function in the PPF dedicated to performing these overrides C. Stability of model outputs Application of scorecards outside the development population scope the proposed model implementation is to firstly use consolidated scorecard for overseas entities. This proposed approach has the limitation of not considering differing systemic economic conditions across different countries. Secondly it is proposed that the consolidated scorecard be used for large companies. In both these instances the scorecards development population differs from the target population and is therefore not in line with industry practice. Employment of a model hierarchy given the broad range of employers in the PPF universe, it is commonplace in the financial services sector to implement a model hierarchy to accommodate multiple information sources available for assigning final credit grades to obligors. To illustrate, where rating agency grades are in existence, they can be preferred over the internal model outputs given the large degree of analysis undertaken by the rating agency when assigning long term issuer ratings. For example credit assessments may be improved due to evaluating risk drivers not contained within published financials such as management quality, contingent liabilities, industry prospects and competitive position within the sector etc. We recommend that a mechanism to incorporate additional information to the credit assessment process is developed, particularly for larger / externally rated companies. The stability of a model considers the degree to which the model output changes over time periods. In the validation analysis received to date this has been analysed based on the degree of grade migrations rather than through the use of any population stability indices. Overall, the analysis shows that the model exhibits acceptable levels of stability, instability occurs during the 2007/08 period, but this is driven by the underlying economic conditions combined with the chosen model calibration method. Overall no deterioration in model discrimination was observed.

7 D. Resilience to manipulation E. An appropriate level of transparency of calculation F. Successful transition from D&B failure scores G. Appropriate Appeals Mechanism H. Flexibility and coverage I. Cost of information requirements on employers The revised suite of models is very heavily reliant on audited financial data published at Companies House. The use of these metrics within the model substantially reduces the degree by which model outputs can be manipulated without fundamentally changing the underlying performance and financials of the business. It is however recognised that a degree of manipulation can still arise through changing the organisational structural and re-mortgaging but that these elements are core to business operations. We understand that The revised scorecards are expected to be published in a transparent manner when they become operational enabling all levy members to fully understand the calculation of their credit scoreband. This was descoped from the requirements of this review. A formal appeals process is still being developed and has not been contained within this validation exercise. Coverage considers the proportion of the population that can be rated by the model. The approach adopted for portfolio segmentation for model development purposes has sought to maximise the coverage of the model to the PPF s population whist at the same time ensuring appropriate model performance and efficiency. The proposed model suite covers 89% of the total population. The threshold for model acceptance based on the PPF success criteria is model coverage of more than 94%. It is noted that the remainder of the population will be scored automatically using the relevant scheme, industry or population averages and that the performance of this approach should continue to be monitored going forwards. Given the inputs to the model are filed accounts, the burden on employers for data collation is anticipated to be light. Should employers decide to disclose further information in an attempt to improve their score that would be their strategic decision and not one that is imposed upon them.

8 2012 PricewaterhouseCoopers LLP. All rights reserved. PricewaterhouseCoopers refers to PricewaterhouseCoopers LLP a limited liability partnership incorporated in England or, as the context requires, other member firms of PricewaterhouseCoopers International Limited, each of which is a separate and independent legal entity.