A response to PRA s consultation paper CP26/17 Model risk management principles for stress testing

Transcription

1 A response to PRA s consultation paper CP26/17 Model risk management principles for stress testing March 2018 Introduction UK Finance is pleased to respond to PRA s consultation paper CP26/17 Model risk management principles for stress testing 1. UK Finance represents nearly 300 of the leading firms providing finance, banking, markets and payments related services in or from the UK. UK Finance was created by combining most of the activities of the Asset Based Finance Association, the British Bankers Association, the Council of Mortgage Lenders, Financial Fraud Action UK, Payments UK and the UK Cards Association. Our members are large and small, national and regional, domestic and international, corporate and mutual, retail and wholesale, physical and virtual, banks and non-banks. Our members customers are individuals, corporates, charities, clubs, associations and government bodies, served domestically and cross-border. These customers access a wide range of financial and advisory products and services, essential to their day-today activities. The interests of our members customers are at the heart of our work. High-level points for consideration by the PRA The development of principles to assist in the identification of model risk in stress testing is helpful to our members. In our response we set out comments on each of the principles. However, the scope and focus of the consultation paper raises further points that we would welcome discussion and further interaction with the PRA before publication of the supervisory statement. Our view is that model risk can exist in all aspects of a model s use. Many models are used both in a BaU environment and in stress testing. We question why it is necessary to have different sets of principles for models based upon their use. The Bank may wish to develop a consistent set of principles for model risk in all models. If not, then we would welcome the principles for stress testing providing a link to other PRA model governance standards. Some of our members may choose to adopt the model risk and governance process for all models, model components or material models as they see fit outside of the stress testing purpose. We would welcome the Bank supporting this flexible approach. We understand that the scope includes models that include an expert judgment. In stress testing the use of expert judgment can be an important component. We think that it can be challenging to quantify the model risk arising from these processes and that controls to manage such risk should be of a different nature than those applicable to more statistically based models. We urge the PRA to remain flexible with regard to this matter and to learn from our members experience as well as its own observations. 1 UK Finance is the trading name of NewTA Limited. Company number: Registered address: 1 Angel Court, London, EC2R 7HJ

2 The PRA s expectations set out in Principle 2.1 is for the board of directors and senior management to understand model capabilities, model limitations, and the potential impact of model uncertainty. We have concerns related to the granularity level of Board oversight please refer to our detailed response. We think that it would be beneficial to make it clear the Board is responsible for the model risk management framework and that its attention should be on the aggregate model output rather than the specificities of individual models which rely on a greater level of subject matter expertise than it would be reasonable for a board member to have. We comment below in more detail on the draft principles.

3 Model risk management principles for stress testing Principle 1 Banks have an established definition of a model and maintain a model inventory P1.1 Definition of a model: Banks should establish their own definition of a model. When identifying models banks are expected to take into consideration: a) Calculation methods or systems that are based on statistical, financial or economic assumptions (e.g. impairment models, income models). b) Calculation mechanisms used to transform a set of parameters or values into a quantitative measure (e.g. scenario expansion models, probability of default models). c) Frameworks or systems where qualitative judgement is applied to generate quantitative results (e.g. where adjustments are made to address known model limitations). d) Calculation mechanisms where outputs of other models are used to calculate financial/risk measures (e.g. expected loss which uses the output of probability of default, loss given default and exposure at default models). In cases where calculation mechanisms are not classified as models, banks should ensure the risks associated with the implementation and use of such calculations are adequately understood, controlled, and documented as part of an established management control. Members recommend alignment with the requirement with SR Letter 11-7 by deleting Frameworks or systems in (c) and replacing with quantitative approaches. One can argue that interpretation of the proposed requirement extends the model definition to include qualitative approaches. We suggest the removal of the requirement relating to the risks associated with calculation mechanisms that are not classified as models. Members seek confirmation that all formulas set out by regulators to calculate an outcome that is used for regulatory reporting of the prudential risk and capital are excluded from the principles. In our view only the modelled inputs that are included. Clarity is also sought as to whether expert judgement based forecasting approaches are captured by the principles; if so, they should include a provision that, for these processes, the concept of independent challenge is different from the one applicable to statistical/mathematical models or scorecards - i.e. that the challenge for expert judgement approaches may be delivered by robust corporate governance controls, including subject matter expert participation in decision making, rather than by a technical independent review by a validation team. As the experience from the implementation of SR11/7 for US banks tells us, clarification of which modelling processes are captured by the rules is important to reduce the uncertainty around compliance and to avoid lengthy debates within banks themselves and between banks and their supervisors. P1.2 Model inventory: Banks should maintain a comprehensive set of information on models implemented for use, under development, or recently retired. The information should clearly identify model owners and users, and should also include all model uses and dependencies, i.e. models that depend or use the output of other models. A designated internal party should be responsible for maintaining the bank-wide inventory of all models. Any variation of a model which requires separate validation and approval should be classified as a separate model. We agree that an inventory should identify and capture the many different business uses of a model as each model could result in different model risk. However, we think that flexibility should be allowed in how these model uses are captured in the inventory.

4 Principle 2 Banks have implemented an effective governance framework, policies, procedures and controls to manage their model risk P2.1 Board of directors and senior management responsibility: The board of directors should establish a framework for the management of model risk and this should be adequately documented. Senior management is responsible for the execution and maintenance of the framework and should designate the roles and responsibilities for the framework to model owners, model users, and control and compliance functions. The board of directors and senior management are expected to provide challenge to model outputs and understand model capabilities, the model limitations, and the potential impact of model uncertainty. We think the Bank could consider reviewing and revising this principle. The requirement for model risk policy to set the roles and responsibilities of control and compliance functions is in our opinion too broad. We recommend that the focus should be to set expectations specifically for control and compliance with respect to the management of model risk. Our members also propose that the PRA amends the current requirement to align with Principle 2 Additional points for banks as proposed BCBS consultative document on Stress testing principles 2 and the EBA s consultation paper on institution s stress testing 4.2 para 25 3, which allow for the stress testing programme to be delegated to senior management and relevant committees. This would make it quite clear that the Board is responsible for the model risk management framework, but not the specificities of individual models. The relevant BCBS Principle states that The board, or an appropriately senior-level governance body, is expected to have an understanding of the material aspects of the stress testing framework that enables it to actively engage in discussions with senior management or senior experts that are responsible for stress testing and challenge key model assumptions, the scenario selection and the assumptions underlying the stress test. We propose the following amendment to the wording of this principle: The board of directors should require senior management to provide challenge to model outputs and understand model capabilities, the model limitations, and the potential impact of model uncertainty. This is consistent with SR 11-7 roles and responsibilities: Model risk governance is provided at the highest level by the board of directors and senior management when they establish a bank-wide approach to model risk management. As part of their overall responsibilities, a bank's board and senior management should establish a strong model risk management framework that fits into the broader risk management of the organization. That framework should be grounded in an understanding of model risk not just for individual models but also in the aggregate. The framework should include standards for model development, implementation, use, and validation. While the board is ultimately responsible, it generally delegates to senior management the responsibility for executing and maintaining an effective model risk management framework. Duties of senior management include establishing adequate policies and procedures and %29.pdf

5 ensuring compliance, assigning competent staff, overseeing model development and implementation, evaluating model results, ensuring effective challenge, reviewing validation and internal audit findings, and taking prompt remedial action when necessary. In the same manner as for other major areas of risk, senior management, directly and through relevant committees, is responsible for regularly reporting to the board on significant model risk, from individual models and in the aggregate, and on compliance with policy. Board members should ensure that the level of model risk is within their tolerance and direct changes where appropriate. These actions will set the tone for the whole organization about the importance of model risk and the need for active model risk management. P2.2 Model risk management policies: These should cover all aspects of model risk management, including model definitions; model development standards; model change; implementation; use; validation; review; and management sign-off. The policies should set out appropriate governance and challenge frameworks, and the roles and responsibilities of model owners, model users, and control and compliance functions. The prioritisation, scope and frequency of validation, review, and monitoring activities should also be set out in the policies. To implement an effective governance framework and manage the model lifecycle effectively, we believe that it is necessary to adopt a risk-based approach based on materiality of the models and risks, with due consideration given to model complexity. Put simply - and to recognise proportionality - an approach that encourages institutions to concentrate their effort on those models that really pose the greatest risk, rather than spread the same effort over a larger number of models is preferable and ensures that management attention is appropriately targeted. While the concepts of materiality and complexity are articulated for certain principles, it would be helpful for the PRA to make clear that these concepts should apply more generally across the principles and model lifecycle. P2.3 Model owners and control functions: Model owners should have accountability for model use and performance. Model owners should be responsible for ensuring that models are appropriately developed, implemented, used as intended, have undergone appropriate validation and approval, and are recorded and maintained in the model inventory. Control staff should have the authority to restrict the use of models and monitor any limits on model use. We note this is very similar with SR11-7. The role of model owner involves ultimate accountability for model use and performance within the framework set by bank policies and procedures. Model owners should be responsible for ensuring that models are properly developed, implemented, and used. The model owner should also ensure that models in use have undergone appropriate validation and approval processes, promptly identify new or changed models, and provide all necessary information for validation activities. We encourage the Bank to set out a clarification of the roles of model owner, model developer and model user. In our view there is a need to distinguish between the model developer and model owner roles. A model developer may not have full control (and therefore cannot always be expected to have full responsibility for how their model is subsequently used. In our view the model owner should be the business manager responsible for deciding to use a model for a specific business purpose.

6 P2.4 Role of Internal Audit (IA): IA should assess the overall effectiveness of the model risk management framework. IA should evaluate and independently verify whether model risk management practices are comprehensive, rigorous, and effective. P2.5 Use of external resources: If external resources are used for any model development, validation, or review activities, banks should be able to verify that these are conducted in accordance with their model risk management standards. Designated internal staff should be responsible for the work delivered by the external party, and should be able to address any issues identified either with model development or as a consequence of model validation. Principle 3 Banks have implemented a robust model development and implementation process and ensure appropriate use of models P3.1 Model purpose and design: The purpose, design, choice of parameters, mathematical theory, and underlying assumptions of a model should be appropriately documented and conceptually sound (appropriate for the intended business purpose), and supported by published research and generally accepted industry practice where appropriate. Particular emphasis should be placed on model limitations and, where possible, model results should be supported by a comparison with alternative theories/approaches, or by assessing the sensitivities of changes in model inputs. We recommend deletion of on model limitations and, where possible and replace by on model limitations and, where appropriate. We recommend addition of where available after to published research. P3.2 Use of data: The data used to develop a model should be assessed for quality and relevance. Where adjustments are made, proxies are used, or where the data are not representative of the bank s portfolio or asset mix, the impact should be justified and documented so that users are aware of the potential model limitations. P3.3 Testing: Appropriate testing of models should be conducted to take into account potential limitations, assess their robustness and stability over time, and across a variety of economic and market conditions, in particular those relating to periods of stress. Testing activities should be appropriately documented. P3.4 Documentation: Banks should have sufficiently detailed model documentation so that an independent third party with relevant expertise is able to understand how the model operates, identify its key assumptions and limitations, and replicate any parameter estimation and model results. Where a bank uses vendor models, it should have appropriate documentation on the approach to be able to validate the model.

7 P3.5 Use of judgement: Any judgements or model overlays that are used to modify the parameters, inputs and/or outputs of a model due to known model limitations should form a part of the development process, should be appropriately understood and documented, and should be subject to review and challenge by independent parties. Members propose to align the requirement with SR 15-18, by adding challenge by independent parties commensurate with the materiality, the impact and the complexity of the approach. The independent party conducting the challenge of the expert judgement can be different from the staff conducting independent validation of models. P3.6 Supporting systems: Model calculations should be implemented in information systems or environments which have been thoroughly tested for this purpose. The findings of any system/implementation tests should be documented. P3.7 Business involvement: Frontline business should play an integral part in the design and testing of models and should challenge the methods, the underlying assumptions, and the output of the models both at inception and on an ongoing basis. Frontline business are the users of the models, who serve as source of feedback and challenge of models, when appropriate. Our members propose to adopt the concepts of model owners and users as mentioned in this Consultation Paper, and we suggest the Bank includes the following clarifying text: Model Users rely on the Model output in performing their responsibilities in the context of their business objectives and requirements. They are responsible for ensuring the Model use is consistent with the Model s intent, and they provide feedback on the ongoing performance of the Model. Effective challenge should be performed by second line of defence, i.e. Model Risk Management. P3.8 Model uncertainty: Banks should demonstrate that model uncertainties are adequately understood, managed, monitored, reported, and accounted for in the results. Where conservatism is used to mitigate model uncertainty, banks should justify and document any such adjustments and demonstrate that the adjustments are intuitive from a business and economic perspective. Members would welcome clarification on the definition and scope of uncertainty. This is a particularly challenging concept as an appropriate assessment of model uncertainty will always be contextual. Does this embrace bias that is often overcome by introducing conservatism, overlays, and adjustments. Or does it embrace volatility that could lead to development of error bands? The PRA may wish to provide some examples and or guidance that it would expect our members to consider in its assessment including: Analysis of the impact of expert overlays (i.e. projections before and after overlays) and Sensitivity analysis of key assumptions (i.e. how projections change when changing key modelling assumptions) This would represent a reasonable and pragmatic approach, as deriving error bands around stress outcome would require several assumptions that in turn generate model error and hence cast doubts as to the reliability and usefulness of the exercise.

8 P3.9 Monitoring: Banks should perform periodic monitoring of model performance with a frequency commensurate with the nature and materiality of the models and risks, with due consideration given to model complexity. Our members would welcome clarification as to the meaning of periodic monitoring of model performance frequent performance for stress testing models. Whilst periodic monitoring is possible, subject to data availability, this is generally done against outcomes observed across the cycle. Stress projections are done against hypothetical, rare scenarios rather than frequently observed outcomes. Monitoring of model performance in stress conditions should therefore be seen as an attestation process that the selected stress assumptions continue to remain reasonable and intuitive. This can be achieved by running sensitivity analysis across a range of scenarios, benchmarking model outputs with what observed in previous stressed periods. This analysis is meaningfully done as part of the model validation process, e.g. yearly, rather than on a more frequent basis and using the longest possible time series not just one quarter or even one year of data. Please consider clarifying and revising the text. Principle 4 Banks undertake appropriate model validation and independent review activities to ensure sound model performance and greater understanding of model uncertainties. P4.1 Scope of validation and review: All model components (inputs, calculations and reporting outputs) should be subject to independent validation for both in-house developed models and vendor models. Any validation work undertaken by model developers and users as well as any material changes to already validated models or overlays should be subject to review by an independent party. The extent of validation and independent review should be appropriate with the overall use, complexity, and materiality of the models or changes to a model. We note that the current wording used in the requirement, i.e. All model components is all encompassing. We would point out that it is not always possible to validate all the inputs and calculations of all vendor models. It would be helpful to have a further dialogue on this matter to ensure there is a common understanding between the PRA and our members. Our members would welcome clarity as to whether the review by an independent party of overlays means review by the model validator. P4.2 Independence: The staff performing model reviews should be independent of the model development process to be able to provide a robust and objective view. The effectiveness of the independent challenge should be judged by the quality of the issues identified and the actions taken by model owners and management to address them. We recommend deletion of the sentence The effectiveness of the independent challenge should be judged by the quality of the issues identified and the actions taken by model owners and management to address them. In our opinion it does not add anything to the guidance. Our members would welcome clarity as to whether this sentence, if retained, encompasses the activities of the Internal Audit function as at P2.4 or whether this is instead setting an expectation for a complementary 2nd line of defence activity to be separately set up to provide oversight on the adequacy and effectiveness of the independent challenge on models, which we would not support. We encourage the Bank to review this principle to avoid any ambiguity and or conflict.

9 P4.3 Staff competence and influence: Banks should consider whether validation staff have: the necessary knowledge, skills, and expertise to perform model validations; an adequate degree of familiarity with the business, product, risk, and intended use of the model; and sufficient influence and stature within the bank to ensure that issues and deficiencies are escalated and addressed in a timely manner. P4.4 Treatment of model issues/deficiencies: When significant model deficiencies and/or errors are identified during the validation process, banks should consider whether the use of models should either be prohibited or only be permitted under strict controls and mitigants. The process of managing identified model issues should include the tracking of the outstanding issues and should be adequately documented. P4.5 Frequency of model validation: Banks should undertake regular revalidation of models to track known limitations and to identify potential new issues. Periodic reviews should be carried out with a frequency and level of rigour commensurate with the overall use, complexity, and materiality of the models. Responsible executive John Perry; Senior Consultant! john.perry@ukfinance.org.uk UK Finance, 5th Floor, 1 Angel Court, London, EC2R 7HJ