Model Risk Management

Transcription

1 Model Risk Management Brian Nappi, Crowe Horwath 2017 Crowe Horwath LLP

2 Agenda Regulatory Perspectives on Model Risk Management Model Basics MRM Audit Considerations MRM Best Practices FAQ s 2017 Crowe Horwath LLP 2

3 Regulatory Perspective 2017 Crowe Horwath LLP 3

4 Regulatory Landscape On April Supervisory Guidance on Model Risk Management was issued by the Federal Reserve (FRB) FRB SR Letter 11-7 OCC bulletin was adopted January 12, 2012 by the Office of the Comptroller of the Currency (OCC) Reflects increased attention placed on model effectiveness by BSA/AML requirements and capital stress testing Applies to all institutions, irrespective of size More than six years since guidance issued, regulators still are seeing widely varying degrees of implementation at Banks, with some comprehensively complying and others at early stages of maturity Crowe Horwath LLP 4

5 Regulators DO Expect Risk-Based MRM Approach Banks MRM programs appropriately vary after implementation Program sophistication should reflect size and complexity of exposures and models used Program should apply appropriate knowledge and resources to significant models: different types of model may require resources with specific knowledge for model types 2017 Crowe Horwath LLP 5

6 Areas of Focus Management of key models such as AML, ALM, ALLL Key man risk Vendor model issues Model selection process Management s knowledge of model theory and construction Vendor back up Data quality and integrity Inventory completeness Sufficient documentation for key stakeholders Shifting focus from program adequacy to active model risk management (Policy compliance) 2017 Crowe Horwath LLP 6

7 Model Basics 2017 Crowe Horwath LLP 7

8 What is a Model? A model is a simplified representation of real world relationships among observed characteristics, values and events. Model Components Input Processing Reporting Assumptions Data May be output from feeder models Transforms inputs into estimates Uses quantitative methods Applies statistical, economic, financial or mathematical theories May use some qualitative approaches or adjustments Translates the model estimates into useful business information 2017 Crowe Horwath LLP 8

9 What is Model Risk? Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports Financial Loss Poor Business Decisions Incorrect or Misused Model Outputs Reputation Damage 2017 Crowe Horwath LLP 9

10 Sources of Model Risk? Model Risk Model Risk arises from Reliance on erroneous model estimates or incorrect interpretation of model results Incorrect use of models Unreliable Model Model Misuse 2017 Crowe Horwath LLP 10

11 Unreliable Models Mathematical Errors Conceptual Errors Erroneous Assumptions Excess Complexity Developmental Data Issues Management Overrides Model Drift 2017 Crowe Horwath LLP 11

12 Model Model Misuse Misuse Input Errors Output Errors Untrained Users Incorrect Results Interpretation Use Outside Model Design Disregarded Limitations Poor Maintenance 2017 Crowe Horwath LLP 12

13 A FEW Model Types Valuation & Pricing Credit Scoring Behavior (AML, anti-fraud) Risk Assessment Forecasting 2017 Crowe Horwath LLP 13

14 Internal Audit Approach and Focus 2017 Crowe Horwath LLP 14

15 Expectations of Internal Audit (SR Letter 11-7) Internal Audit should assess the overall effectiveness of the model risk management framework Internal Audit Role is not to duplicate model risk management activities Role is to evaluate whether model risk management framework and implementation is comprehensive, rigorous, and effective. If some internal audit staff perform certain validation activities, then they should not be involved in the assessment of the overall model risk management framework. Guidance has extensive detail on internal audit responsibilities 2017 Crowe Horwath LLP 15

16 Expectations of Internal Audit (SR Letter 11-7) Internal Audit A bank's internal audit function should assess the overall effectiveness of the model risk management framework, including the framework's ability to address both types of model risk described in Section III, for individual models and in the aggregate. Findings from internal audit related to models should be documented and reported to the board or its appropriately delegated agent. Banks should ensure that internal audit operates with the proper incentives, has appropriate skills, and has adequate stature in the organization to assist in model risk management. Internal audit's role is not to duplicate model risk management activities. Instead, its role is to evaluate whether model risk management is comprehensive, rigorous, and effective. To accomplish this evaluation, internal audit staff should possess sufficient expertise in relevant modeling concepts as well as their use in particular business lines. If some internal audit staff perform certain validation activities, then they should not be involved in the assessment of the overall model risk management framework. Internal audit should verify that acceptable policies are in place and that model owners and control groups comply with those policies. Internal audit should also verify records of model use and validation to test whether validations are performed in a timely manner and whether models are subject to controls that appropriately account for any weaknesses in validation activities. Accuracy and completeness of the model inventory should be assessed. In addition, processes for establishing and monitoring limits on model usage should be evaluated. Internal audit should determine whether procedures for updating models are clearly documented, and test whether those procedures are being carried out as specified. Internal audit should check that model owners and control groups are meeting documentation standards, including risk reporting. Additionally, internal audit should perform assessments of supporting operational systems and evaluate the reliability of data used by models. Internal audit also has an important role in ensuring that validation work is conducted properly and that appropriate effective challenge is being carried out. It should evaluate the objectivity, competence, and organizational standing of the key validation participants, with the ultimate goal of ascertaining whether those participants have the right incentives to discover and report deficiencies. Internal audit should review validation activities conducted by internal and external parties with the same rigor to see if those activities are being conducted in accordance with this guidance Crowe Horwath LLP 16

17 MRM Internal Audit Planning Considerations Build technical skills to understand and test controls around models Determine how you build model risk assessment into your riskbased audit universe & plan Look for models during risk assessments and incorporate impact on area risk Place sufficient focus on auditing model governance Include controls over development, implementation and use of significant business models in audit scope for relevant areas Leverage work done by others (validators and risk management personnel) 2017 Crowe Horwath LLP 17

18 Internal Audit Approach Applied A comprehensive Internal Audit approach evaluating an institution's model risk management process and use of models should be aligned with regulatory guidance. Crowe has developed a framework to capture regulatory expectations that we use to ensure we consider key aspects in developing our audit scope Crowe Horwath LLP 18

19 Audit considerations Model Inventory Clearly defined and defendable inventory of models Completeness and Accuracy Prescriptive methodology to assess model risks Model Development, Implementation, and Use Clear understanding of model inputs, data integration, and system functionality Assumptions & limitations Assurance for black box models Monitoring processes Model Validation Quantifying risk and exposure of gaps and limitations Corrective action to gaps and limitations Difference between validation and tuning and optimization 2017 Crowe Horwath LLP 19

20 Audit considerations Model Reporting and Governance Board & Sr. Management Oversight Policies and Procedures Compliance Organizational Structure for managing Model Risk Management Framework Model Definition Model Inventory Model playbook consistent development approach Documentation Standards Risk rating (complexity, uncertainty, assumptions uncertainty, breadth of use, impact) Validation Cycle Third Party Management 2017 Crowe Horwath LLP 20

21 Common Audit Issues Errors in model inventory Incomplete inventories Policies approved but not implemented, so numerous policy violations on start up Missing documentation for vendor model implementation Incomplete or past-due validations and reviews Lack of performance monitoring Lack of model risk reporting and monitoring Lack of supporting framework elements to ensure consistent implementation Lack of documentation for management adjustments/overlays 2017 Crowe Horwath LLP 21

22 Best Practices 2017 Crowe Horwath LLP 22

23 Comprehensive Model Risk Frameworks are Developed Framework assesses model risk for individual models and in aggregate Framework includes standards for Model identification Model inventory/tracking Model development, implementation & use Model Validation Model Maintenance and Updates Model Governance Senior management develops, maintains and implements framework 2017 Crowe Horwath LLP 23

24 Best Practices Data governance is a priority processes to identify key data and ensure it is complete and accurate have been implemented Model owners monitor performance on an ongoing basis, and tune models when appropriate Key stakeholders are educated to help them understand their role and key expectations Documentation for vendor models includes developmental information, including bank s support for configuration decisions and assumptions Validation is risk-based timing and extent of validation activities are determined based on risk, performance, and changes to the model or to the environment Regular monitoring to ensure program is dynamic and kept up-todate 2017 Crowe Horwath LLP 24

25 Model Validation, Review and Testing 2017 Crowe Horwath LLP 25

26 What is Model Validation Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. It helps reduce model risk by identifying model errors, recommended corrective actions, and appropriate model use. Key Considerations: Should assess overall reliability. Should assess impact of assumptions and identify limitations Should consider all model components input, processing and reporting Should be completed by parties with independence from development and use Should be completed by knowledgeable, skilled and expert practitioners 2017 Crowe Horwath LLP 26

27 What is Model Validation Model validation should not be considered a discrete event that occurs on a fixed schedule, even if risk-based. After the initial (pre-implementation) validation, the scope and frequency for review and validation activity is generally determined by risk Should be completed prior to first use of model, when results have high impact Should be completed on a minimum frequency post-production to ID deterioration Annual reviews should be completed to assess continued reliability Changes in model or environment may trigger partial or full re-validations 2017 Crowe Horwath LLP 27

28 What is Model Validation Pre-implementation validation is expected Supervisory Guidance on Model Risk Management requires an annual review of all models to determine whether testing or adjustments are needed. High risk models are typically validated every 1-2 years. However, some aspects may not need to be retested each cycle Pre- Implementation Validation Periodic Validation Procedures Annual Review Process 2017 Crowe Horwath LLP 28

29 Frequently Asked Questions 2017 Crowe Horwath LLP 29

30 What are common implementation challenges? Translating concepts into active management it s easy to develop policies and procedures, but harder to roll out Making MRM a first line issue (engaging management) Deciding what is a model Lack of quantitative and statistical skills in development, validation, and audit Setting performance standards Assessing model risk in aggregate terms 2017 Crowe Horwath LLP 30

31 How can I determine what is a model and what is a tool? Model identification continues to be an area of debate and disagreement. Why is it so confusing? Spreadsheets can be models, but not all complex spreadsheets are models Owners are not reliable in self-identifying models. Even subject matter experts may disagree ID the quantitative systems used & determine if They use assumptions They apply theories Results are an approximation or uncertain value. Controls are still appropriate for tools, just not the full set of MRM controls 2017 Crowe Horwath LLP 31

32 How do I know if my inventory is complete? Regulators are looking for controls over model inventory completeness to be in place. How to get a complete inventory? Ask for self identification by model owners (and get a certification from LOB leaders) ID all significant quantitative systems used & assess whether they are models Develop a list of typical models and check for these in the appropriate business areas Include as part of risk assessments Look for Missing Models 2017 Crowe Horwath LLP 32

33 Do I have to do anything special for vendor models? Vendor models should be subject to the Bank s model risk requirements Apply good third-party risk management practices. Robust processes should be in place for vendor model selection. Banks may not have access to application code, so must assess model reliability through other means, such as: Benchmarking Sensitivity analysis Customization choices and assumptions used in running vendor models should be documented and supported Even with vendor models, typically the Bank is inputting assumptions and should have good support for configuration and assumption decisions 2017 Crowe Horwath LLP 33

34 If we are buying a vendor model, what should we obtain from the vendor? Documentation explaining components, design, and intended use Test results or certification showing model is operating as intended Documentation of model s limitations & where use may be a problem Documentation of assumptions and related support, allowing Bank to determine whether they are applicable Communication regarding modifications (change control processes in place regarding code for the model software) 2017 Crowe Horwath LLP 34

35 Who are key stakeholders and what are their responsibilities? Business units manage model risk for models they use All groups are responsible for documentation and tracking of activities Model Owners development, implementation and use ensure models are suitable for use identify and control changes maintain information needed to conduct validation Model Control (Risk Control Staff or committee) Risk measurement, limits and monitoring Manage independent validation and review process, including resource selection Restrict use of models and monitor limits on usage ID exceptions to policy, with compensating mechanisms such as timelines for completion Model Users Policy compliance 2017 Crowe Horwath LLP 35

36 Do we need to establish an MRM committee? There is no right answer for a correct organizational structure Many banks with a large number of complex models do have committees Other banks handle model risk as part of its risk committee activities Structures vary from embedded MRM activities in the business to dedicated MRM departments 2017 Crowe Horwath LLP 36