KPMG Model Risk Management Survey

Transcription

1 KPMG Model Risk Management Survey Discussion and Summary IACPM November 8, 2013

2 Background Model Risk Governance and Validation: A (Very) Brief History

3 Financial Models A long, long time ago A few basic models were used Interest Rate Risk (Balance Sheet Value measure and Net Interest Income/Margin) VaR for Trading Book Simple valuation models/tools Risk Ratings and Transition Matrices Portfolio Construction and Asset Allocation Loss Forecasting / Reserving Actuarial Models and Tables (Insurance) 3

4 Financial Models Over time, models have grown in number and increased their importance Expanded modeling use includes areas such as: Derivative and Financial Instrument Pricing, Risk, and Valuation (and much more complex derivatives) Securitization (cashflow and waterfall; highly structured products, synthetics) Credit Decisioning/Underwriting Credit Portfolio Management Credit Loss Modeling (PD/LGD/EAD; ALLL; OTTI) Operational Risk Capital (Economic Capital, CCAR/Capital Adequacy and Forecasting, Stress Testing) Risks associated with trading (Duration, Convexity; VaR, cvar) Counterparty Risk and Exposure (EE, PFE, EAD; CVA) Financial Reporting (CVA/DVA, Fair Value Balance Sheet) 4

5 Financial Models Largest Institutions Goldman Sachs, Morgan Stanley, JPMorgan Chase, Bank of America Merrill Lynch, Citigroup, Wells Fargo, Thousands of models, including some highly complex and interrelated, others very specific in their use Large, Complex Institutions Fannie Mae, Freddie Mac, FHLBs GE, Coca-Cola, AIG, MetLife, Prudential, State Street Banks > $50 Billion (SunTrust, Regions, Capital One, BBVA Compass) 100+ models/systems/tools Other Large Companies Financial Reporting Models, especially around derivatives or investments Financial Forecasting Models Insurance actuarial models and economic capital Commodity and FX hedging (exposure, hedge accounting, derivative valuation) 5

6 Model Governance Issues & Concerns Frequently-Seen Critical Regulatory Concerns on Model Risk Management: 1. An incomplete model inventory; 2. Lack of a robust method to update the model inventory on a regular basis; 3. Lack of independence of the model validation department; 4. Model validation failed to demonstrate effective challenge ; 5. Lack of data used in the model development process to support review/validation; 6. Lack of developmental evidence to substantiate model assumptions; 7. Lack of an explanation to support the application of expert judgment and model overrides; 8. Lack of a transparent process to generate important inputs for a model; (e.g. a bank fails to demonstrate how it determines the magnitude of PD shocks in a macroeconomic stress testing model.) 9. Fail to include relevant risk drivers in the model; this is particular common in macroeconomic stress testing. 10. Models have not been validated. 11.Failure to maintain comprehensive and up-to-date model documentation. 6

7 KPMG Survey: Objectives and methodology

8 Model Risk Governance Survey Introduction Introduction KPMG undertook a survey to confirm where industry practices were established and where there were areas of divergence. Areas of practical application remain in the interpretation of the guidance, and will require further consistent clarity from regulators. Until such time practices will vary across institutions, for example: Practical questions around which systems, processes, tools, calculations, and applications should be considered models in the sense of the interagency regulatory guidance, and which should not? How should model risk be assessed, both for individual models and in the aggregate given that practices and regulatory expectations are evolving, creating various points of uncertainty? Practical model validation issues including: At what point does a model update or a recalibration require re-validation? Is full validation required if no significant or material changes have been made to a model since its previous validation? Institutions also have governance standards concerns including: What are the implicit independence requirements on model risk management functions, and what exactly does independence mean; What classification and materiality criteria are standard requirements of all institutions and which criteria are optional depending on an individual profile; What actions would satisfactorily demonstrate the required level of executive management and Board engagement; and, How and under what circumstances might models not passing validation be approved for use? 8

9 Model Risk Governance Survey Objectives and methodology Objectives To identify and understand the following industry-applied model risk management practices in relation to regulatory model risk management guidance contained in OCC / Fed SR 11-7: - Model definition scope - Model classification - Model risk management governance - Validation process practices Model development and implementation issues Key attributes of model validation program and issues - Resources and staffing - Risk assessment of models - Regulatory issues and future trends Methodology Web-based survey conducted during mid 2013 among financial services executives responsible for model risk management. Total respondents: 60+ The respondents included a wide representation of financial institutions with varying asset sizes (less than $100B to over $500B): Global banks International - Non-U.S. banks** Global systemically important financial institutions G-SIFI Regional U.S banks Insurance companies * Global bank operating in the U.S under regulatory guidance and not necessarily defined as G-SIFI. ** Bank headquartered outside the U.S., with US Operations 9

10 Executive summary

11 Model Risk Governance Survey Executive summary Model definition A. Model definition The regulatory model definition prescribed in the guidance is consistently applied by respondents irrespective of size, complexity, and geographical footprint: Majority of respondents report that regulators have provided feedback on the definition (e.g., not broad enough) and/or have emphasized certain characteristics expanding scope. In-scope inventory includes percent of the following traditional models: risk management, regulatory and economic capital impact, stress testing and scenario analysis, and valuation. Divergence exists across institutions on including / excluding the following: product development, underwriting and pricing, compliance and marketing/targeting, client asset management, decision support/monitoring and, strategic / other business planning / forecasting models: Only 58 percent include rule based decision models (e.g., fraud) and 70 percent exclude maintaining formal control structure models 79 percent believe there are clear criteria/definitions with the institution to distinguish in/out of scope models. Institutions typically maintain a centralized model inventory, however this may vary based on the nature of institution, numerous distributed lines of business, model types, geographical footprint etc. Most utilize in-house/proprietary tools, databases, or spreadsheets to facilitate the model inventory management: Vendor provided and other tools are used to a lesser degree. Report an ongoing effort to further rationalize model inventory over the next 12 months For example include regulatory models and anti-money laundering (AML)/fraud models which have traditionally while excluding budget and forecasting, and economic scenario generation models. 11

12 Model Risk Governance Survey Executive summary Model classification B. Model classification Model classification methods commonly applied include one or more of the following: Materiality based: Financial or regulatory materiality / threshold defined by institution (e.g., Tier 1 model represents material financial statement impact or High model represents significant regulatory impact) Risk or factor based: Risk/factors generally translated into a tier or high, medium or low classification (e.g., high represents a new model or with significant model changes, or complex with high operational/process risk) Materiality/risk based combination: Combination of risk based factor, materiality and judgmental overlay. Model reliance, complexity, regulatory and financial impact are most common criteria used to determine model validation priority classification. The financial impact is the most important criteria followed by regulatory impact especially for large banks. The model tiering/classification methods are consistently applied across regional banks given the benefits of central control and a limited geographical footprint. Model classifications are mostly reviewed by institutions on a periodic basis (e.g., annually most common) or upon a significant model change/update or in reaction to other internally defined trigger event. Validation practices and standards vary based on the model classification (e.g., low rated model may only be subject to periodic monitoring): Respondents evenly divided on work effort associated with models not subject to a full independent review or outcome analysis i.e., may result in an ultra conservative approach or disproportionate attention on low risk models Varying practices may include higher documentation, back testing standards and level of staff assigned. 12

13 Model Risk Governance Survey Executive summary Governance C. Model risk management governance (MRMG) MRMG responsibility predominantly rests with Enterprise Risk Management (ERM) or Chief Risk Officer (CRO). Some respondents report this function falling under internal audit, credit or the business unit. Responsibility for assigning model classification generally rests with the independent model risk management (MRM) group or a combination of functions (e.g., business unit and MRM). MRMG policies clearly address the appropriate roles and responsibilities assigned to each function (e.g., model owners, developers, users, validation, compliance, etc) The primary roles of the independent MRM team are as follows: Establish model governance and validation and compliance program including policy and procedures Approve the models after development and validation, and control model use (i.e., sign-off on model) Review the model from a governance perspective Most significant actions taken by MRMG resulting from the changing regulatory environment include: Increased number of model validation staff (e.g., greater quantitative experience and education) Updated model inventory (e.g., fresh review of the inventory constituents vs. model definition / regulator input) Review of model risk governance infrastructure (e.g., policy, procedures, and use of technology) Increased scrutiny from senior management Increased urgency to validate regulatory models. The role of internal audit (i.e., 3rd line) has changed for most respondents in particular regional banks. Significant work still remains to enhance key less mature program attributes (e.g., model change management policy / procedures, and ongoing model monitoring / tracking program. 13

14 Model Risk Governance Survey Executive summary Model development practices D. Model development practices Most common model development and implementation issues include: Failure of model documentation to meet internal policy standard Insufficient model development life cycle documentation Limited explanation to support the application of judgment and model overrides Transparency of the models and how their output impacts business decisions Insufficient data used in the model development process to support review/validation especially in global banks Scheduling model validations with model users in a timely manner particularly for regional banks. Regulator expectations/emphasis regarding model development and implementation practices: Reduced implementations without validations, outcomes analysis, and/or documentation Developers gaining a better understanding of model use, data quality, justifying model overlays, and metrics Improved enterprise-wide model governance (e.g., clear lines between developer, validator and model approver) Enhanced documentation with emphasis on developmental evidence and sensitivity analysis Increased focus on pre-implementation validation (i.e., rigorous with thorough testing and approval process) First line of defense to clearly explain the data applicability and cleaning process More effective challenge by the MRM group during development. 14

15 Model Risk Governance Survey Executive summary Model validation practices E. Model validation practices Common model validation issues include: Input data and assumptions (e.g., data integrity and reliability from certain data sources, heavy reliance on management s subjective assumptions, etc.) Challenge scheduling validations in a timely manner Models used for purposes other than what was originally intended when designed and implemented Shortage of experienced and qualified staff to build and validate the models Incomplete model inventory documentation. Model finding and severity rating definitions (e.g., Satisfactory, needs improvement, unsatisfactory) are consistently applied by respondents across models. Half the respondents indicate that the validation scope and approach differs for internally developed vs. external 3rd party vendor models (e.g., depending on access to documentation and level of transparency to code, assumptions etc). Model performance is monitored on a regular basis (i.e., to meet the ongoing model performance monitoring and tracking requirements). MRMG staffing needs have increased for all respondents over the last 12 to18 months: Primarily for regulatory compliance purposes Resources (e.g., validators and developers) increased depending on maturity of program In many cases a new MRMG group, sub-function recently created. 15

16 Model Risk Governance Survey Executive summary Model risk assessment F. Model risk assessment The majority of respondents have not established a model risk appetite and associated limits. Less than half of respondents measure model risk and establish a reserve/buffer or provision in some form (e.g., set aside a model risk reserve as a component of regulatory or economic capital). Of those respondents that reserve, a 2-3 percent buffer on average is established using the following approaches: Subjective qualitative calculation included in regulatory or economic capital Component included in operational risk using a qualitative based metric (e.g., number and severity of issues) Judgmental qualitative add-on on top of regulatory or economic capital Include model uncertainty/imprecision incorporated in capital as part of CCAR/DFAST stress testing process May be based on model attributes (i.e., materiality, number of validations completed, findings from validations, severity of issues, level of model risk assessment/quality grade, or whether systemic models). Model risk aggregation is more common at larger institutions. Models assigned a model risk buffer/reserve include: Regulatory capital (including allowance for loan losses), economic capital and valuation models A low percentage of respondents reserve for all models. Respondents expect more regulatory pressure to quantify risk on an aggregated level. Banks are challenged to develop a consistent quantitative approach across model risk measures and types. Most respondents consider this area to be a work in progress. 16

17 Model Risk Governance Survey Executive summary Staffing G. Staffing and resources MRM staffing resources (e.g., developers/validators) has increased for all respondents the last months. Key drivers for resource changes include: Enhancement of overall MRM program Expanded model definition scope (e.g., increased types and number of models) Increased demand on ongoing monitoring and outcomes analysis. Model validation staff are segregated by majority of institutions by business unit, specialty, process, or model type (e.g., valuation, credit, Basel, risk models). Respondents report that model developers are generally not independent from users (i.e. this may limit the degree of effective challenge prior to validation). Most respondents don t have a formal dedicated model validation staff/ model user training program. Common challenges faced with scheduling and completing validations in a timely manner include: Pressure from model user to undertake and complete validation Insufficient number of staff Insufficient documentation by model owners Increased volume of models requiring validation Unresolved open issues and exceptions 17

18 2013 KPMG LLP, a Delaware limited liability partnership and the U.S. member firm of the KPMG network of independent member firms affiliated with KPMG International Cooperative ( KPMG International ), a Swiss entity. All rights reserved. NDPPS The KPMG name, logo and cutting through complexity are registered trademarks or trademarks of KPMG International. Contact Details: Mark J. Nowakowski Principal, Financial Risk Management KPMG LLP (404) mnowakowski@kpmg.com