INTERAGENCY GUIDANCE ON THE ADVANCED MEASUREMENT APPROACHES FOR OPERATIONAL RISK Robert Rell February 29, 2012 Disclaimer: The views expressed do not necessarily reflect the views of the Federal Reserve Bank of Philadelphia or the Federal Reserve System.
The purpose of risk management is not to eliminate risk, but to manage it in a prudent manner.
DEFINING OPERATIONAL RISK Risk of monetary losses resulting from inadequate or failed internal processes, people, and systems, or from external events, such as natural disasters. Includes legal risk Operational risks are complex and quantification methods are still evolving
ADVANCED MEASUREMENT APPROACH (AMA) FOR OPERATIONAL RISK UNDER BASEL II Advanced approaches rule (Dec 2007) Purpose of AMA is to enhance operational risk measurement and management Under the AMA, a banking organization will use its internal model-subject to supervisory approval-to determine its regulatory capital requirement for operational risk. Consistent with sound and rapidly evolving industry practices
ADVANCED MEASUREMENT APPROACH (AMA) FOR OPERATIONAL RISK UNDER BASEL II Provides greater sensitivity (in contrast to simple approaches) Attempts to balance need for flexibility (to foster continued innovation) with the need for consistency of application
ADVANCED MEASUREMENT APPROACH (AMA) FOR OPERATIONAL RISK UNDER BASEL II In order for a banking organization's AMA to be approved, a number of supervisory standards must be met. Standards can broadly be grouped into three categories: corporate governance, data, and quantification Framework flexibility Measurement and management programs, processes, and tools Appropriate relative to bank s activities, business environment, and internal controls
INTERAGENCY GUIDANCE Issued June 3, 2011 Agencies expect operational risk discipline will evolve and converge toward more narrow range of practices Based on industry research, experience, and observed best practices Discusses certain common implementation issues and challenges and considerations for addressing Focuses on four required AMA elements Internal operational loss event data External operational loss event data Scenario analysis Business environment and internal control factors
INTERAGENCY GUIDANCE Attempt to make supervisory objectives clearer Prescriptive where appropriate, but generally allows for considerable flexibility Will likely evolve based on visitations and industry developments http://www.federalreserve.gov/bankinforeg/srletters/sr1108a1.pdf
GOVERNANCE ELEMENTS OF THE AMA Independent firm-wide Operational Risk Management Function (ORMF) Design, implementation, and oversight Stature commensurate with bank s profile Line of Business Management Oversight Responsible for day-to-day risk management Independent Testing and Verification (e.g., audit) Banks should be prepared to demonstrate that their operational risk governance structures are independent, have appropriate stature within the organization, and are consistent with an effective system of controls and oversight.
CHART FROM RMA Source: The Value of Clear Roles and Responsibilities in the Management of Operational Risk - RMA Journal Feb 2012
GOVERNANCE Board and senior management oversight roles should be detailed and communicated clearly Independent enterprise-wide operational risk framework and function with clear delineation Policies and procedures for all aspects of the operational risk framework Reporting of relevant operational risk exposures, losses, risk indicators to board and management in easily understood manner Sound internal control environment Frequent restructuring of ORMF is a red flag
Quantification Systems Internal Data External Data Scenario Analysis Business Environment & Internal Control Factors Operational Risk Capital Calculation
Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted. Albert Einstein
INTERNAL OPERATIONAL LOSS EVENT DATA Gross operational loss amounts, dates, recoveries, and relevant causal information Operational risk data and assessment systems must include a historical observation period of at least 5 years for its internal data Consider longer if not including tail events For quantification, many banks currently use a loss distribution approach (LDA) Frequency Severity
INTERNAL OPERATIONAL LOSS EVENT DATA Scrutiny when bank excludes internal data from the estimation of operational risk severity, particularly the exclusion of tail events Consider both impact of external data and scenario analysis for meaningful estimates of exposures Bank permitted to refrain from collecting internal data for individual operational losses below established dollar thresholds Documented support to demonstrate thresholds are reasonable, do not exclude important data, and permit the bank to capture substantially all the dollar value of the bank s operational losses.
LEGAL LOSS DATA Presents challenges given time lag between initiation and settlement Can significantly impact exposure estimates To address these potential differences and ensure that a bank s operations risk capital reflects its risk profile, a bank should include legal losses in its quantification processes using a date no later than the date a legal reserve is established. Written policies to describe practices
EXTERNAL OPERATIONAL LOSS EVENT DATA Occurring at organizations other than the bank Can provide useful information on potential areas of risk exposures based on industry loss experience Can be useful in scenario analysis and BEICF process Inclusion of external data with other data elements can support development of a comprehensive risk profile
EXTERNAL OPERATIONAL LOSS EVENT DATA Commonly obtained from publicly available sources or consortia Data from different sources can have varied characteristics Publicly sourced data Generally more descriptive information on individual operational loss events and their underlying causes Challenge of inherent reporting bias, tendency of publically reported losses to focus only on larger, more notable losses. Banks should addresses these biases in their methodologies
EXTERNAL OPERATIONAL LOSS EVENT DATA Consortia data Typically less descriptive Broader range of operational loss events Not subject to same reporting bias, but banks may face challenges in determining data relevance and scaling. Banks should demonstrate that the external data they use are relevant to their risk profiles and appropriate for use in their AMA frameworks
EXTERNAL OPERATIONAL LOSS EVENT DATA Carefully consider and adequately document how incorporated into quantification systems Supervisors will closely scrutinize a bank s approach for combining internal data and external data at the observation level, and will analyze a bank s statistical evidence and rationale for why such an approach is valid. Weighting scheme should have well-documented empirical support, including sensitivity analysis
EXTERNAL DATA USE IN OPERATIONAL RISK MEASUREMENT External data can be used in a benchmark approach Separate model from base internal data model Use as comparative data Some may not be relevant to particular bank s risk profile External data filtering or scaling methodologies to compensate Clear policies around exceptions criteria
EXTERNAL DATA USE IN OPERATIONAL RISK MEASUREMENT When size differs from external dataset representation it may be appropriate to scale Bank must provide empirical support demonstrating that its scaling methodology is credible, transparent, systematic, and verifiable. Same for third parties
When models turn on, brains turn off. Til Schulman
SCENARIO ANALYSIS Systematic process of obtaining expert opinions from business managers and risk management experts to derive reasoned assessments of the likelihood and loss impact of plausible, high-severity operational losses Forward-looking view that complements historical data Allow for better identification and preparation for risk exposures
SCENARIO ANALYSIS Exercises for subject matter experts to identify potential operational events and their impacts Skill and expertise of facilitators and participants Responsive to internal and external environment changes ORMF oversight, business line and subject matter representation Mitigation of bias Overconfidence, motivational bias, availability bias, partition dependence Justification for loss frequency and severity estimates Clearly defined, repeatable, and transparent
SCENARIO ANALYSIS Given the subjective nature of scenario analysis, banks should implement mechanisms for identifying and mitigating biases inherent in the scenario development process High quality documentation of the reason and rationale Robust challenge process Process to evaluate and improve upon past scenario workshops
SCENARIO ANALYSIS - CHALLENGES Difficult to mix scenario data and observational data in a credible manner Supervisors will closely scrutinize a bank s approach to mixing internal and scenario data at the observation level, and will review statistical evidence confirming that such an approach is valid
SCENARIO ANALYSIS - BENCHMARKING May result in adjustment to base model Critical to demonstrate the credibility of the benchmark model through validation and appropriate documentation Demonstrate that: Scenario output can be credibly and transparently translated into an estimate for the bank s units of measures For given unit of measure, the risk exposures can be appropriately estimated using internal and relevant external data
SCENARIO ANALYSIS - BENCHMARKING Method for comparing benchmark vs. confidence interval should incorporate a range of possible outcomes, such as the calculation of a confidence interval around the point estimate of the base model Two possible results: Falls within confidence interval not statistically different than base model Scrutiny as nears limits of confidence interval (e.g. 95 percentile) Falls outside of confidence interval Investigate credibility of models
SCENARIO ANALYSIS BASE MODEL Reduction in exposure estimates acceptable only in extremely limited circumstances Not consistent with conservative risk assessment Scenario analysis as the base model Rare cases of insufficient internal data and relevant external data to derive UOM Documented Address paucity of data
BUSINESS ENVIRONMENT AND INTERNAL CONTROL FACTORS BEICFs are indicators of a bank s operational risk profile that reflect a current and forward-looking assessment of the bank s underlying business-risk factors and internal control environment Forward looking tools that complement the other data elements in the framework ORMF should be actively involved in development and monitoring Business line management should implement and use BEICFS as a component of day-to-day operational risk management
BUSINESS ENVIRONMENT AND INTERNAL CONTROL FACTORS Include risk and control assessments, key risk indicators, and audit evaluations Consistency across business lines may facilitate aggregation and reporting of risk driver Reporting within business lines should be appropriate and include both the identified risks and the corresponding controls aimed at mitigating those risks. Board reports Clear policy around the reporting of the results of the assessment process
BUSINESS ENVIRONMENT AND INTERNAL CONTROL FACTORS BEICFs are typically incorporated in the quantification process as indirect inputs to inform other data elements or determine ex post adjustments Bank must periodically compare the results of its prior BEICF assessments against its actual operational loses in the intervening period Need to recalibrate?
INDEPENDENT REVIEW - VALIDATION Requires that bank validates, on an ongoing basis, its advanced systems. Operational risk management processes Operational risk data and assessment systems Operational risk quantification systems Validation of AMA framework must include Evaluation of the conceptual soundness of the advanced systems Ongoing monitoring process that includes verification of processes and benchmarking Outcomes analysis process that includes back-testing
INDEPENDENT REVIEW - VALIDATION Formal policies Commensurate with size and complexity Independent, or be subject to an independent review of its adequacy and effectiveness Ensure individuals performing are not biased due to involvement with development Credible capital estimate?
VALIDATION GOVERNANCE AND DATA ELEMENTS Conceptual framework appropriate for size and complexity? Ongoing monitoring to assess whether framework was implemented effectively, remains appropriate, and is performing as intended Capture of internal and external data is complete Scenario and BEICF data are well supported and not bias Risk monitoring is effective Appropriate remediation is undertaken if deficiencies exist Validations must incorporate outcomes analysis
VALIDATION OF QUANTIFICATION SYSTEMS Ensure quantification systems generate credible estimates that reflect profile Validation of model inputs, outputs, assumptions, and methodology Ensuring conceptual soundness of system and that underlying theory and logic remain sound and appropriate Periodic evaluation of appropriateness of assumptions, parameters, inputs, outputs, and methodology Including comparisons of model to other models
INTERNAL AUDIT Requires internal audit function (independent of business line) that at least annually assesses the effectiveness of controls around system and reports findings to board (or committee) May be overlap between a bank s validation and audit activities Independent of the advanced systems development, implementation, and operation ORMF may perform validation work, provided that the work is reviewed by an independent party Some banks validate internal loss data for a given business unit using support from an independent party within the same business unit, supplemented with an ORMF review
INTERNAL AUDIT Some banks use the internal audit function to validate non-quantitative aspects of advanced systems This could present a conflict of interest--or at least the appearance thereof--in that a bank s internal audit function is expected to assess the controls, including validation, related to the advanced systems Objectivity of the review could be compromised If internal audit staff reviews validation work that was performed by other, distinct internal audit staff, the bank should be prepared to demonstrate that such an arrangement does not compromise the independence of the review. Any such arrangement would be subject to heightened supervisory scrutiny.
How well a company manages operational risk has everything to do with how resilient it is in a crisis and how adept at avoiding one.