Session 56, Model Governance: What Could Possibly Go Wrong? Part II. Moderator: David R.W. Payne, MAAA, FCAS

Transcription

1 Session 56, Model Governance: What Could Possibly Go Wrong? Part II Moderator: David R.W. Payne, MAAA, FCAS Presenter: Dwayne Allen Husbands, FSA, MAAA David R.W. Payne, MAAA, FCAS Chad R. Runchey, FSA, MAAA

2 Model governance What could possibly go wrong? Day 2 Valuation Actuary Symposium Hollywood, FL August 29-30, 2016

3 Today s presenters Chad Runchey Chad.Runchey@ey.com David Paul David.Paul1@ey.com Dwayne Husbands Dwayne.Husbands@ey.com James Collingwood James.Collingwood@ey.com Page 1

4 Model Governance What could possibly go wrong? Day 2 agenda Model Risk Management - Day 1 overview and recap Presentation - Model validation Exercise 3 - Fleetville Insurance Model validation case study Closing remarks Page 2

5 Model risk management Day 1 overview and recap Page 3

6 Model risk management framework First line of defense (Model owners) Model life cycle Business purpose Model development Model implementation Model operation Performance monitoring Model change management Second line of defense (Model governance and validation) Model governance Enterprise governance Model review components Conceptual soundness Data and IT infrastructure Process and controls Independent testing Roles and responsibilities First line owns model life cycle and related activities Second line establishes model risk management framework and standards, provides reviews and challenges, and can perform independent model testing Third line performs independent testing and verification of first and second line activities Third line of defense (Internal audit) Page 4

7 Model risk management framework Responsibilities first line of defense (model owners) Objective: Manage the model risk of the organization by ensuring that models are developed, used and maintained consistent with enterprise policies. Focus on thoughtful and transparent model development, well controlled and tested model implementation, rigorous change management procedures and ongoing performance monitoring. Business purpose and model development Model operation and performance monitoring Clear purpose for the model Well documented rationale for key modeling decisions, including alternatives considered Use of developmental evidence and supporting analysis for key decisions Operates model in a manner consistent with intended use Regular reviews of model performance to confirm fit for purpose Monitoring includes confirmation of key assumptions and modeling choices Model implementation Transparency Model change management Clearly defined and comprehensive testing of calculations prior to use in production environment Appropriate reviews and sign-offs by senior management Formal controls over changes to models, including adequate testing to maintain validation integrity Formal governance process prior to change implementation Robust assumption management and review process Page 5

8 Model risk management framework Responsibilities second line of defense (model governance and validation) Objective: Manage the model risk of the organization by establishing and implementing a model risk management policy. Key roles include maintaining and monitoring model inventory, performing independent model validation and providing effecting challenge throughout the model development process. Enterprise governance Model validation Establish model risk management policy and governance body Perform regular validation on models, depending on prioritization Define models and maintain model inventory Determine prioritization approach and define requirements for model life cycle Procedures for second line usually focus on challenging judgment in model development process Detailed testing of calculations may be included Set standards Effective challenge Understand model risk Determine documentation standards and templates to assist first line of defense implementation Determine standards for model validators, including expectations around specific tests and approaches Develop a view of the key risk to the organization from models Determine potential impact for highest risk models and determine mitigation approaches Page 6

9 Model risk management framework Responsibilities third line of defense (internal audit) Objective: To ensure that the first and second lines of defense are complying with the MRM policies within the organization. First line review components Second line review components Confirm model documentation requirements are being followed throughout model development life cycle including development, implementation and performance monitoring Ensure all models included in reviewed processes are included on model inventory Confirmation Review model validation reports and confirm compliance with policy and presence of effective challenge Confirm model governance committee charters, agendas and meeting minutes are compliant with firm expectations Confirm testing performed during model development confirms with policy Confirm the execution of the model risk policy and validation schedules/calendar Page 7

10 Model validation Page 8

11 Model validation components 1 Conceptual soundness Applicability of model theory to use Consistency with industry practice Completeness and appropriateness of drivers and macro variables Linkage to applicable scenarios 2 Documentation and governance Model transparency and use Effective challenge by model owners and senior management Model risk mitigation and management Documentation completeness 3 Model performance and integrity Back-testing model performance on historical data Replication and re-performance of key model development Sensitivity analysis Benchmarking to industry sources/models Effective challenge 4 Implementation Assessment of SDLC process and controls End user computing controls Select replication of ex ante model modules Model change control procedures 5 Data quality Data sources and quality review (e.g., data dictionaries, data flows, existing data quality infrastructure) Data quality validation and testing (e.g., data anomaly testing) Variable generation logic (e.g., pseudo code) Application of SDLC process and controls Validation goes beyond testing the calculations and, in fact, may not include independent calculation replication if model owners/developers have performed adequate testing. Page 9

12 Independent validation scope options Adjusting model validation scope Validation scope can be tailored to each situation. The example below represents an indicative evaluation, which allows management to gain an understanding of the potential uncertainty caused by the use of these models as developed. Scoping may be performed by model class level or model-by-model. Review component Focus and depth options Overview Conceptual soundness Review methodology and compare to statistical and actuarial theory Quantify impact of alternative methodology choices for all potentially material items Review the conceptual soundness, including the developmental evidence Review key assumptions/limitations Assess applicability of model methodology to models in scope Documentation and governance Confirm that documentation meets minimum enterprise standards and governance is followed Detailed review of documentation, including review of support; include review of model change compliance Review model documentation Assess model risk mitigation process and framework Model performance and integrity Review model owner analysis for sensitivity of model and impact to key metrics Independently run the model under stressed and boundary assumptions/ conditions and review results Review model performance on a select basis High level assessment of model uncertainty Implementation Review implementation controls and detailed testing performed by model owner Independent testing of production model to confirm calculation accuracy Understand operational environment and controls for models in production Replication of model outputs Data quality Review data checks performed by model owner, including evidence of sign-off Detailed review of data sources, including reconciliation with original sources; confirm proper data usage Review documentation related to data sourcing and controls Page 10

13 Model validation approaches Model performance and integrity Effective model validation requires specific approaches that are relevant for the type and complexity of model. Sample validation approaches include, but are not limited to: Stress and scenario testing Reverse stress testing Back testing Sensitivity testing Simulation/convergence test Profit and loss attribution Challenger/benchmark models Replication Boundary tests Page 11

14 Exercise 3 Page 12

15 Exercise 3 Fleetville Insurance model validation case study Instructions Teams from Day 1 will be paired together and assigned roles; 1) Model Validators and 2) Model Developers Both groups are to review the document Fleetville Life Economic Capital Financial Model Overview Model Validation teams are responsible for generating questions and challenge the Model Development teams following their review of the model overview document Model Development teams are responsible for developing support/ explanations and remediation plans for their model in anticipation of questions from the Model Validation team (20 minutes) Conduct a role play discussion between the two teams (15 minutes) Teams are to switch roles and conduct another role play discussion (10 minutes) Page 13

16 Exercise 3 Fleetville Insurance model validation case study (continued) Model validation tips Pay attention to: Judgement decisions Model omissions Model errors Methodology considerations, including alternate approaches Analytics, sensitivities, etc. to support model decisions, methodology, etc. Never assume! Could this have been done differently? What would be the impact if you made this change? Page 14

17 Closing remarks Page 15

18 EY Assurance Tax Transactions Advisory About EY EY is a global leader in assurance, tax, transaction and advisory services. The insights and quality services we deliver help build trust and confidence in the capital markets and in economies the world over. We develop outstanding leaders who team to deliver on our promises to all of our stakeholders. In so doing, we play a critical role in building a better working world for our people, for our clients and for our communities. EY refers to the global organization, and may refer to one or more, of the member firms of Ernst & Young Global Limited, each of which is a separate legal entity. Ernst & Young Global Limited, a UK company limited by guarantee, does not provide services to clients. For more information about our organization, please visit ey.com. EY is a leader in serving the global financial services marketplace Nearly 43,000 EY financial services professionals around the world provide integrated assurance, tax, transaction and advisory services to our asset management, banking, capital markets and insurance clients. In the Americas, EY is the only public accounting organization with a separate business unit dedicated to the financial services marketplace. Created in 2000, the Americas Financial Services Organization today includes more than 6,900 professionals at member firms in over 50 locations throughout the US, the Caribbean and Latin America. EY professionals in our financial services practices worldwide align with key global industry groups, including EY s Global Wealth & Asset Management Center, Global Banking & Capital Markets Center, Global Insurance Center and Global Private Equity Center, which act as hubs for sharing industry-focused knowledge on current and emerging trends and regulations in order to help our clients address key issues. Our practitioners span many disciplines and provide a well-rounded understanding of business issues and challenges, as well as integrated services to our clients. With a global presence and industry-focused advice, EY s financial services professionals provide high-quality assurance, tax, transaction and advisory services, including operations, process improvement, risk and technology, to financial services companies worldwide EYGM Limited. All Rights Reserved ED none This material has been prepared for general informational purposes only and is not intended to be relied upon as accounting, tax, or other professional advice. Please refer to your advisors for specific advice. ey.com