Session 49 PD, Model Validation. Moderator: Sebastien Cimon Gagnon, FSA, CERA

Transcription

1 Session 49 PD, Model Validation Moderator: Sebastien Cimon Gagnon, FSA, CERA Presenters: Sebastien Cimon Gagnon, FSA, CERA Katherine M. Papillon-Rodrigue, ASA, CERA, MAAA Jeffrey S. Schlinsog, FSA, MAAA

2 Session 49: Model Validation Sebastien Cimon Gagnon Katherine Papillon Rodrigue Jeffrey Schlinsog

3 Overview of Model Validation Sebastien Cimon Gagnon, FSA, CERA 2015 Towers Watson. All rights reserved.

4 Agenda Intro to Model Validation Why Model Validation Current industry challenges Where companies are vs. best practices Other reasons Proposed Model Validation Framework High level review of current state Model assessment and documentation Independent model testing Remediation & maintenance Lessons Learned 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only. 3

5 Intro Intro to Model Validation 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

6 Model Risk Management in an ERM Framework Capital Management Risk Appetite & Limits Link to Business Strategy Risk Culture and Governance Risk Identification & Assessment Risk Management, Monitoring & Reporting Risk Measurement (models, stress & scenario tests) Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

7 Model Risk Management in an ERM Framework Capital Management Risk Appetite & Limits Link to Business Strategy Risk Culture and Governance Risk Identification & Assessment Risk Management, Monitoring & Reporting Risk Measurement (models, stress & scenario tests) Model Risk Management Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

8 Model Validation within Model Risk Management Model Development Model Validation Model Risk Management Elements Model Governance Model Use Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

9 Intro Why Model Validation 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

10 Current Challenges Increasing Reliance on Models Focus on Risk Management Sophisticated Policyholder Assumptions Black Box Calculations Increasing Regulatory Requirements Shorter Reporting Cycles More Complex Products Modeling Errors Leading to Restatements U.S. Life Industry Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

11 Historic Siloed Structure Corporate Finance Annuity Life Pension Variable Annuity Fixed Annuity UL / VUL Perm Institutional Annuities 401k Each sub-unit does their own thing: Builds their own models (often on different systems) Updates their models Generates production runs Analyzes their results Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

12 Inefficient Modeling Processes General Ledger Input Information Multiple Input Locations and Formats Model Build/ Compression Calculation Engine #1 Multiple Output Locations and Formats Input Information Input Information Multiple Input Locations and Formats Multiple Input Locations and Formats Model Build/ Compression Model Build/ Compression Calculation Engine #2 Calculation Engine #3 Multiple Output Locations and Formats Multiple Output Locations and Formats Management Reports Financial Statements Manual Input and Adjustments Engines & Analytics Limited Manual Reporting & Output Adjustments CFOs/CROs/Chief Actuaries are challenged with having confidence in the above process 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only. 11

13 Other reasons for Model Validation Regulatory requirements US ORSA (Effective January 1st, 2015) Sarbanes-Oxley (SOX) SR 11-7 Supervisory Guidance on Model Risk Management Systemically Important Financial Institution (SIFI) designation Solvency II Internal Model Approval Process (IMAP) ASOPs ASOP 46 Risk Evaluation in Enterprise Risk Management ASOP Exposure Draft - Modeling 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only. 12

14 Proposed Model Validation Framework 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

15 Model Validation Framework A targeted review of your current models and modeling processes consisting of the following steps. This must be tailored to your organization Phase 1 High Level Review of Current State Phase 2 Model Assessment & Documentation Phase 3 Independent Model Testing Phase 4 Remediation & Maintenance Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

16 Phase 1 High Level Review of Current State Discuss current modeling needs and uses Discuss target state or internal/external standards for your models Develop an inventory of models Define materiality thresholds Discuss perceived weaknesses and future expectations for the model 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only. 15

17 Phase 2 Model Assessment and Documentation Phase 2A Documentation of Current State Phase 2B Documentation of Gaps Assess and document each of your modeling steps leveraging existing documentation when available: Inputs Output Engine Reporting Analytics Compare the current state of your models to the target state Document gaps and determine potential corrective actions The documentation of these gaps is done in an Excel tool, the Review Log, that keeps track of every gap identified Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

18 Phase 3 Independent Model Testing Phase 3A Single Cell Testing Define the approach used for single cell testing Can involve parallel testing using separate software or independent assessment of your current software Select a sample of representative contracts for which parallel testing should be performed Phase 3B Aggregate Testing and Assumption Review Perform and document various aggregate tests: Reconciliation of model to actual inforce liability and asset data Assessment of inforce model building process Sensitivity testing for the key model drivers Static/dynamic validations Extreme scenarios and outliers Provide a thorough independent assumption review Review of key assumptions relative to experience studies and industry standards Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

19 Phase 4 Remediation & Maintenance High level summary report for senior management covering: Model purpose Users Overall processes Key findings Proposed remediation plan Redesigning modeling processes and controls with regards to: Model change tracking Centralized assumption storage Model governance and validation Documentation maintenance Discussions lead to detailed Remediation and Maintenance Plans Stream Activity 15 Oct 22 Oct 29 Oct 5 Nov 12 Nov 19 Nov 26 Nov 3 Dec 10 Dec 17 Dec 24 Dec 31 Dec 7 Jan 14 Jan Week#6 Week#7 Week#8 Week#9 Week#10 Week#11 Week#12 Week#13 Week#14 Week#14 Week#14 Week#14 Week#14 Week# Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only. 18

20 Lessons Learned 2015 Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

21 Lessons Learned From Prior Work The most overlooked step in a model validation project is project planning; it is important to have a proven model validation framework Start small to test your framework and templates, then you can expand to company wide initiatives Heavy documentation and processes are hard to maintain Degree of model validation varies depending on the risk and complexity of the business/model A Review Log often becomes the key document that everyone focuses on A model steward helps to ensure modeling best practices related to documentation, validation and governance Towers Watson. All rights reserved. Proprietary and Confidential. For Towers Watson and Towers Watson client use only.

22 Session 49: Model Validation Katherine Papillon-Rodrigue Model Validation Manager Life & Annuity Symposium May 5, 2015

23 Key Items from Strong Governance Framework Understanding the model inventory Understanding the company model risk exposure Having a clear Model Risk Policy 22

24 Elements of a Model Risk Policy Clear definition of what constitute a model Clear definition of the roles and responsibilities Frequency of model validation Thorough description of the validation process Impacts of the validation outcome 23

25 Elements of a Model Risk Policy Clear definition of what constitute a model Projection systems Valuation systems Assumptions End users tools Clear definition of the roles and responsibilities Business unit and corporate function Enterprise Risk Management Internal and external audit Regulatory oversight 24

26 Elements of a Model Risk Policy Frequency of model validation Periodic review Full validation Trigger for re-validation Commissioning/Decommissioning of models Thorough description of the validation process Impacts of the validation outcome Communication and acknowledgment of model status Risk acceptance Remediation plan for findings 25

27 Implementation Challenges from Modeling Owner s Perspective Culture change Additional responsibilities Lack of institutional knowledge for legacy models 26

28 Implementation Challenges from Validator s Perspective Culture change Evolution of process and structure Periodic review and refinement of the Model Risk Policy IT structure and licensing Knowledge and experience combination needed Line of business / product Model type Platform experience 27

29 Session 49 Model Validation Life & Annuity Symposium Jeff Schlinsog May 5, 2015

30 Model Validation Discussion Topics Governance Documentation Effective Challenge Maturity Pitfalls PwC Towers Watson. All rights reserved. 29 Proprietary and Confidential For Towers

31 Model Validation Governance Model Risk Committee Often reports up to Risk Committee Audit Committee Role Membership Model steward Model owners Model users Executive management Agenda MRM policy execution PwC Towers Watson. All rights reserved. 30 Proprietary and Confidential For Towers

32 Model Validation Model Steward Establish standards for model Development Testing Usage Responsible for execution of model risk management policy Model and model risk definitions Assessment of model risk Standards for model development and data quality in relation to model inputs Validation Implementation, modification and use Governance and controls PwC Towers Watson. All rights reserved. 31 Proprietary and Confidential For Towers

33 Model Validation Documentation - Content Statement of purpose Model design/ methodology Approximations Limitations Testing protocol Calibration Assumptions Sources Methodologies PwC Towers Watson. All rights reserved. 32 Proprietary and Confidential For Towers

34 Model Validation Documentation Best Practices Documentation starts at the inception of a model Content is known in advance Common contents Use of templates Documentation is peer reviewed for accuracy and completeness Final product is stored in a single location accessible to all users PwC Towers Watson. All rights reserved. 33 Proprietary and Confidential For Towers

35 Model Validation Effective Challenge Independence Independent of the model owner/user Not necessarily a third party Competence Experienced in the model subject matter Aware of industry practices and approaches Follow-up Sufficient influence for assessment and remediation PwC Towers Watson. All rights reserved. 34 Proprietary and Confidential For Towers

36 Model Validation Maturity Assessment Current state of model risk management framework can be assessed on a maturity scale Model changes are regression tested to ensure accuracy before release Periodic baselining of results Documented evidence of model approval process Theoretical justification Testing results Appropriate sign-off Limited to financial models No model inventory or assessment of riskiness of models Independent review of in scope models, possibly by internal audit. Requires design and implementation of validation tests, including: Replication using other models or external sources of information Comparison of model output to actual events Assessment of model controls Results of the exercise are clearly documented, including a clear description of the test, the stated purpose, test results, and recommendations for improvement Board and senior management take responsibility for understanding nature and level of model risk, and for fostering an appropriate culture. Governance framework outlines good practice concerning model purpose, scope, design, specification, build, testing, use, roles, responsibilities and standards in risk modeling. Robust Model Validation Process including escalation of key findings to appropriate authority. Planning of future development and validation activities. Operational 3 Lines of Defense Integrated with ERM framework Peer Review Control Focus Independent Assessment Robust MRM Framework PwC Towers Watson. All rights reserved. 35 Proprietary and Confidential For Towers

37 Model Validation Pitfalls Inadequate documentation Lack of resources Skill set Effective challenge Immature framework Day 2 items Unreasonable expectations Deferred maintenance / future issues PwC Towers Watson. All rights reserved. 36 Proprietary and Confidential For Towers