Model Governance: Is YOUR Company There Yet? Intersection of Model Governance and Technology. Moderator: Joseph N. Soga, ASA, MAAA

Transcription

1 Model Governance: Is YOUR Company There Yet? Intersection of Model Governance and Technology Moderator: Joseph N. Soga, ASA, MAAA Presenters: Andrew Ching Ng, FSA, MAAA David Halldorson Darin G. Zimmerman, FSA, MAAA

2 Intersection of Model Governance and Technology Model Governance Is Your Company There Yet? August 31, 2016 Darin Zimmerman VP Actuarial Integrity, Transamerica

3 Agenda Introduction A Word about Controls Transamerica History with Model Governance Inventory of Models and Risks Approvals and signoffs Documentation Conclusion 2

4 Agenda

5 A Word About Controls It is incredibly hard to get actuaries to appreciate the value of controls We are financial engineers at heart and engineers are always trying to improve efficiency Faster Cheaper Less Waste Controls are engineered inefficiency Slower More costly Redundant Do this to prevent costly mistakes! It is the essence of the insurance premium: Pay a relatively small cost now to avoid a ruinous cost later

6 Transamerica History w/ Model Governance Started with Model Validation Inventory, then categorize our models into low, medium, and high risk models We needed a way to track this Validate methodology, assumptions, documentation, control procedures and then document findings We needed a way to track this, too

7 Inventory of Models and Risks Thus, the Risk Model Catalog was born: It has three Components The Model Inventory The SLIME log The Assumption and Methodology Repository

8 Inventory of Models and Risks Additional Definitions of a model a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information. Supervisory Guidance on Model Risk Management, Board of Governors of the Federal Reserve System, April 4, From Second Exposure Draft Proposed Actuarial Standard of Practice: Modeling Section 2. Definitions The terms below are defined for use in this actuarial standard of practice. 2.4 Implementation An executable form of a model. 2.8 Model A representation of relationships among variables, entities, or events using statistical, financial, economic, mathematical, or scientific concepts and equations. Models are used to help explain a system, to study the effects of different components, and to derive estimates and guide decisions. A model consists of: (1) a specification, (2) an implementation, and (3) one or more model runs Model Run The output of a model derived from a given set of input Specification A description of a model that identifies the inputs and their interactions with each other to produce output through logic, algorithms, or a set of mathematical formulas.

9 Inventory of Models and Risks The Model Inventory Captures Essential Model information Model Name Risk Rating (based on complexity, novelty, and inforce size) Model Owner and Model Supervisor Products, Riders Model Groupings

10 Inventory of Models and Risks The SLIME Log is NOT list of Problems to be fixed. SLIME log is Opportunities to increase precision Simplifications Limitations (in the modeling platform) Issues (as in Model Control Issues with run procedures) Maintenance (update assumption, new regulations) Errors This has been renamed, Financial Gaps Actuarial Error: Does model do what it was intended to do? Accounting Error: Is the number materially correct?

11 Inventory of Models and Risks The Assumptions and Methodology Repository This is Visionary thinking in our Modernization efforts to reduce model error risk Not just an inventory, but a source for populating models at runtime to ensure we are using the properly authorized assumption set for a particular run date. Three Critical Elements 1. Name 2. Review 3. Implementation

12 Transamerica History w/ Model Governance Next we remediated Errors and Gaps which gave us validated models Now what? How do we keep them this way? Recognized the need to keep models safe We also needed a formal change management process We pulled in a pure IT guy to crib IT best practices This lead to an automated workflow

13 Approvals and Signoffs

14 Inventory of Models and Risks We had to choose our words carefully Management wanted a secure environment In the short term we could only deliver a secure repository Significant IT involvement to achieve the former Trying to get to our hedging system

15 Approvals and Signoffs Started with breaking the process into 4 phases:

16 Approvals and Signoffs

17 Approvals and Signoffs CVC is an acronym for: Component Validation Control Use Beyond Compare to verify each file Supposed to change and did (GOOD) Not Supposed to change and didn t (GOOD) Changed when wasn t supposed to (BAD) Didn t change when it was supposed to (BAD) Compare Spreadsheets with Excel 13 Compare other MS files with Beyond Compare Use other platform tools for Axis, ALFA, MoSes, ArcVal, Poly Maintain database of reports for routine and Non routine

18 Approvals and Signoffs SOX Controls: Change Management or Production? IT Changes to Admin Systems & Record Layout Data Model Start Date, Policy Inventory, Asset inventory, Bridge Scenario, Sensitivity Multipliers, RBC Factors Non routine Change Management Assumptions Economic Scenario Files Updating Expense & Mortality Tables Expenses, Mortality, Morbidity, Surrender, Static & Dynamic Lapse, Competitor Rate, Crediting Strategy, Reinvestment Strategy, Long term Mean Reversion Logic Routine Production Set up DAC Spreadsheets Product Data File Setup, User Defined Function Coding, Database Coding, Changes to Spreadsheet Formulas Changes to Macros and VBA Formulas Changes to Table Names & Dimensions

19 Documentation

20 Conclusion Three years in, we still have a ways to go We ve made very modest technology investments Used widely available, widely understood tools

21 Thank you!

22 Intersection of Model Governance and Technology Andrew Ng David Halldorson 2016 SOA Model Governance Seminar August 31 st, 2016 Hollywood, FL

23 Definition of model Background Governance Technology Intersection A quantitative method or system that utilizes theories, methodologies and assumptions to represent, imitate, emulate or portray actual business outcomes Input components Data, assumptions Processing components Quantitative methods transform inputs into output Output components Mathematical representations of reality to help achieve a business application s objectives 2016 SOA Model Governance Seminar August 31,

24 Background Governance Technology Intersection Model risks Uncertainty within models invites for operational and financial risk. E.g., models not governed by locked down IT controls pose an operational risk decisions based on inadequate outputs can lead to a financial risk Two primary reasons leading to model risk *: The model may have fundamental errors and may produce inaccurate outputs when viewed against the design objective and intended business uses. The model may be used incorrectly or inappropriately * (SR Letter 11-7) - Guidance on Model Risk Management. Board of Governors of the Federal Reserve System 2016 SOA Model Governance Seminar August 31,

25 Model governance objective Background Governance Technology Intersection Ensure suitable production and application of model outputs using risk based approach framework with requirements governing all model lifecycle activities. An effective model governance framework should aim to: Enhance the consistency and quality of model results Manage risks inherent with models Improve overall efficiency and optimize deployment of modeling resources Improve communication relating to model usage throughout organization 2016 SOA Model Governance Seminar August 31,

26 Background Governance Technology Intersection Key elements of model governance Scope & applications Governance structure Model inventory and risk assessment Standards and policies governing model lifecycle activities Documentation Governance compliance reporting process 2016 SOA Model Governance Seminar August 31,

27 Proper use of technology allows Background Governance Technology Intersection Easy access of information and effective data management Creation of more complex and robust models More dynamic modeling with greater details and precision Expanded modeling capacity and possibility Comprehensive testing and validation Automation of processes Stronger security and controls Efficient deployment of computing resource Improved communication, transparency, and information sharing 2016 SOA Model Governance Seminar August 31,

28 Background Governance Technology Intersection Technology can enable better model governance Validation Approvals Releases Operation & Use Development Initiation / Change Request Technology is embedded in all phases of a model s lifecycle Performance Monitoring Retirement 2016 SOA Model Governance Seminar August 31,

29 Background Governance Technology Intersection IT Governance What can we learn from evolving IT governance and Software Development Lifecycles (SDLC)? Waterfall SDLC Example Requirements Sign-offs, Documentation, Toll-gates Design Implementation Verification Maintenance 2016 SOA Model Governance Seminar August 31,

30 Background Governance Technology Intersection IT Governance Some other SLDC methodologies in use include: iterative waterfall, waterfall with overlapping phases, agile (extreme programming, test driven development, scrum) Agile Scrum Example 2016 SOA Model Governance Seminar August 31,

31 Background Governance Technology Intersection Technology can automate governance process Documentation Approvals & Sign-Offs Releases & Version Controls Computing Resource Scheduling Governance Policies & Standards Inventory of models and risks Technology can avoid manual implementation of complex model governance Workflow Planning Compliance Reporting 2016 SOA Model Governance Seminar August 31,