Model Validation Eric Holmquist Michael Karibian, CGMA, CMA

Transcription

1 Model Validation Eric Holmquist Managing Director, ERM National Practice FIS EGRC Michael Karibian, CGMA, CMA Corporate Treasurer Midland States Bank

2 FIS: World s Largest Banking & Payments Provider Founded 1964 Public Company NYSE: FIS Headquarters North America Major Offices Asia Australia Europe Latin America Rated #1 FinTech 100 #3 Barron s 500 BPO Innovator of the Year FIS Clients FIS Offices FIS Data centers FIS Offshore BPO delivery centers S&P 500 Member $6.1B+ Revenue $1.2B+ Cash Flow 40,000+ Employees 14,000+ Clients 100+ Countries Slide 2

3 Midland States Bank Midland States Bank s Wealth Management Group has landed the bank on the 2014 Nifty 50 Top 50 Banks for Trust Revenue Growth by Bank Director Magazine. From 2008 through mid-2104, Midland successfully grew the bank, including 8 acquisitions, 6 de novo locations and a 12-fold increase in our wealth management assets under administration. Slide 3

4 Agenda Regulatory standards Model governance Model validation Final thoughts Q&A Slide 4

5 Model Risk Management Essentially, all models are wrong, but some are useful. - George E. P. Box Slide 5

6 Model Risk Management OCC Supervisory Guidance on Model Risk Management Slide 6

7 Model Risk Management OCC Supervisory Guidance on Model Risk Management Fed SR 11-7 Guidance on Model Risk Management Slide 7

8 Model Risk Management OCC Supervisory Guidance on Model Risk Management Fed SR 11-7 Guidance on Model Risk Management Fed BCC 13-5 Applying the Requirement for Conservatism to the Parameters in the Advanced Approaches Slide 8

9 Model Risk Management OCC Supervisory Guidance on Model Risk Management Fed SR 11-7 Guidance on Model Risk Management Fed BCC 13-5 Applying the Requirement for Conservatism to the Parameters in the Advanced Approaches Fed BCC 14-1 Model Risk Management under Basel AMA Slide 9

10 Model Risk Management Model risk management is fundamentally about one thing, the one thing that is central to all of risk management Uncovering assumptions Slide 10

11 Keys To Successful Model Risk Management Model inventory Documentation Assumptions Formulae Data sources and uses Business use Independent validation Clear policies for model development Slide 11

12 Model Risk Management Slide 12

13 Model Inventory List of all business critical models with model owners Identify business criticality level Determine a basic risk rating Focus testing and validation efforts on high-risk or business critical models Yes, spreadsheets can be models. Very dangerous ones in fact Slide 13

14 Model Definition a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information." * In other words: Uses data, formulae & assumptions Provides quantitative output Is influential to business decision making *Source: Section III, Overview of Model Risk Management, Supervisory Guidance on Model Risk Management (SR 11-7) Slide 14

15 Model Documentation Assumptions: about data, usage, purpose, criticality, validation, redundancy. Do you have a consensus? Formulae: Key calculations, use caution with black boxes Data sources and uses: where does the data come from, what fields are used, what comes out, is it right? Business use: Is everyone clear on what decisions are being made based on this model? Slide 15

16 Model Validation Must be an independent party, but independent doesn t have to mean outside the company Validation done by model owner is good, but secondary Critical models should be validated annually No internally developed black boxes EVER! Vendor black boxes require special validation, may have to rely on vendor provided independent certifications Validation should include model owner attestation on model assumptions and testing Slide 16

17 Model risk management policies Must have a clear process for the development of models that ensures clear business ownership, full transparency on logic and assumptions, change management protocols, full documentation and periodic validation (business critical). No model may be deployed until the documentation is complete, ever, under any circumstances. Beware of death by spreadsheet. Think about buying before building. Remember, trust me is the fastest way to invite trouble. The board is ultimately responsible for policy ownership, even though it generally delegates the responsibility to senior management. The board should ensure the level of risk is within their tolerance! Slide 17

18 Model Risk Management Slide 18

19 Internal or External Model Validation vs. Internal External Slide 19

20 Establishing a Validation Scope Ensure appropriate documentation exists Establish a clear scope for validation How deeply will data sources be validated (back to source or reasonableness only?) Will calculations be individually validated and back-tested, or tested for reasonableness only? Will output be back-tested or evaluated for reasonableness only? What exactly will be delivered as a result of the validation? How long will the validation take, start to completed report? How much staff time will be necessary during the process? Will the validation include any peer comparison or other benchmarking? Review business purpose, data sources and uses, key assumptions Slide 20

21 Validation Process Considerations Validate sampling of data elements within model back to source system Back-testing key formulae and documenting results Back-testing output data and documenting results Possibly back-testing data at several points in time Documenting source data, assumptions and key formulae for review by the business owner and other stakeholders Testing and/or verifying key assumptions within the model for accuracy Comparing historical outputs for reasonableness of trending (do the outputs over time make sense given the source data?) Analyzing model procedures as to accuracy and completeness Review of staff running the models for knowledge, training and usage Review published interpretations of the output data for accuracy and appropriateness Is there overlap with other internal models, and are redundant outputs consistent? Slide 21

22 Third Party Models Must have either: Vendor documentation that allows for back-testing of the data Third party certification of the model validity Lacking either of these options, find a new vendor. You will be challenged by the regulators at some point on this topic, especially the larger organization. Slide 22

23 Validation Results Consider who will receive the validation details and findings As soon as possible, Enterprise Risk Management should develop the technical ability to understand and interpret these validations in order to consult with the business on any implications Slide 23

24 Final Thoughts Risk lies in the assumptions Documentation is everything Not everyone is going to be happy about having their model torn apart, oh well Shame on you if you allow a model release without documentation this is inexcusable Model validation must be independent of the model owner Lastly, make sure your Board isn t surprised by a regulator who requests your dividend be delayed due to a model issue! Slide 24

25 Model Validation Eric Holmquist Managing Director FIS, ERM National Practice Leader Michael Karibian, CGMA, CMA Corporate Treasurer Midland States Bank Slide 25