Model Governance Framework for

Transcription

1 Model Governance Framework for Analytics A Recipe for Success SAS Analytics Rome, Italy Dr. Jarrad Hee Head of Group Capital and Portfolio Analytics Nov 2016

2 2 Disclaimer The views presented in this deck are entirely my own and may not necessarily reflect that of Emirates NBD.

3 Emirates NBD at a Glance 3

4 Emirates NBD is the Largest Bank in the UAE 4

5 Emirates NBD is among top 3 banks in the GCC by Operating Income, Net Profit, Loans and Deposits in

6 Contents (Today s Menu) 1. Increasing Attention on Models and Analytics 2. End-to-End Model Process a) Components b) The Process c) The Considerations 3. Governance Framework and the Challenges

7 7 What s Cooking in the Analytics Space The last 2 decades saw advancements in decision sciences and technology that led to the phenomenal growth and sophistication in the use of models (predictive analytics) / algorithms. Developments in the business environment have also accelerated the adoption: Regulatory developments Basel II and III more granular assessment of risk and capital, stress testing Accounting standards IFRS9 forward looking impairment / provisions Competitive pressures Processing efficiency faster loan approvals (decision models), cross selling Targeting accuracy Facebook, Online shopping to boarder security, inspection of containers by customs Program trading algorithms to execute trades it is ubiquitous These models, if developed correctly and operated appropriately, would serve organizations well in optimizing risk, return, profitability, market share

8 8 The Menu Today Today s Special The focus of today s discussion will be on the risk and impairment models used in financial services industry, more specifically, use of analytics in commercial banking Basel II Advanced Internal Ratings Based (AIRB) approach for risk component models IFRS9 Forward looking impairment models to estimate Expected Credit Loss (ECL) or Lifetime Expected Loss (LEL) The approaches above have a common theme requiring a strong model governance framework for managing ongoing use of models To have a good model governance framework, we need to first understand the components in the modeling process from start to finish (in use)

9 Contents (Today s Menu) 1. Increasing Attention on Models and Analytics 2. End-to-End Model Process a) Components b) The Process c) The Considerations 3. Governance Framework and the Challenges

10 10 Components in the End-to-End Model Process Understanding the components in the end-to-end value chain of model development and usage process is the starting point for building a robust model governance framework Raw Data Data Preparation Model Development Model Validation Model Implementation In-Use Production Performance Monitoring = n k=0 t 1 n 1Y n k xk a n k = k=0 n t 1 n 1Y n k xk a n k Robust governance framework required Without a proper governance framework, the use of models to make decisions becomes a high risk endeavor, in addition to potential breaches in regulatory requirements

11 11 Using a Cooking Metaphor to Illustrate A good governance framework is like following a well written recipe / methodology to prepare a complex dish Auditors / Regulator Inspectors Raw Data Data Preparation Model Development Model Validation Model Implementation = n 1 k=0 t n 1Y n k xk a n k n = k=0 t 1 n 1Y n k xk a n k In-Use (Production) Performance Monitoring Selection of Ingredients Cleaning / Processing Cooking - Modelling Assessing Fit for Purpose Serving Putting into Production Consuming Using Outputs Feedback Quality Control Quality Quantity Meets requirements Relevant Clean Appropriateness of technique Adapt to ingredients Professional judgement Independent assessment Meets the requirements / expectations Approval process Getting the model ready for production Testing - UAT Serve the intended purpose How good How bad How reliable Insights on each component (area) would require consideration during the development of the governance framework

12 12 The Component Raw Data Raw data Basic ingredients for modelling Quality data Consistent, valid, reliable constructs. Supporting usage of quality data requires: Robust enterprise data architecture (within the IT infrastructure) Enforcement of data standards (including control processes and ownership) Training for data originators and users Quantity of data Sufficient history / coverage. More data points supports robust decisions outcomes Data capture systems ongoing to build history Storage / archiving requirements Maintenance / house keeping Data is a basic ingredient in the model development process and good data contributes to the robustness of the model

13 13 The Component Data Preparation Data preparation Cleaning and chopping for modelling Relevance of data (internal and external sources) Data gaps, errors, cleanliness Data assessment Detail analysis of the underlying data to assess if it meets the requirements for modelling Perform data visual assessment (scatterplots, dispersion etc.) Data analytics tests (coverage, gaps, outliers etc.) Data diagnostic test (normality, stationarity etc.) Well prepared data is the critical starting point for modelling Garbage in garbage out

14 14 The Component Model Development Model development Cooking the ingredients = n k=0 t 1 n 1Y n k xk a n k Discipline which requires highly skilled analytics professional using techniques suited to data and constraints Specialized techniques / methodologies may be required when faced with challenging data Experienced modelers to tease out the underlying patterns and use judgement Fusion of analytics and knowledge of credit (for credit, impairment models) Fit with bank s model architecture / landscape Communicating in technical and non-technical language Developers should not work in isolation broader communication with stakeholders Documenting the process accurately especially rationale or assumptions on adaptations Model version control and standardized naming convention Impact assessment of new or existing Model developers are the first line of defense

15 15 The Component Model Validation = n k=0 t 1 n 1Y n k xk a n k Model validation Independent assessment of the quality of the model Validator to independently and formally assess fitness of the model as 2 nd line of defense Determine if the data (including definitions) is appropriate for the intended purpose Assess if the methodological approach meets the intended requirements, conduct and verify performance tests If meets minimum performance standards Holistic process that covers the system view of the model risk Determine appropriateness of the model operating space policies, IT, user training, output usage, fit with model architecture / landscape Determine appropriateness of solutions to model issue resolution plan as models are not perfect Post model validation, the model needs to approved by appropriately delegated authority for use External validators can provide the required expertise to assess the model, but equally important to have an internal independent team to continue with ongoing validation work

16 16 The Component Model Implementation Model implementation Putting it together for use Setting the model in place to support the business Appropriate sign off on the computation and outputs Technical documentation of the system implementation Adequate level of test cases for UAT and tolerance levels Operationalizing the model Change management efforts and training do not underestimate the effort Policy updates and operational process changes Manage impact on stakeholders With a changing portfolio of models to manage, it is critical that the bank has a disciplined model version control in place

17 17 The Component In-Use (Production) Model production Using the outputs Model being used as part of business as usual No models are perfect understanding model risk or limitations Model controls and management in place Overlay / overrides based on judgement only those who are authorized Correct and proper use prevent gaming / abuse of the models Use of built-in intelligence to manage errors or abuses Information capture Data to be stored for current and future use Having an automated model selection process can reduce model selection error

18 18 The Component Performance Monitoring Performance monitoring How are the models doing Regular monitoring of models Perform a range of tests to determine accuracy and reliability Quantitative and qualitative assessment Correct usage e.g. Control Self Assessment (CSA) Annual validation To be performed by independent unit annually (2 nd line of defense) Determine model fit for use Reported and endorsed by senior management Performance monitoring provides early warning on the potential model weakness. Annual validation provides a rigorous assessment to determine root cause issues and meet regulatory expectations

19 19 The Component Auditors / Regulators Auditor and regulator review Internal audit 3 rd line of defense and auditors Provides an additional oversight layer Governance to ensure proper processes are adopted Identify weakness External auditors increasingly involved particularly for IFRS9 impairment models Regulatory authorities or central banks Provides guidance on standards or directive Assessment of institution level of compliance Determine if level of sophistication appropriate with other similar institutions Critical for banks to demonstrate governance process is appropriately implemented and adds credibility to its governance process

20 Contents (Today s Menu) 1. Increasing Attention on Models and Analytics 2. End-to-End Model Process a) Components b) The Process c) The Considerations 3. Governance Framework and Challenges

21 Raw Data Data Preparation Model Development Model Validation Model Implementation In-Use (Production) Performance Monitoring Regulator 21 Governing Policies (Internal) 1. Risk Management Framework 2. Risk Appetite Framework 3. Data Governance Standards 4. Model Governance Framework 5. Model Development Standards 6. Model Validation Standards 7. Model Performance Monitoring Standards 8. Model Documentation Guidelines 9. IT Architecture Framework Critical Need Important / Necessary Working Requirement Useful

22 Raw Data Data Preparation Model Development Model Validation Model Implementation In-Use (Production) Performance Monitoring Regulator 22 Governing Policies (Internal) 10. IT Implementation and Testing Framework 11. Credit Model Policy and Framework 12. Corporate Credit Policies 13. Retail Credit Policies 14. Write Off and Impairment Policies 15. Model Overrides Policy 16 Internal Audit Framework The internal policies provide a clear, consistent and objective approach for governing the use of models (formal), to ensure that model risks are appropriately managed

23 Raw Data Data Preparation Model Development Model Validation Model Implementation In-Use (Production) Performance Monitoring Regulator 23 Directives / Guidelines (External) 1. Basel Frameworks 2. Central Bank / Regulator / Prudential Authority 3. Federation or Union of Banks Guidelines 4. Global Standards Authorities e.g. IFRS9 5. Vendor Model Development Highlights Critical Requirements Important / Necessary Working Requirement Useful External directives provide the required standards for banks globally or domestically to have alignment of approach and a level playing field

24 24 Challenges Monitor Hire Effectively monitoring of the model use and escalate for resolution Finding the right caliber of staff for development through to execution Iterative Putting the governance framework in place and making it work effectively can take time to achieve Implement Develop Developing clearly written frameworks which are in line with international practices yet appropriate for the local context They key to addressing the challenges starts with hiring the right individuals

25 25 Key Takeaways Understanding the steps in the end-to-end model development and usage process Banks should have the appropriate policies / frameworks in place that address each of the process components

26 Thank You 26