Enhancing Risk and Internal Audit effectiveness using KNIME. KNIME Fall Summit November 2017

Transcription

1 Enhancing Risk and Internal Audit effectiveness using KNIME KNIME Fall Summit November 2017

2 Background Project Problem Solution Outcome 2 1 st Line of Defence Leadership Functions Reports to Executive Management Chief Executive Officer Execution of strategy, management of risk, operations Board of Directors GUIDE Leadership Chief Risk Officer 2 nd Line of Defence Risk Function ~1-2% of FTE Enable efficient, effective governance of significant risks/opportunities Functions Help management identify, assess, respond to and monitor threats Includes financial, strategic, regulatory (compliance), operational and reputational risks. FS specific functions for credit, market, liquidity risks Reports to The CEO and sometimes to the Board Risk Committee CHECK 3 rd Line of Defence Internal Audit ~1% of FTE CHECK Leadership Chief Audit Executive Functions Reports to Independent, objective assurance/guidance re achievement of objectives Better practice administratively to CEO, functionally to Board Audit Committee [in the US, this is law for publicly traded companies]

3 Background Project 1/3 Problem Solution Outcome 3 Company: Large mutual - loans, deposits, insurance, etc. Project: Home loan review decided on data driven approach brainstormed and identified 15 hypotheses Recalculation - full population, daily recalculation of interest (16 months) Checking - master data quality Checking - application of product rules Checking - adherence to customer instructions (more detail on this later) Data: Customer master (incl. account linkages), daily account balances, daily transactions, CRM CRM was free text - so we used KNIME NLP nodes (BoW, Doc Vector, TF, ) Combined output from NLP with structured data they blended

4 Background Project 2/3 Problem Solution Outcome 4 Dataset Content # Records 1 Loan Account Balances Account ID, $ values, dates, etc. 69m 2 Loan Account Transactions Account ID, Transaction ID, Teller ID, descriptions, categories, $ values, dates, etc. 8m 3 Deposit Account Balances Account ID, $ values, dates, etc. 638m 4 Deposit Account Transactions Account ID, Transaction ID, Teller ID, descriptions, categories, $ values, dates, etc. 9m 5 Customer / Account Relationships Customer ID, Account ID, various date and status fields, names, addresses, etc. 82m 6 Actual Account Links Loan Account ID, Deposit Account ID, various date and status fields 65k 7 Product Codes Product code ID, product code description 1k 8 Link Eligibility Deposit product code ID, Home Loan product code ID, link eligibility flag 1k 9 CRM Free text in flat files, customer / account IDs (somewhere), date and interaction text 404k 806m 10 Loan application archive Loan application ID, new home loan account ID, offset deposit account ID, date 51k 800m structured records, average of ~20 columns each 400k free text records resulted in document vector with > 1,000 columns That s reasonably big, but we ve all seen that before Particular about keeping data in bank environment Could only provide laptop: 8G RAM, 256GB SATA SSD Analyses all ran, albeit slowly (24hrs for complex calcs) small number of errors - modified workflows KNIME handled memory management and disk management within this constrained environment If migrate the workflows to run in production environment, technical specs would not be an issue

5 Background Project 3/3 Problem Solution Outcome 5 Import Workflows Analysis Workflow Classify Workflow (more on this later)

6 Background Project Problem 1/2 Solution Outcome 6 One of the 15 hypotheses was deemed high risk Allow members to link eligible deposit and home loan accounts - reducing interest Historical problems In Oz recently, many public acknowledgements by banks regarding failure to correctly provide for offset linkages - financial (recalculate & compensate) and reputational damage. E.g.: $230k - Mutual, $58m Regional bank, $70m Major bank With the data blended, we then tried to identify expected offset linkages that may not have been properly setup: CRM - Keyword searches (including fuzzy) Linkages - matching on eligibility Balances - checking dollar value thresholds etc.

7 Background Project Problem 2/2 Solution Outcome 7 Scale of potential linkages found ( >4,000 links and 7,000 CRM data points) -> manual investigation would be a significant task We knew that many of the exceptions were false positives Typical audit approach: select a random sample not optimal Alternate approach: manual profiling / extrapolation - because of nature of NLP output (vector), false positives could not easily be identified this way Audit team is forward thinking, so we suggested a further alternate we explained it as predicting the past classification

8 Background Project Problem Solution 1/2 Outcome 8 KNIME enabled us to solve this: 1. Selected a sample of exceptions 2. Performed a high-level manual review and classification (e.g. keep, maybe, drop) 3. Fed the results into a model in particular, we used a combination of algorithms and a forward feature selection loop, then identified the combination of features and algorithm that yielded the best model score 4. Used the selected model to predict the outcome of the remaining exceptions

9 Background Project Problem Solution 2/2 Outcome 9

10 Background Project Problem Solution Outcome 10 >90% decrease in the number of exceptions - i.e. eliminate noise > less investigation effort Remainder then passed onto relevant business area for further investigation Model accuracy: ~70% Can be defended, far better than random audit sample Savings: 295 took ~ 65 hrs => ~ 0.2 hrs each => remove 3,800 => ~ saving of 100 days (~$50k)

11 Background Project Problem Solution Outcome 11 Some obscure challenges Auditors identify gaps in people s work -> some then try to reciprocate and find flaws in the auditor s work. We found duplicate clients in the master data by using string distances, etc. AML risk because of transaction thresholds at client level (e.g. client deposits > AUD10k/day -> must report) Risk manager is responsible for this, and their analytics did not pick this up. Him: how do you verify the veracity of the software code, given that is open source? Me: (in my head) we found it, you didn t, how do you verify your proprietary tool? IT guy: if it is open source, what if someone has coded a backdoor? Me: if it is proprietary, what if someone has coded a backdoor? IT guy: hmmm will have to check with the security team Security guy (to IT guy): Did you really ask that question? Do you know that the other tool that we use, XXX, is in the same area of the Gartner quadrant as this one?

12 Background Project Problem Solution Outcome 12 A final note Such a significant step for the folk that I deal with, that it can take a while to get people to understand what the possibilities are, because they can t get over the fact that it exists in an open source tool Me: (showing the client some of the capabilities) this analytics platform is open source Client: no, that can t be surely there must be a catch? Me: there is, but you have to try first

13 Yusuf Moolla riskinsights.com.au Mobile: +61 (0) Usage Limitations: This document is intended solely for presentation by Risk Insights to participants of the KNIME Fall Summit, held in Austin TX during November It is not intended for and should not be used by any other person or entity. No reliance is to be placed, in any manner or for any purpose, on this document. We disclaim all responsibility for any loss or liability that may be suffered or that may incur arising from, relating to or in any way connected with the content of this document.