Audit the Future: Using Audit Analysis to Predictively Manage Future Risks. Dan Zitting, CPA, CISA, GRCA Chief Product Officer, ACL

Transcription

1 Audit the Future: Using Audit Analysis to Predictively Manage Future Risks Dan Zitting, CPA, CISA, GRCA Chief Product Officer, ACL

2 I Hear Unbelievable Stories Every Day A savvy ACL user last year landed himself in the news... Unfortunately, the money was of course already long gone.

3 The Challenge We Had... Right Inside ACL We have all the data analytics power we could ever possibly need... but we are still talking about the past Laurie Schultz, CEO of ACL

4 Today s Fundamental Challenge How do we audit what is likely to happen in the future instead of what already happened in the past? Three key building blocks: 1. Evolving from traditional auditing to timely assurance with data automation 2. Incorporate three analytical approaches to make audit techniques predictive rather than reactive 3. Integrate predictive analysis into risk assessment process for true risk intelligence

5

6 Building Block #1 Evolving to Timely Assurance with Data Automation

7 ACL s Audit Analytics Capability Model

8 Our Journey to Timely Assurance Cornell University

9 An Evolving Vision Data Analytics for unit audits Baby steps Scripted Analytics for audits and monitoring Think bigger Transform into leading, data-driven assurance, risk, and advisory function Continuous Assurance Evolve

10 Where to Start? Data Analytics specialist Training for all staff Setting goals/expectations People Establish practices Quality control Measuring use of analytics Process Technology

11 Leveraging Analytics Efforts Ad-hoc analytics for unit audits Leverage routines across audits Scripting routines for enhanced efficiencies and consistency Procurement Cards Disbursements/Vendors Research Expenditures Human Resources IT Security Payroll General Ledger Capital Assets Building a script Library

12 Building Confidence Showcase projects Audit metrics Reduced audit cycle times More consistent results Increased value Requests for additional projects Having a seat at the table

13 Learning from Mistakes Don t underestimate the challenges obtaining data access; also opportunities to build stakeholder confidence in IA practices, objectives, etc. Data Protection Plan (unit WISP) How does audit protect my data? Data Access Agreements What is audit doing with my data? Go beyond Trust in Audit (whether assumed, through authority, or otherwise)

14 Finding the Sweet Spot Shifted to purposeful Teaming model with management Bring management to the table early, throughout the process Allow management to help shape program focus areas, without loss of IA independence and objectivity Risk/Value Matrix

15 Transformed into Timely Assurance New branding highlighted key elements: Timeliness Key to being relevant to management Assurance Avoids audit trigger word Engages 2 nd Line of Defense functions Tailored, innovative program

16 Maturing the Program Developed workflows and trigger notifications Established expectations and processes for remediation Remediation down to 17 days Quarterly reviews of remediation efforts Addressing issues timely, proper remediation, sufficient documentation, etc. Reporting to senior leadership and board Drives accountability, reinforces value

17 Next Steps: Risk Indicators & Scoring Identified/Developed analytics aligned with institutional risk areas/priorities Integrate with ERM program Composite risk scoring Incorporating Timely Assurance and unit audit results Building unit risk profiles

18 Building Block #2 Making Audit Analysis Predictive

19 Please, no more hype about predictive analytics

20

21 3 Simple Approaches to Predictive Analysis Statistical analysis and basic forecasting Automated KRI monitoring Scenario modeling and simulation analysis

22 Approach #1 - Statistical analysis and basic forecasting Fundamentally, using historical to forecast what will happen in the future Example: We currently have $10m of sales pipeline and historically we close 40%, so I forecast $4m of revenue in the next quarter Statistics like standard deviation, confidence intervals, etc. can make forecasts more accurate

23 Statistical analysis of historical data to forecast future risk

24 Approach #2 - Continuous KRI monitoring Without predicting actual outcomes, we can monitor predictive indicators as early warning system Key is defining those indicators that do indeed have predictive value relative to business objectives

25 KPIs versus KRis KPIs: Evaluate performance against objectives Tend to be reporting (retrospective) in nature KRIs: The precursor to a positive KPI, used to govern performance Predictive in nature to provide early warning sign of missing a KPI

26 Implementing Good KRis KRI Quality Validation Criteria Contemplated KRI Does it support the achievement of a specific KPI? Is it predictive in nature relative to the corresponding KPI? Can the periodicity be shorter than the corresponding KPI? Does it quantify a specific risk(s) in the ERM/ORM risk universe? Can it be automated with well governed data? Potential KRI #1 Yes Yes Yes Yes Yes Potential KRI #2 Yes Yes Yes No No Potential KRI #3 No No No Yes Yes A very useful and efficient KRI Should the risk universe be updated? Is this really a properly quantifiable KRI? Is something missing in our strategy/performance objectives?

27 Data Automation for KRIs KRIs compiled monthly in Excel tend not to be sustainable, consider professional solution in time Data quality always highlighted as the biggest issue in KRI projects, especially as they tend to come from multiple data sources Key considerations for automation: Data cleansing and blending Central control/governance over underlying data model Scheduling automation End user ability to define reports and metrics

28 KRI Workflow Automation Provides governance over: Defining KRI thresholds (risk appetite) Ensuring that when thresholds are crossed someone is notified and appropriate actions are taken Risk assessments and frameworks are appropriately updated to reflect change in risk levels

29 A Case Study in Achieving Real-Time Risk Intelligence My First KRIs The program of a major US government entity

30 Enterprise risk assessment driving KRI selection

31 Defining triggers and workflows

32 Approach #3 - Scenario modeling and simulation analysis Actually models the possible outcomes of a risk scenario to predict what will most likely happen Achieved by modelling the scenario then running simulations where the possible inputs changed Running many simulations results in a distribution that is a picture of prediction

33 Risk Outcome Prediction in a Picture

34 Case Study in Scenario Modelling A case study in leveraging basic data analytics, an investigation of concerns at the Arizona State Lottery: om/en/aboutus/news/2018/03/15/12/01 /musl-releases-rng-report

35 Background Lottery Incident On October 9, November 15 and November 21, 2017, the same Arizona Pick 3 numbers (8-0-4) were selected using AZ RNG1. Internal audit examined the pre-post and post-draw tests to determine if the results were within expected norms.

36 Investigation Objective and Scope An independent statistical expert was engaged to evaluate the probability and frequency that three numbers, each ranging from zero to nine, would repeat within a draw population of 742 events. The purpose of this exercise was to determine whether actual Pick 3 results were consistent with predicted outcomes.

37 Enter basic scenario modelling... Out of a population of 742 Arizona Pick 3 lottery drawings, the winning combinations of and each came up 5 times. Intuitively there surely is a problem perhaps machine is broken or someone is committing a fraud? Fundamentally only three steps: 1. We must model the scenario random drawings of three numbers, each one from Next, run that scenario 1000 (or 10K/1M/etc.) times to see what a distribution of possible draw results looks like 3. Compare our real world concerning result (2 combinations that each recurred 5 times) to this full range of outcomes to see just how likely our real world scenario actually was

38 Our Scenario Modelling Driven Results In Conclusion: Most importantly... entire risk analysis, scenario modelling and all, done in an afternoon. Our scenario risk modelling of 1000 potential outcomes, what does indeed intuitively seem to be a severe risk issue is probably not. The repeating lottery numbers generated by AZ RNG1 do indeed look to be within the expected statistical norms

39 Investigation Conclusion Furthermore, and this is the key, if we go back to our full set of 742 observations, it is not unreasonable to have a small set of three-digit numbers occur 3, 4, or even 5 times among the set of winners. In our data, no (set of) number(s) occurred more than 5 times, with only two numbers occurring exactly 5 times (8-0-4 and 9-1-9). Thus, it is within the realm of what might occur naturally for to have occurred 5 times between 7/15 and 11/17.

40 Other scenario modeling and simulation examples: Global commodity trader predict fraud potential Global tech company predicted impact of price increases on customer churn

41 Building Block #3 Incorporating Predictive Analysis for Risk Intelligence

42 Predictive Automation of Risk Assessments

43 Predictive Automation of Risk Assessments

44 Achieving Predictive Risk Automation... Real-Time Risk Intelligence

45 Self Assessment Risk Prediction Capabilities

46 Predictive Auditing Maturity Curve Level 0 Traditional, manual audit techniques evaluating past events Level 1 Ad-hoc analysis using desktop data tools to evaluate one off audit items Level 2 Achieving Timely Assurance through data automation techniques in key risk and assurance areas Level 3 Using predictive techniques of statistical analysis & forecasting, KRI automation, and/or scenario modelling for future facing assurance Level 4 Integrated predictive analysis and enterprise risk assessment for real-time risk intelligence

47 Thank you. Linkedin: Dan Zitting

48 Audit the Future: Using Audit Analysis to Predictively Manage Future Risks Is your entire audit plan focused on what already happened in the past? Forward-thinking audit teams use digital analysis techniques to predictively assess risk levels, then get ahead to control and manage future risks. Organizations have implemented immediately actionable methodologies and models to enable internal audit to directly support the CEO in navigating the risk landscape of future years. In this session, participants will: Explore the principles of risk prediction and how to systematically assess future risk levels. Learn about skills, models, and technologies used to quantify and communicate risk predictions. Evaluate example risk prediction dashboards and reports to consider what might resonate with your organization's management and board. Self-assess your current capabilities and gaps to develop an impactful risk prediction program.