Beyond Rules-Based Tests: The "Seven Wonder" Counter-Fraud Statistical Modeling Tests. Vincent Walden, CFE, CPA, CITP

Transcription

1 Beyond Rules-Based Tests: The "Seven Wonder" Counter-Fraud Statistical Modeling Tests Vincent Walden, CFE, CPA, CITP 2015 Association of Certified Fraud Examiners, Inc.

2 Beyond Rules-Based Tests: The Seven Wonder Counter-Fraud Statistical Modeling Tests 2015 ACFE Annual Conference

3 Today s Topics Big risks require big data thinking. What do we mean by beyond rules-based tests? The seven wonder statistical tests and examples Effective deployment considerations What s ahead cognitive computing Page 3

4 Page 4 Big Risks Require Big Data Thinking

5 The Forensic Data Analytics Landscape Making the Business Case for Action The regulators are upping their game. Be ready the regulators are investing in advanced monitoring technology. Big risks require big data thinking. New approaches to counter fraud and compliance monitoring, beyond simple rules-based tests Compliance fatigue? Analytics can help. Analytics can help improve efficiency and program effectiveness to help compliance functions audit and monitor smarter saving both time and valuable resources. Page 5

6 Upping Their Game: SEC Priorities Around Forensic Data Analytics U.S. SEC Chair Mary Jo White, prepared testimony before the Senate Appropriations Subcommittee, May 14, 2014 Page 6

7 Global Forensic Data Analytics Survey Big Risks Require Big Data Thinking ( 63% of companies agree they need to do more to improve their anti-fraud/anti-bribery procedures, including the use of forensic data analytics (FDA). 68% of companies responded that the board of directors is a top beneficiary of FDA. 72% of companies say that big data technologies can play a key role in fraud prevention and detection, yet very few are deploying it for FDA. Only 11% reported the use of statistical analysis and data-mining packages (SPSS, SAS, R, Strata) for use in forensic data analytics. Single biggest FDA challenge: getting the right tools or expertise for FDA

8 What s the difference between rules-based and statistical analysis techniques? CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP 2013

9 Gartner s Data Analytics Maturity Framework Value What happened? Descriptive Analytics Why did it happen? Diagnostic Analytics What will happen? Predictive Analytics What should we do? Prescriptive Analytics Statistics-based tests Rules-based tests Sophistication CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP

10 Rules-Based vs. Statistical-Based Tests Rules Based Statistics Based Effective for fraud detection and prevention ü ü Used most commonly in controls monitoring Requires hypothesis of fraud being conducted Associated with matching, querying, sorting, filtering Generally easy to understand and interpret ü ü ü ü Allows the data to define itself without a hypothesis or pre-defined criteria Looks for outliers, anomalies, or similarities Effective for both structured and unstructured data Typically results in fewer false positives and higher detection rates ü ü ü ü 10 FIDS Unitized Revenue Recognition Confidential Internal Use Only

11 Forensic Data Analytics Maturity Model Beyond Traditional Rules-Based Tests Consider All Four Quadrants Low Detection Rate High Structured Data Matching, Grouping, Ordering, Joining, Filtering Traditional Rules-Based Queries and Analytics Anomaly Detection, Clustering Risk Ranking Statistical-Based Analysis Unstructured Data Keyword Search Traditional Keyword Searching Data Visualization, Drill-Down into Data, Text Mining Data Visualization and Text Mining High False Positive Rate Low CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP

12 Seven Wonder Statistical Analysis Tests for Counter Fraud 1. Predictive modeling 2. Text analytics/concept extraction 3. K-Means clustering 4. Entity analytics 5. Pattern matching and social network analysis 6. Statistical anomaly detection 7. Jackknife resampling Page 12

13 Predictive Modeling Find Me More Like This Profile existing known suspect transactions/activities. Understand key variables that drive the outcome. Finds statistically similar transactions based on a confidence interval. Model continuously improves based on human validation of results strong alignment with continuous monitoring programs. Page 13

14 Case Example: Predictive Modeling Challenge: Analyze 400,000 transactions for suspected bribery payments per DOJ subpoena. 1. Team reviewed 2,000 transactions from ledger data (text comments, amounts, dates, etc.). Identified 400 suspicious and 1,600 non-suspicious entries 2. Created statistical model: Is Suspicious/Is Not Suspicious 3. Applied model to remaining 398,000 additional transactions 4. Identified 14,000 new suspicious transactions With confidence over 95% similar to Is Suspicious Identified over $8 million in highly suspicious payments. Methodology accepted by the DOJ for this case. CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP

15 Predictive Modeling Focus on the Variables That Matter Most Perform Variable Analysis Dependent Variable (Improper Payment) These three variables were this highest drivers of suspicious transactions These variables were less important when predicting suspicious transactions. Client should focus resources on monitoring efforts for the three leading drivers, which accounts for 80% of the predictive value. Independent Variables CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP

16 Text Analytics What is text analytics? Uses natural language processing, heuristic rules, and statistical techniques to reveal conceptual meaning and sentiments in text. Extracts concepts from text and categorizes them What are the benefits? Makes unstructured qualitative data more quantifiable Combine with data mining to improve the predictive power of the models that are created.

17 Rationalization Often Lives in the Text T&E descriptions Payment descriptions and instant message Social media Contracts Presentations User documents Page 17

18 Text Mining: Concept Extraction From the T&E data, we identified over 500 noun or noun-phase concepts from the expense descriptions. We analyzed these concepts and identified 30 concepts that warrant additional audit inquiry based on the word description or amounts. Recommendation: Utilize concept analysis to identify suspect payment descriptions. Page 18

19 Text Mining: Sentiment Analysis Sentiment by Category of Keywords Sentiment by Keyword and Person Page 19 19

20 Text Mining: Text Link Analysis and Word Clouds Page 20

21 K-Means Clustering Categorical Analysis to Find Anomalies K-means clustering is a popular statistical method for partitioning data into groups that share a common set of quantitative attributes. This is a method of organizing data in a sensible, yet objective way for subsequent analysis. Re-clustering is often performed to look at the same set of data in different ways. Page 21

22 Entity Analytics Run demo CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP

23 Entity Analytics What is entity analytics? An entity could be an individual, vehicle, vessel, etc. Resolve like entities, even when the entities do not share key values (e.g. ID number). The result is more accurate analytics, based on correctly resolved entities. Where would you use it? Where data quality and model accuracy are critical Financial services such as banking and insurance know your customer and AML Sanctions and watchlist screening Is this one person or two? CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP

24 Named Entity Recognition Named-entity recognition (NER) seeks to locate and classify elements in text into pre-defined categories. Example: Jim bought 300 shares of Acme Corp. in [Jim] Person bought 300 shares of [Acme Corp.] Organization in [2006] Time. Much like concept extraction, the results can be examined within accounting or expense transactions for any unusual or unexpected interactions by occurrence or aggregate monetary value. Page 24

25 Pattern Matching and Social Network Analysis What is social network analysis? Provides a way to analyze groupings Where would you use it? Primarily used to identify and target influential group leaders Identifies individual groups and relationships between individuals Third-party due diligence, know your customer or vendor Identifies core groups and leader(s) within them Improve segmentation accuracy by adding new social network attributes. Analyze influence spheres amongst Twitter users. CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP

26 Pattern Matching and Social Network Analysis Example Hypothesis: A discrete vendor should not share bank accounts with other vendors or employees. Found an instance Recommendation: Analyze relationships with multiple, discrete vendors linking to one bank account. Review for duplicate vendors. Review for high-risk travel agencies and hidden relationships. Page 26

27 Social Network Analysis, Continued Who Is Talking to Whom? The first 48 hours: Live server log files pulled for early case assessments Understanding a complex organization s true organization chart: identification of relationships, versus activities, amongst actors Filtered Page 27 27

28 Anomaly Detection Anomaly detection identifies unusual cases, or outliers, that do not conform to patterns of normal data. For example: In this graph, records A, B, and C would be anomalous. Page 28 28

29 Statistical Anomaly Detection (Letting Anomalies Identify Themselves) Statistically driven anomalies based on patterns in the data Incorporate statistical anomaly detection as an extra measure to identify potentially improper payments across the entire population, based on highest anomaly score. Page 29

30 Jackknife Resampling Systematically recomputes the statistic estimate, leaving out one or more observations at a time from the sample set Used to quality assure the model and/or find outliers by removing one transaction at a time to see if it makes a meaningful difference to the model Page 30

31 Page 31 Effective Deployment Considerations

32 Understanding Your Data Is the data available? Does the data cover the relevant attributes? Is the data messy? Is there enough data? Is domain expertise on the data available?? CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP 2013

33 Selecting the Best Predictive Modeling Projects Ease of obtaining relevant data High Invest Minimally Postpone or Avoid Priority Projects Cautiously Accept Low Business Importance/Impact/ROI High CONFIDENTIAL INFORMATION: Do not disclose Ernst & Young LLP 2013

34 No One Tool Does It All Visualization: Detect fraud within a business process. Detect Respond Statistical: Apply fraud insights to take action in real time when it matters. Case management: Turn fraud intelligence into action. Investigate Discover Pattern and link: Uncover hidden fraud and relationships. Page 34

35 Counter Fraud Framework A Process View unstructured structured Monitoring & Detection Tools Investigation Tools Pattern matching FINANCIAL ACCOUNTING DATA MASTER & REFERENCE DATA Library of counter fraud tests Text mining & advanced search Statistical & predictive INTERNAL RISK ELEMENTS EXTERNAL, SOCIAL MEDIA DATA Repeat the process: continuous auditing Big data processing platform Triage, stop payment and/or sample audit selection VISUALIZATION & RISK RANKING Case manager, task delegation and data refresh/ scripting automation Audit, shared services, compliance Investigative mindset Page 35

36 Five Success Factors in Deploying FDA 1. Focus on the low-hanging fruit; the priority of the first project matters. 2. Go beyond traditional rules-based tests incorporate big data thinking. 3. Communicate: Share information on early successes across departments/business units to gain broad support. 4. Leadership gets it funded, but interpretation of the results by experienced or trained professionals make the program successful. 5. Enterprise-wide deployment takes time; don t expect overnight adoption. Page 36

37 Forensic Data Analytics Deployment Models Solution as a Service (SaaS): Client logs into secure platform hosted by EY and conducts analysis and case management. Mobile support: EY sends a high-powered mobile kit to client site and performs analysis on-site with client. On-site deployment: Client purchases necessary hardware/ software. Service provider familiar with the business assists with configuration, test design, and training. Page 37

38 Compliance Analytics Vision Recap Internal Sources Data Governance and Collaboration Beneficiaries Shared svcs. data feeds ERP systems Entity analytics Predictive analytics Behavioral analytics Internal Audit Master & reference data Security intelligence Internal reports & communications External Sources Social media feeds Sanctions & watchlists News feeds & adverse media Context analytics Business intelligence Decision management Content management Compliance Analytics Platform Flexible approaches, reports, and capabilities for each beneficiary Changing risks requires flexible tools Knowing who is who is key to identifying patterns & opportunities Reduced false positives, better ROI Cross-enterprise view of exposures Expedient audits/investigations Data transparency, no black box Case management Forensic analysis Text analytics Social network analysis GeoSpatial analytics Big data, scalable platform, delivered on desktop or mobile device Legal & Compliance (including M&A) Shared Services & Finance BU Leadership & Corporate Other beneficiaries Page 38

39 Page 39 What s Ahead: Cognitive Computing

40 About Cognitive Computing Definition and Discussion Cognitive computing makes a new class of problems computable. It addresses complex situations that are characterized by ambiguity and uncertainty; in other words, it handles human kinds of problems. To respond to the fluid nature of users understanding of their problems, the cognitive computing system offers a synthesis not just of information sources but of influences, contexts, and insights. To do this, systems often need to weigh conflicting evidence and suggest an answer that is best rather than right. Cognitive computing systems make context computable. They identify and extract context features such as hour, location, task, history, or profile to present an information set that is appropriate for an individual or for a dependent application engaged in a specific process at a specific time and place. Page 40

41 Uses Link Analysis and Entity Extraction to Document Decision and Relationships Page 41

42 Entity Relationships Page 42

43 Watson-Like Cognitive Anti-Fraud Questions What investigations involved more than five suspects? What job title is most likely to commit travel and entertainment abuse fraud? What contract terms pose the highest risk of abusive trade discounts? In what division are we at risk of improper revenue recognition? Page 43

44 Questions and Discussion Vincent Walden Partner, EY Fraud Investigation & Dispute Services New York Page 44

45 Beyond Rules-Based Tests: The "Seven Wonder" Counter-Fraud Statistical Modeling Tests Vincent Walden, CFE, CPA, CITP 2015 Association of Certified Fraud Examiners, Inc.