Business White Paper Analytics of rare events

Transcription

1 Analytics of rare events Delivering value through predictive analytics

2 Table of contents Definition of rare events 1 Threshold for rarity 3 Realities of the digital universe 3 Considerations for businesses 4 Data volume and velocity 4 Rare event predictions 4 Static and dynamic predictions 5 About the author 6 Our constantly expanding digital universe means there are enormous amounts of data to process and derive insights from; however, certain events are still rare leaving us with little information to predict their occurrences. Definition of rare events A rare event happens with a low frequency but is very significant in the nature and scope of its impact. The event s impact varies based on the type of organization individual, business, industry, or government. While rare events from nature have a pervasive impact, rare events, for example, at a factory shop floor may affect only the business. Predicting a rare-event occurrence is important for businesses to prepare for the eventuality. Obvious examples include: Societal Wars, coups, epidemics Macroeconomic Large economic depressions, economic and shocks, or market crashes Natural disasters Tsunamis, hurricanes, earthquakes, or asteroid impacts Industrial Machine failure, catastrophic nuclear power plant failure Marketing Response to mass mailing campaigns, customer churn Finance Fraudulent card transactions, borrower bankruptcy, and others Examples of applying rare-events prediction and detection include: Predicting machine failure The number of occasions when a machine breaks down is significantly lower than the number of times when the machine was fit for use. But predicting machine failures, particularly for expensive machines, is important for business continuity. And manufacturers of these machines want to offer financially viable warranty and support services. Customer Churn All industries suffer from voluntary churn in their customer base. This is particularly true for the telecommunications industry. While the installed customer base could be large and network larger, the number of customers actually churning could be quite small. Detecting fraud Only a small percentage of the millions of credit card transactions are fraudulent. Detecting potentially fraudulent transactions based on metadata and transaction history could raise flags. Then transactions are either blocked or queued for manual cardholder confirmations before processing. Identify visitors to an e-commerce store as potential buyers Conversion percentages in retail e-commerce stores are typically very low often in low single digits. Identifying site visitors as potential buyers is a typical case of recognizing a rare event. 1

3 Detecting network intrusion In large corporations, numerous devices are on the network. And these networks have many users connected on both sides of the firewall. Intrusions are actions that attempt to bypass security mechanisms of the overall network traffic. But failure in predicting them could cause catastrophic damage to a business s interests. Medical diagnostics When classifying pixels in mammogram images as normal or abnormal (cancerous), only a very small fraction of the entire image is abnormal. In statistical terms, datasets containing rare events produce a histogram of the response variable that is extremely right-skewed showing a large value for rare events. Or the histogram would display extremely left-skewed with a small value for rare events. Figure 1 below, shows a right-skewed distribution with a tracking density curve. The rare event here would occur at the extreme end of the distribution s tail. Such frequency distributions have a long tail. 1 Simply said, a probability distribution has a long tail if a larger share of the population rests within its tail than it does in a normal distribution. The long tail was popularized by Chris Anderson in an October 2004, Wired Magazine article. Figure 1. Rare event shows a right-skewed distribution 2

4 Threshold for rarity A question that s often asked is whether there is a cut-off for recognizing events to be rare. Rareness of events, however, is not defined as a set count or a percentage. Also, the cut-off would differ by the specific use case. Some researchers state that rare events occur in frequencies that range from 0.1% to 10%.2 Figure 2 below shows the relative rareness of events in various use cases. Percentage of occurrences ANNUAL CUSTOMER CHURN IN TELECOM 3 10% GLOBAL RETAIL ECOMMERCE CONVERSION RATES 4 2-3% NETWORK INTRUSIONS 5 1-2% UNAUTHORIZED CREDIT CARD TRANSACTIONS % MACHINE FAILURES % Figure 2. Occurrence of rare events in varied scenarios 1 Chris Anderson, The Long Tail. New York: Hyperion Books, July Army High Performance Computing Research Center, Data Mining for Analysis of Rare Events: A Case Study in Security, Financial and Medical Applications, Aleksandar Lazarević et al, May Source: Database Marketing Institute, Churn reduction in the telecom industry, Arthur Middleton Hughes, October Source: Smart Insights, Ecommerce conversion rates, Dave Chaffey, April Source: University of Minnesota, Data Mining for Network Intrusion Detection, Paul Dokas et al, Source: pymnts.com, A tale of Realities of the digital universe IDC s latest Digital Universe study says: Like the physical universe, the digital universe is large by 2020 containing nearly as many digital bits as there are stars in the universe. It is doubling in size every two years, and by 2020 the digital universe the data we create and copy annually will reach 44 zettabytes or 44 trillion gigabytes. 8 If rare events are as infrequent as presented above, then is the problem with insufficient counts assuaged by the abundance of data? This is really not so. The amount of data on rare events is often hard to acquire in enough counts so much so that it s akin to finding a needle in a haystack. In our recent experience of predicting failure events for haul trucks used in mining operations, we noticed that on-board sensors to capture information on engine, payload, tires, and electric drive systems yielded a dataset with 300 million records a month s worth of data for 20 different haul trucks. Yet, the count of downtime events was in the low 2 digits. So, despite the abundance of sensor data, the amount of which is available to understand its relationship with actual down events is significantly low. two fraud stats, August Source: From a predictive model developed by DXC for a partner in the mining industry 3

5 Assessing data volume and velocity Predicting rare events Operationalizing prediction Figure 3. Rare event analysis for businesses Considerations for businesses Assessing data volume and velocity: This will have implications on the technological infrastructure required to store and process data. Making predictions: With the infrastructure in place, algorithms to predict need to be developed. Operationalizing predictions: Integrating predictions with business processes is when the benefits begin to show. Data volume and velocity Developing algorithms to predict occurrences of rare events would require analyzing large volumes of historical data and new data as it streams in. Analyzing large volumes of historical and streaming data requires significant secondary storage, memory, and processing power. Also, streaming data has bounded storage limits, meaning that with the volume and velocity considerations, a real-time, single-pass analysis could be necessary. Single-pass data analysis means that data streaming-in is available only once for algorithmic considerations. These requirements influence an organization s technology choice. The first five instances mentioned previously are typical streaming data scenarios. The sixth example, classifying pixels in mammogram images, describes detecting rare events in a static dataset. Rare event predictions Time-independent predictions This provides a probability for the rare event s occurrence, but without referencing time elapsed since the last rare event. For example, earthquake predictions estimate the probability of an earthquake as a function of specific geographical parameters. This can help build better structures in earthquake regions. However, since it doesn t provide a time component, it doesn t help in timing occurrences for making evacuation decisions. 9 8 IDC, The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things, April

6 Time-dependent predictions This gives a probability that considers the time elapsed since the last event to calculate the probability for an event s next occurrence within a specified time period. In the earthquake example, this is a prediction of the probability of large earthquake happening in California for a 30-year period starting in Such predictions contain uncertainties, including historical data from a few events, which occur under variable conditions. So to say that an earthquake will occur within the next six months with a probability 90 percent, is still not enough for a mass evacuation of San Francisco. However, some preparatory measures may be taken. 9 Deterministic predictions A prediction made with high confidence, specific time frames, and locations is precise and can affect decisions for organizing an advanced evacuation. Hurricanes, tornadoes, and blizzards are examples roughly in this category. 9 The maturity in our understanding of the phenomenon and the science behind its occurrence, influences the accuracy in predicting it. It s important to evaluate the accuracy of predictive algorithms. If a scenario characterized by 1 percent rare events, is that even a trivial classifier that categorizes every data point to the majority class can achieve 99 percent accuracy. Model performance here, should be evaluated as a trade-off between the number of actual rare events correctly detected and the number of events from the majority class that are incorrectly identified as rare events. Static and dynamic predictions Assuming there is adequate storage data, operationalizing analytics would require evaluating data as it comes in. And given the high velocity of incoming information, this is challenging. In the network intrusion detection example, it s about detecting evidence of malicious traffic shortly after occurrence, from the enormous, and mostly innocuous, data that s constantly flowing in. Also, traditional event logging Operationalizing predictive algorithms can be achieved through: Static algorithms applied in real time Algorithms are developed offline on static datasets and applied in real time. Such datasets are usually large tables best stored in databases with high levels of compression, such as DXC Technology Vertica or Hadoop. The machine learning algorithms can be integrated indatabase in DXC Vertica using C++/ R UDFs. The number of R packages available significantly extend Vertica s capabilities. 9 University of Edinburgh, Is the reliable prediction of individual earthquakes a realistic scientific goal?, Ian Main, February Wikipedia, Concept drift. 5

7 Dynamic algorithms applied in real time This uses the premise of concept drift, which recognizes statistical properties of streaming data can change over time.10 A static approach make predictions less accurate over time. This requires real-time, single-pass analysis using tools like Apache Spark, an open-source framework that sits on the Hadoop Distributed File System. Spark extends Hadoop s core functionality by providing in-memory cluster computation, and a stream-handling framework. About the author Bharathan Shamasundar Learn more at [ /analytics] Bharathan Shamasundar is a data scientist with 11 years of delivery and consulting experience spanning marketing, finance & accounting, and business operations. His industry experience includes financial services, technology, and e-commerce. Bharathan works on Big Data projects, and his interests include machine learning, which focuses on analytics of streaming data. About DXC DXC Technology (NYSE: DXC) is the world s leading independent, end-to-end IT services company, helping clients harness the power of innovation to thrive on change. Created by the merger of CSC and the Enterprise Services business of Hewlett Packard Enterprise, DXC Technology serves nearly 6,000 private and public sector clients across 70 countries. The company s technology independence, global talent and extensive partner alliance combine to deliver powerful next-generation IT services and solutions. DXC Technology is recognized among the best corporate citizens globally. For more information, visit DXC Technology Company. All rights reserved. DXC_HPE-4AA6-2645ENW. March, 2017