wipro.com Data Anarchy The out-of-control data babble

Transcription

1 wipro.com Data Anarchy The out-of-control data babble

2 Everyone can relate to this: when a flight is delayed, there are angry passengers. Some have missed their connecting flights, others their business appointments, some will have to put up with delayed or missed medical examinations, and a few will unfortunately be late for dad s 75 th birthday. On the other side, flight delays result in increased crew, maintenance, ground assistance and fuel costs aside from impacting competitiveness and safety issues. The hidden impact of delayed flights on the economy is immense. One reliable estimate in 2015 put the cost of worldwide delays the previous year at US$ 25 billion i. What does it take to predict a flight delay so that passengers can be warned, other flight options made available and the delay itself fixed to stop its dominoes effect? It is the kind of problem that should have data scientists rolling up their sleeves in delight. But as it turns out, it is their worst nightmare. They spend 80% of their time cleaning and caging data before it can be used ii. As the volume of data grows thanks to sensors, the web, mobile networks, documents, , chats, CRM and corporate data bases businesses are waking up to the fact that not all of it is helping unlock information or intelligence. Data anarchy is growing and it calls for new ways of collecting, cleaning and using data that help increase efficiency and insight. Let s get back to our key question: What does it take to predict flight delays? Simplistically, the minimum data needed would comprise airline information, flight numbers, departure and arrival airports, aircraft types and condition, crew availability, pilot contracts, fleet plans, load factors, ground support and infrastructure availability, delays in arrival of connecting flights, flight cancellations by other airlines, air congestion parameters along the route, weather forecasts, capacity induced delay thresholds, etc. It sounds simple enough: collect the data and parse it using algorithms to spews out insight, predictions and prescriptions. But it isn t quite as simple. If you go over the data requirement once more, a glaring factor becomes apparent: airlines don t own or control all the required data. It comes from disparate partners, sources and systems. Airport names can be tricky, data metrics and formats are different and some data is unusable iii. 1

3 A not-so-good shift in data quality Here is the sum and substance of the problem: gone are the days in the 1980s, before the Internet, when companies were monolithic and did everything in-house. In the pre-internet era, all they had to do was dive into their own enterprise for data. Then, data storage was relatively expensive, so enterprises exercised restrained over what they measured and stored. It made things easy. This changed in the 90s. Economies of major nations started to saturate. Growth began to shrink. Enterprises looked for markets outside their domestic boundaries. Fortunately, the developing world craved goods and services to fuel further growth. There was an explosion of global supply chains. Enterprises were now working with data flowing in from different sources. None of it was under their control. It brought reliability and speed of analysis into question. Then towards the late 90s, the Internet exploded. In the last 25 years, the Internet has become a source of chaotic, disorderly information from social platforms, blogs, audio and video. Enterprises now also need to look at opinions for data. It is a double-edged sword: enterprises can get a direct sense of customer sentiment, however the data is inconsistent and gloriously unstructured (many would add unreliable to the list of grievances). Today, we have to data exchanged between people on the Internet, mobile phones and the Internet of Things (IoT). To add to the woes, because of hacking, phishing, unethical marketing and a whole slew of digital ills, people have begun to mask personal information or even misrepresent it. But the dependency of enterprises on data to reduce risk, stem commercial crime, improve safety, avert waste, optimize operations, drive better customer experience and improve products is immense. They must forge on, regardless. This means addressing a key issue: how do you manage the transportation of these huge volumes of data? Putting a petabyte of data on cloud for analytics is expensive and time consuming. Putting it on hard drives and transporting it is no better. There is the risk of confidential/ business critical data being compromised. 2

4 The bullwhip effect Managing data volumes is one side of the problem. The other side is data quality, especially bad data, and its impact. Bad data triggers the famous bull whip effect. For example, if retailers in a supply chain report a demand that is wrong by 20 % (that is, instead of 10 units of demand they give is 12 units) at their outlets, then the impact of such a demand at the distribution center level is an extra 100 units if there are 50 shops being served by it. If there are 50 distribution centers, the extra demand would be 5,000 units. Imagine the impact of this false demand on production schedules, material import and the entire supply chain. As the data travels up the value chain, decisions go alarmingly awry. However, one way to prevent this is by inhibiting the data anarchy at the data collection points itself. Pre-processing to the rescue From a practitioner s perspective, responsible data management calls for addressing the data glut itself. The solution lies in applying filters to data collection. Think of it as turning down the data spigot to an optimal flow so that what comes out of the pipeline is exactly what you want. The next step is to pre-process the data at the collection site before scientists take it over for analysis. This means sifting and isolating the data that has impact and discarding that which is irrelevant. This substantially reduces the burden on transporting the data and quickens the analytical process. Pre-processing also needs to be intelligent so that it sends the right samples. Samples that are biased one way or the other lead to flawed analysis. No business can afford to mess around with its data. The reliance on data as a source of economic value is going to continue to grow. The volume of data available for collection is also predicted to grow exponentially. Sensors are becoming cheaper, data networks are proliferating and the cost of storage is coming down. The case to be selective about what data to generate, capture, filter and pre-process is becoming stronger. These decisions require a blend of data science, domain knowledge, a good handle on business imperatives and even some felicity with investment hacks would be handy. As important as a broad notion of why data is being collected is the knowledge of sophisticated tools like R, Julia, Pandas, RedShift, NetworkX, etc. Some serious extended nerdery, by way of custom tools that address business-specific challenges, becomes even more vital. Responsible data scientists are the ones that will make the 3

5 intelligent shift to filtering and pre-processing data, turning it into what it is: a business asset directly related to value creation. For the others, it will be a disappointment to watch insights blur into a chaotic data landscape. References i ii iii Data like aircraft age may be difficult to use as some components may be older than the aircraft, a result of maintenance activity 4

6 About the author Dr. Chetan has a PhD in Algorithms and is a passionate data scientist. He is currently part of the Chief Technology Office involved in tackling challenging algorithm design problems in the field of Applied AI. About Wipro Wipro Ltd. (NYSE:WIT) is a leading information technology, consulting and business process services company that delivers solutions to enable its clients do business better. Wipro delivers winning business outcomes through its deep industry experience and a 360 degree view of "Business through Technology. By combining digital strategy, customer centric design, advanced analytics and product engineering approach, Wipro helps its clients create successful and adaptive businesses. A company recognized globally for its comprehensive portfolio of services, strong commitment to sustainability and good corporate citizenship, Wipro has a dedicated workforce of over 160,000, serving clients in 175+ cities across 6 continents. For more information, please visit wipro.com or write to us at info@wipro.com 5

7 DO BUSINESS BETTER CONSULTING SYSTEM INTEGRATION BUSINESS PROCESS SERVICES WIPRO TECHNOLOGIES, DODDAKANNELLI, SARJAPUR ROAD, BANGALORE , INDIA. TEL : +91 (80) , FAX : +91 (80) Wipro LTD 2016 IND/B&T/MAY-JUN2016 No part of this booklet may be reproduced in any form by any electronic or mechanical means (including photocopying, recording and printing) without permission in writing from the publisher, except for reading and browsing via the world wide web. Users are not permitted to mount this booklet on any network server.