The Road to Becoming a Visionary Big Data Analytics Organization

Transcription

1 CHECKLIST NO. 4 OF 4 TDWI CHECKLIST REPORT 2016 The Road to Becoming a Visionary Big Data Analytics Organization By David Stodder Sponsored by: c1 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org tdwi.org

2 CHECKLIST NO. 4 OF TDWI CHECKLIST REPORT The Road to Becoming a Visionary Big Data Analytics Organization By David Stodder TABLE OF CONTENTS 2 FOREWORD Best Practices for Becoming a Visionary Organization with Big Data Analytics 3 NUMBER ONE Drive Continuous Innovation Through Application of Big Data Analytics 3 NUMBER TWO Increase Business Agility with Big Data Analytics 4 NUMBER THREE Develop a Data Management Strategy That Delivers Detailed, Diverse Data 4 NUMBER FOUR Unify Data Architecture and Integrate Analytics to Support Smart Expansion 5 NUMBER FIVE Improve Effectiveness of Data Governance as Big Data Analytics Matures 5 SERIES SUMMARY 6 BACKGROUND TO THE SERIES 7 ABOUT OUR SPONSOR 7 ABOUT THE AUTHOR 7 ABOUT TDWI RESEARCH 555 S Renton Village Place, Ste. 700 Renton, WA T F E info@tdwi.org tdwi.org 2016 by TDWI, a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or part are prohibited except by written permission. requests or feedback to info@tdwi.org. Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies. 1 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org

3 FOREWORD BEST PRACTICES FOR BECOMING A VISIONARY ORGANIZATION WITH BIG DATA ANALYTICS Author s Note: TDWI has created a series of four Checklist Reports that detail best practices for getting from one stage of big data maturity to the next. Please see Background to the Series on page 6 for more information. TDWI identifies visionary organizations as those at the highest stage of big data analytics maturity. It is, by definition, an exclusive community; our research finds that most organizations are traveling in the right direction but few have reached the visionary level. This checklist is focused on the key issues to address for reaching the top level of maturity. Visionary Improve data governance Pioneering firms primarily in e-commerce, search, and social networking were the original visionaries in big data analytics. In trying to establish new business models and innovate with big data volume, variety, and velocity, many of these companies developed their own technologies and data science practices, which then became the basis for many of the technologies and frameworks in the Apache Hadoop ecosystem. Organizations across nearly all industries today are at least experimenting with, if not fully deploying, systems that are based on the Hadoop ecosystem. However, most are developing and deploying big data analytics with a combination of Apache open source project technologies and frameworks plus commercial off-the-shelf tools, analytics platforms, and data management systems. One of the biggest challenges as well as opportunities for organizations maturing toward the visionary stage is to knit together their variety of technologies and platforms into a unified architecture that helps them reach objectives faster and more efficiently. Visionary organizations understand that analytics is vital to everything they do, from executive decisions to daily operational actions involving customer engagement, supply chain management, and financial performance planning. They must protect the experimental aspects of analytics because these drive business innovation; learning from projects that do not go as planned is part of maturation. At the same time, organizations need to bring standardization and governance to bear so that analytics can safely and effectively become part of their activities and decision processes. This checklist discusses issues critical to achieving that balance. Determining the right questions to ask with big data analytics and conceptualizing the desired business outcomes of projects are challenges at every level of maturity. There is no fast track through the analytics life cycle, the entirety of which requires thought, discussion, and creativity. What visionary organizations are able to do, however, is execute on ideas and plans faster with solid technologies, practices, and governance (see Figure 1). Drive continuous innovation Increase business agility Manage for diverse data Figure 1. Five steps for the visionary organization. Unify architecture, integrate analytics 2 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org

4 NUMBER ONE DRIVE CONTINUOUS INNOVATION THROUGH APPLICATION OF BIG DATA ANALYTICS NUMBER TWO INCREASE BUSINESS AGILITY WITH BIG DATA ANALYTICS Visionary organizations seek to be analytics driven. Rather than have isolated analytics projects happening here and there without knowledge of each other, visionary organizations want to infuse analytics in everything from strategic decisions to daily operational actions by both humans and automated applications. They see the potential of analytics projects to benefit the entire organization; they want high-impact analytics that realize tangible insights from big data and move the organization to the industry forefront. Led by corporate executives, organizations at this level see continuous innovation in data science and big data analytics as vital to improving competitive advantage. The first place many organizations focus with big data analytics is customer intelligence. Organizations want to analyze the variety of customer behavior data generated across sales and marketing channels and examine how it relates to transaction data so they can understand trends, influences, and buying patterns. Initial goals are to derive insights that will help improve customer experiences and engagements, sharpen marketing campaigns, and increase personalization so that cross-sell and up-sell offers are more successful. As organizations become more mature with big data analytics, they share customer insights with other parts of the organization beyond marketing. Visionary firms can form innovation teams to focus use of big data analytics on development of new products and services, including those that stretch the boundaries of the organization s existing business model. Fully mature, visionary organizations use big data analytics often involving exploratory discovery into new data sources such as social media networks, wearables, and the Internet of Things (IoT) to enter new markets with data-informed products and services. From an organizational perspective, investment in data science personnel, big data technologies, and business user skills (i.e., for analyzing data independently) is a key issue for supporting continuous innovation through big data analytics. Throughout this checklist series, we have emphasized the need for collaboration between business and IT; this becomes ever more vital as organizations mature in their use of big data analytics. Business and IT together can guide the organization beyond fragmented projects and investments to a more strategic, enterprise-level perspective. With concerted leadership, organizations are able to address both short-term needs and long-term priorities in the expansion of big data analytics. Innovative organizations need analytics to achieve one overarching objective: supreme agility. Faster decision cycles, competitive pressures, and adjustment to the unexpected all drive organizations to improve business agility. Organizations are turning to big data analytics to mine new sources, including real-time data streams, to sense change, adjust personnel behavior and automated systems, and take advantage of unanticipated opportunities. Agility puts a premium on using analytics for real-time and predictive insights that enable organizations to move quickly in response to events and be proactive when trends are recognized. Organizations at the visionary stage of big data analytics maturity must demonstrate excellence in developing predictive models applying a variety of analytics techniques to examine numerous variables across data sources and draw insights from data relationships that help predict the future. Such organizations can use state-of-the-art distributed processing and front-end technologies to enable fast, flexible use of data visualization so that data scientists as well as business users can see data and analytics from different perspectives and spot actionable trends and patterns. To attain agility, visionary organizations need personnel with skills in applying a range of analytics for predictive and real-time insights, including text and stream analytics for unstructured or semistructured data. It is equally critical for organizations to assemble teams that match personnel skilled in analytics techniques with experts in business domains of interest such as customer behavior, financial performance, fraud detection, and population health. Guided by both business and analytical expertise, visionary organizations can embed analytics into business applications and processes to track the flow of big data and automatically uncover important events, data relationships, and patterns. Visionary organizations require flexible underlying data systems, not cement legacy systems mired in complex and hard-to-change programs, rules, and limits on the types of data that the systems can manage. Organizations need to evaluate existing systems and IT procedures to ensure they don t hinder flexible big data analytics. If they are employing newer Hadoop ecosystem technologies, organizations need to make sure that development of these systems is layered and modular and does not repeat errors of the past through poor documentation and code and rules that are difficult to change. 3 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org

5 NUMBER THREE DEVELOP A DATA MANAGEMENT STRATEGY THAT DELIVERS DETAILED, DIVERSE DATA NUMBER FOUR UNIFY DATA ARCHITECTURE AND INTEGRATE ANALYTICS TO SUPPORT SMART EXPANSION Big data is exploding not just in sheer volume but in granularity. Detailed data generated by online behavior, IoT sensors, and streams are adding quintillions of bytes per day. What would have been too costly or difficult to collect and access is now within reach. In addition, geospatial location and mobility are providing new attributes for location and movement; many organizations want to enrich and integrate their existing data with these new attributes to gain new analytical insights. Big data analytics can give organizations an unprecedented opportunity to gain finer-grained and potentially more accurate views of what is important, such as (depending on the industry) customer activity across channels, effectiveness of patient care, weather and soil conditions for agriculture, fraudulent activity, and manufacturing quality. Organizations seeking to progress to the mature, visionary state with big data analytics need to develop a technology strategy that sets out how the organization will realize value through analytics from ever-increasing data volume and detail. The first question organizations need to address is simple: Why do we need to do this? One rising objective for many is to monetize big data by developing revenue-generating, information-based services. Both business-to-business (B2B) and business-to-consumer (B2C) relationships can be enhanced when organizations tap data assets to develop dashboards and analytics that communicate helpful insights. Leading firms have successfully packaged information for clients or given clients access through applications to their data sources to run analytics. Mobile applications, which generate location, movement, behavioral, and transactional data, could provide a focus for data- or analytics-as-a-service offerings that are made possible by big data analytics. Whether your organization has plans to monetize data and analytics or not, it is important to prepare for data access and management that is less oriented toward traditional development of samples, cubes, and small data marts and more toward enablement of analytics that requires frequent iterations through massive, detailed data. Aggregations will always have their place for BI and limited analytics, but organizations at the mature, visionary stage of big data analytics will find that an increasing number of projects involve predictive, discovery, and other advanced analytics that sift through all the data. These will serve business objectives that demand a higher level of accuracy and sensitivity to detail. A unified data architecture and better integration of the analytics life cycle are critical for organizations aspiring to the highest level of big data analytics maturity. However, as users and data scientists reach out to multiple and diverse data to achieve goals, it is easy for data and analytics chaos to grow. Even visionary organizations can become overwhelmed to the point where the promise of big data turns into a burden of having too much data that is under attack by an uncontrolled onslaught of analytics. It is important for organizations to achieve unification through architecture and support carefully planned analytics life cycles so that they can expand big data analytics more rapidly and easily. In the previous checklist, we noted that organizations at the corporate adoption level of maturity are cognizant of the problems caused by a lack of unity in how they develop and deploy big data analytics but have not taken steps to increase it. Visionary-level organizations bring business and IT leadership together to champion the value of unified architecture and provide investment, a road map, and the leadership to bridge division and move forward. With greater attention to a unified architecture, visionary organizations can be more comfortable with extensibility into new data and new functionality for analytics and visualization. The architecture and infrastructure are set up to support it. Areas where organizations should focus to improve unity include: Analytics ecosystem and life cycle: Organizations typically have diverse analytics projects, each of which has its own life cycle of problem determination; data preparation; model building, validation, and deployment; performance monitoring; and evaluation of results. Organizations should examine where duplication exists and where groups could be working in greater concert. They can begin to build awareness of how projects can benefit by ensuring reuse, repeatability, and documentation so others can move faster. Big data management and processing: Organizations at the corporate adoption and visionary levels of maturity typically have a heterogeneous group of relational, Hadoop, and NoSQL environments. Each may be sensible for certain projects and data types, but without a unified data architecture, they can become difficult and costly to maintain and grow. Organizations need to focus on integration to harmonize platforms for better data flow and analytics across them. They should ensure that each platform is truly the best choice for the workload. Extensibility for smarter growth: With a unified architecture and analytics life cycle, organizations can plan for extensibility rather than simply add to a hodgepodge of technologies. Organizations should create a road map that envisions how the architecture and infrastructure can be extended efficiently, including through use of cloud computing to accelerate support for dynamic demand. 4 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org

6 NUMBER FIVE IMPROVE EFFECTIVENESS OF DATA GOVERNANCE AS BIG DATA ANALYTICS MATURES SERIES SUMMARY Throughout this checklist series, TDWI has advocated strongly that organizations make governance part of their maturation with big data analytics. Governance is vital for safeguarding sensitive data and properly adhering to regulations. Given that there are always new sources of data for analytics (such as data from IoT devices) and that the only constant with regulations is change, enterprises cannot rest with data governance. Organizations should monitor and measure their data governance effectiveness. It is important to review policies and rules to make sure that they are up-to-date, accomplish their purpose, and are well administered. Data governance rules and policies must be as tight as necessary but no tighter; they must be balanced with users needs for data access to perform analytics and data discovery. If rules are unnecessarily tight, users will work around them by setting up their own shadow systems outside governance controls, which could expose data and the organization to governance problems. Organizations at the visionary level of maturity need to evaluate and deploy software that enables them to standardize and automate governance, ensure that users tools and applications apply the rules, track data lineage, monitor derived data s transformation and use, and adjust rules as conditions change. In TDWI Checklist: Becoming an Analytically Mature Organization in a Big Data Age (Checklist No. 3), we discussed that organizations at the corporate adoption level of maturity are cognizant of the need to extend governance beyond monitoring the data to also cover big data analytics model deployment. Visionary organizations can work through Centers of Excellence (CoEs; covered in Checklist No. 3) and governance committees to establish guidance and best practices for data scientists and developers as they build and execute models. They can encourage values such as reuse and consistency so that model development and deployment are more efficient and build on the organization s body of experience. Visionary organizations should make it a priority to include governance as a core part of their emerging unified architectures, particularly as they extend them beyond traditional systems to data lakes and cloud computing. Organizations at the visionary level of maturity integrate data governance with data stewardship, which is focused on raising the quality of data and analytics content. Visionary organizations can deploy data catalogs, glossaries, and master data management tools to manage shared definitions and increase overall big data quality and consistency. We have reached the conclusion of our four-part checklist series on achieving big data analytics maturity. The journey has taken us through how organizations can evolve from the beginning stages nascent and pre-adoption to the first real plateau, early adoption. We outlined practices for building a strong foundation of accomplishment in each of the five dimensions organization, data management, infrastructure, analytics, and governance critical to success with big data and analytics. We then discussed how organizations can move across the chasm to corporate adoption. With a head of steam from successes at the corporate adoption stage, the best and brightest organizations are ready to take the last step up to the mature, visionary level and become big data and analytics leaders in their respective industries. A key point, however, is that your organization can gain business value at every stage. Maturing with big data analytics is a process of developing greater knowledge about data and how analysis of the data can inform decisions and actions. Just as every step delivers value, the process of acquiring knowledge returns benefits at every stage and must continue, especially as new sources and types of analytics emerge. Creativity and innovation will find new technological ways of unleashing the power of data and analytics. The challenge is to harness that power. 5 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org

7 BACKGROUND TO THE SERIES TDWI research indicates that organizations view analytics as an opportunity to help in better decision making, to better understand customers, and to improve business performance. Some also see it as a way to generate revenue. At the same time, many organizations are collecting increasing amounts of disparate data. In fact, many are collecting more than they can manage or analyze, yet they realize that big data and data analysis can provide an important strategic competitive advantage. According to TDWI research, interest is growing for big data solutions. In early 2014, TDWI created a big data maturity model to help organizations understand how their big data and big data analytics deployments compared to those of their peers and how they could advance with analytics. The maturity model consists of five stages: nascent, pre-adoption, early adoption, corporate adoption, and mature/visionary. The model also includes an online assessment tool that measures the maturity of a big data program in an objective way across five dimensions key to deriving value from big data and analytics. These dimensions are: organization, data management, infrastructure, analytics, and governance. Approximately 600 organizations have completed the TDWI assessment to date. The vast majority of respondents (86%) are in the pre-adoption and early adoption stages of big data maturity. Slightly less than 10% have moved past the chasm that separates early adoption from corporate adoption (see Figure 2). Please see the other TDWI Checklist Reports in this four-part series detailing best practices for getting from one stage of big data maturity to the next: Five Best Practices for Getting Started with Big Data Analytics (Checklist No. 1) Five Keys to Moving Your Big Data Analytics Program Forward (Checklist No. 2) Becoming an Analytically Mature Organization in a Big Data Age (Checklist No. 3) 60% 50% 40% 30% 20% 10% 0% Percent of Respondents in Each Stage of Maturity (n=600) CHASM Nascent Pre-adoption Early adoption Corporate Mature/ adoption Visionary Figure 2. Percent of respondents in each stage of big data maturity (n=600), from the TDWI Big Data Maturity Model, online at 6 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org

8 ABOUT OUR SPONSOR ABOUT THE AUTHOR Hewlett Packard Enterprise The rapid explosion of big data at enterprises globally is changing the landscape of business and IT. Hewlett Packard Enterprise offers the only big data platform designed to harness 100% of your relevant data fast and at scale, regardless of its source. The platform offers a wide range of deployment options including on premise, in the cloud, or on Hadoop. See how you can extract more business value from all your data to deliver new insights and gain competitive advantage. Learn more at DAVID STODDER is the senior director of TDWI Research for business intelligence (BI). He focuses on providing research-based insight and best practices for organizations implementing BI, analytics, performance management, data discovery, data visualization, and related technologies and methods. He is the author of TDWI Best Practices Reports and Checklist Reports on data discovery, data visualization, customer analytics in the age of social media, BI/data warehouse agility, mobile BI, and information management. He has chaired TDWI conferences on BI agility and big data analytics. Stodder has provided thought leadership on BI, information management, and IT management for more than two decades. He has served as vice president and research director with Ventana Research, and he was the founding chief editor of Intelligent Enterprise, where he served as editorial director for nine years. You can reach him by (dstodder@tdwi.org), on Twitter (@dbstodder), and on LinkedIn (linkedin.com/in/davidstodder). ABOUT TDWI RESEARCH TDWI Research provides research and advice for BI professionals worldwide. TDWI Research focuses exclusively on BI/DW issues and teams up with industry practitioners to deliver both broad and deep understanding of the business and technical issues surrounding the deployment of business intelligence and data warehousing solutions. TDWI Research offers reports, commentary, and inquiry services via a worldwide membership program and provides custom research, benchmarking, and strategic planning services to user and vendor organizations. 555 S Renton Village Place, Ste. 700 Renton, WA T F E info@tdwi.org tdwi.org 7 TDWI RESEARCH CHECKLIST NO. 4 tdwi.org