IDC MaturityScape Benchmark: Big Data and Analytics in the United States

Transcription

1 IDC MaturityScape Benchmark IDC MaturityScape Benchmark: Big Data and Analytics in the United States Dan Vesset Suya Xiong IDC MATURITYSCAPE BENCHMARK FIGURE FIGURE 1 Big Data and Analytics Maturity Distribution Across Stages n = 150 Source: IDC's Big Data and Analytics MaturityScape Benchmark Survey, 2015 Figure 1 presents the maturity distribution across the IDC MaturityScape stages from the simplest ad hoc stage to the advanced, systematized optimized stage. December 2015, IDC #US

2 IDC OPINION As the evidence mounts for the competitive edge demonstrated by organizations embracing datadriven decision making, Big Data and analytics (BDA) has become top agenda items for a growing number of executives. Many business and IT leaders are challenged to employ BDA solutions to create competitive advantage or derive desired outcomes through better and faster decision-making processes. The BDA capabilities are represented by a complex interplay of technology, data, processes, and people looking to derive value from data. Many organizations, however, do not yet have the BDA maturity to address the range of requirements needed to capitalize on all of their enterprise data assets and to deploy analytics pervasively to optimize operational, tactical, and strategic decisions. To help IT and business leaders plan and prioritize opportunities brought by BDA, IDC has developed the BDA maturity model, identifying the five stages of BDA maturity and the critical competencies, measures, outcomes, and actions that are required for organizations to improve the BDA discipline in their organizations. IDC has also established a series of benchmark studies on BDA maturity to offer a systematic and rigorous approach for organizations to work with IDC to assess their BDA competency and maturity. The current research provides updated benchmark data to help organizations identify the key capabilities that distinguish organizations whose BDA efforts have met or exceeded their overall expectations from their peers whose BDA efforts have fallen short. Key findings of this benchmark study include: The majority of companies surveyed have yet to establish and leverage BDA capabilities and expertise at the enterprise level (i.e., managed maturity) and are still at the opportunistic and repeatable stages. Organizations need to effectively evolve their BDA abilities as BDA capabilities are becoming increasingly critical to achieving desired outcomes. Organizations able to achieve greater business outcomes tend to have greater BDA maturity across all five maturity dimensions. This correlation highlights the need to improve BDA maturity by focusing on the most critical aspects of the enterprise's business needs, target outcomes, and competitive environment. IN THIS STUDY This study presents the results of IDC's 2015 Big Data and Analytics MaturityScape Benchmark Survey and should be viewed as a supplement to IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015). Together, they provide a comprehensive overview of the IDC's Big Data and analytics maturity model: The study presents the current state of the market based on quantitative research. The results in this study will enable organizations to work with IDC to assess their BDA maturity level against industry benchmarks and make data-driven decisions based on their organizational goals. In IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015), we identified the stages, dimensions, outcomes, and actions that organizations should consider to effectively develop BDA competency. This study updates the BDA maturity model presented in IDC MaturityScape: Big Data and Analytics - A Guide to Unlocking Information Assets (IDC #239771, March 2013). This document enables organizations to answer the following questions: How will BDA affect the way we do business? 2015 IDC #US

3 Where are we on a maturity scale for BDA competencies in terms of the business needs? Where are our peers? How should we establish standards for pursuing BDA initiatives? What's the path to encourage and improve intra- and intergroup collaboration in promoting and encouraging the use of BDA solutions? How can we maximize returns on investment (ROIs) in Big Data, data warehousing, business intelligence, advanced analytics, and related technology, people, and processes? SITUATION OVERVIEW As business and IT leaders are challenged to address today's most pressing challenges and opportunities, they are looking for ways to transform how their enterprises acquire, manage, and leverage data for competitive advantage. BDA, as a new generation of technologies and architectures designed to economically extract value from data, have become top agenda items for a growing number of executives. However, the plethora of technology choices; the range of analytics, technology, and management skills; and the amount of hype made it difficult for many organizations to prioritize resource allocations toward BDA projects and coordinate all the moving parts to implement successfully a cohesive BDA strategy. IDC believes that the hype surrounding Big Data has subsided, and a more pragmatic approach to all data and analytics is taking root. IDC market forecasts point to the expectation that BDA will remain one of the top investment priorities for organizations of all sizes for the foreseeable future. It is becoming increasingly critical for business and technology leaders to focus on improving the maturity of their organization's BDA capabilities to create competitive advantage in the marketplace or to fulfill their organization's mission. IDC's Big Data and Analytics MaturityScape enables businesses to assess their organizations' competencies to leverage and manage the BDA solutions with respect to five key dimensions: vision, technology, data, people, and process. Each dimension is targeted at a key aspect of BDA mastery and can be assessed as an independently measure as well as in conjunction with other dimension. Stages of IDC's Big Data and Analytics MaturityScape IDC's Big Data and Analytics MaturityScape consists of five stages: ad hoc, opportunistic, repeatable, managed, and optimized. For each stage, the IDC's Big Data and Analytics MaturityScape Benchmark addresses how capabilities for a particular dimension need to change to draw maximum benefit from the extraordinary power of data. The key characteristics of the five maturity stages are: Ad hoc experimental: Ad hoc, siloed pilot projects, undefined processes, and individual effort Opportunistic intentional: Defined requirements and processes, unbudgeted funding, project management, and resource allocation inefficiency Repeatable accepted: Recurring projects, budgeted and funded program management, documented strategy and processes, and stakeholder buy-in Managed measured: Project, process, and program measurement that influence investment decisions and standards emerge Optimized operationalized: Continuous and coordinated Big Data and analytics process improvement and value realization 2015 IDC #US

4 The five maturity stages are described in greater detail in IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015). Refer to Figure 4 in the Appendix section for a visual representation of these stages. Survey Findings: Maturity Distribution Across Stages Refer back to Figure 1 to see the aggregated maturity distribution across all five Big Data and analytics maturity dimensions. The highlights of IDC's 2015 Big Data and Analytics MaturityScape Benchmark Survey are as follows: More than half (53.6%) of respondents indicated that their organizations were at the opportunistic and repeatable stage. These organizations have already had some defined requirements and process around BDA projects. Data collection, monitoring, and integration processes are in place at the department level, but consistent data governance and security practices have not been established. Business value realized remains localized because of the lack of coordinated use of BDA capabilities. There needs to be enterprise support for measurement tools and methods to drive outcomes and effectiveness, which can be achieved when organizations move to higher maturity levels. A small percentage of respondents (19.1%) are at the managed stage, where their organizations had experienced the emergence of BDA program standards. These organizations have developed a cross-business unit-level BDA strategy, and there is an enterprisewide budget with upper management support. IDC believes that these organizations are starting to enjoy the competitive advantage brought by BDA. About one-fourth of survey respondents (23.4%) are at the optimized stage. Organizations at this stage have already established an enterprisewide, documented, and accepted BDA strategy. We suspect that the percentage of United States-based organizations at this level of BDA maturity is this high because of the way in which Big Data and analytics were defined in this research. In other words, a comprehensive definition encompassing decision support and decision automation processes, technology, people, and data was used (rather than a narrow definition only focused on the extreme cases of Big Data). BDA processes are categorized into performance management, operational intelligence, and exploration and discovery, with appropriate support, staffing, technology, and funding for each. These organizations leverage BDA solutions and their predictive capabilities to capitalize on new opportunities and to mitigate risk. Very few (3.8%) of organizations were still at the least mature, ad hoc, stage. These organizations lack basic BDA strategy and management interest or support. BDA processes in those organizations are focused primarily on creating information repositories with access only to siloed information therein, which results in narrow knowledge and limited effect on business outcomes. These results provide a consolidated current view of the overall BDA maturity in the United States. Successful deployment and the use of BDA solutions depends on a multipronged approach guided by a strategy that accounts for not just technology but also talent and capital resources, business and IT processes, and the data. To help organizations understand the underlying strengths and weaknesses of their BDA capabilities and compare them with their peers, we also measured maturity across the five key dimensions of the Big Data and Analytics MaturityScape. Dimensions of IDC's Big Data and Analytics MaturityScape To view the opportunities and challenges more clearly as IT moves through the various stages of Big Data and analytics maturity, organizations need to understand the five critical dimensions. Note that the dimensions of IDC's Big Data and Analytics MaturityScape and their characteristics at each stage of maturity are highlighted in IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015): 2015 IDC #US

5 Vision includes attributes such as strategy, capital and operational budgets, performance metrics, sponsorship, and project and program justification. Technology includes attributes such as the appropriateness, integration, support for standards, and performance of technology and IT architecture to all the relevant BDA workloads. Data includes attributes such as the quality, relevance, availability, reliability, governance, security, and accessibility of multistructured data. People includes attributes such as technology deployment and management, data analysis, analytic application, report, dashboard development skills, and intra- and intergroup collaboration, as well as organizational structure, leadership, training, and cultural readiness. Process includes attributes such as the processes of data collection, consolidation, integration, analysis, information dissemination and consumption, and decision making. Survey Findings: Maturity Distribution Across Dimensions Figure 2 illustrates the maturity distributions for each of the five BDA maturity dimensions. We see a consistency of maturity distribution patterns for ad hoc and optimized stages across the five dimension. However, there are some distribution variations across the middle three maturity dimensions. For example, while 14.9% of all of the surveyed organizations identified themselves as being at the managed stage for data, almost two times that percentage (26.0%) claimed to be at the same stage for vision. Similarly, 20.4% of the organizations claimed to be at the opportunistic stage for process dimension, but 31.2% of the organizations were at the same stage for their people dimension IDC #US

6 FIGURE 2 Big Data and Analytics Individual Dimensions Maturity Distribution Dashboard n = 150 Source: IDC's Big Data and Analytics MaturityScape Benchmark Survey, 2015 In addition, within each organization, the maturity levels for dimensions are different: Organizations have their own strengths and weaknesses in BDA. With respect to the data dimension, majority of respondents (59.1%) fall into the opportunistic and repeatable stages; these enterprises lack consistent and enterprisewide data quality processes. Similarly, within the people dimension, 56.2% fall within the opportunistic and repeatable stages: if these organizations' current BDA skills, collaboration, and training structures don't mature further, it is unlikely that they will be able to create consistent data-driven value over the long term. The process dimension is in a less mature stage, with 47.4% of respondents in opportunistic or repeatable stages. Improving BDA maturity is a multidimensional journey that is needed to adapt current organizational processes, technology, and skills to the requirement of the enterprise's business strategy, target outcomes, and competitive environment. Understanding all the five BDA MaturityScape dimensions is 2015 IDC #US

7 important because organizations often focus only on one specific measure, instead of prioritizing their resources and investments to create alignment in the overall organizationwide ability to support decision-making processes. For instance, organizations with deep expertise in technology can find themselves unprepared for the efforts needed to change organizational behavior and those with ample human resources for analytics may find they are lacking access to relevant, on-time data. These figures offer a more focused way to think about the migratory issues involved in moving from one stage to the next. IDC suggests that the least advanced dimension should never be more than one level of maturity behind the most advanced dimension. Improving Big Data and Analytics Maturity: Survivors and Thrivers The IDC MaturityScape framework is structured in part not only to identify what a particular level of maturity requires but also to enable leaders to assess just how much maturity they need at any given point in time. As part of IDC's Big Data and Analytics MaturityScape Benchmark study, we asked organizations to self-assess their approach to vision, technology, data, people, and process in the context of their overall ability to leverage BDA solutions (see Figure 3). Based on this result, we segmented organizations into two categories: Survivors: Organizations where the benefits of recent BDA project, in aggregate, did not meet expectations or resulted in no benefits Thrivers: Organizations where the benefits of recent BDA projects, in aggregate, met or exceeded expectations Note that here we excluded organizations that either had unquantified benefits or where BDA deployments were too new to allow for the assessment of benefits from this categorization. Survey Findings: Comparison of Survivors and Thrivers Figure 3 demonstrates the comparison of BDA maturity between survivors and thrivers. These results demonstrate that thrivers gravitate toward higher maturity levels. Almost half of thrivers (47.4%) fell into the managed and optimized stages of BDA maturity, while the organizations classified as survivors had 39.4% in the managed or optimized stages. Conversely, about one-third (34.2%) of survivors were categorized as ad hoc or opportunistic, while just one-fourth (25.5%) of thrivers were in the ad hoc and opportunistic stages. When examining thrivers versus survivors for the five dimensions of BDA maturity, we see that thrivers have higher representation in the managed and optimized stages, with lower representation in the ad hoc and opportunistic stages. The organizations had a quantifiable advantage when they actively pursued BDA solutions with long-term vision and well-defined process. IDC believes that results presented in Figure 3 represent an opportunity to improve communications between business and IT leaders around the themes and challenges that BDA presents as we've described in IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015) IDC #US

8 FIGURE 3 Big Data and Analytics Dimensions Dashboard: Comparison of Survivors and Thrivers n = 119 Source: IDC's Big Data and Analytics MaturityScape Benchmark Survey, 2015 ESSENTIAL GUIDANCE Having effective, mature BDA abilities is precursor for creating innovative, data-driven organizations that are capable of assisting in the achievement of desired outcomes. For IT executives in world-class enterprises, knowing their BDA maturity level is essential to enable them to understand what the IT organization is capable of delivering, grasp what maturity level is required to achieve target business objectives, and identify the next steps to closing that maturity gap. In IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015), we discussed that the ability to manage and analyze Big Data and derive value from these activities, as well as to measure and improve them, will increasingly define an organization's ability to compete or service its constituents. This benchmark study confirms this finding IDC #US

9 Table 1 depicts the most important factors that define thrivers with respect to their BDA capabilities. TABLE 1 Top 5 Traits of Thrivers' Big Data and Analytics Maturity Dimension Vision Data Technology People Process Trait A BDA strategy and data-driven culture across the enterprise Data quality governed by centralized processes, metrics, and methods BDA deployments based on an architecture governed by a central architecture board Collaboration processes among staff to share relevant data, metrics, and best practices Data management and analysis processes within the organization are defined, measured, and managed based on clearly understood metrics Source: IDC's Big Data and Analytics MaturityScape Benchmark Survey, 2015 Based on the findings in this study, we suggest that organizations should have a balanced approach to investing across all five BDA maturity dimensions. Our interactions with customers suggest that overinvesting in one or some BDA capabilities at the expense of others could be a roadblock or at least an inhibitor to an organization's goal of achieving higher levels of BDA maturity. Table 2 provides a set of steps for progressing from stage to stage, in each of the five dimensions of IDC's Big Data and Analytics MaturityScape IDC #US

10 TABLE 2 IDC MaturityScape: Big Data and Analytics Progressing Through the Stages Stage Ad hoc Outcome: Proof-of-concept or pilot projects. Value through new knowledge and learning. Guidance Launch a proof-of-concept or pilot project using existing resources. Use existing data with the recognition that it may be incomplete and may lack the necessary quality, which will require substantial manual data preparation effort. Expect significant manual effort to tune the system to reach desired performance. Lack of all the necessary functionality will require workaround and scope control to achieve desired project outcome. Do not spend significant time and effort to try to gain executive support before establishing initial project proof points. Look for support from colleagues with specialized BDA skills. Focus on information access projects within a specific business domain. Opportunistic Outcome: Cross-functional deployment. Knowledge value grows; business value opportunities become visible. Establish a department-level BDA strategy. Although disconnected from the rest of the organization, it will guide the next round of project investments. Budget for localized projects that have midmanagement support. Begin to integrate data from multiple sources to move toward high levels of trust in the information coming from the BDA system. Deploy new technology (on-premise or in the cloud) that enables more timely access to information because of system performance improvements. Establish a BDA team with skills in existing and newly deployed technologies and that coordinates work with colleagues in business. Allow data scientists to experiment to uncover new insights IDC #US

11 TABLE 2 IDC MaturityScape: Big Data and Analytics Progressing Through the Stages Stage Repeatable Outcome: Persistent use. Business value is realized but remains localized to business units. Guidance Develop cross-departmental business-unit-level BDA strategies. Budget for business unit needs. Perform a localized cost-benefit analysis for BDA projects. Continue to expand the availability of and to integrate internal multistructured data sources. Be aware that data governance policies and procedures will be difficult to implement at a single business unit level. Expand the availability of fit-for-purpose technology with the understanding that initial adoption will be selective. Minimize shelfware or unutilized subscription licenses via staged rollouts. Assign, train, and hire staff based on the BDA strategy. Augment existing skills with specialized external service providers. Begin to monitor and document decision processes and decision outcomes. Ensure the BDA team has representatives from all stakeholder groups to facilitate collaboration. Managed Outcome: Predictable outcomes. New product and service opportunities transition to business plans. IT efficiency is demonstrated. Assign an executive-level leader to coordinate the development of a crossbusiness-unit BDA strategy. Establish metrics and methodology for data governance and metrics by which BDA processes, staff, and outcomes are measured. Deploy fit-for-purpose and workload-optimized technology. Incorporate predictive analytics into technology performance monitoring and management processes. Enable broad technology adoption by ensuring that an appropriate technology pricing structure is negotiated with IT vendors. Establish executive support and develop plans for a centralized BDA team. Provide support for decentralized analytics staff within business groups. Optimized Outcome: Continuous learning and improvement. Previously unattainable business value is continuously produced. Develop an enterprisewide BDA strategy that is championed by a C-level executive. Budget for BDA projects. Make available tools and methodology for business case development and performance and outcomes measurement. Make available information about all the data sources for users with the appropriate security rights. Maximize use of the workload-optimized system, automated system performance management, and dynamic scalability features of BDA technology. Regularly provide training to all the BDA technology, analytics, and business staff. Maximize BDA staff centralization for functions such as data integration, systems management, and report and dashboard development. Ensure that both performance management and experimentation and discovery processes are supported with appropriate staffing, technology, and funding. Employ decision management techniques to enable continuous process improvement and integration of analytics into business processes. Source: IDC, IDC #US

12 LEARN MORE Related Research IDC FutureScape: Worldwide Big Data and Analytics 2016 Predictions (IDC #259835, November 2015) Worldwide Business Analytics Services Forecast, (IDC #US , November 2015) Worldwide Big Data Technology and Services Forecast, (IDC #259532, October 2015) IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015) IDC MarketScape: Worldwide Business Analytics Services 2015 Vendor Assessment (IDC #255276, April 2015) IDC's Worldwide Big Data Taxonomy, 2015 (IDC #254052, March 2015) Appendix Figure 4 is a graphic representation of the five stages of Big Data and analytics maturity. The five maturity stages are described in greater detail in IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015) IDC #US

13 FIGURE 4 IDC's Big Data and Analytics MaturityScape Stage Overview Source: IDC's Big Data and Analytics MaturityScape Benchmark Survey, 2015 Methodology The results in this study are based on IDC's Big Data and Analytics MaturityScape Benchmark Survey of 150 organizations in the United States, conducted in August The survey, executed online, was based on a structured questionnaire of 30 questions. These survey questions were focused on the five dimensions of IDC's Big Data and Analytics MaturityScape. For each dimension, we created a set of questions to assess the level of capability/maturity for the dimension. Note: All numbers in this document may not be exact due to rounding. Survey Respondent Segmentation The survey respondents segmented as followed: All of the respondents were from the United States. 60.0% of the respondents were from organizations with 5,000+employees, and the rest (40.0%) of the respondents were from organizations with 1,000 4,999 employees. Respondents were from 20 different industries: 20.0% of them in professional services industry (legal, accounting, engineering, etc.) and 16.0% of respondents in retail. All of the respondents were responsible for their organizations' BDA initiatives. 24.7% of the respondents were CIOs or vice presidents of IT in the organizations IDC #US

14 SYNOPSIS This IDC study presents the results of IDC's 2015 Big Data and Analytics MaturityScape Benchmark Survey and should be viewed as a supplement to IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015). Together, they provide a comprehensive overview of the IDC's Big Data and analytics maturity model: The study presents the results from quantitative research that will enable organizations to work with IDC to assess their BDA maturity level against industry benchmarks and make datadriven decisions based on their organizational goals. In IDC MaturityScape: Big Data and Analytics (IDC #255138, June 2015), we identifies the stages, dimensions, outcomes, and actions that organizations should consider to effectively develop BDA competency. It updates the BDA maturity model presented in IDC MaturityScape: Big Data and Analytics - A Guide to Unlocking Information Assets (IDC #239771, March 2013). This document enables organizations to answer the following questions: How will BDA affect the way we do business? Where are we on a maturity scale for BDA competencies in terms of the business needs? Where are our peers? How should we establish standards for pursuing BDA initiatives? What's the path to encourage and improve intra- and intergroup collaboration in promoting and encouraging the use of BDA solutions? How can we maximize returns on investment in Big Data, data warehousing, business intelligence, advanced analytics, and related technology, people, and processes? "The BDA capabilities are represented by a complex interplay of technology, data, processes, and people looking to derive value from data," says Dan Vesset, program vice president of IDC's Business Analytics research. "However, most organizations still haven't taken full advantage of their BDA capabilities. To harness the possibilities of BDA, organizations will find the need to improve across some or many dimensions of Big Data and Analytics maturity model." 2015 IDC #US

15 About IDC International Data Corporation (IDC) is the premier global provider of market intelligence, advisory services, and events for the information technology, telecommunications and consumer technology markets. IDC helps IT professionals, business executives, and the investment community make factbased decisions on technology purchases and business strategy. More than 1,100 IDC analysts provide global, regional, and local expertise on technology and industry opportunities and trends in over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology media, research, and events company. Global Headquarters 5 Speen Street Framingham, MA USA idc-insights-community.com Copyright Notice This IDC research document was published as part of an IDC continuous intelligence service, providing written research, analyst interactions, telebriefings, and conferences. Visit to learn more about IDC subscription and consulting services. To view a list of IDC offices worldwide, visit Please contact the IDC Hotline at , ext (or ) or sales@idc.com for information on applying the price of this document toward the purchase of an IDC service or for information on additional copies or Web rights. Copyright 2015 IDC. Reproduction is forbidden unless authorized. All rights reserved.