2018 Big Data Trends: Liberate, Integrate & Trust

Transcription

1 2018 Big Data Trends: Liberate, Integrate & Trust

2 Executive Summary Syncsort conducted its fourth annual survey of IT professionals working with Big Data to get a real-world perspective on the opportunities and challenges facing enterprises as we enter Almost 200 IT professionals, all of whom have some involvement with Big Data, participated, representing a wide range of industries. It s clear that what had been new, unfamiliar technology when we first started the survey, is now becoming an integral part of many IT landscapes. More than 40% of respondents report being in production with Hadoop or Spark, and more than 30% say they are engaged in a proof of concept or pilot program. We wanted to uncover what priorities are driving these organizations. What challenges are the toughest to overcome? How are they going about it? What data is essential? And what benefits are companies seeing in their bottom line? Read on to learn what they told us.

3 What is your current role? 24.0% Data Architect CIO CTO COO 4.7% Business Intelligence/ Data Analyst 7.3% 12.5% Other 11.5% Data Scientist 20.8% Developer 15.1% IT Manager Other IT Executive 4.2%

4 19.8% Financial Services 13.0% 13.0% Healthcare Information Other What industry is your company in? 8.9% 8.3% 7.3% Government Retail 6.8% Insurance Data Services Telecommunications Travel & Hospitality Non-profit Software or Software as a Service Transportation 4.2% 4.2% 4.2% 3.6% 3.6% 3.1%

5 Data Lake Use Cases The distributed architectures of Hadoop and Spark are especially adept at data integration and manipulation, whether that takes the form of traditional ETL, offloading data and workloads from legacy systems, filling an active archive, or simply blending disparate data types. The insights from analytics that drive Big Data projects have the hightech plumbing of distributed integration powering them. 70.8% ETL 63.5% Advanced/ Predictive Analytics 39.5% Data Blending 60.4% Real-time Analytics Active Archive 30.2% What Data Lake use cases are of most interest to you? Offload Data and/or Workloads from Legacy Systems such as EDW or Mainframe 29.2% Data Discovery & Visualization Other 0.5% Clickstream Analysis & Social Media Data Internet of Things 22.9% 16.7% 53.1% 45.3% Operational Analytics

6 RDBMS Enterprise Data Warehouse NoSQL databases Files from Third-Party Data Providers or Partners Cloud Repositories Mainframe 31.8% Web/mobile/social media 30.2% AIX Power Systems 30.2% IBM i 17.7% Machines/ sensors 16.1% Other 0.5% 46.4% 45.3% 40.6% What data sources are/will be used to populate your data lake What s Filling the Data Lake 69.3% 62.5% A wide variety of data is filling data lakes, including data from mainframes, relational databases and enterprise data warehouses, along with relatively newer streaming and NoSQL data. In addition, as more organizations leverage cloud as a deployment platform, it is also gaining importance as a data source. Pulling all data in from across the enterprise is key to getting the benefits promised by this technology. Leaving any data source behind means a gap in knowledge, a blind spot in corporate insight. Putting ALL enterprise data together, blending and integrating it, then analyzing that holistic view provides the greatest benefits from Big Data implementations.

7 59.9% Increase business user productivity with more readily available data across the organization 59.4% Increase operational efficiency and reduce costs Do you see Hadoop/Spark as a way to 55.7% 54.7% 55.2% Take advantage of nextgeneration analytics Increase revenue and accelerate growth based on better insights Support a cost-effective storage/data archive strategy Many Benefits to the Business An implementation of Big Data technology provides a wide variety of business benefits. Data Lakes are helping businesses cut costs through operational efficiencies, costeffective storage strategies, and avoiding compliance fines. This technology is also boosting revenue by increasing productivity, extending the capabilities of existing infrastructure, increasing agility, and providing better business insights with advanced analytics. Increase business/ IT agility 41.7% Complement/ extend your existing large EDW or mainframe investments 32.8% Facilitate regulatory compliance by retaining data long term Free-up mainframe resources and reduce costs 30.2% 21.4%

8 Another, often overlooked, benefit of Big Data projects is getting a better understanding of capacity.over 90% of respondents found some capacity management value from Data Lake projects. How important is the ability to collect data across traditional and Next-Gen platforms to optimize capacity planning and management? 20.1% Somewhat 51.0% 1.5% Not 5.7% Slightly 21.6% Very

9 Top Challenges of Implementing a Data Lake While Big Data projects help organizations overcome challenges and open up new opportunities in their business, they also come with their own set of hurdles. Challenges around data quality and data governance top the list this year, followed closely by the skills shortage, which had been the #1 challenge cited in our survey for the prior three years. The difficulty keeping up with rapid change in technology, as well as keeping the data lake fresh, round out the top 5. The more Big Data workloads become productionized, the clearer the need for ensuring data quality and including the data lake in data governance initiatives. Organizations will also use more tools that leverage the skills of the staff they already have to keep the cost of implementation and maintenance down. Please rank the challenges/barriers for implementing a Data Lake (1 = most significant, 5 = least significant) Data quality: Assessing and improving the quality of data as it enters and/or resides in the data lake Skills/Staff: Need to learn a new set of skills and tools and Hadoop programmers are difficult to find and/or too expensive Data Governance: Including the data lake in data governance initiatives and meeting regulatory compliance mandates Rapid change: Compute frameworks and tools continue to evolve and it s difficult to keep up with the latest technologies Cost: Hadoop/Spark may be more % 15.6% 24.5% 20.3% 24.5% 18.3% 20.8% 15.6% 32.8% 13.0% Fresh data: Difficult to keep data lake up-todate with changes made on other platforms 8.3% 27.6% 23.0% 25.0% 16.2% Mainframe: Difficult to move mainframe data in and out of Hadoop/Spark Data movement: Difficult to move data in and out of Hadoop/Spark 8.9% 22.9% 27.1% 28.7% 12.5% expensive than we think 16.2% 13.5% 18.3% 27.6% 24.5% Connectivity: Ability to integrate 19.8% 20.9% 13.5% 20.3% 18.8% 21.9% 21.9% 17.7% 17.7% 24.0% 14.1% 23.4% 19.2% 29.2% 17.7% with existing data sources/applications 12.5% 16.7% 29.7% 24.0% 17.2% Uncertainty: I don t know what I don t know 13.5% 15.1% 26.5% 25.6% 19.3%

10 Keeping Data Fresh When we asked a follow up question specifically about keeping data in sync with data sources after it s been loaded into the Data Lake, over 75% of respondents say they have difficulty with this, especially when the source is mainframe data. How difficult is it to keep your Data Lake in sync with changing data sources, particularly mainframe sources? The only way to keep insights accurate over time is to continually refresh data in the lake with up-to-date source data. Filling the Data Lake isn t a one-time job. Streaming data sources pour in all the time. Transactional sources change with each transaction. That data needs to go in and get integrated with the rest of the data at a reasonable speed, or the Data Lake becomes a stagnant, stale source of old news. 11.5% Not difficult 11.5% My company doesn t have a mainframe 12.5% Very difficult 64.6% Somewhat difficult

11 Old Meets New Data from so-called legacy platforms including the mainframe and IBM i (aka AS/400) play a significant role in the data lake. Organizations that have relied on these systems for decades know that leaving this valuable data stranded would severely limit their analytics. How valuable would it be for your company to access and integrate mainframe data into Hadoop for real-time analytics? 10.9% 4.7% Not 15.1% Very My company doesn t have a mainframe 25.5% Somewhat 43.8% Key point If you ve got a mainframe, you want that data in your data lake. Over 97% of respondents with mainframes believe its valuable to access and integrate that data in the data lake for real-time analytics.

12 How important is it for you to access and integrate IBM i data into the Data Lake for analytics? Key point Over 90% of organizations that have IBM i say it is somewhat to very valuable to integrate that data with Hadoop. 39.1% Somewhat valuable 24.0% My company doesn t have IBM i systems 22.9% Very 8.3% Not Important 5.7%

13 Using Big Data to Fund Big Data Leveraging Hadoop and Spark to take the load off traditional systems proves valuable not only to drive bigger insights, but also to fund the new Big Data technology itself. More than 90% of respondents found value in this strategy. How valuable is optimizing traditional/legacy systems to save money so you can remix your IT spend to support Big Data projects? 18.2% Very valuable 40.6% 31.3% Somewhat valuable Slightly valuable 11.3% 2.9% Not

14 5 Big Data Trends for 2018 More Enterprise Data Flows into the Data Lake. Demands for data across the organization will continue to rise as business users enjoy the downstream benefits of early Hadoop deployments. Data silos will continue to be broken as data from traditional systems blends with newer sources like sensors. Data Quality Improves. Organizations need to trust the data they are analyzing in order to trust the resulting insights for decision-making. Expanded use of data lakes will drive an increased emphasis on improving data quality whether that s when the data is entering, or once its within the data lake itself. Data Governance Expands. As regulatory compliance becomes a top concern in the board room, the scope of data governance will expand to include the data that is stored and analyzed in the data lake. Organizations will initiate processes and employ technology that allows them to understand what their data is, where it has been, and who has been touching it. Data Lakes Will be Fresher. Coinciding with the influx of enterprise data and the emphasis on quality, organizations will implement solutions to ensure their data lakes stay up to date even if the source of the data is hard to access. Big Data Will be Stronger than Ever. The names and technologies might change (e.g., Hadoop is giving way to Spark), but initiatives involving the processing of massive data volumes for greater insights are not going away. Cost savings from optimizing expensive traditional platforms like mainframes and EDWs, as well as revenue increases driven by better decision-making, will be funneled back to fund more Big Data projects.

15 Spotlight on Data Quality Special With the importance of Data Quality rising to the top of the list of challenges, we dug deeper into the survey results to identify additional insights in this area. The greater the data diversity, the more important the need for data quality. Over 60% of respondents said that storing enterprise-wide data in the data lake is most critical to supporting their business, with an average of four sources populating the platform. However, survey participants who identified five or more sources were four-times more likely to name data quality as a critical factor for successfully implementing and maintaining a data lake.

16 Financial Services and Insurance are most focused on Data Governance and Data Quality. Given the number of regulations that Financial Services and Insurance companies are subject to and the serious ramifications of non-compliance, it s not surprising that survey respondents in those industries are more likely to have data governance and data quality on their minds. When asked to identify their priorities for implementing and maintaining their data lake, participants from these industries were much more likely to rank Data Quality as important. Nearly 60% said ensuring the data quality of the data inside the data lake is of top importance, compared with 40% of respondents from other industries. Financial Services and Insurance practitioners were also twice as likely to report significant challenges with including the data lake in data governance initiatives (35% vs. 16% in other industries). Not everyone is making the connection between quality and business benefits. We took a closer look at the responses from participants who did not include data quality as a top priority for their data lake, and the results were surprising. This population identified: Advanced/predictive analytics as their top data lake use case (70%) Taking advantage of next-generation analytics as their #1 most expected Big Data business benefit (66%) These results present a paradox: Participants who did not regard data quality as a high-priority concern also expressed the most interest in analytically-intensive data lake uses which are, of course, dependent on proper data quality. The apparent disconnect between desired data lake analytics and the relative importance of data quality among these participants is a cause for concern, as there is increased reliance by executives on the insights generated from Big Data to drive the business. We anticipate that this gap will close significantly in 2018.

17 About Syncsort Syncsort is a trusted enterprise software provider and the global leader in Big Iron to Big Data solutions. More than 6,000 organizations, including 84 of the Fortune 100, use the company s products to solve their most complex data management challenges, on premise and in the cloud. Syncsort helps customers optimize traditional data systems and deliver mission-critical data from these systems to next-generation analytic environments. Its Big Iron to Big Data portfolio now features the #1 high availability product for IBM i Power Systems, powerful cross-platform capacity management, best-inclass mainframe app and machine data access & integration, and market-leading data quality capabilities. Rediscover Syncsort at Syncsort Incorporated. All rights reserved. All other company and product names used herein may be the trademarks of their respective companies.