Engaging in Big Data Transformation in the GCC

Transcription

1 Sponsored by: IBM Author: Megha Kumar December 2015 Engaging in Big Data Transformation in the GCC IDC Opinion In a rapidly evolving IT ecosystem, "transformation" and in some cases "disruption" is changing the way industries utilize technologies and interact with customers. Whether they are enterprises or mid-sized market players, organizations are being inundated by an ecosystem that is cluttered with devices, applications, and processes, and is becoming increasingly competitive and overwhelmed with data. As much of a cliché as it may be to say "data" is the new currency, it is important to note that it is the value and insights that can be derived from this information that is critical. It is this "insight" that big data analytics intends to harvest in order to aid decision making and improve organizational agility. Enterprises within the Gulf Cooperation Council (GCC) countries of the UAE, Qatar, Kuwait, Bahrain, and Oman acknowledge the need for improved decision making. However, barring select early adopters within the region's telecommunications, financial, energy, and retail sectors, many enterprises are of the opinion that their data volumes are not high enough to pursue big data projects. Another view focuses on the costs associated with these engagements, with many enterprises believing their current analytical solutions should continue to suffice to meet their business needs. Mid-market organizations, on the other hand, tend to have the notion that big data analytics solutions can only be utilized by large enterprises since they have more data and more importantly the budgets to deploy big data analytics solutions and infrastructures. IDC Page 1

2 A lot of these perceptions around the feasibility of pursuing big data projects by large and midmarket organizations have to do with the way big data technologies tend to be defined, a general lack of understanding of relevant use cases for big data, and an inherent lack of awareness of cost-effective alternatives for big data analytics infrastructure. IDC believes that both large and mid-market organizations can utilize big data analytics to gain quality insights, enhance decision making, and improve service/product innovation. By leveraging high-value insights, sectors will encounter disruption as the rate of competition and innovation increases. Enterprise incumbents may even face increased competition from their mid-market counterparts as they use information to innovate and transform themselves. Defining Big Data and Analytics IDC defines big data and analytics technologies as being a "new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data by enabling high-velocity capture, discovery, and/or analysis." It essentially encompasses the attributes of the four Vs volume, velocity, variety, and value. To be classed as "big data", the volume of data being collected must be over a certain terabyte (TB) (IDC defines this at over 100 TB) or growing at a rapid rate of over 60% annually. Another key attribute is that of velocity, which means the data being captured must have a stream or change that is at or above certain gigabytes-per-second level (IDC currently states this at or above 60GBps). This takes into account instances where the data is flowing in from real-time or near real-time sources such as weather, financial transactions, and buyer behavior patterns. Big data analytics projects can leverage data from multiple sources and formats to gain better insights, which means that variety is a crucial attribute. The final V value is the critical aspect for any big data deployment, since the underlying necessity or driver for big data is that of being able to drive business value. This need for business value will help define the metrics, KPI, data sources, formats, and reporting for any big data project. It is important to note that big data analytics goes beyond traditional analytics of analyzing structured data; indeed, it is able to integrate and incorporate multiple data sources, be they structured, semi-structured, or completely unstructured data sets. Since big data projects involve multiple data sources, a critical element that needs to be addressed is data quality. The integrity and veracity of the data is crucial for the overall trustworthiness of the "analytics" process. Poor quality/incorrect data can have severe cost and time implications, making veracity a critical factor for big data engagements. Organizations need to be able to trust the IDC Page 2

3 data and the analysis they are undertaking, and veracity will therefore impact the overall "business value" of the analysis. FIGURE 1 IDC Definition of Big Data Analytics Source: IDC, 2015 Use Cases for Big Data Analytics There are several relevant use cases for big data analytics that are applicable across various sectors:» Improving business process optimization: Organizations can leverage data from across the company and from multiple sources and then engage in real-time optimization of business processes. This is particularly relevant within discrete and process manufacturing, insurance, and logistics.» Improving the customer experience and enhancing product/service innovation: Companies within industries such as banking, retail, healthcare, transportation, and telecommunications can use multi-channel data to innovate new products and services that can be targeted at individual customers or use the data to push customized solutions and offerings. IDC Page 3

4 » Predicting profitability: Companies are able to utilize big data analytics to predict costs and output and forecast profitability. This is critical for firms within the agriculture, retail, and manufacturing sectors.» Improving the citizen experience: In a region such as GCC, there is a major focus around Smart Cities, and governments (be they local or federal) can utilize data from a wide range of sources from traffic to weather to smart meters and other sensors to improve citizen services, safety, and experiences. It also opens up opportunities from public-private partnerships and innovation.» Gaining higher wallet share: With increased competitiveness among the GCC's banks, it is imperative that they understand the needs of their clients. Undertaking a proper analysis of their clients current wallet share will enable banks to better position their solutions and identify cross-sell opportunities. This will also enable them to establish a better relationship with their customers, offer improved business value for their clients, and gain higher wallet share. This use case for banks can also apply to multiple industries such as telecommunications, retail, transportation etc Big data analytics use cases can also cover security and threat management, fraud detection, customer churn analytics, profit-margin enhancement, healthcare analytics, and social media analytics, among others. With big data analytics, companies are able to do things more efficiently (time and costs) or are able to innovate in terms of their products, services, and processes. FIGURE 2 Types of Analytics Source: IDC, 2015 IDC Page 4

5 Using big data analytics can encompass various types of analytics, and organizations need to decide which types of analytics would best suit their business goals. Analytics has moved beyond standard descriptive and diagnostics analytics to predictive and prescriptive analytics, and more recently it has moved to the next level by using cognitive computing. Organizations can use a combination of analytics but they should first decide on what they intend to achieve, determine what types of data they need to analyze, and, most importantly, reach a consensus with the business on key performance indicators (KPIs). Big Data Analytics and Organizations in the GCC Organizations in the GCC (Figure 3) do acknowledge that utilizing big data makes them increasingly agile, enabling them to improve decision making both in terms of time and value. They are also able to better understand their customers. This allows them to innovate and improve products and services. Lastly, organizations in the GCC expressed that utilizing big data analytics would be beneficial in terms of engaging in predictive analysis, whereby they can predict possible outcomes based on the data or situation at hand. However, when it comes to big data analytics in the GCC region, there tends to be a major barrier due to the relative misunderstanding of the term "big". Many organizations both at the enterprise and mid-market level feel they do not have enough terabytes of data to utilize big data analytics technologies. Aside from volume, there also tends to be a perception that the technology backend needed to pursue big data analytics is expensive, and with rationalizing IT budgets in the region it becomes difficult to build a business use case. Organizations of any size can use big data analytics and get around the terabyte and cost conundrum. Both enterprise and mid-market organizations need to realize that they can use multiple data sources to engage in big data analytics and that it does not always have to be about large volumes of data in terabyte terms; big data can also encompass data that is growing at an exponential rate annually. For example, organizations can seek to engage in determining market demand or customer analysis by considering their internal data sources and combining it with data sources from the likes of social media feeds such as Twitter and Facebook or data that is available from governments (demographics, socioeconomic mix, weather, etc.), or at times will have to consider procuring the data that they need from alternative sources. IDC Page 5

6 FIGURE 3 Top Three Big Data Analytics Use Cases for Organizations in the GCC 60% 50% 46% Better decision making Predictive analysis Understand customer behavior Source: IDC CIO Summit 2015, N=120 When it comes to building a business case for advanced analytics such as predictive, prescriptive, or cognitive analytics, "response time" and the definition of business value are both critical. Organizations must decide whether their current analytics tools are able to meet the demands of the "business" and if the time taken for the analysis is fast enough to be agile and responsive to their customer needs. When it comes to the cost associated with deploying big data technologies, organizations in the GCC have access to a plethora of solutions both on-premise and in the cloud where they can avail "big data as a service". The majority of big data analytics solutions that are available use components such as Hadoop and Spark and run on Linux platforms, enabling organizations to not only scale out their compute power but also gain advantages around flexibility and cost. Infrastructure to Support Big Data Analytics Deployments Using big data analytics technologies involves analyzing and gaining insights from massive volumes of structured or unstructured data. This level of agility requires organizations to deploy proper scalable and flexible infrastructures that are powerful, reliable, and, most importantly, cost effective. IDC Page 6

7 To be able to extract insights from data, organizations need to consider two core architectures Hadoop and Spark. Both these architectures are open source, enable improved scalability and flexibility, and are cost effective.» Hadoop is essentially a distributed computing and storage model that provides storage for massive volumes of data and is able to handle multiple tasks since it essentially distributes the processing across clusters of computers or several nodes. This can usually be done on specialized hardware and, more importantly, it is fault tolerant (i.e., if one node fails, the workload or process is redirected to ensure stable compute power). Most importantly, by using Hadoop organizations can pre-process their data so they can store the data in any format and leverage scalability by being able to add additional nodes.» Spark is essentially a distributed computing data processing tool. Unlike Hadoop, it does not come with a distributed storage system and usually leverages this from Hadoop, though other solutions are available. Spark's advantage is that it allows for faster computing power, much faster than even Hadoop when it comes to both batch processing operations and even in-memory analytics. Both Hadoop and Spark can work independently of each other. To understand which is more suitable, an organization needs to take into account two major concepts "data in rest" and "data in motion". Essentially, "data in rest" considers the analysis of data that has been collected from various sources and a subsequent decision or action is taken at a separate time. In the concept of "data in motion", organizations engage in real-time analysis and make a decision or action in real time. Organizations will need to decide what end goal they are trying to achieve with their big data analytics projects and by doing so they will be able to determine if they need real-time insights and in-memory processing or they can alternatively engage in batch processing. Due to the fact that Spark is able to support real-time analysis, it is a far more robust and reliable architecture to utitlize for analysis of data as it "streams" into applications/databases from multiple sources and this is done at a faster pace than Hadoop. This makes Spark more appealing for use cases such as fraud detection for financial institutions, improved patient monitoring for hospitals, targeted advertising for media firms, and IT and network monitoring for telecommunications, government, and transportation organizations. However, Hadoop has been around longer and therefore has a better grade of security and support. IDC Page 7

8 FIGURE 4 Big Data Analytics Process Chain Source:IDC,2015 For these two big data architectures to work effectively, organizations will need to consider the right type of business platform that can support Hadoop and Spark with the computing power that is needed. Given that both Hadoop and Spark are open source, a Linux-based computing platform would make sense for an organization pursuing big data analytics. A Linuxbased solution is not only applicable for very large enterprises but makes absolute sense for mid-market enterprises with tighter IT budgets. In order to undertake a broad set of analysis, both Hadoop and Spark depend on having scalable computing clusters and Linux provides the best value, especially when organizations have huge volumes of structured and unstructured data that they need to manage. It should be noted that many High-Performance Computing environments rely on Linux servers to be able to provide them with stability, scalability, and interoperability. Given that the solutions are open source, it allows for customizations as well enabling organizations to introduce features as and when needed. All these advantages, can be leveraged by mid-market enterprises to undertake big data analytics on commodity hardware, thereby negating any previous perceptions of big data analytics projects being too expensive. By using a Linux-based environment, mid-market organizations can easily undertake basic reporting to visualization to engaging in-memory and cognitive computing. IDC Page 8

9 Essential Guidance This section aims to provide guidance to organizations on some of the aspects that they should consider before engaging in a big data analytics project.» Take stock of your data: Organizations need to understand what type and quality of data is available to them and whether this data is capable of providing the insights that they seek. Alternatively, mid-market organizations should consider other data sources if need be.» Engage with lines of business: Before engaging in a big data analytics project, IT needs to reach an agreement with the business on what the end goal they wish to achieve is and have the business define the right KPIs. Rather than IT taking all the responsibility, they need to engage with lines of business, especially when it comes to data governance. LoBs should define what can or cannot be accessed since this can have a major impact on compliance and data privacy.» Address security: Given the variety and volume of data that will be utilized, organizations should address the need to secure their data and ensure data privacy.» Consider the associated costs: Mid-market organizations acknowledge that they can benefit from using big data, and such projects can be cost effective when open source frameworks and Linux-based platforms are utilized.» Form partnerships to address the skills challenge: Skills in general and those associated with big data analytics and Linux in particular will remain a challenge in the GCC. To counter this reality, organizations should consider teaming up with the right partners to support their implementations and provide the training that would be needed to drive their big data projects. Copyright Notice: External Publication of IDC Information and Data Any IDC information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC vice president or country manager. A draft of the proposed document should accompany any such request. IDC reserves the right to deny approval of external usage for any reason. Copyright 2015 IDC. Reproduction without written permission is completely forbidden. IDC Page 9