Towards The Real-Time Enterprise

Transcription

1 EXECUTIVE BRIEFING PAPER Towards The Real-Time Enterprise How A Cloud-Native Application Architecture Provides An Essential Foundation For Digital Transformation January 2019

2 Table Of Contents About This Document This paper provides an overview of the trend towards the adoption of cloud-native application architectures by industry leaders as they embark on various digital transformation journeys. It is intended for technology executives at all levels of Global 2000 enterprises and government agencies. Deep understanding of modern cloud and application technologies is not required, although a general awareness of cloud computing, enterprise application architectures and big data is assumed. 2

3 Digital Transformation: it s Real and it s Happening The term digital transformation can mean different things to different people but, however you define it, there s little doubt that large enterprises embracing digital strategies are having a profound impact on the way that we do business, and the way that we as consumers live our lives. For the purposes of this paper, we are going to look at digital transformation through the lens of Global 2000 enterprises that are grappling with the need to run their businesses smarter, taking advantage of more data and needing to make decisions in real-time (or as close to real-time as is realistic). And that means transforming their businesses to operate more like technology companies, taking advantage of all available digital assets in their ecosystem. From a technological standpoint, we can break down digital transformation elements into a number of mega-trends that we need to embrace in our business systems. $ Mobile Business Big Data Cloud Computing Internet of Things (IoT) Artificial Intelligence Figure 1: Mega-trends influencing digital transformation Mobile business Providing responsive mobile experiences featuring real-time personalization can lead to increased share of wallet and improved customer loyalty. Big data The ability to derive insights from disparate data sources in near-real time provides a terrific opportunity to improve customer experience (CX) in many industries. The move to the cloud Estimates vary, but according to a LogicMonitor survey 1, the leading factor driving greater public cloud engagement is enterprises engaging in digital transformation (63%). 1 Source: LogicMonitor, Cloud Vision 2020: The Future of the Cloud Study, January

4 Internet of Things (IoT) 5G is expected to be a huge driver for innovation with IoT, from smart cities and factories, to self-driving cars and packages that track themselves. The possibilities are pretty much endless. AI and Machine Learning The opportunity to build systems that can help make your business run better without human intervention is a compelling vision that few will be able to ignore. Of course, enterprises are not undertaking digital transformation journeys simply because their C-Suite executives see these trends happening in the industry. The business benefits can be truly significant. Later in this paper we will explore examples of digital transformation where enterprises are realizing value in areas such as: Deepening engagement with customers Reducing risk Increasing market share Improving developer productivity Improving time to market Reducing infrastructure costs Most importantly, we also need to recognize that if we don t reshape our businesses to take advantage of these trends and become a real-time enterprise, we run the risk of getting left behind since our competitors, existing and/or new, most certainly will. Foundations for a Real-Time Enterprise So how do we go about building and/or modernizing our business systems to be able to take advantage of these mega-trends and move towards becoming a real-time enterprise? These new or modernized systems have a number of requirements. They must be: Data-centric: able to process large amounts of data in real-time High performance Able to scale up and down elastically Fully resilient and self-healing Able to make efficient use of all the compute resources available To accomplish all of these simultaneously, we need cloud-style infrastructure paired with a cloud-native application architecture. 4

5 Cloud-style infrastructure provides a great starting point for this architecture. Cloud-style in this context does not necessarily mean using the public cloud: containerization technologies such as Docker and orchestration technologies such as Kubernetes enable many of the capabilities required to deliver on these attributes in an on-premise / hybrid cloud / private cloud environment. But having a cloud-style infrastructure alone is not sufficient: the applications that run in this infrastructure must be built using a modern, cloud-native, data-centric architecture. When considering that, as Anne Thomas from Gartner points out: traditional application architectures and platforms are obsolete 2. The software infrastructure required to support these modern systems is very different from the legacy Java EE and.net based architectures that most enterprises have run their businesses on for the past 15 years. At Lightbend, based on our experience with many organizations transforming themselves, we firmly believe that the foundation for a real-time enterprise is by necessity cloud-native. That means investing both in cloud style infrastructure and in building cloud-native applications using a modern, data-centric application architecture featuring Reactive microservices and supporting big data at speed. Figure 2: Cloud-native architectural foundation for a real-time enterprise Let s look at this in some more detail. 2 Source: Gartner, Inc. Market Guide for Application Platforms, November market-guide-application-platforms 5

6 Cloud-Style Infrastructure According to Forrester: As firms prepare for 2019, we predict that the biggest cloud providers will get bigger, containers will reshape every cloud platform, and the combination of lowcost infrastructure plus high-value development services will firmly establish cloud as the new enterprise digital application platform... 3 It s not hard to see why this is the case. Cloud style infrastructure allows you to have a very flexible, very dynamic, very elastic, very resilient, very responsive infrastructure, allowing optimum use of resources. The economics of the cloud are compelling in terms of infrastructure costs, management costs, operational costs, and, in many cases, data protection and security considerations. In a hybrid cloud or private cloud scenario, containerizing applications and then automating the deployment, scaling, and management of these applications using container orchestration software such as Kubernetes can provide an enterprise with many of the benefits typically associated with the public cloud, but with additional controls and flexibility. But what about the applications themselves? If all you are doing is lifting and shifting your legacy applications into your public, private or hybrid cloud, the potential gains are going to be seriously limited. The cloud by its very nature delivers a distributed computing environment. Traditional monolithic, 3-tier architectures were never designed to run as distributed systems. To genuinely take full advantage of cloud style infrastructure, we need the applications to be built using a true cloud-native application architecture. Reactive Microservices The trend towards using a microservices-based architecture for systems design is significant and it is happening for good business reasons. By having small, discrete components that can be developed, deployed and scaled independently, microservices-based applications are able to take advantage of cloud economics. Developers can build and deploy services much more rapidly. Lightbend frameworks Akka, Play and Lagom as well as Spring Boot, Node.js and Go can be and are used by many for this purpose. 3 Source: Forrester, Inc., Cloud Computing Comes Of Age As The Foundation For Enterprise Digital Transformation com/blogs/predictions-2019-cloud-computing/ 6

7 But as we consider building true cloud-native applications, the requirement is for systems of microservices that work together delivering high-performance, scalability and resilience. This is why this class of applications should be built using Reactive microservices. So what does Reactive mean in the context of systems of microservices? The definition of Reactive was codified by Lightbend co-founder and CTO, Jonas Bonér, in the Reactive Manifesto. The manifesto, which now has more than 24,000 signatories, has the goal of defining a common vocabulary, both in terms of business values and technical concepts, to make it easier for developers, users, businesses, and vendors to discuss, collaborate, and innovate around this new class of applications. responsive elastic resilient message-driven Figure 3: Principles of reactive systems To summarize, reactive systems are: Responsive high-performance, providing instant feedback based on user and API interactions in all circumstances. So that means that Reactive systems need to be responsive under load at peak periods, which is why they need to be highly Elastic able to scale up and down predictably and elastically on-demand, across cores, nodes, and clusters. Reactive systems also need to be responsive in cases of failure, which is much more common in distributed systems, meaning that they need to be highly Resilient self-healing: recovering and repairing themselves automatically for seamless business continuity. And the fundamental underlying architectural tenet of being able to support elasticity while maintaining high resilience is that the systems should be Message-Driven processing messages in parallel, asynchronously, without blocking, ensuring loose coupling, isolation and location transparency. I.e. services can run anywhere and can communicate with other services without having to know where they are physically - an essential for cloud-native design. 7

8 Reactive systems are designed to be able to take full advantage of cloud-style infrastructure, enabling you to handle very, very large amounts of data at high speed in real-time or near real-time in an elastic, resilient manner while using an optimal amount of compute resources. Big Data at Speed While many enterprises are seeing genuine benefits from their investment in big data in terms of being able to make smarter decisions about their business, the battle now is about how to make those decisions in real-time 4 (or near real-time). Forrester agrees: The next frontier of competitive advantage is the speed with which you can extract value from data. Mike Gualtieri, VP, Principal Analyst Forrester Research, January 2018 The key is to be able to harness big data and process it at high speeds, often as the data is being generated or entering your ecosystem - not necessarily when it has been dropped into a data lake. Lightbend s Dr. Dean Wampler provides context for this in the introduction to the O Reilly report, Fast Data Architectures for Streaming Applications: 5 Until recently, big data systems have been batch oriented, where data is captured in distributed file systems or databases and then processed in batches or studied interactively, as in data warehousing scenarios. Now, exclusive reliance on batch-mode processing, where data arrives without immediate extraction of valuable information, is a competitive disadvantage. Hence, big data systems are evolving to be more stream oriented, where data is processed as it arrives, leading to so-called fast data systems that ingest and process continuous, potentially infinite data streams. Streaming applications often run forever and process a never-ending amount of data therefore they require even higher levels of resilience, elasticity and availability than traditional big data applications. These demands require fast data architectures to become more like Reactive microservices architectures. Conversely, as we have explored, applications built using microservices have an increasing requirement to process large amounts of data. Hence, there is a unification happening between big data and microservice architectures: 4 Real-time in this context typically means sub-second response 5 Source: Fast Data Architectures for Streaming Applications: Getting Answers Now from Data Sets that Never End Dr. Dean Wampler, O Reilly,

9 Reactive Microservices Fast Data Applications Big Data Increasing requirements for scalability & resilience Streaming data pipelines are being served by Microservices Increasing requirements for streaming vs. batch While there is undoubtedly a lot of complexity in designing, building and running fast data systems, forward-looking enterprises are embracing the challenge - there simply is no alternative on the path to becoming a real-time enterprise. Completing the Picture Many of the challenges related to running these complex cloud-native systems can be obviated to a large extent by choosing appropriate platforms that deliver management, monitoring, and security. This can make a massive difference for DevOps and Operations teams compared to starting with the raw open source upstream distributions of the infrastructure software that form the basis for the majority cloud-native operation. From the cloud style infrastructure side, enterprises are adopting management and automation platforms such as Red Hat OpenShift Container Platform, Mesosphere DC/OS and Pivotal Container Service as a means to support hybrid and multi-cloud infrastructure management. From the cloud-native application architecture standpoint, application platforms should deliver management and monitoring tools that provide visibility into the various components and the distributed applications themselves - this becomes vital to the successful deployment and ongoing health of a running system. Building a system of Reactive microservices featuring parallel, asynchronous, non-blocking, loosely coupled, and location transparent microservices is difficult enough. Add in the requirement to deal with real-time data streaming and you have a whole new world for developers. Cloud-native application platforms such as the Lightbend Platform provide advanced tooling for developers to significantly ameliorate the challenges of building such systems and dramatically improve developer productivity. 9

10 Business Benefits Until now, we ve discussed supporting digital transformation with modern applications or systems in pretty abstract terms. There are many, many use cases where the cloud-native architecture that we have described is essential. A few of the most common use cases that we have seen with our customers at Lightbend include: Real-time Analytics IoT IoT E-Commerce Real-time Financial Processes Application Modernization Of course, none of these digital transformation use cases get funded without anticipated business benefits to back up the investment. Having worked with over one hundred Global 2000 enterprises undergoing digital transformation, Lightbend has a pretty good handle on the business cases that support these projects being green lighted. We can summarize the most popular in six categories: Deepen Customer Engagement Example: HPE is transforming their customer experience with near real-time insights from their customers data centers where they have over 20 billion sensors deployed sending trillions of metrics each day to InfoSight, providing analytics on petabytes of telemetry data. Analyzing this vast amount of data in near real-time provides HPE the ability to monitor infrastructure, predict possible problems and recommend ways to enhance performance. 10

11 Increase Market Share Example: The world s leading coffee retailer with more than 24,000 retail stores in 70 countries implemented a new universal commerce platform to support the use of mobile rewards, mobile ordering, mobile payments and personalization across all stores, irrespective of whether they were owned by the retailer or by their partners. The expected result is a 200% increase in revenue. (XXX over what period??? XXX) Reduce Risk Example: Capital One is one of the major retail banks in the US but they also are the number one provider of auto loans outside of the car manufacturers own finance arms. Their previous loan process took on average about 55 hours, in large part to ensure that they managed the risk associated with the loan. After re-architecting, they are now able to apply 12 machine learning models concurrently to the data as it s streaming through the platform, completing the process in under one second. This allows them not just to be able to provide a better customer experience and being able to provide the more tailored loans to meet their customer requirements, but also to dramatically reduce the risk associated with the loan portfolio. Accelerate Time to Value Example: A major provider of financial management solutions that serves approximately 50 million customers worldwide wanted to take advantage of the huge amount of data available across its different platforms to differentiate their offering and improve customer experience. At the same time, they had a goal of moving to the cloud to reduce infrastructure costs. By redesigning the backend of one of their major solutions using Reactive microservices, they were able to implement the whole new system and deploy it into the cloud in less than six months. Decrease Compute Costs Example: PayPal re-architected one of their core systems according to Reactive principles such that the system was able to be completely asynchronous and was able to maximize the use of all the resources of the the underlying servers at its peak. The legacy system required 1,200 virtual machines to process a billion events per day, but, after re-architecting, they were able to reduce this down to just eight virtual machines. [Not all use cases will behave as beautifully as this! But what we can say is that a Reactive architecture will always give you the best possible use of the underlying physical resources in your infrastructure.] 11

12 Boost Developer Productivity Example: Verizon is one of the world s largest telecommunication providers and their e-commerce site, verizonwireless.com, is one of the most popular sites whenever major phones, such as the Apple iphone X, are launched. Verizon wanted to significantly enhance the capabilities of their e-commerce system, but, because of the legacy system architecture, it took the Verizon team on average two months to roll out any new features on the system. At the same time, they were aware that popular launches bring a significant risk to their system. And so they re-architected a major component of their e-commerce flows into a reactive architecture, realizing a 40% reduction in their development costs. Getting There It is not a coincidence that the enterprises realizing these benefits of digital transformation are adopting the cloud-native enterprise architecture outlined earlier. They have a number of things in common: They were able to design for a cloud-style infrastructure Their applications follow the design principles outlined in the Reactive Manifesto Scala or Java are the core languages used for development They use Lightbend open source frameworks to support elasticity, scalability, agility, resilience and high-performance They make extensive use of other relevant open source frameworks such as Apache Spark and Apache Kafka for the Fast Data components of their systems Designing, building and running true cloud-native systems is a non-trivial exercise. Lightbend Platform was created to make this a much easier proposition for Global 2000 enterprises. Lightbend Platform Lightbend Platform has been designed to deliver the application architecture for cloud-native applications together with the advanced tooling, management and monitoring required for developers and operators to design, build and run these complex systems successfully in production, at scale. 12

13 Lightbend Platform includes: Distributed Systems Architecture Event-Driven Reactive Microservices Real-TIme Streaming & Machine Learning End-To-End Monitoring & Management Security & Risk Mitigation Distributed Systems Architecture Cloud-native infrastructure like Kubernetes requires a cloud-native application architecture to be truly performant. Lightbend Platform enables you to deliver responsive, globally consistent systems at scale, backed by self-healing protocols that automatically protect your business systems from code to cluster. Event-Driven Reactive Microservices One service by itself is nothing Reactive Microservices come in systems. Lightbend Platform enables teams to deliver scalable, self-healing systems of microservices, web and application services with the right persistence strategies in place for handling unexpected issues gracefully. Real-time Streaming & Machine Learning Remaining relevant as a business means extracting value from data in real-time. Lightbend Platform delivers the ability to handle never-ending data streams at scale, so you can offer new user experiences and use Machine Learning for continuous improvements all in real time. End-to-End Monitoring & Management Finding bugs once in production is the stuff of nightmares. Lightbend Platform provides system-wide observability of your Akka, Lagom, Play, and Scala applications from Day 1, so you can verify regularly in development and operate with confidence. Security & Risk Mitigation Ensuring the security and global compliance of your distributed systems can be difficult. Lightbend Platform helps you protect your code and continuously comply with legal requirements such as GDPR and includes integration with source code analysis for application security. Cloud-native applications built with the Lightbend Platform offer high performance, true elastic scalability and ultra-high resiliency with unparalleled responsiveness, dramatic improvements in resource utilization and the ability to be able to process big data in real-time. 13

14 Summary To compete in a fast-paced digital economy, businesses need to become real-time enterprises. This journey requires investment in digital transformation and in developing, deploying, and scaling new cloud-native applications quickly and cost-effectively. The software architectures that Global 2000 enterprises have been running their businesses on for the past fifteen years were not designed for the cloud and will not support this next generation of applications. This sea change allows not just data, but information and context to flow through an organization, be served up on-demand and allow better business decisions to be taken automatically. Building and running these applications requires a full cloud-native architecture: a flexible cloud-style infrastructure a modern reactive microservices-based application architecture the ability to process large quantities of data in real-time The appropriate application architecture for a real-time enterprise should be: Data-centric: able to process large amounts of data in real-time High performance Able to scale up and down elastically Fully resilient and self-healing Able to make efficient use of all the compute resources available Lightbend Platform delivers all of these and was designed to be able to run on any public, hybrid, or private cloud infrastructure. Lightbend-based applications are truly cloud-native in design and Lightbend s open source technologies are used by millions of developers, many of whom are building cloud-native applications. Most enterprises deploying Lightbend-based systems follow this cloud-native pattern. These systems are large, business-critical cloud-native applications with deployment requirements perfect for cloud style infrastructure, delivering real business value such as: Deepening engagement with customers Increasing market share Improving time to market Reducing risk Improving developer productivity Reducing infrastructure costs We look forward to joining you and your team on the road towards your real-time enterprise. 14

15 Lightbend provides the leading Reactive application development platform for building distributed systems. Based on a message-driven runtime, these distributed systems, which include microservices and fast data applications, can scale effortlessly on multi-core and cloud computing architectures. Many of the most admired brands around the globe are transforming their businesses with our platform, engaging billions of users every day through software that is changing the world. Lightbend, Inc. 625 Market Street, 10th Floor, San Francisco, CA