Building a Data Lake on AWS

Transcription

1 Partner Network EBOOK: Building a Data Lake on AWS

2 Contents What is a Data Lake? Benefits of a Data Lake on AWS Building a Data Lake On AWS Featured Data Lake Partner Bronze Drum Consulting Case Study:Rosetta Analytics Getting Started

3 What is a Data Lake? Organizations are tasked with managing greater volumes of data, from more sources, and containing more types of data than ever before. In the face of massive, heterogeneous volumes of data, many organizations are finding that to deliver timely business insights, they need a storage and analytics solution that offers more speed and flexibility than legacy systems. A Data Lake is a new and increasingly popular way to store and analyze data that addresses many of these challenges by allowing an organization to store all of their data in one, centralized repository. Since data can be stored in its original form, there is no need to convert it to a predefined schema before ingestion, giving you a way to store all of your data, both structured and unstructured, with minimal lead time. With a Data Lake on AWS, you no longer need to know what questions you want to ask of your data before you store it, giving you a flexible platform for data analysis. At the heart of an AWS-based Data Lake is Amazon Simple Storage Service (S3), which provides secure, cost-effective, durable, and scalable cloud storage. AWS also offers an extensive set of services to help you provide strong security for your Data Lake, including access controls, virtual isolation, 256-bit encryption, logging and monitoring, and more. There s a variety of ways to transfer data to your Data Lake, including through services such as Amazon Kinesis, which enables you to ingest data in real-time, AWS Import/ Export, a service where you can send a portable storage device with your data to AWS, AWS Import/Export Snowball, a secure appliance AWS sends you for ingesting data in batches, AWS Storage Gateway, which enables you to connect on-premises software appliances with your AWS Cloud-base storage, or through AWS Direct Connect, which gives you dedicated network connectivity between your data center and AWS. With a Data Lake on AWS, you no longer need to know what questions you want to ask of your data before you store it, giving you a flexible platform for data analysis. EBOOK: BUILDING A DATA LAKE ON AWS 2

4 Benefits of a Data Lake on AWS There are a variety of benefits to hosting your Data Lake on AWS, including: Cost-Effective Data Storage Amazon S3 provides cost-effective and durable storage, allowing you to store nearly unlimited amounts of data of any type, from any source. Because storing data in Amazon S3 doesn t require upfront transformations, you have the flexibility to apply schemas for data analysis on demand. This enables you to more easily answer new questions as they come up and improve the time-to-value. Easy Data Collection and Ingestion There s a variety of ways to ingest data into your Data Lake, including services such as Amazon Kinesis, which enables you to ingest data in real-time; AWS Snowball, a secure appliance AWS sends you for ingesting data in batches; AWS Storage Gateway, which enables you to connect on-premises software appliances with your AWS Cloud-based storage; or through AWS Direct Connect, which gives you dedicated network connectivity between your data center and AWS. Security and Compliance When hosting your Data Lake on AWS, you gain access to a highly secure cloud infrastructure and a deep suite of security offerings designed to keep your data secure. As an AWS customer, you will benefit from a data center and network architecture built to meet the requirements of the most security-sensitive organizations. AWS also actively manages dozens of compliance programs in its infrastructure, helping organizations to easily meet compliance standards such as PCI DSS, HIPAA, and FedRAMP. Most Complete Platform for Big Data AWS gives you fast access to flexible and low cost IT resources, so you can rapidly scale virtually any big data application including data warehousing, clickstream analytics, fraud detection, recommendation engines, event-driven ETL, serverless computing, and internetofthings processing. With AWS, you don t need to make large, upfront investments in time and money to build and maintain infrastructure. Instead, you can provision exactly the right type and size of resources you need to power big data analytics applications. EBOOK: BUILDING A DATA LAKE ON AWS 3

5 Catalog & Search Access & User Interface Access and search metadata Give your users easy and secure access Glue Athena API Gateway Identity & Access Management Cognito Data Ingestion Processing & Analytics Get your data into S3 quickly and securely Use of predictive and prescriptive analytics to gain better understanding Kinesis Central Storage Machine Learning Secure, cost-effective storage in Amazon S3 Direct Connect QuickSight S3 Snowball EMR Database Migration Service Redshift Protect and Secure Use entitlements to ensure data is secure and users identities are verified Identity & Access Management Security Token Service CloudWatch CloudTrail Key Management Service

6 Bronze Drum Consulting Bronze Drum brings deep experience and high-octane brainpower to your capital markets challenge. AWS Partner focused exclusively on the AWS platform and serving capital markets customers. We re distinguished by our seasoned team distinguished by long experience with building solutions for financial services on AWS. Our most recent new team member, David Secrest is highly skilled in analysis and solving unsolvable puzzles. Formerly a Princeton University Doctoral Fellow in Particle Physics leads our data modeling and application team. We focus on designing the critical combination of capabilities firms need to succeed with Big Data: the DevOps workflow for applications, scale-out clusters for Deep Learning, and the Data Lakes and continuous analysis that powers the worlds most innovate firms. Further, we provide customers with the option to access insights as needed, by developing API Gateway solutions that enable simple, secure access, to critical insights.

7 Capital Markets Deep Learning Powered by Serverless Capital Markets Data Lake This green-field environment was designed from the to be cost effective from day one yet to efficiently store and analyze from terabytes to petabytes to exabytes of traditional capital markets data, as well as alternative data. Critical Success Factors: Security, Elasticity, Cost Efficiency, Performance This Data Lake solution needed to deliver insights by querying very large data sets, yet to be cost efficient as a discovery, research, and modeling solution. The team needed to prove the value of the models early, and then be able to scale them out cost-effectively. Security. Bronze Drum selected Advanced Technologies that minimize exposure to user and configuration error. The less people need to configure and manage solutions, the simpler it can be to manage the attack surface area and configuration of a solution. Simplicity of operation and ease of use. Two data scientists with a background as researchers in artificial intelligence needed to focus on analysis not running and wrangling large scale clusters. Be Ambitious Bronze Drum set the ambitious goal of designing a solution that could operate with very little user intervention, yet at the same time, highly personalized to the the needs of the end users. Combine ease of use, personalization, with the need to scale out to thousands of nodes and you ve just described the kind of challenge Bronze Drum s team loves to tackle. Bronze Drum designed a Continuous Delivery solution that enabled the data scientists to check-in code and then see it seamlessly deployed on immutable containers running Torch 7 and custom code and data sets. In English? They designed a solution that automatically deploys custom code written by the data scientists, then creates a container and then run that container on thousands of nodes in parallel. To achieve this Bronze Drum created events that enable steps in a process to trigger subsequent events, and to enrich the data passed to the next event. In this way, the automation, the personalization, and the scale could be achieved, while the data scientists could focus on their code, training data, and analyzing the results of the training. Bronze Drum selected CodeCommit, Codebuild, CodePipeline, and AWS Batch for this high performance compute solution. By leveraging AWS Spot pricing, the company is able to save on average 40 to 90% of on-demand pricing, enabling the data scientists to analyze more data when prices are best. The data lake powers the data sets analyze by the Deep Learning process, and is queried by both portfolio managers and data scientists. By leveraging S3 storage, Amazon Athena, AWS Lambda, Kinesis and Glue, the solution is able to acquire Bloomberg data, as well as nontraditional data sets and continuously process the data. A traditional data warehouse didn t fit how we worked at all. We needed a data lake as we retain very large data sets that are continuously processed. Bronze Drum s severless data warehouse on AWS enables us to perform ad hoc analytic queries at any scale--often for pennies. Our Data Scientists to focus on training our deep learning engine--not wrangling with data from sources like Bloomberg. Combined with a Continuous Delivery pipeline, our data scientists can check-in code and it is seamlessly deployed, and processes the latest data sets. Both business users and data scientists can query the massive historical data warehouse on demand, performing sophisticated queries on a cluster that scales seamlessly. A further benefit is that storing large historical data sets is cost effective, yet analytic queries of billions of records can be performed when and as needed at just $5.00 per TB of data processed. Many queries cost just pennies!

8 Amazon Web Service Website Big Data on AWS Building a Data Lake on AWS (Video) About AWS For 10 years, Amazon Web Services has been the world s most comprehensive and broadly adopted Cloud platform. AWS offers over 70 fully featured services for compute, storage, databases, analytics, mobile, Internet of Things (IoT) and enterprise applications from 33 Availability Zones (AZs) across 13 geographic regions in the U.S., Australia, Brazil, China, Germany, Ireland, Japan, Korea, and Singapore. AWS services are trusted by more than a million active customers around the world including the fastest growing startups, largest enterprises, and leading government agencies to power their infrastructure, make them more agile, and lower costs. To learn more about AWS, visit aws.amazon.com.

9 Partner Network 2017, Amazon Web Services, Inc. or its affiliates. All rights reserved. (c) Bronze Drum Consulting, Inc. All Rights Reserved.