2016 Big Data. Mejores pra cticas en AWS

Transcription

1 2016 Big Data. Mejores pra cticas en AWS Javier Ros, Solution Architect Jun, , Web Services, Inc. or its Affiliates. All rights reserved.

2 Agenda Big Data challenges Design Patterns on AWS RavenPack. Big Data for Financial Applications Shopping cart

3 Big Data Challenges Variety Velocity Volume

4 Simplify Big Data Processing data ingest / collect store process / analyze consume / visualize answers Time to Answer (Latency) Throughput Cost

5 On-Demand Big Data Analytics Young Huang. Director, Big Data Analytics. We were able to save about 90% over the EC2 ondemand cost

6 Clickstream Analysis Suneel Sajnani. Senior VP of Enterprise Technology Kinesis and Spark to process more than 30TB per day

7 Event-driven Extract, Transform, Load (ETL) Brian Filppu. Director of Business Intelligence Kinesis, Lambda and EMR for 16 million events per day

8 Smart Applications Joe Emison. Founder & Chief Technology Officer Machine Learning democratizes the process of building predictive models. It's easy and fast to use, and has machine-learning best practices encapsulated in the product, which lets us deliver results significantly faster than in the past.

9 June 2, 2016 RavenPack AWS Summit Madrid Mapping the World s Big Data for Financial Applications Jose Luis Cruz Operations Manager jlcruz@ravenpack.com

10 What is RavenPack? Current Use Cases in the Cloud What s Next?

11 RavenPack at a Glance RavenPack delivers big data analytics to financial professionals 80% of big data is unstructured Only 29% of decisions are based on big data. 80% 29% ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel:

12 RavenPack at a Glance Top hedge funds and investment banks use RavenPack for trading and risk management RavenPack processes hundreds of thousands of documents each day We produce machine readable analytics for each document in real time <250ms Archive of +300 million documents, over +20 years ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel:

13 Use Case: Realtime Classification Classic Model Collectors Classifier RT Feed Files RDBMS Snapshots 6 Servers, 19 KVM virtual machines Limited Storage - Expensive to Upgrade Multiple Points of Failure ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel:

14 Use Case: Realtime Classification Cloud Model using AWS CloudFormation to model the Stack Unlimited, Distributed Storage Easy redundancy, failover and backup Collectors Classifiers RT Feed S3 DynamoDB RDS EC2 AWS CloudFormation Snapshots ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel: Kinesis CloudSearch Redshift 14

15 Use Case: History Classification Classic Model Same Limited Set of Servers, Same RDBMS Can affect Realtime System, Backups Full archive, 4-6 Classifiers 6 weeks! Classifiers Classifiers RDBMS Files ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel:

16 Use Case: History Classification Cloud Model using AWS Servers on Demand, Distributed Storage Independent of Realtime System Full archive, 100 Classifiers from 6 weeks to 3 days!... DynamoDB RDS Availability Classifiers Zone Availability Zone EC2 AWS CloudFormation Coordinator ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel: S3 Redshift 16

17 Future: Incorporating Structured Data Structured BIG DATA available: Consensus and estimates Online purchases Bank and credit card transactions Satellite photographical information Can improve current analytics or create new ones EMR EC2 Kinesis Challenges Amount of data available Mapping all those different datasets Solution: Kinesis + RedShift + EMR ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel: S3 Redshift 17

18 Future: Self-Service Data Download a Custom Slice of Analytics Data Provide a Web-API and Web Service Let client specify parameters Data Set and Time Range Entities and Events Filters Leverage RedShift and S3 Compression and Multiple Output Formats EC2 API Gateway AWS Lambda ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel: S3 Redshift 18

19 Future: The RavenPack Cloud Let Clients upload Proprietary Content to the Virtual Private Cloud (VPC) Internal documents / research , Instant Messaging CRM, bug tracking system Client Support Calls transcriptions... Provision Computing and Storage Resources on a Per Project Basis View Private Analytics in Isolation or Alongside Standard RavenPack Analytic DataSets Everything Goes Away when Project Completes ravenpack.com info@ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel: DynamoDB CloudSearch S3 EC2 RDS AWS CloudFormation Redshift 19

20 Thanks for listening! Jose Luis Cruz: ravenpack.com AMERICAS Tel: (646) EMEA-APAC Tel: +44 (0) RavenPack International

21 Shopping Cart

22 Shopping cart

23 Business Metrics Time to buy Time to cancel Number of sales Sales per country % buy

24 Architecture client Cart event Redshift QuickSight Cart event API Server Kinesis S3 EMR S3 mobile client Machine Learning

25 Customer events { } type : productadded, timestamp : , customer : 5, cart : , product : { } type : productremoved, timestamp : , customer : 5, cart : , product : { } type : cartbuy, timestamp : , customer : 5, cart : productlist : [34, 253] { } type : cartdiscard, timestamp : , customer : 5, cart : , productlist : [2353, 1355, 1234]

26 Kinesis Firehose

27 Kinesis Firehose

28 Architecture AWS Data Pipeline Cart event client Redshift QuickSight Cart event API Server Kinesis S3 EMR S3 mobile client Machine Learning

29 AWS Data Pipeline

30 AWS Elastic MapReduce

31 Pig script DATA = LOAD 's3://shoppingcart-summit/streams/$inputdate/*' USING JsonLoader('type:chararray,timestamp:int,customer:int,cart:long,product:chararray, productlist: chararray'); DATA2 = FILTER DATA BY type is not null; CARTS = GROUP DATA2 BY cart; CARTDATA = FOREACH CARTS { LOGIN = FILTER DATA2 BY type == 'login'; ADDED = FILTER DATA2 BY type == 'productadded'; REMOVED = FILTER DATA2 BY type == 'productremoved'; BUY = FILTER DATA2 BY type == 'cartbuy'; GENERATE MAX(DATA2.customer) AS customer, group AS cart, MAX(DATA2.timestamp)-MIN(DATA2.timestamp) AS duration, IsEmpty(BUY) AS buy, COUNT_STAR(ADDED) AS added, COUNT_STAR(REMOVED) AS removed, MAX(DATA2.timestamp)-MAX(ADDED.timestamp) AS thinking, MIN(LOGIN.timestamp) AS timestamp, '\"\"'; }; STORE CARTDATA INTO 's3://shoppingcart-summit/redshift/$inputdate/' USING PigStorage(',');

32 AWS Quicksight

33 Architecture AWS Data Pipeline Cart event client Redshift QuickSight Cart event API Server Kinesis S3 EMR S3 mobile client Machine Learning

34 Machine learning and smart applications Machine learning is the technology that automatically finds patterns in your data and uses them to make predictions for new data points as they become available Your data + machine learning = smart applications

35 Introducing Machine Learning Easy to use, managed machine learning service built for developers Robust, powerful machine learning technology based on s internal systems Create models using your data already stored in the AWS cloud Deploy models to production in seconds

36 Building smart applications with ML Train model Evaluate and optimize Retrieve predictions - Create a Datasource object pointing to the shopping cart processed data - Explore and understand your data - Transform data and train your model

37 Building smart applications with ML Train model Evaluate and optimize Retrieve predictions - Understand model quality - Adjust model interpretation

38 Explore model quality

39 Fine-tune model interpretation

40 Building smart applications with ML Train model Evaluate and optimize Retrieve predictions - Batch predictions - Real-time predictions

41 Real-time predictions for interactive applications Query for predictions with ML real-time API Your application ml = boto3.client('machinelearning') prediction = ml.predict( MLModelId='ml-dZxbrDXAstA', Record={ 'customer': '4634', cart': ' ', }, PredictEndpoint=' )

42 Architecture AWS Data Pipeline Cart event client Redshift QuickSight Cart event API Server Kinesis S3 EMR S3 mobile client Machine Learning

43