FLINK IN ZALANDO S WORLD OF MICROSERVICES JAVIER LOPEZ MIHAIL VIERU

Transcription

1 FLINK IN ZALANDO S WORLD OF MICROSERVICES JAVIER LOPEZ MIHAIL VIERU

2 AGENDA Zalando s Microservices Architecture Saiki - Data Integration and Distribution at Scale Flink in a Microservices World Stream Processing Use Cases: o o Business Process Monitoring Continuous ETL Future Work 2

3 ABOUT US Mihail Vieru Big Data Engineer, Business Intelligence Javier López Big Data Engineer, Business Intelligence 3

4 4

5 One of Europe's largest online fashion retailers 15 countries ~19 million active customers ~3 billion revenue ,500 brands 150,000+ products 11,000+ employees in Europe 5

6 ZALANDO TECHNOLOGY TECHNOLOGISTS Rapidly growing international team 6

7 VINTAGE ARCHITECTURE

8 VINTAGE BUSINESS INTELLIGENCE Dev Business Logic Business Logic Business Logic Business Logic DBA Database Database Database Database Classical ETL process BI Data Warehouse (DWH) 8

9 VINTAGE BUSINESS INTELLIGENCE Exasol DWH Oracle 9

10 RADICAL AGILITY

11 RADICAL AGILITY AUTONOMY MASTERY PURPOSE 11

12 RADICAL AGILITY - AUTONOMY 12 Technologies Operations Teams

13 REST API REST API REST API REST API REST API SUPPORTING AUTONOMY: MICROSERVICES Business Logic Business Logic Business Logic Database Database Database Business Logic Business Logic Database Database 13

14 REST API SUPPORTING AUTONOMY: MICROSERVICES Team A Business Logic public Internet REST API Business Logic Team B Database Database Applications communicate using REST APIs Databases hidden behind the walls of AWS VPC 14

15 REST API SUPPORTING AUTONOMY: MICROSERVICES Team A Business Logic public Internet REST API Business Logic Team B Database Database Classical ETL process is impossible! 15

16 App A REST API App B REST API App C REST API App D REST API SUPPORTING AUTONOMY: MICROSERVICES Business Logic Business Logic Business Logic Business Logic Database Database Database Database Business Intelligence 16

17 SAIKI

18 SAIKI DATA PLATFORM App A App B App C App D BI Data Warehouse SAIKI 18

19 SAIKI DATA INTEGRATION & DISTRIBUTION App A App B App C App D BI Data Warehouse E.g. Forecast DB SAIKI REST API Exporter Stream Processing via Apache Flink Data Lake. AWS S3 19

20 Data sources Extraction SAIKI SUMMARY Data sources Connections Data sources Technologies Data Delivery B E F O R E Δ J D B C A F T E R REST 20

21 FLINK IN A MICROSERVICES WORLD

22 OPPORTUNITIES FOR NEXT GEN BI Cloud Computing - Distributed ETL - Scale Access to Real Time Data - All teams publish data to central event bus Hub for Data Teams - Data Lake provides distributed access and fine grained security - Data can be transformed (aggregated, joined, etc.) before delivering it to data teams Semi-Structured Data General-purpose data processing engines like Flink or Spark let you define own data types and functions. - Fabian Hueske, dataartisans 22

23 THE RIGHT FIT STREAM PROCESSING 23

24 THE RIGHT FIT STREAM PROCESSING ENGINE Candidates: Storm & Samza ruled out because of batch processing requirement 24

25 25

26 26 SPARK VS. FLINK DIFFERENCES Feature Apache Spark Apache Flink Processing mode micro-batching tuple at a time Temporal processing support processing time event time, ingestion time, processing time Latency seconds sub-second Back pressure handling manual configuration implicit, through system architecture State access full state scan for each microbatch value lookup by key Operator library neutral ++ (split, windowbycount..) Support neutral ++ (mailing list, direct contact & support from data Artisans)

27 APACHE FLINK true stream processing framework process events at a consistently high rate with low latency scalable great community and on-site support from Berlin/ Europe university graduates with Flink skills 27

28 FLINK ON AWS - OUR APPLIANCE MASTER ELB EC2 Docker Flink Shadow Master EC2 Docker Flink Master WORKERS ELB EC2 Docker Flink Worker EC2 Docker Flink Worker EC2 Docker Flink Worker 28

29 USE CASES

30 BUSINESS PROCESS MONITORING

31 BUSINESS PROCESS A business process is in its simplest form a chain of correlated events: start event completion event ORDER_CREATE ALL_PARCELS_SHIPPED D Business Events from the whole Zalando platform flow through Saiki => opportunity to process those streams in near real time 31

32 REAL-TIME BUSINESS PROCESS MONITORING Check if business processes in the Zalando platform work Analyze data on the fly: o Order velocities o Delivery velocities o Control SLAs of correlated events, e.g. parcel sent out after order 32

33 ARCHITECTURE BPM Operational Systems App A App B Nakadi Event Bus Kafka2Kafka Unified Log Stream Processing Elasticsearch App C Saiki BPM Cfg Service UI Alert Svc OAUTH PUBLIC INTERNET 33

34 34 HOW WE USE FLINK IN BPM Event Types; 1 Event Type -> 1 Kafka topic Analyze processes with correlated event types (Join & Union) Enrich data based on business rules Sliding Windows (1min to 48hrs) for Platform Snapshots State for alert metadata Generation and processing of Complex Events (CEP lib)

35 STREAMING ETL

36 Extract Transform Load (ETL) Traditional ETL process: Batch processing No real time ETL tools Heavy processing on the storage side 36

37 WHAT CHANGED WITH RADICAL AGILITY? Data comes in a semi-structured format (JSON payload) Data is distributed in separate Kafka topics There would be peak times, meaning that the data flow will increase by several factors Data sources number increased by several factors 37

38 ARCHITECTURE STREAMING ETL Operational Systems App A App B App C Importer Oracle DWH Nakadi Event Bus Kafka2Kafka ` Unified Log Exporter Stream Processing Saiki Streaming ETL 38

39 HOW WE (WOULD) USE FLINK IN STREAMING ETL Transformation of complex payloads into simple ones for easier consumption in Oracle DWH Combine several topics based on Business Rules (Union, Join) Pre-Aggregate data to improve performance in the generation of reports (Windows, State) Data cleansing Data validation 39

40 FUTURE USE CASES

41 COMPLEX EVENT PROCESSING FOR BPM Cont. example business process: Multiple PARCEL_SHIPPED events per order Generate complex event ALL_PARCELS_SHIPPED, when all PARCEL_SHIPPED events received (CEP lib, State) 41

42 DEPLOYMENTS FROM OTHER BI TEAMS Flink Jobs from other BI Teams Requirements: manage and control deployments isolation of data flows 42 o prevent different jobs from writing to the same sink resource management in Flink o share cluster resources among concurrently running jobs StreamSQL would significantly lower the entry barrier

43 REPLACE KAFKA2KAFKA COMPONENT Python app extracts events from REST API Nakadi Event Bus writes them to our Kafka cluster Idea: Create Nakadi consumer/producer to enable stream processing with Flink to other internal users 43 (first POC done)

44 OTHER FUTURE TOPICS New use cases for Real Time Analytics/ BI o o Sales monitoring Price monitoring Fraud detection for payments (evaluation) Contact customer according to variable event pattern (evaluation) 44

45 CONCLUSION Flink proved to be the right fit for our current stream processing use cases. It enables us to build Zalando s Next Gen BI platform. 45

46 THANK YOU