Building event-driven Microservices with Kafka Ecosystem

Size: px

Start display at page:

Download "Building event-driven Microservices with Kafka Ecosystem"

Piers Ray
5 years ago
Views:

wordpress.com BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M.

1 Building event-driven Microservices with Kafka Ecosystem Guido Schmutz London, guidoschmutz.wordpress.com BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH

2 Guido Schmutz Working at Trivadis for more than 21 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: Slideshare: Twitter: gschmutz

3 Agenda 1. Where do we come from? 2. What are Microservices? 3. Why not Event Driven? 4. What about streaming sources? 5. What about integrating legacy applications? 6. What about (historical) data analytics? 7. Why Kafka for Event-Driven Microservices? 8. Summary

4 Where do we come from?

5 Traditional Approach Customer Fat Client App Data Storage Customer UI GUI Customer BO Data Shop Backend Application Shop Rich UI Search Facade Order DAO Shared Database Shop UI GUI UI Logic Customer DAO sync request/response Order Facade Business Product DAO Data

6 SOA Approach Contract-first Web Services Shop Web App Business Activity Service SOAP Business Entity Service SOAP Data Storage Technical layers offer their own interfaces Shop UI GUI UI Logic Custer BAS Customer BES SOAP Customer Database Reuse on each level Shop UI App SOAP Search BAS Payment BES Order DAO SOAP Lower layer often wraps legacy code Shop UI GUI UI Logic SOAP Order BAS Product Customer BES DAO SOAP Product DAO Order BES Order and Product DB

7 Virtualized SOA Approach Service Virtualization Layer Shop Web App Business Activity Service Business Entity Service Data Storage Shop UI GUI UI Logic SOAP SOAP Service Bus Custer BAS Customer BES SOAP Customer Database Shop UI App SOAP Payment BES Order DAO SOAP Shop UI GUI UI Logic Search BAS SOAP Product Customer BES DAO SOAP Order and Product DB Order BAS Order BES

8 What are Microservices?

9 What are Microservices? Tightly Scoped behind clear interfaces Responsible for managing their own data (not necessarily the infrastructure) Should be highly decoupled Independently deployable, self-contained and autonomous SOA done right?!

10 Microservice Approach Shop Web App Tightly Scoped REST Responsible for managing their own data Customer Highly Customer decoupled API Customer Logic Independently deployable, self-contained Order and autonomous Microservice REST Shop UI GUI UI Logic Order API Order Logic Order Product Microservice REST Product API Product Logic Product Stock Microservice REST Stock API Stock Logic Stock

Shop UI GUI UI Logic Order API Order Logic Order Product Microservice REST

11 Microservice Approach with API Gateway Customer Microservice REST Customer API Customer Logic Customer Shop Web App API Gateway Order Microservice REST Shop UI GUI UI Logic Order API Order Logic Order Product Microservice REST Product API Product Logic Product Stock Microservice REST Stock API Stock Logic Stock

12 Synchronous World of Request-Response leads to tight, point-to-point couplings Service 1 Service 2 Service 3 Service 4 API Logic API Logic API Logic API Logic State State State State problem in lower end of chain have a ripple effect on other service crash of service overloaded service / slow response time change of interface Service 5 API Logic Service 6 API Logic Service 7 API Logic State State State

13 Why not Event-Driven?

14 3 mechanisms through which services interact Request-Driven (Imperative) Event Driven (Functional) Command Order IPad boolean order(ipad) Service Logic Event Consume OrderEvent{iPad} Query Retrieve my Orders List<Orders> getallorders(for) State Event Publish OrderValidatedEvent{iPad}

Event-Driven (Async) Microservice Approach Customer Microservice REST Customer API Customer Logic Customer Shop Web App API Gateway Order Microservice Shop UI GUI UI Logic REST Order API Order

15 Event-Driven (Async) Microservice Approach Customer Microservice REST Customer API Customer Logic Customer Shop Web App API Gateway Order Microservice Shop UI GUI UI Logic REST Order API Order Logic Order Event Store Product Microservice REST Product API Product Logic Product sync request/response async, event pub/sub async request/response Stock Microservice REST Stock API Stock Logic Stock

16 What about streaming sources?

How to work with streaming data sources Customer Microservice REST Customer API Customer Logic Customer Shop Web App Order Microservice Shop UI GUI UI Logic REST Order API Order Logic Order Event

17 How to work with streaming data sources Customer Microservice REST Customer API Customer Logic Customer Shop Web App Order Microservice Shop UI GUI UI Logic REST Order API Order Logic Order Event Store Product Microservice Location Mobile Apps REST Product API Product Logic Product Social Weather Data Event Stream Stock Microservice Click stream Sensor Data REST Stock API Stock Logic Stock

18 Streaming Analytics Architecture BI Tools SQL Location Social Click stream Mobile Apps Weather Data Sensor Data Event Stream Event Store Event Stream Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Stream Analytics Reference / Models Results Dashboard Search Service Search / Explore Online & Mobile Apps Microservice Cluster Service Event Stream Microservice State API

19 What about integrating legacy applications?

Integrate existing systems through CDC Customer Fat Client App Data Store Integration Stream Processing Customer UI Customer BO CDC Customer CDC Connector Stream Processor Results Capture changes

20 Integrate existing systems through CDC Customer Fat Client App Data Store Integration Stream Processing Customer UI Customer BO CDC Customer CDC Connector Stream Processor Results Capture changes directly on database Change Data Capture (CDC) => think like a global database trigger Transform existing systems to event producer Event Store Reference / Models Microservice Logic Dashboard State

Results Search Service Search / Explore Online & Mobile Apps Location Mobile Apps Reference / Models Dashboard

21 Integrate existing systems through CDC BI Tools Billing & Ordering SQL CRM / Profile Marketing Campaigns Change Data Capture Event Store Event Stream Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Stream Processor Results Search Service Search / Explore Online & Mobile Apps Location Mobile Apps Reference / Models Dashboard Social Weather Data Microservice Cluster Service Click stream Sensor Data Event Stream Event Stream Microservice State API

22 What about (historical) data analytics?

File Import / SQL Import Event Store Data Flow Event Stream Raw Refined Hadoop Clusterd Hadoop

Stream Processor State Microservice Cluster API SQL Export SQL Search Search Service Service

23 Streaming & (Big) Data Analytics Architecture Billing & Ordering CRM / Profile Marketing Campaigns Location Social Click stream Mobile Apps Weather Data Sensor Data Change Data Capture Event Stream File Import / SQL Import Event Store Data Flow Event Stream Raw Refined Hadoop Clusterd Hadoop Cluster Big Data Cluster Storage Storage Results Parallel Processing Stream Processing Cluster Stream Processor State Microservice Cluster API SQL Export SQL Search Search Service Service Enterprise Data Warehouse BI Tools Search / Explore Online & Mobile Apps Event Stream Microservice State API

24 Why Kafka for Event-Driven Microservices?

25 Apache Kafka A Streaming Platform High-Level Architecture Scale-Out Architecture Distributed Log at the Core Logs do not (necessarily) forget

26 Apache Kafka Schema Registry Producer Kafka avro Kafka Broker Kafka avro Consumer avro schema Schema Registry avro schema

27 Apache Kafka scalable message processing and more! Stream Processing Source Connector Kafka RESTful trucking_ driver Kafka Broker Kafka RESTful Sink Connector Schema Registry

28 Change Data Capture (CDC) Legacy Microservice RDBMS cdc-source trucking_ driver Customer Topic elasticsearc h-sink NoSQL

29 Demo Customer Microservice REST Customer API Customer Logic Customer Kafka Order Microservice REST Order API Order Logic Order Customer

30 Hold Data for Long-Term Data Retention 1. Never Broker 1 2. Time based (TTL) log.retention.{ms minutes hours} 3. Size based log.retention.bytes 4. Log compaction based (entries with same key are removed): kafka-topics.sh --zookeeper zk:2181 \ --create --topic customers \ --replication-factor 1 \ --partitions 1 \ --config cleanup.policy=compact Producer 1 Broker 2 Broker 3

31 Topic Viewed as Event Stream or State Stream (Change Log) Event Stream ,Peter,Muster,3010,Berne ,Paul,Steffen,8001,Zurich ,Lisa,Meier,3043,Ittigen ,Peter,Muster,3015,Berne ,Lisa,Steffen,8001,Zurich ,Peter,Muster,3052,Zollikofen State Stream (Change Log Stream) ,Peter,Muster,3010,Berne ,Paul,Steffen,8001,Zurich ,Lisa,Meier,3043,Ittigen ,Peter,Muster,3015,Berne ,Lisa,Steffen,8001,Zurich ,Peter,Muster,3052,Zollikofen

32 Keep Topics in Compacted Form K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Offset Key Value Compaction Offset Key Value K1 K3 K4 K5 K2 K6 V4 V5 V7 V9 V10 V11 K1 K2 K3 K4 K5 K6 V4 V10 V5 V7 V9 V11 V3 V6 V8 V1 V2

33 Keep Topics both in Original and Compacted Form OR Producer TX Kafka Broker A B C B A E K A E A C K Producer Kafka Broker A B C B A E K A E A C K B E A C K B E A C K Consume and Produce

34 Enrich Stream with Static Data with Kafka Streams Order Topic Consume and Produce Join Order & Customer Topic Customer Topic Custome r Handler Customer Table Customer Local State

35 Summary

36 Summary Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture File Import / SQL Import Data Flow Hadoop Clusterd Hadoop Cluster Big Data Raw Refined Storage Storage Results Parallel Processing SQL Export SQL Enterprise Data Warehouse BI Tools Location Social Click stream Mobile Apps Weather Data Sensor Data Data Flow Event Hub Event Stream Stream Processing Stream Processor State Microservices API Search Search Service Service Search / Explore Online & Mobile Apps Event Stream Microservice State API

37 Summary service autonomy is key in a Microservices Architecture! not all communication need to be synchronous => separate into commands events queries Kafka is well suited as an event broker / event store brings many more interesting features beyond just message passing

38 References Microservices Blog Series, Ben Stopford, Confluent: Apache Kafka for Microservices: A Confluent Online Talk Series: Turning the database inside-out with Apache Samza, Martin Kleppmann, Con Event sourcing, CQRS, stream processing and Apache Kafka: What s the connection?, Neha Narkhede, Confluent: Immutability Changes Everything, Pat Helland, Salesforce: Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:

39 Technology on its own won't help you. You need to know how to use it properly.

Introduction to Stream Processing

Introduction to Stream Processing Introduction to Processing Guido Schmutz DOAG Big Data 2018 20.9.2018 @gschmutz BASEL BERN BRUGG DÜSSELDORF HAMBURG KOPENHAGEN LAUSANNE guidoschmutz.wordpress.com FRANKFURT A.M. FREIBURG I.BR. GENF MÜNCHEN