Integration Reloaded Integration Solutions Based on Reactive Principles. Red Expert Alliance

Transcription

1 Integration Reloaded Integration Solutions Based on Reactive Principles

2 About: Arturo Viveros Arturo Viveros Principal Middleware Architect at Sysco MW Oracle ACE Oracle Dev Champion Blogs:

3 About: Sven Bernhardt Sven Bernhardt Senior Solution Architect at OPITZ CONSULTING Deutschland GmbH Oracle ACE Blogs:

4 Members Seite 4

5 Agenda Status Quo Integration Reactive Patterns Architectural thoughts Use case Summary

6 What is integration about? Simplified: Get Data from System A and send it to System B X X System A System B Message/Protocol Transformation

7 What is integration about? System B System A Message Splitting Routing System C

8 ESB solutions as basic architecture element global internal specific External Interfaces Proxy Servics Integration Logic Internal Interfaces Business Interfaces Database

9 Limited and expensive Scalability Loadbalancer ESB 1 ESB 2 ESB 3 Database

10 Caught in a technology Trap SOA Cloud Service Fixed Technologies Set { dialect : JSON } <XML />

11 Limited agility Hard to implement Continuous Integration / Continuous Delivery Complex infrastructure Missing automated testing capabilities Expert knowledge needed Implementation Deployment Error analysis

12 But the world is getting more complex!

13 So we re Constantly Dealing with Overstretched Integration Components / Patterns ESB JMS Application Server JCA Adapter SOAP / XML Etc.

14 Question How to deal with this new complexity efficiently? (By the way it doesn t mean that everything we did before was wrong!)

15 Agenda Status Quo Integration Reactive Patterns Architectural thoughts Use case Summary

16 Reactive Principles Value Responsive Form Elastic Resilient Means Message Driven

17 Actor model autonomous Source Actor Sink Async - Boundary Advantages: Useful for realizing reactive patterns Fault tolerant Distributed execution Scalability

18 Adopting Reactive Patterns (i.e. Ivy Pattern)

19 Actor hierarchies autonomous Source Actor Sink

20 Actor Balancing Actor Actor Queue Master Topic Actor Actor Queue Actor Router Queue Actor

21 Reactive Patterns: Backpressure Systems should be designed and implemented to react assertively under sustained load This is often neglected although such conditions are usually expected and highly predictable Negotiating back pressure usually requires some sort of Flow Control Which in simpler terms is the ability to say hey, slow down!!, or even stop it!!

22 Non-Blocking Backpressure Concept autonomous Source Actor Sink Give me 20 messages! Give me 3 messages!

23 Serveral ways to handle Backpressure Dynamic push / pull Static / Dynamic Quotas Buffers Ack / Nack Throttling Caching Circuit breaker Scaling

24 Reactive Patterns: Circuit Breaker Used in the industry to protect electronic components In software development Protect resources Help resources to auto-recover from failures Prevent hanging operations

25 Reactive Patterns: Bulkhead Used in the industry to partition a ship into sections In software development Segragate resources Protect limited resources from being exhausted

26 Avoiding single points of failure is key for reactiveness

27 Important Concept: a chain is no stronger than its weakest link

28 Agenda Status Quo Integration Reactive Patterns Architectural thoughts Use case Summary

29 Implementing Reactive Patterns with Akka Akka Streams Ideal for Fast Data Seamless Integration Abstraction to configure end-to-end Backpressure Fully Asynchronous & non-blocking Bulkheads & Circuit Breaker patterns are also supported

30 Implementing Reactive Pattern with Kafka Distributed streaming platform Dynamic durable buffer Horizontally scalable Load distribution No data loss settings (Producer, Consumer, Broker) Quotas

31 Next-gen Integration Platform Sysco reactive Integration platform

32 Demo: Backpressure Fast Producer, Fast Consumers Fast Producer, Fast and Slow Consumers Fast Producer, Fast and Slow Consumers with buffering: OverflowStrategy.dropNew Fast Producer, Fast and Slow Consumers with buffering: OverflowStrategy.backpressure

33 Implement Integration with Serverless/FaaS Function-as-a-Service (FaaS) Atomar step within a certain business capability Example: Bicycle rental - Process incoming Damage events Serverless No need to care about servers Scale-out functions automatically on-demand You only pay processing time

34 Implement Integration with Serverless/FaaS X System A System B f(x) getdata f(x) transformdata f(x) senddata

35 When is FaaS adequate for integration? Typical use cases: Batch processing, which takes place on regular schedules Event processing Considerations to make: Number of invocations Keep runtime short Serverless Platforms will terminate long-running Functions Don t block HW resources for too long

36 Integration with Fn Project Implement functions F(x) getdata F(x) transformdata F(x) senddata Orchestrate functions using Fn Flow Alternatively: Use Topics in between the functions

37 Agenda Status Quo Integration Reactive Patterns Architectural thoughts Use case Summary

38 New digital business models in bicycle rental [Rainer Klute]

39 Happy Path: Renting a bike Scan a QR Codes at bike rental station with an mobile app Book the rental bike Open lock at rental station Remove the bicycle from the station and drive off

40 Use Case: Bonus Action After renting a bike for a fee for the 10th time you get a 20% discount and can get a free coffee at the next Mermaid Coffee during your tour at our expense.

41 What does this mean for the application? Connection to CRM and ERP necessary Calculation of different cost rates on the basis of historical data Connection of all branches of the external partner (Mermaid Coffee) Near real-time - cyclists must still have rented a bike to redeem their coffee Payment flows between partners, so that a high degree of robustness is necessary

42 Solution approach: Actors & Reactive Principles Get data from ERP Get data from CRM Inform Partner Buffer Enable bonuses? Topic Inform Custome r Topic

43 Use Case: Damage 3 out of 5 bikes at the same rental station fell over in 2 minutes???

44 Vandalism is the Hulk of our business model [Marcus Pink]

45 Solution approach: Fast Data & Integration Damage events Process Damage claims Topic Analyze Events Buffer New Vandalism event Topic Inform technician

46 Agenda Status Quo Integration Reactive Patterns Architectural thoughts Use case Summary

47 Summary Integration is still a relevant discipline nowadys, but the way how to implement it has chnged Existing integration approaches based on ESB solutions are still valid, but Businesses getting disrupted, are volatile and techologies is emerging rapidly, so the architecture needs to consider this Architectures must address this accordingly

48