Watch your communication How communication drives design

Transcription

1

2 Watch your communication How communication drives design Uwe Friedrichsen (codecentric AG) GOTO Berlin Berlin, 13. November 2016

3 @ufried Uwe Friedrichsen

4 The one-size-fits-all trap

5 Standardization as a virtue

6 We see that a lot One programming language One framework One build system One type of runtime unit One data store One type of database access One application server One... One communication paradigm

7 Why are we doing it?

8 Reducing complexity Avoid intellectual overload Easier to understand Easier to change Faster to change Easier to operate Less error-prone Cheaper

9 These are valid and important reasons

10 ... but that is just one side of the coin

11 The complexity of a solution must match the complexity of the problem to solve

12 If your task is to build a box from pre-cut boards, only using a hammer might be adequate

13 If your task is also to paint the box you just built, only having a hammer might be a problem

14 Now imagine building a whole house just with a hammer

15 Yet, in IT we often tend to build whole houses just with a hammer because boxes can be built with a hammer

16 Complexity mismatch means Increased accidental complexity Harder to understand Harder to change Slower to change Harder to operate More error-prone More expensive

17 Problem complexity Essential complexity Complexity Adds indirect accidental complexity Adds direct accidental complexity Ideal solution complexity

18 Problem complexity Complexity More tooling than needed to get the job done well Increased accidental complexity Overly complex solution

19 Problem complexity Complexity Increased accidental complexity Not enough tooling to get the job done well Over- simplified solution

20 Overall Solution Complexity := Essential Problem Complexity + Solution Complexity Mismatch Solution Complexity Mismatch a.k.a. Accidental Complexity Be aware that mismatch in both directions adds to overall solution complexity

21 Problem complexity Essential complexity Complexity Aim for the least complex solution with respect to the problem you have to solve (Yes, it is about trade-offs as always in architecture)

22 Communication paradigms

23 Communication paradigms Dimensions of communication Synchronous vs. asynchronous Request/response vs. message passing Commands, Events and Documents Sender/Receiver dependency

24 Communication paradigms Typical communication patterns Synchronous request/response Asynchronous commands (with known receiver) Asynchronous events (without known receiver)

25 The synchronous request/response everywhere trap

26 Why are we doing it?

27 Remember the one-size-fits-all trap?

28 Yes, but why this paradigm?

29 Why synchronous R/R? It s how the internet works The library support is better It is easier to reason about Matches the way we consciously think Matches the way we learned to do design

30 Do not forget the other side of the coin

31 The flipside of the coin HTTP was designed for a specific use case Access hypermedia content Not an ubiquitous communication solution The world outside a process is different Non-determinism kills easy reasoning Latency creep kills easy reasoning Solution mismatch adds complexity Compromises functional encapsulation Changes and extensions become more complicated Longer activation paths compromise robustness

32 Designing communication

33 An (invidious) example

34 Order fulfillment

35 Let us begin with the well-known synchronous request/response design...

36 Order fulfillment Initial version with limited functionality Order includes Items Amounts Price per item Total price Credit card payment Only non-digital items

37 Online Shop Coordinate Order Fulfillment Payment Credit Card Warehouse Shipment Credit Card Provider Warehouse System <Own > <Foreign >

38 Order fulfillment Add coupons Order additionally includes Optional coupon ID Optional remaining coupon value Coupon is already charged against total price Validate coupon

39 Online Shop Coordinate Order Fulfillment Coupon Payment Coordinate Credit Card Warehouse Shipment Coupon Management Credit Card Provider Warehouse System <Own > <Foreign >

40 Order fulfillment Add promotions Order additionally includes Optional promotion ID Promotion is already applied to order Notify Campaign Management System about promotion usage

41 Online Shop Coordinate Order Fulfillment Campaign Management Promotion Coupon Payment Coordinate Credit Card Warehouse Shipment Coupon Management Credit Card Provider Warehouse System <Own > <Foreign >

42 Order fulfillment Add customer accounts Order additionally includes Payment type (Credit card/account) Charge customer account

43 Online Shop Coordinate Order Fulfillment Campaign Management Promotion Coupon Payment Coordinate Credit Card Warehouse Shipment Coupon Management Account service Credit Card Provider Warehouse System Accounts Receivables <Own > <Foreign >

44 Order fulfillment Add PayPal Order additionally includes More payment types (Credit card/paypal/account) Payment via PayPal

45 Online Shop Coordinate Order Fulfillment Campaign Management Promotion Payment Coordinate Coupon Warehouse Shipment Coupon Management Account service Payment provider Credit Card PayPal Warehouse System Accounts Receivables Credit Card Provider PayPal <Own > <Foreign >

46 Order fulfillment Add digital music library New shipment method Activate music assets in digital music library Notify customer via including access link Digital music library has its own API

47 Online Shop Coordinate Order Fulfillment Server Campaign Management Promotion Payment Coordinate Coupon Coordinate Warehouse Shipment Notify Cust. Music Coupon Management Account service Payment provider Credit Card PayPal Warehouse System Music Library Accounts Receivables Credit Card Provider PayPal <Own > <Foreign >

48 Order fulfillment Add digital video library New digital shipment channel Activate video assets in digital video library Notify customer via including access link Digital video library has its own API Different from the music library API

49 Online Shop Coordinate Order Fulfillment Server Campaign Management Promotion Payment Coordinate Coupon Coordinate Warehouse Shipment Notify Cust. Assets Coupon Management Account service Payment provider Credit Card PayPal Warehouse System Music Library Video Library Accounts Receivables Credit Card Provider PayPal <Own > <Foreign >

50 Order fulfillment Add e-book library New digital shipment channel Activate e-books in e-book library Notify customer via including access link E-Book library has its own API Different from the other asset library APIs

51 Online Shop Coordinate Order Fulfillment Server Campaign Management Promotion Payment Coordinate Coupon Coordinate Warehouse Shipment Notify Cust. Assets E-Book Library Coupon Management Account service Payment provider Credit Card PayPal Warehouse System Music Library Video Library Accounts Receivables Credit Card Provider PayPal <Own > <Foreign >

52 Order fulfillment Add bonus card Order additionally includes Optional bonus card ID Potential discount is already applied to order Notify Loyalty Management System about customer purchase

53 Online Shop Loyalty Management Coordinate Order Fulfillment Server Campaign Management Loyalty Promotion Payment Coordinate Coupon Coordinate Warehouse Shipment Notify Cust. Assets E-Book Library Coupon Management Account service Payment provider Credit Card PayPal Warehouse System Music Library Video Library Accounts Receivables Credit Card Provider PayPal <Own > <Foreign >

54 Order fulfillment More requirements to come

55 Synchronous request/response findings Divide and conquer design Loyalty Management Tight service coupling Coordinate including cross-domain coupling Campaign Loyalty Payment Management Promotion Coupon Long activation paths Coordinate Online Shop Order Fulfillment Coordinate Warehouse Shipment Notify Cust. Assets Server E-Book Library Latency creep Coupon Account Payment Credit Card Warehouse Management Availability Music Library service erosion provider PayPal System Underlying design assumption is 100% availability of all services used Video Library Complex orchestration logic in services Accounts Credit Card Receivables Provider including cross-service transactions New downstream services require changes of upstream services PayPal including knowledge about how to access downstream services <Own > in upstream <Foreign services >

56 Let us repeat the design using asynchronous events...

57 Order fulfillment

58 Order fulfillment Initial version with limited functionality Order includes Items Amounts Price per item Total price Credit card payment Only non-digital items

59 Order confirmed Payment authorized Warehouse Warehouse System Online Shop Credit Card Credit Card Provider <Own > <Foreign > <Event> Payment failed

60 Order fulfillment Add coupons Order additionally includes Optional coupon ID Optional remaining coupon value Coupon is already charged against total price Validate coupon

61 Order confirmed Payment authorized Warehouse Warehouse System Online Shop Credit Card Credit Card Provider Coupon Coupon Management <Own > <Foreign > <Event> Payment failed

62 Order fulfillment Add promotions Order additionally includes Optional promotion ID Promotion is already applied to order Notify Campaign Management System about promotion usage

63 Order confirmed Payment authorized Warehouse Warehouse System Online Shop <Own > <Foreign > Credit Card Credit Card Provider Coupon Coupon Management Promotion Campaign Management <Event> Payment failed

64 Order fulfillment Add customer accounts Order additionally includes Payment type (Credit card/account) Charge customer account

65 Order confirmed Online Shop <Own > <Foreign > Payment authorized Account service Accounts Receivables Credit Card Credit Card Provider Warehouse Warehouse System Coupon Coupon Management Promotion Campaign Management <Event> Payment failed

66 Order fulfillment Add PayPal Order additionally includes More payment types (Credit card/paypal/account) Payment via PayPal

67 Order confirmed Online Shop <Own > <Foreign > <Event> Payment authorized Account service Accounts Receivables Credit Card Credit Card Provider PayPal PayPal Warehouse Warehouse System Coupon Coupon Management Promotion Campaign Management Payment failed

68 Order fulfillment Add digital music library New shipment method Activate music assets in digital music library Notify customer via including access link Digital music library has its own API

69 Order confirmed Payment authorized Warehouse Digital asset provisioned Account service Accounts Receivables Warehouse System Coupon Music Library Music Library Online Shop Credit Card Credit Card Provider Coupon Management Promotion Notification Server <Own > <Foreign > PayPal Campaign Management <Event> PayPal Payment failed

70 Order fulfillment Add digital video library New digital shipment channel Activate video assets in digital video library Notify customer via including access link Digital video library has its own API Different from the music library API

71 Order confirmed Payment authorized Warehouse Digital asset provisioned Account service Accounts Receivables Warehouse System Coupon Music Library Music Library Online Shop Credit Card Credit Card Provider Coupon Management Promotion Video Library Video Library Notification Server <Own > <Foreign > PayPal Campaign Management <Event> PayPal Payment failed

72 Order fulfillment Add e-book library New digital shipment channel Activate e-books in e-book library Notify customer via including access link E-Book library has its own API Different from the other asset library APIs

73 Order confirmed Payment authorized Warehouse Digital asset provisioned Account service Accounts Receivables Warehouse System Coupon Music Library Music Library Online Shop Credit Card Credit Card Provider Coupon Management Promotion Video Library Video Library Notification Server <Own > <Foreign > PayPal Campaign Management E-Book Library <Event> PayPal E-Book Library Payment failed

74 Order fulfillment Add bonus card Order additionally includes Optional bonus card ID Promotion is already applied to order Notify Loyalty Management System about customer purchase

75 Order confirmed Payment authorized Warehouse Digital asset provisioned Account service Accounts Receivables Warehouse System Coupon Music Library Music Library Online Shop Credit Card Credit Card Provider Coupon Management Promotion Video Library Video Library Notification Server <Own > <Foreign > <Event> PayPal PayPal Payment failed Campaign Management Bonus Card Loyalty Management E-Book Library E-Book Library

76 And how do we figure out if a business transaction eventually completes?

77 s are responsible to eventually succeed or fail for good, usually incorporating a supervision/escalation hierarchy for that Order fulfillment supervisor Track flow of events. Reschedule events in case of failure Order confirmed Payment authorized Warehouse Digital asset provisioned Account service Accounts Receivables Warehouse System Coupon Music Library Music Library Online Shop Credit Card Credit Card Provider Coupon Management Promotion Video Library Video Library Notification Server <Own > <Foreign > <Event> PayPal PayPal Payment failed Campaign Management Bonus Card Loyalty Management E-Book Library E-Book Library

78 s are responsible to eventually succeed or fail for good, usually incorporating a supervision/escalation hierarchy for that Order confirmed Payment authorized Go with the flow design Online Shop <Own > <Foreign > Account service Accounts Receivables Credit Card Credit Card Provider Warehouse Warehouse System Coupon Coupon Management Promotion Music Library Music Library Video Library Video Library Digital asset provisioned Notification Server Campaign No central coordination PayPal and cross-service transactions Management Instead functional compensation logic in external supervision E-Book Library service Bonus Card PayPal New services can be integrated without touching E-Book existing services Library s only focus on their task (no knowledge about other services required) Loyalty Payment failed Management <Event> Order fulfillment supervisor Asynchronous events findings s and domains decoupled Very short activation paths High responsiveness High availability No design assumptions about availability of services Coordination via choreography Track flow of events Reschedule events in case of failure

79 Didn t we miss any special cases that would lead to very different findings?

80 Some special cases Synchronous request/response design Current order status E.g., pass status through call chain Maybe additionally extend order fulfillment service to store status in an status information database Payment failed E.g., stop processing in fulfillment service and send back appropriate status to online shop Cashing coupon failed E.g., extend payment service to place task in desk officer s inbox Integrate new video library (different API) Adapt shipment service to new API Make sure not to disrupt other service aspects

81 Some special cases Asynchronous events design Current order status E.g., track events with a new service and store them in an status information database Payment failed E.g., track dedicated event with a new service that notifies customer and/or online shop Cashing coupon failed E.g., send out dedicated event that a new service subscribes to New service then places task in desk officer s inbox Integrate new video library (different API) Implement new service Replace existing service with new service

82 Feels a lot like recurring patterns

83 What did we learn so far?

84 Comparing the paradigms Decomposition Vertically divide-and-conquer vs. horizontally go-with-the-flow Coordination Coordination logic/services and orchestration vs. event chains and choreography Transactions Built-in transaction handling vs. external supervision Error handling Built into service vs. escalation/supervision strategy Separation of concerns Multiple responsibilities service vs. single responsibility services Encapsulation Domain logic distributed across services vs. domain logic in one place Complexity A draw

85 Different communication paradigms lead to very different service designs which means different options to address a given problem

86 Related areas and limitations

87 Mixing communication paradigms

88 Mixing paradigms Always an option Note the trade-off between design purity and solution mismatch complexity A (very) simple heuristic Single functionality All-or-nothing characteristic Single domain Ø Synchronous request/response Multiple functionalities Flow/Process characteristic Multiple domains Ø Asynchronous messaging

89 What if the paradigm is fixed?

90 Fixed paradigm You can simulate other paradigms E.g., simulate sync. r/r with async. messaging E.g., simulate async. commands with sync. r/r Will support the service design topic Apart from that it is a (lame) compromise

91 What about user interaction?

92 User interaction Interaction with user client is usually r/r Use web sockets if available Use the mediator pattern with timeouts Could be implemented via a BFF service Complement with processing status page

93 Can we apply the same patterns inside process boundaries?

94 Inside process boundaries Have the process context in mind Without threads If your downstream call dies, you are also dead Usually synchronous r/r is appropriate Otherwise single-threaded scheduler is needed With threads Multi-process light Better multi-threading resilience possible More design options

95 Wrap-up The one-size-fits-all trap Communication paradigms The synchronous-r/r-everywhere trap Designing communication Influence on service design Different communication paradigms lead to very different service designs

96 There is no one-size-fits-all solution

97 @ufried Uwe Friedrichsen

98

99