Erlang and Scalability

Transcription

1 Erlang and Scalability Percona Performance 2009 Jan Henry Nystrom Course Introduction Course Course Author 2007

2 Introduction Scalability Killers Design Decisions Language and Yours Thinking Scalable/Parallel Code for the correct case Rules of Thumb Scalability in the small: SMP Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 2

3 Scalability Killers Synchronization Resource contention Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 3

4 Scalability Killers Synchronization Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 3

5 Design Decisions No sharing Processes Encapsulation No implicit synchronization Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 4

6 Design Decisions No implicit synchronization Spawn always succeed Sending always succeed Random access message buffer Fire and forget unless you need the synchronization Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 5

7 Design Decisions Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

8 Design Decisions Concurrency oriented programming Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

9 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

10 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

11 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Sets the focus firmly on the concurrent tasks Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

12 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Sets the focus firmly on the concurrent tasks Code for the correct case Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

13 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Sets the focus firmly on the concurrent tasks Code for the correct case Clear Code Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

14 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Sets the focus firmly on the concurrent tasks Code for the correct case Clear Code Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

15 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Sets the focus firmly on the concurrent tasks Code for the correct case Clear Code Clarity is King! Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

16 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Sets the focus firmly on the concurrent tasks Code for the correct case Clear Code Clarity is King! Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

17 Design Decisions Concurrency oriented programming Concurrency support an integral part of the language Distribution support Sets the focus firmly on the concurrent tasks Code for the correct case Clear Code Clarity is King! I rather try to get clear code correct than correct code clear Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 6

18 Thinking Scalable/Parallel 0 Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 7

19 Thinking Scalable/Parallel List length: Obviously Linear 4 : But not when you have n processors? Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 7

20 Thinking Scalable/Parallel List length: O(logN) with sufficient processors Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 8

21 Thinking Scalable/Parallel List length: O(logN) with sufficient processors Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 8

22 Thinking Scalable/Parallel In the Erlang setting Do not introduce unneeded synchronization Remember processes are cheap Do not introduce unneeded synchronization A terminated process is all garbage Do not introduce unneeded synchronization Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 9

23 Code for the Correct Case Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 10

24 Code for the Correct Case set timer request set timer request set timer request Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 10

25 Code for the Correct Case set timer request set timer release timer check answer set timer release timer check answer request request answer release timer check Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 10

26 Code for the Correct Case set timer request set timer release timer check answer set timer release timer check answer request request answer release timer check Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 10

27 Code for the Correct Case Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 11

28 Code for the Correct Case set timer request request request Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 11

29 Code for the Correct Case set timer request request request answer Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 11

30 Rules of Thumb Rule 1 - All independent tasks should be processes Rule 2 - Do not invent concurrency that is not there! f() g() h() Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 12

31 Rules of Thumb Rule 1 - All independent tasks should be processes Rule 2 - Do not invent concurrency that is not there! f() g() h() Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 12

32 Rules of Thumb Rule 1 - All independent tasks should be processes Rule 2 - Do not invent concurrency that is not there! f() g() h(g(f())) h() Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 12

33 Rules of Thumb Rule 1 - All independent tasks should be processes Rule 2 - Do not invent concurrency that is not there! f() g() h() h(g(f())) h(g(f())) h(g(f())) h(g(f())) Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 12

34 Scalability in the small: SMP Erlang SMP Credo SMP should be transparent to the programmer in much the same way as Erlang Distribution You shouldn t have to think about it...but sometimes you must Use SMP mainly for stuff that you d make concurrent anyway Erlang uses concurrency as a structuring principle Model for the natural concurrency in your problem Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 13

35 Scalability in the small: SMP Erlang on multicore Big bang benchmark on Sunfire T2000 SMP prototype 97, First OTP release May 06. Mid -06 benchmark mimicking call handling (axdmark) on the (experimental) SMP emulator. Observed speedup/core: 0.95 First Ericsson product (TGC) released on SMP Erlang in Q scheduler Simultaneous processes 16 schedulers Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 14

36 Scalability in the small: SMP Case Study: Telephony Gateway Controller AXE TGC Mediates between legacy telephony and multimedia networks. Hugely complex state machines + massive concurrency. Developed in Erlang. Multicore version shipped to customer Q207. GW GW Porting from 1-core PPC to 2-core Intel took < 1 man-year (including testing). GW Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 15

37 Scalability in the small: SMP Case Study: Telephony Gateway Controller Traffic scenario IS/GCP 1slot/board IS/GEP Dual core One core running 2slots/board IS/GEP Dual core Two cores running 2slots/board AXD CPB5 AXD CPB6 POTS-POTS / AGW X call/sec 2.3X call/sec One core used 4.3X call/sec OTP R11_3 beta +patches 0.4X call/sec 2.1X call/sec ISUP-ISUP /Inter MGW 3.6X call/sec 7.7X call/sec One core used 13X call/sec OTP R11_3 beta +patches 1.55X call/sec 7.6X call/sec ISUP-ISUP /Intra MGW 5.5X call/sec 26X call/sec 3.17X call/sec 14X call/sec Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 16

38 Scalability in the small: SMP Case Study: Telephony Gateway Controller Traffic scenario IS/GCP 1slot/board IS/GEP Dual core One core running 2slots/board IS/GEP Dual core Two cores running 2slots/board AXD CPB5 AXD CPB6 POTS-POTS / AGW X call/sec 2.3X call/sec One core used 4.3X call/sec OTP R11_3 beta +patches 0.4X call/sec 2.1X call/sec ISUP-ISUP /Inter MGW 3.6X call/sec 7.7X call/sec One core used 13X call/sec OTP R11_3 beta +patches 1.55X call/sec 7.6X call/sec ISUP-ISUP /Intra MGW 5.5X call/sec 26X call/sec 3.17X call/sec 14X call/sec Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 16

39 Scalability in the small: SMP 5.00 Speedup on 4 Hyper Threaded Pentium4 Speddup # Schedulers Chatty 1000 processes created Each process randomly sends req/recieves ack from all other processes Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 17

40 Scalability in the small: SMP non-smp VM Erlang VM run queue Scheduler Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 18

41 Scalability in the small: SMP Current SMP VM OTP R11/R12 Erlang VM Scheduler #1 run queue Scheduler #2 Scheduler #N Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 19

42 Scalability in the small: SMP New SMP VM OTP R13 Erlang VM Scheduler #1run queue Released 21th April Scheduler #2 run queue migration logic Scheduler #N run queue Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 20

43 Scalability in the small: SMP Multiple run queues Speedup: Ca 0.43 * 32 cores Memory allocation locks dominate... Single run queue Speedup of Big Bang on a Tilera Tile64 chip (R13A) 1000 processes, all talking to each other Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 21

44 Scalability in the small: SMP Shift in Bottlenecks All scalable Erlang systems were stress tested for CPU usage for network usage With SMP hardware we must stress test for memory usage In the typical SMP system, the bottleneck has shifted from the CPU to the memory Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 22

45 Scalability in the small: SMP Death by a thousand cuts Many requests that generate short spikes in memory usage Limit or serialize those requests More on this in coming paper from CTO Ulf Wiger loop(state) -> receive {request, typea, Data} -> Data1 = allocate_lots_of_memory(data), a_server! {request, typea, self()}, receive {answer, Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 23

46 Questions??? Erlang and Scalability Percona Performance Conference , Erlang Training and Consulting 24