Guagua: An Iterative Computing Framework on Hadoop. Zhang Pengshan(David), PayPal

Size: px

Start display at page:

Download "Guagua: An Iterative Computing Framework on Hadoop. Zhang Pengshan(David), PayPal"

Todd Harmon
5 years ago
Views:

2 Guagua: An Iterative Computing Framework on Hadoop Zhang Pengshan(David), PayPal

3 AGENDA Introduction Distributed Neural Network Algorithm What is Guagua? Guagua Advanced Features Shifu on Guagua Future Plans

4 AGENDA Introduction Distributed Neural Network Algorithm What is Guagua? Guagua Advanced Features Shifu on Guagua Future Plans

5 ALIPAY vs. PAYPAL Q: Where is risk control in PayPal? A: Risk control is everywhere in paypal.com.

6 FRAUD TYPES IN PAYPAL Account Take Over Fraud Types in PayPal Stolen Financials Credit Cards INR/SNAD INR: SNAD: Item Not Received Significantly Not as Described

7 RISK CONTROL IN PAYPAL Models Rules Agents

8 RISK MODELING IN PAYPAL MODELING CHALLENGES Thousands of Features Algorithms (LR, NN, DT) Big Training Data SLA (Online) Simulation

9 RISK MODELING IN PAYPAL MODELING CHALLENGES Thousands of Features Algorithms (LR, NN, DT) Big Training Data SLA (Online) Simulation Q: How to train models with TB data and thousands of features?

10 AGENDA Introduction Distributed Neural Network Algorithm What is Guagua? Guagua Advanced Features Shifu on Guagua Future Plans

11 DISTRIBUTED NEURAL NETWORK ALGORITHM* GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] ACCUMULATE GRADIENTS UPDATE WEIGHTS 1st iteration WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] ACCUMULATE GRADIENTS UPDATE WEIGHTS 2nd iteration WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] * Distributed batch gradient descent algorithm.

12 DISTRIBUTED NEURAL NETWORK ALGORITHM GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] ACCUMULATE GRADIENTS UPDATE WEIGHTS 1st iteration WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] GRADIENTS: DOUBLE [] ACCUMULATE GRADIENTS UPDATE WEIGHTS 2nd iteration WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] WEIGHTS: DOUBLE [] Q: How to implement it?

13 WHY NOT MAHOUT OR SPARK? Mahout Spark No distributed logistic regression & neural network. Iterative through Hadoop jobs, bad performance. No independent Spark cluster. Hadoop cluster is still 1.0 based, not YARN. Q: How to implement it in Hadoop?

14 POSSIBLE SOLUTIONS Pros Cons Hadoop YARN Flexible framework for framework Self resource management Alpha PayPal Clusters: Hadoop Extra fault tolerance, splits, UI Hadoop MapReduce Works well on all Hadoop versions Mature computing model Internal fault tolerance, splits, UI Different computing model How to do iterative coordination?

15 AGENDA Introduction Distributed Neural Network Algorithm What is Guagua? Guagua Advanced Features Shifu on Guagua Future Plans

16 ITERATIVE COMPUTING MODEL IN GUAGUA WORKER RESULT WORKER RESULT WORKER RESULT 1st iteration MASTER RESULT MASTER RESULT MASTER RESULT WORKER RESULT WORKER RESULT WORKER RESULT 2nd iteration MASTER RESULT MASTER RESULT MASTER RESULT Guagua is a framework over such iterative computing model, compared with Hadoop 1.0 over MapReduce.

17 GUAGUA API Computable Computable

18 GUAGUA OVERVIEW - s core CORE Fault Tolerance Coordination Iterative Computing Framework MapReduce Adapter (For Hadoop 1.0) YARN Adapter (For Hadoop 2.0) Consistent Client Distributed Neural Network Application

19 PLUGGABLE, SCALABLE INTERCEPTORS Fault Tolerance Interceptor Fault Tolerance Interceptor ZooKeeper Coordinator ZooKeeper Coordinator Perf Interceptor Perf Interceptor Timer Timer User Defined Interceptors User Defined Interceptors Computation Computation * These two graphs are aspects for each iteration.

20 GUAGUA RUNTIME : Mapper (Container) : Mapper (Container) : Mapper (Container) : Mapper (Container) : Mapper (Container) : Mapper (Container)

21 GUAGUA RUNTIME : Mapper (Container) REGISTER REGISTER : Mapper (Container) : Mapper (Container) REGISTER ZooKeeper Cluster REGISTER : Mapper (Container) REGISTER : Mapper (Container) REGISTER : Mapper (Container) 1. is listening znodes of workers. 2. s are listening znode of master.

22 GUAGUA RUNTIME : Mapper (Container) UPDATE ITER UPDATE ITER : Mapper (Container) : Mapper (Container) UPDATE ITER ZooKeeper Cluster UPDATE IITER : Mapper (Container) UPDATE ITER : Mapper (Container) UPDATE ITER : Mapper (Container) 1. Data is loaded in worker memory in the first iteration. 2. Whole process is done when reaches maximal iteration or halt condition is triggered.

23 AGENDA Introduction Distributed Neural Network Algorithm What is Guagua? Guagua Advanced Features Shifu on Guagua Future Plans

24 FAULT TOLERANCE n

25 FAULT TOLERANCE n * The same on workers.

26 STRAGGLER MITIGATION 1 2 3

27 STRAGGLER MITIGATION 1 2 3

28 STRAGGLER MITIGATION 1 2 3

29 PROGRESS AND STATE REPORT 0.86 = 432/501 (Current Iteration) / (Total Iteration)

30 GUAGUA UNIT

31 AGENDA Introduction Distributed Neural Network Algorithm What is Guagua? Guagua Advanced Features Shifu on Guagua Future Plans

32 WHAT IS SHIFU? Shifu* is an open-source, end-to-end machine learning and data mining framework built on top of Hadoop. NEW INIT STATS VARSELECT EVAL NORMALIZE POSTTRAIN TRAIN Built on Guagua *Want to try Shifu? Please visit

33 SHIFU ON GUAGUA (TRAIN STEP) Computable Interceptor Computable Gradients NN AbstractComputable BasicInterceptor Weights NN NNOutput GUAGUA API SHIFU CODE ENCOG CODE

34 SHIFU NN vs. SPARK LR Run Time Comparison Time Shifu-NN Spark-LR Iteration Shifu-NN: 1102*20*1 Netw ork, 319 Mappers * 1G Spark-LR: 1102 features, 120 executors * 3G

35 SHIFU NN BENCHMARK RESULTS Run Time (Seconds) Time(Seconds) G 375G 500G 625G 750G 875G 1000G Size of Input All data are located in memory. At most w e used 2400 mappers. 20 epochs are used.

36 AGENDA Introduction Distributed Neural Network Algorithm What is Guagua? Guagua Advanced Features Shifu on Guagua Future Plans

37 WHAT S NEXT? More open source docs Support more (distributed) machine learning algorithms Improve YARN (Beta) implementation Support more input formats Big model support Deep learning support

38 Q&A

39 APPENDIX Website Guagua issue website Shifu & Guagua source code:

41 @InfoQ infoqchina

BIG DATA AND HADOOP DEVELOPER

BIG DATA AND HADOOP DEVELOPER Approximate Duration - 60 Hrs Classes + 30 hrs Lab work + 20 hrs Assessment = 110 Hrs + 50 hrs Project Total duration of course = 160 hrs Lesson 00 - Course Introduction 0.1