Apache Mesos. Delivering mixed batch & real-time data infrastructure , Galway, Ireland

Transcription

1 Apache Mesos Delivering mixed batch & real-time data infrastructure , Galway, Ireland Naoise Dunne, Insight Centre for Data Analytics Michael Hausenblas, Mesosphere Inc.

2 Types of Workloads batch streaming PaaS MapReduce 2

3

4 Mesos Intro

5 Apache Mesos A top-level ASF project A cluster resource negotiator Scalable to 10,000s of nodes but also useful for a handful of nodes Fault-tolerant, battle-tested An SDK for distributed apps Native Docker support 5

6 What is a Data Center Scheduler? Schedulers run your Distributed Apps An operating system kernel for the cloud Schedulers coordinate execution of work on cluster

7 A Quick History of Schedulers

8 History of Datacenter Schedulers 2010 Nexus (Mesos) 2003 Slurm 2008 Hadoop released 2004 Google Borg Google filesystem 2014 Google Omega paper 2010 Spark Paper Hadoop started 2004 mapreduce paper Quick history of distributed schedulers Mesos Released 2011 Mesos Paper Hadoop Yarn Kubernetes

9 Hadoop - Original O/S Scheduler Linked datat m/r job Monolithic scheduler: Original open source datacenter scheduler jobs are batched and executed Designed only to run Mapreduce jobs No concurrency between apps Evolving into yarn Linked datat app Linked datat m/r job Hadoop hadoop- resource management mesos slave mesos slave Linux Server Linux Server

10 Mesos - a Great Leap Forward 2 level scheduler : More flexible Can Schedule many kinds of applications Frameworks (such as spark) are delegated the per application scheduling Mesos responsible for resource distribution between applications and enforcing overall fairness Very modular, due to 2 level scheduling. frameworks manage apps as they like Linked datat app Linked data job Hadoop M/R job framework marathon framework spark framework chronos Mesos - scheduler jobs Mesos Mesos - resource management mesos slave mesos slave Linux Server Linux Server

11 How Mesos Works

12 Mesos Architecture 12

13 Mesos Resources resource == anything a task/executor consumes in order to do their work standard resources: cpu, mem, disk, ports DRF

14 2015 Mesosphere, Inc. 1 4

15 2015 Mesosphere, Inc. 1 5

16 2015 Mesosphere, Inc. 1 6

17 2015 Mesosphere, Inc. 1 7

18 2015 Mesosphere, Inc. 1 8

19 2015 Mesosphere, Inc. 19

20 2015 Mesosphere, Inc. 20

21 2015 Mesosphere, Inc. 21

22 2015 Mesosphere, Inc. 22

23 2015 Mesosphere, Inc. 23

24 2015 Mesosphere, Inc. 24

25 2015 Mesosphere, Inc. 25

26 2015 Mesosphere, Inc. 26

27 2015 Mesosphere, Inc. 27

28 2015 Mesosphere, Inc. 28

29 2015 Mesosphere, Inc. 29

30 Benefits of using a Scheduler Efficiency - best use of computing resources Agility - change your application mix with no turnaround Scalability - grow to the current demand of your app Modularity - 2 level schedulers have plugin frameworks that allow quick repurposing of core and no reliance on one vendor (more later)

31 Mesos Ecosystem

32 Applications work with frameworks to get resources they need Mesos Ecosystem Mesos Resources Monitor cpu mem disk Managed by Mesos OS Monitor graphx Graph Jobs Spark Fwk Chronos Fwk Mesos - scheduler short jobs Datastores HDT, Neo4J Granatum Revealed Frameworks Negotiate with mesos to run their jobs Marathon Framework Mesos - scheduler long run jobs Mesos Mesos - resource management mesos client Docker Linux Server mesos client Docker Linux Server mesos client Docker Linux Server Docker manages isolation on Linux servers

33 Mesos Ecosystem We need HDFS for large storage on Spark Jobs Need Mesos DNS for service discovery Marathon can now use HDFS to store large Dependencies HDFS Chronos Fwk Spark Fwk Mesos - scheduler short jobs Mesos DNS Marathon fwk Mesos - scheduler long run jobs Docker Registry Mesos Zookeeper you will need docker reg for marathon Mesos - resource management mesos client Mesos & frameworks needs zookeeper Docker mesos client Docker mesos client Linux Server Linux Server Linux Server DCOS DCOS DCOS Docker Universe/Multi verse To run mesos you will need dcos or glue

34 Datacenter schedulers: Why? Schedulers help you focus on your own work and not the infrastructure. its great to be able to focus on what it is you want to be doing rather than worrying about how do you get what it is you need in order to be able to get stuff done - John Wilkes (Google)

35 Mesos Best Practices

36 Mesos Best Practices Discovery Orchestration Composition

37 Discovery

38 Orchestration

39 Orchestration

40 Orchestration

41 Composition Marathon: apps and groups Kubernetes: pods and services Reusability, affinity and loose coupling

42 Monitoring

43 Monitoring

44 Enter DCOS

45 Local OS vs. Distributed OS 45

46 DCOS, A Distributed Operating System kernel (Apache Mesos, written in C++) scales to 10,000 of nodes fault-tolerant in all components, rolling upgrades throughout containers first class citizens (LXC, Docker) local OS per node (+container enabled) scheduling (long-lived, batch) service discovery, monitoring, logging, debugging 46

47 DCOS High Level Overview Any Service or Container Your favorite services, container formats, and those yet to come Mesosphere DCOS Runs distributed apps anywhere as simply as running apps on your laptop Any Infrastructure Build apps once on DCOS, and run it anywhere 47

48 DCOS Benefits Run stateless services such as Web servers, app servers (via Marathon) and stateful services like Spark, Kafka, HDFS, Cassandra, ArangoDB etc. together on one cluster Dynamic partitioning of your cluster, depending on your needs (business requirements) Increased utilization (10% 80% and more) 48

49 DCOS Architecture 49

50 It s demo time

51

52 See Also

53 See Also 53

54 Q&A