for the NFV World with.io
Speaker Donald Hunter Principal Engineer in the Chief Technology and Architecture Office at Cisco. Lead the MEF OpenLSO project which uses.io as a reference implementation for big data analytics in the MEF LSO Framework. Historical focus has been software architecture leadership for element management systems, diagnostics and network provisioning applications in Cisco's product portfolio.
Introducing.io
What is? brings together a number of open source technologies to provide a simple, scalable, open, big data analytics Platform for Network Data Linux Foundation Collaborative Project based on the Apache ecosystem
Why? There are a bewildering number of big data technologies out there, so how do you decide what to use? We've evaluated and chosen the best tools, based on technical capability and community support. combines them to streamline the process of developing data processing applications.
ODL Logstash OpenBPM pmacct Telemetry Data Distribution Processing Real -time Stream Batch File Store Query SQL Query OLAP Cube Search/ Lucene NoSQL Visualisation and Exploration Data Exploration Metric Visualisation Event Visualisation Time Series Applications Unmanaged App Unmanaged App Managed App Managed App Simple, scalable open data platform Provides a common set of services for developing analytics applications Accelerates the process of developing big data analytics applications whilst significantly reducing the TCO provides a platform for convergence of network data analytics Plugins Platform Services: Installation, Mgmt, Security, Data Privacy App Packaging and Mgmt Producer API Consumer API
ODL Logstash OpenBPM pmacct Telemetry Data Distribution Processing Real -time Stream Batch File Store Query SQL Query OLAP Cube Search/ Lucene NoSQL Visualisation and Exploration Data Exploration Metric Visualisation Event Visualisation Time Series Applications Unmanaged App Unmanaged App Managed App Managed App decouples data aggregation from data analysis Support for near-real-time stream processing and in-depth batch analysis on massive datasets Consuming applications can be either platform apps developed for or client apps integrated with Client apps can use one of several structured query interfaces or consume streams directly. Plugins Platform Services: Installation, Mgmt, Security, Data Privacy App Packaging and Mgmt Leverages best current practise in big data analytics Producer API Consumer API
Capabilities Platform for data aggregation, distribution, processing and storage Automated installation, creation, and configuration Openstack, AWS and baremetal Typical install ~1hr Modular install Open producer and consumer APIs Avro platform schema Plugins for Logstash, pmacct, OpenBMP, OpenDaylight, Cisco XR-telemetry, bulk ingest Data distribution Apache Kafka Data store: Automated data partitioning and storage (HDFS) OpenTSDB time series analysis Hbase - NoSQL Support for batch and stream processing: Apache Spark Batch and Spark Streaming Jupyter notebook server for app prototyping and data exploration Impala-based SQL query support Grafana for time series visualisation application packaging management and dashboard
The console provides a dashboard across all components in a cluster Inbuilt platform test agents verify the operation of all components Active platform testing verifies the end-to-end data pipeline Console
for NFV
Perf and fault info defined by e.g. 3GPP0 Proprietary Perf and Fault Information OSS CFS Layer (and RFS layer) Perf and fault info related to a network service instance EMS ( VR Perf and Fault Info related to a Indicator Notifications NFVO (Network Services M ( VR Perf and Fault info VR Perf Metrics VR Perf TCAs VR Resource alarms Current ETSI NFV architecture defines a limited set of capabilities within the orchestration stack to monitor the deployed services and underlying infrastructure (Virtualisation VIM (Virtual Resource Note that ETSI NFV has not yet defined what the performance metrics are. A new work item will start to do this soon. (Hardware layer) ETSI NFV specifications on related topics: REL004 Report on Active Monitoring and Failure Detection
Perf and fault info defined by e.g. 3GPP0 Proprietary Perf and Fault Information OSS CFS Layer (and RFS layer) Perf and fault info related to a network service instance EMS ( (Virtualisation VR Perf and Fault Info related to a Indicator Notifications NFVO (Network Services M ( VIM (Virtual Resource VR Perf and Fault info VR Perf Metrics VR Perf TCAs VR Resource alarms Note that ETSI NFV has not yet defined what the performance metrics are. A new work item will start to do this soon. Challenges: More expansive monitoring and analysis capabilities are needed These require a big data analytics approach Requires different underlying technologies than those used within the existing stack (Hardware layer) ETSI NFV specifications on related topics: REL004 Report on Active Monitoring and Failure Detection
Perf and fault info defined by e.g. 3GPP0 OSS CFS Layer (and RFS layer) Perf and fault info related to a network service instance NFVO (Network Services How to integrate big data analytics into NFV? Proprietary Perf and Fault Information EMS ( VR Perf and Fault Info related to a Indicator Notifications M ( VR Perf and Fault info VR Perf Metrics VR Perf TCAs VR Resource alarms (Virtualisation VIM (Virtual Resource Note that ETSI NFV has not yet defined what the performance metrics are. A new work item will start to do this soon. (Hardware layer) ETSI NFV specifications on related topics: REL004 Report on Active Monitoring and Failure Detection
Data Aggregation Perf and fault info defined by e.g. 3GPP0 Proprietary Perf and Fault Information Perf and fault info related to a network service instance EMS ( (Virtualisation OSS CFS Layer (and RFS layer) VR Perf and Fault Info related to a Indicator Notifications NFVO (Network Services M ( VIM (Virtual Resource VR Perf and Fault info VR Perf Metrics VR Perf TCAs VR Resource alarms Log/Event Agg Perf Agg Telemetry Agg 1. All data aggregated and published to a) logs/events, metrics and telemetry b) Across all domains; from infrastructure, from the services, and from the orchestration and control stack c) Data types may be sub divided by topics d) Multiple aggregators may be used for a single data type (Hardware layer)
Applications Perf and fault info defined by e.g. 3GPP0 Proprietary Perf and Fault Information Perf and fault info related to a network service instance EMS ( OSS CFS Layer (and RFS layer) VR Perf and Fault Info related to a Indicator Notifications NFVO (Network Services M ( VR Perf and Fault info VR Perf Metrics VR Perf TCAs VR Resource alarms Event Log/Event Agg Perf Perf Agg Security Telemetry Agg 2. Analysis functions are implemented as big data applications on or as applications that take a data stream from This will not remove the need for monitoring and analysis within the orchestration stack (Virtualisation VIM (Virtual Resource (Hardware layer)
Operational Context Perf and fault info defined by e.g. 3GPP0 Proprietary Perf and Fault Information Perf and fault info related to a network service instance EMS ( OSS CFS Layer (and RFS layer) VR Perf and Fault Info related to a NFVO (Network Services M ( VR Perf Metrics VR Perf TCAs VR Resource alarms Real-time Inventory Event Perf Security 3. The orchestration and control state provides the context required for the big data analytics applications to provide meaningful insight We refer to this context as the Real-time Inventory Indicator Notifications VR Perf and Fault info Log/Event Agg Perf Agg Telemetry Agg (Virtualisation VIM (Virtual Resource (Hardware layer)
Closed-loop Control Perf and fault info defined by e.g. 3GPP0 Perf and fault info related to a network service instance EMS ( OSS CFS Layer (and RFS layer) VR Perf and Fault Info related to a NFVO (Network Services Real-time Inventory Event Perf Security 4. The output from the analytics applications may then be used to optimize deployed services through feedback to the orchestration and control functions. Proprietary Perf and Fault Information M ( VR Perf Metrics VR Perf TCAs VR Resource alarms Indicator Notifications VR Perf and Fault info Log/Event Agg Perf Agg Telemetry Agg (Virtualisation VIM (Virtual Resource (Hardware layer)
Multiple Feedback Loops Plan & Provision Offline feedback loop Use cases: Capacity planning Peering planning Cache placement Use cases: Traffic engineering: network optimisation Demand placement Workload placement Design Optimise Analyse Use cases: Service assurance Security operations Real-time feedback loop Orchestrate Monitor
Data Infrastructure Orchestration Applied to NFV VIM NFVO M Network Control Inventory Context Open Source Custom Licensed Applications Open Data Platform () Data Aggregators State Related as loosely coupled systems Logs Alerts Metrics Telemetry Data Sources User Access Aggregation Core Data Center
Questions?