Test Resource Management Center Big Data Analytics / Knowledge Management Architecture Framework Overview

Size: px

Start display at page:

Download "Test Resource Management Center Big Data Analytics / Knowledge Management Architecture Framework Overview"

Shavonne Quinn
5 years ago
Views:

1 Test Resource Management Center Big Data Analytics / Knowledge Management Architecture Framework Overview Ed Powell Test Resource Management Center Presented at the ITEA 34th International Test and Evaluation Symposium (2017) Oct 4, 2017, Reston, VA

2 T&E Need for Big Data Analytics/KM Capability Most investment has been on test needs, but this is an evaluation need. Need an evaluation strategy to not only analyze today's questions but future questions, even after fielding. T&E quality is inadequate for our needs More data is being collected than can be properly analyzed Only a tiny fraction of data is looked at Only simplistic analysis is being done on a small fraction of data No global view of the collected data is ever done No systematic anomaly detection, trend analysis, regression analysis, causality analysis, pattern recognition, simulation/test comparisons, perceived truth/ground truth comparisons are being done. T&E timeliness is inadequate for our needs Analyst retrieval of test data in some cases takes more than a week Sometimes it s easier (though not cheaper) to just re-run a test rather than find old data that may answer your question Long data ingest times prevent proper debriefing of test participants after a test is over, since their statements cannot be correlated with data in real time. T&E dollars are being spent unnecessarily More tests than necessary are being done, sometimes at enormous expense No cross-program lessons learned are being made, except anecdotally A systematic approach to Big Data Analytics and Knowledge Management (an architecture) is required to address these three serious issues. 2

3 3 Architecture Framework Purpose Understand the domain of Big Data Analytics and Knowledge Management as it relates to Test and Evaluation Needs Identify deficiencies in current T&E data analysis and knowledge management practices Identify commercial and open source software and hardware that could address these deficiencies Create a Roadmap for investment and deployment for these technologies Clearly identify the end state we are looking to achieve Identify clear benefits to acquisition programs Identify necessary S&T and development activities to get us to the end state Identify timeline for technology integration and deployment Socialize the architecture framework and the roadmap throughout the T&E community and receive feedback. Adjust as necessary to gain the support of the bulk of the T&E community. Create a set of implementation guidelines that will allow multiple independent developers to create valuable elements that integrates seamlessly Baseline for future coordinated infrastructure investments

4 What is Big Data Analytics? The use of advanced statistical analytic techniques in a parallel processing high-performance computing environment against very large diverse data sets that include different types of data Allows analysts to make better and faster decisions using data that was previously inaccessible or unusable Previously under-utilized data sources can be analyzed to gain new insights resulting in significantly better and faster decisions Instead of analyzing small chunks of data, Big Data Analytics can give the analyst a broad view of the system, allowing the discovery of unknown unknowns. Most important (and relevant to T&E) big data analytics techniques: Anomaly Detection Did something go wrong? Causality Detection What contributed to it? Trend What s happening over time? Predicting Equipment Function and Failure When will something go wrong? Regression How is today s data different than the past? Data Set Comparison Is test repeatable? Is the simulation the same as the test? Is the perceived truth the same as the ground truth? Pattern Recognition Are there hidden relationships in the data set? 4

DoD T&E Big Data and Knowledge Management Vision 5 Result: T&E data used more effectively &

Big Data Analytics to efficiently handle & securely share T&E data Organize T&E data to

execution & automated search scenarios that cannot occur today Use modern mechanisms to

5 DoD T&E Big Data and Knowledge Management Vision 5 Result: T&E data used more effectively & efficiently during acquisition The primary product of T&E is data & knowledge Embrace KM & Big Data Analytics to efficiently handle & securely share T&E data Organize T&E data to build knowledge across all DoD acquisitions Federate distributed data repositories to enable execution & automated search scenarios that cannot occur today Use modern mechanisms to enable collaboration between SMEs in government and industry Fundamental Functions Performed by KM and BDA

Big Data Analytics Overview (OV-1) Individual Range Range Augmentation Virtualized Big Data Tools Some Processing Some Tiered MILS Enhance Ingest Current Range Infrastructure Tools Ingest

6 Big Data Analytics Overview (OV-1) Individual Range Range Augmentation Virtualized Big Data Tools Some Processing Some Tiered MILS Enhance Ingest Current Range Infrastructure Tools Ingest Capabilities Quick-Look Working Files Cloud-Based Big Data Analytics and Knowledge Management System Regional Analytics Capability Regional Analytics Capability Virtualized Big Data Tools Processing Tiered MILS Data Scientists Virtualized Big Data Tools Processing Tiered MILS Data Scientists Video Data Reports Application Repository Schedule Info Regional Analytics Capability Virtualized Big Data Tools Processing Tiered MILS Data Scientists Audio Imagery Integrated Scalable Cost-Effective State-of-the-Art

7 Big Data and TENA Relationship: The Big Data Analytics Architecture is an Extension of TENA Into the Analytic World Seamless Integration Event Data Is Ingested into Big Data Enterprise System Individual Range Current Range Infrastructure Tools Ingest Capabilities Range Augmentation Virtualized Big Data Tools Some Processing Some Tiered MLS Enhance Ingest Quick-Look Working Files 7

Big Data Architecture Overview Applications Alerts Setup, Configure, and Manage Workflow Policies Prioritization Load Balancing Define Metadata Computing Resources Provisioning Configuration

UIs Quick-Look Real-Time Continuous Analytic Services Tools Alerting Scheduling/Automation Legacy Tools AI Tools Simulation Generate Reports Audio/Video Build Queries T&E Specific Custom BDA Services

8 Big Data Architecture Overview Applications Alerts Setup, Configure, and Manage Workflow Policies Prioritization Load Balancing Define Metadata Computing Resources Provisioning Configuration Fault/Recovery Schema License Cloud VM Library Create IDE SDK 2D/3D/Anim Design Reports Display Reports Big Data User Interface Display Alerts Create Automated Products Customized Displays Customized UIs Quick-Look Real-Time Continuous Analytic Services Tools Alerting Scheduling/Automation Legacy Tools AI Tools Simulation Generate Reports Audio/Video Build Queries T&E Specific Custom BDA Services Anomaly Detection Trend Causality Detection Regression Ground Truth Comparison Pattern Recognition Customization Extract- Transform- Load TENA Range Protocols Streams Micro-batch Mega-batch Parallel Structured Unstructured Audio/Video Data Sources Verify Transform Add Metadata Index Scripting Share Messaging Publish/Subscribe Transfer Structured Data Engine SQL Services Structured Database Working Sets Streaming Data Lifecycle Data Packages Data Mining Statistics Query Engine Federated access for both Structured and Unstructured Data Tables Virtualized Tools Workflow Crawl/Index Store Graph-Based Enforce Policies Pipeline Tagging Retrieve Key-Value Store COO/DR Administrative Ontologies Versioning Serve Warehouse Operations Organization Unstructured Data Engine Unstructured/Semi-Structured Database (Hadoop) Metadata Core Abstraction Layer () Infrastructure as a Service Platform as a Service as a Service Replication Machine Learning DB Admin Transform Catalog MPP Programming and Execution Engine C/R/U/D User-Defined Analytic Plugins Filter Sort Summarize Parallelize Optimize Schema Config Mgmt Verify Metadata Consistency Simulation as a Service Archive Tools Sync Data/Video Spatio-temporal Search Resource Mgmt Virtualized Legacy Tools Hypervisor Authenticate Authorize Access Control Enforce Policies Enforce Workflow Threat Detection Intrusion Detection Active Defenses Audit Encryption TRMC-Developed COTS/GOTS Hardware/Network Range HW/SW Computers Range Computing and Raw Files Flat Files Range Databases Databases Distributed File System Massively Parallel Tiered Computing,, and Network Infrastructure At Multiple Independent Levels of Remote Data Computing Replication Resources UC S TS SAP SAR MILS Secure Cloud 8

9 9 Big Data Hardware Hardware means a complementary and integrated combination of both processing and storage to perform the required analytic and knowledge management functions For the Regional Analytics Capabilities, hardware will be deployed focused on long-term storage and broad and deep multi-program analytics At each range, there are four basic options for hardware 1. Rely Primarily on Hardware 2. Hybrid of Hardware and Hardware 3. Rely Primarily on Range Hardware Where investments have already been made 4. Minimal Hardware Deployment and Integration Where we only track what data is where and not provide deep analytic capabilities. Since the amount of T&E data is increasing exponentially, purchasing and deploying hardware must be a continuous process that never ends. This fact needs to be impressed upon our Service partners and Congress.

Big Data Hardware/ Architecture Configuration 1 Rely

Regional Analytics Capability Data Ingest & Processing

Infrastructure Network MPP MILS Tiered Computing,, and

and Infrastructure Tools Databases High-speed data

10 Big Data Hardware/ Architecture Configuration 1 Rely on Hardware Range Instrumentation Local to a Range Regional Analytics Capability Data Ingest & Processing ETL MPP MILS Tiered Computing,, and Network Infrastructure Network MPP MILS Tiered Computing,, and Network Infrastructure Raw Files Flat Files Computing and Infrastructure Tools Databases High-speed data interconnect Hardware Hardware High-speed data interconnect User Tools on 10

11 11 Range Instrumentation Big Data Hardware/ Architecture Configuration 2 - Hybrid And Hardware Local to a Range Regional Analytics Capability Data Ingest & Processing ETL Tools Network MPP MILS Tiered Computing,, and Network Infrastructure Raw Files Flat Files Database Databases s Databases Databases Databases Databases Computing and Infrastructure High-speed data interconnect Hardware Hardware High-speed data interconnect

12 12 Range Instrumentation Big Data Hardware/ Architecture Configuration 3 - Rely Primarily on Range Hardware Local to a Range Management/ Regional Analytics Capability Data Ingest & Processing ETL Tools Network MPP MILS Tiered Computing,, and Network Infrastructure Raw Files Flat Files Database Databases s Databases Databases Databases Databases Computing and Infrastructure JMETC Network Working Set Hardware Hardware High-speed data interconnect

13 Big Data Hardware/ Architecture Configuration 4 Index Local Data Only Range Instrumentation Local to a Range Management/ Regional Analytics Capability Data Ingest & Processing ETL Tools Network MPP MILS Tiered Computing,, and Network Infrastructure Raw Files Raw Files Flat Files Flat Files Databases Databases Computing and Infrastructure JMETC Network Working Set Hardware Hardware High-speed data interconnect

14 Federated / Architecture Regional Analytics Capability Ranges MPP MILS Tiered Computing,, and Network Infrastructure High-speed data interconnect Hardware

15 Architecture: Notional MILS and CDS Enterprise Big Data Classification C Classification B Classification A Med-Speed Med-Speed Long-Term Med-Speed Long-Term Long-Term Regional Analytics Capability MILS-CDS MLS Database Med-Speed Long-Term

Big Data Analytics Maturity Model Similar to the software maturity model (the Capability Maturity

scale from no capability to a capability that is fully integrated and tailored to an organization s

INNOVATIVE NONEXISTENT Level 0 No big data analytics capabilities IMMATURE Level 1 Isolated big

analytics usage is part of mission critical applications only Full benefits are not understood by a

ad hoc approaches Big data analytics development and deployment is constrained, yet departments

centralized into larger groups Management understands and supports big data analytics for strategic

of its future growth plan Big data analytics software framework supports rapid response Big data

16 Big Data Analytics Maturity Model Similar to the software maturity model (the Capability Maturity Model Integration or CMMI), big data analytics capability in any organization can be evaluated on a scale from no capability to a capability that is fully integrated and tailored to an organization s needs. INNOVATIVE NONEXISTENT Level 0 No big data analytics capabilities IMMATURE Level 1 Isolated big data analytics use Unsophisticated tools and practices predominate AWARE Level 2 Some predictive analytics usage is part of mission critical applications only Full benefits are not understood by a majority in the agency INFORMED Level 3 Big data analytics usage consists primarily of tactical and ad hoc approaches Big data analytics development and deployment is constrained, yet departments have their own experts and/or initiatives EMPOWERED Level 4 Big data analytics talent is centralized into larger groups Management understands and supports big data analytics for strategic value, thus brining units into alignment Level 5 Agency is committed to big data analytics as part of its future growth plan Big data analytics software framework supports rapid response Big data analytical output integrated seamlessly into user applications and workflow Adapted from Etches et al., Analytic Technology Industry Roundtable Study: Analytics and Use Cases, published November 2016 by The Mitre Corp.

17 17 Summary An architecture specifies a technical plan for solving complex problems Document requirements and design constraints Identify sub-systems that need to interoperate Determine areas where standardization is needed Understand impact of current capability gaps & limitations Inform investment priorities The Big Data / Knowledge Management architecture framework provides context for the Big Data investment roadmap. Needs: Integrated local data Cloud analytics capability Big Data Tools Trained data scientist workforce Architecture standardization and buy-in will ensure collaboration and investments align to solve this common problem Our goal is to advance the T&E Community s Big Data Analytics Maturity Level

An Enterprise Approach to Evaluating Complex Systems Using Big Data Analytics

An Enterprise Approach to Evaluating Complex Systems Using Big Data Analytics Ryan Norman TRMC Initiative Lead for Big Data & Knowledge Management PM, TENA SDA ryan.t.norman.civ@mail.mil Presented at ITEA