QualiMaster Project Presentation

Transcription

1 QualiMaster Project Presentation QualiMaster Consortium QualiMaster Information Package (D7.2) March 2014

2 Motivation Promise of high volume data stream analysis: New and faster insights from in-depth analysis of detailed data (high volume data streams) from multiple sources (multiple perspectives/factors) Some road blocks: Processing pipelines over very high volume data are already a challenge (volume, changing frequencies/bursts, noisiness) Processing infrastructures have to be configured for worst case (wasting of capacities) Systematic approaches for exploiting the potential of hardwarebased solutions are missing Value adding but resource demanding algorithms can often only be run offline (delay of information for decisions) QualiMaster Project, GA Project Presentation, March

3 Project Vision Vision of QualiMaster : make high volume real-time data processing a highly opportunistic process that flexibly exploits data sources reconfigurable hardware families of approximate algorithms in a configurable, demand-driven and adaptive way. QualiMaster Project, GA Project Presentation, March

4 The Big Picture Market-based real-time systemic risk monitoring based on in-depth co-dependence modeling and monitoring combined with use of social Web data for supporting and stabilizing the prediction of systemic risk A data processing infrastructure mastering reactive, proactive, and reflective adaptation based on available computation sources, load, and needs Pipieline Adaptation Financial Applications Big Data Analysis of financial data streams (hundreds of millions messages per second), social web sources (e.g. Twitter with about 5000 Tweets per second) and other sources relevant for financial risk analysis Systematic exploitation of reconfigurable hardware for stream data processing Configurable Hardware Algorithms & Quality Quality modeling for data stream processing pipelines and exploitation of algorithmic families with different quality/ performance trade-offs QualiMaster Project, GA Project Presentation, March

5 QualiMaster Project Goal Create a configurable data processing infrastructure for analytics over real-time, high volume, and bursty data streams which autonomously masters reactive, proactive, and reflective adaptations based on available computation sources, load, and needs Via runtime adaptation of data processing pipelines by Algorithmic families with different quality/performance trade-offs Exploitation of reconfigurable hardware for stream processing QualiMaster Project, GA Project Presentation, March

6 The QualiMaster Consortium Focussed: 5 Partners from 3 countries (DE, UK, EL) Balanced: 2 SMEs, three academic partners Complementing competences L3S Research Center, Leibniz Universität Hannover Maxeler Technologies Limited Stiftung Universität Hildesheim Telecommunication Systems Institute Spring Techno GmbH & Co KG QualiMaster Project, GA Project Presentation, March

7 Expected Outcomes Models and tool support for pipeline configuration - Based on algorithm families - Considering constraints - Defining adaptation space Novel methods for dynamic quality adaptation of data processing pipelines - Reactive, proactive and reflective adaptation - Also covering cross-pipeline settings Families of scalable data processing algorithms - Different quality/efficiency trade-offs - Well-defined quality parameters - Focus: algorithmic classes Techniques and tools for translating and optimizing data stream processing algorithms into reconfigurable hardware QualiMaster Project, GA Project Presentation, March

8 Expected Outcomes Configurable data processing software infrastructure Provides platform for adaptive pipeline execution Exploits general-purpose as well as reconfigurable hardware + Tools + algorithm management Instantiation for financial domain QualiMaster Project, GA Project Presentation, March

9 Financial Use cases Systemic Risk Analysis (Core Use Case) Motivation: Systemic risk is key factor for stability of financial markets Strong correlation of financial markets increases risk Aim: Real-time systemic risk monitoring Markets co-dependency modeling and monitoring Use of social Web data for supporting and stabilizing the prediction of systemic risk Risk Assessment for institutional financial clients Also uses multi-variant and multi-market approach (different target) Pre-trading risk analysis Real-time, real money trading risk assessment QualiMaster Project, GA Project Presentation, March

10 Main Challenges in the Project Research challenges in individual research areas Bridging Disciplinary Gaps Bringing stream processing (steps) to hardware Coupling adaptation & algorithmic families (algorithm development and evaluation vs. system building)... Really understanding the Application Requirements Capturing the complexity and requirements of Systemic Risk Analysis Going beyond existing solutions Integration into running infrastructure + evaluation QualiMaster Project, GA Project Presentation, March

11 Research Challenges - Data: Big Data challenges: Volume: terabytes of aggregated data Velocity: data volume is growing at a high rate Variety: heterogeneous sources (structures/unstructured) Real-time processing challenges: Latency: analysis performed on live streams with minimum latency Uncertainty: need to cope with the incompletence of the data (Social) Web data challenges: Noise: finding and filtering relevant data on the web Quality: data on the web is generated without quality control Privacy: exploiting user generated data without breaking privacy QualiMaster Project, GA Project Presentation, March

12 Research Challenges - Algorithms: Quality-aware adaptation Identifying, modeling and measuring quality parameters of algorithms Identifying, modeling and measuring trade-offs (cost vs. quality) Quality propagation, end-to-end quality requirements and cost constraints Reactive, proactive and reflective adaptations Combination of software and hardware Unified quality-aware model for software and hardware alternatives Identifying of what should/could be translated to hardware, and when? QualiMaster Project, GA Project Presentation, March

13 The Cloud Computing Approach: Web data crawling Filtering/ Sampling Sentiment analysis Trend predictions more load more computational resources less load less computational resources QualiMaster Project, GA Project Presentation, March

14 The QualiMaster Approach: Configurable and adaptable data processing pipelines alternative execution paths with different quality and cost constraints Web data crawling Filtering/ Sampling Sentiment analysis Trend predictions QualiMaster Project, GA Project Presentation, March

15 The QualiMaster Approach: Configurable and adaptable data processing pipelines alternative execution paths with different quality and cost constraints low load execution path x high quality output Web data crawling Filtering/ Sampling Sentiment analysis Trend predictions QualiMaster Project, GA Project Presentation, March

16 The QualiMaster Approach: Configurable and adaptable data processing pipelines alternative execution paths with different quality and cost constraints High load execution path y lower quality (bounded error) Web data crawling Filtering/ Sampling Sentiment analysis Trend predictions QualiMaster Project, GA Project Presentation, March

17 The QualiMaster Approach: Configurable and adaptable data processing pipelines alternative execution paths with different quality and cost constraints Can (part of) this be translated to hardware? Web data crawling Filtering/ Sampling Sentiment analysis Trend predictions QualiMaster Project, GA Project Presentation, March

18 Development Process: Iterative and incremental development process (RUP) Initial phase: requirements and design Two major iterations Final phase Evaluation Integration Initialization Phase Development 1st Iteration 2nd Iteration Finalization Phase QualiMaster Project, GA Project Presentation, March

19 Miletones Project structured by 6 Milestones QualiMaster Milestones MS1 Foundations 6 MS2 QualiMaster Approaches and Basic Infrastructure 12 MS3 Core Building Blocks for the QualiMaster Infrastructure 19 MS4 Interim QualiMaster Technology & Infrastructure 24 MS5 Advanced Building Blocks for the QualiMaster Infrastructure 31 MS6 Final QualiMaster Technology & Infrastructure 36 QualiMaster Project, GA Project Presentation, March

20 Milestones: Foundations MS1 Approach and Basic Infrastructure MS2 Initialization Phase Core components MS3 Interim Infrastructure MS4 Advanced Components MS5 Final Infrastructure MS6 1st Iteration 2nd Iteration Finalization Phase QualiMaster Project, GA Project Presentation, March

21 Stay Informed! QualiMaster Project, GA Project Presentation, March