DICE: Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements

Transcription

1 DICE: Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements Daniel Pop Institute e-austria Timisoara, Romania Project Coordinator: Giuliano Casale Imperial College London, UK DICE Horizon 2020 Project Grant Agreement no Funded by the Horizon 2020 Framework Programme of the European Union

2 Motivation o Software market rapidly shifting to Big Data 32% compound annual growth rate in EU through % Big data projects are successful [CapGemini 2015] o ICT-9 call focused on SW quality assurance (QA) ISTAG: call to define environments for understanding the consequences of different implementation alternatives (e.g. quality, robustness, performance, maintenance, evolvability,...) o QA evolving too slowly compared to the technology trends (Big data, Cloud, DevOps...) DICE aims at closing the gap DICE 09/22/2015 DICE RIA - Overview 2

3 DICE 09/22/2015 DICE RIA - Overview 3 Overview and goals o MDE often features quality assurance (QA) techniques for developers o How should quality-aware MDE support dataintensive software systems? o Existing models and QA techniques largely ignore properties of data o Characterize the behavior of new technologies o DICE: a quality-aware MDE methodology inspired by DevOps for data-intensive cloud applications

4 DICE Project o Horizon 2020 Research & Innovation Action (RIA) Quality-Aware Development for Big Data applications Feb Jan 2018, 4M Euros budget 9 partners (Academia & SMEs), 7 EU countries DICE 09/22/2015 DICE RIA - Overview 4

5 High-Level Objectives o Tackling skill shortage and steep learning curves Data-aware methods, models, and OSS tools o Shorter time to market for Big Data applications Cost reduction, without sacrificing product quality o Decrease development and testing costs Select optimal architectures that can meet SLAs o Reduce number and severity of quality incidents Iterative refinement of application design DICE 09/22/2015 DICE RIA - Overview 5

6 Some Challenges in Big Data o Lack of quality-aware development for Big Data o How to described in MDE Big Data technologies o Spark, Hadoop/MapReduce, Storm, Cassandra,... o Cloud storage, auto-scaling, private/public/hybrid,... o Today no QA toolchain can help reasoning on data-intensive applications o What if I double memory? o What if I parallelize more the application? DICE 09/22/2015 DICE RIA - Overview 6

7 DICE 09/22/2015 DICE RIA - Overview 7 in a DevOps fashion o Software development methods are evolving o DevOps closes the gap between Dev and Ops From agile development to agile delivery Lean release cycles with automated tests and tools Deep modelling of systems is the key to automation Agile DevOps Development Business Dev Ops

8 DICE 09/22/2015 DICE RIA - Overview 8 Demonstrators Case study Domain Features & Challenges Distributed dataintensive media system (ATC) Big Data for e- Government (Netfective) Geo-fencing (Prodevelop) News & Media Social media E-Gov application Maritime sector Large-scale software Data velocities Data volumes Data granularity Multiple data sources and channels Privacy Data volumes Legacy data Data consolidation Data stores Privacy Forecasting and data analysis Vessels movements Safety requirements Streaming & CEP Geographical information

9 Bringing QA and DevOps together SPE Requirements SLAs Cost Tradeoffs Compare Alternatives DICE Profiling Load testing User behaviour Regression Incident Analysis Bottleneck Identification Root Cause Analysis Monitoring Deployment Capacity Management DICE 09/22/2015 DICE 09/22/2015 Adaptation DICE RIA - Overview 9

10 DICE 09/22/2015 DICE RIA - Overview 10 DevOps in DICE: Measurement incident report Ops Deployment & CI DIA Node 1 MySQL chef release NoSQL Dev jenkins (performance unit tests) Users DIA Node 2 S3 monitoring and incident report

11 DICE 09/22/2015 DICE RIA - Overview 11 DevOps in DICE: Early-stage MDE early-stage quality assessment Ops Deployment & CI incident report DIA Node 1 MySQL chef release NoSQL Dev jenkins (performance unit tests) Users DIA Node 2 S3 monitoring and incident report

12 Quality Dimensions o Reliability Availability Fault-tolerance o Efficiency Performance Costs o Safety & Privacy Verification (e.g., deadlines) Data protection DICE 09/22/2015 DICE RIA - Overview 12

13 DICE 09/22/2015 DICE RIA - Overview 13 Quality-Aware MDE UML MARTE profile, UML DAM profile, Palladio, Usage Profile System Behaviour Failure Probability

14 DICE 09/22/2015 DICE RIA - Overview 14 Quality-Aware MDE Platform-Indep. Model Domain Models Simulation Tools MARTE Architecture Model QA Models Platform Description Platform-Specific Model Cost Optimization Tools Code stub generation

15 DICE 09/22/2015 DICE RIA - Overview 15 DevOps in DICE: Enhancement continuous quality engineering ( shared system view via MDE) release Ops Deployment & CI incident report chef DIA Node 1 MySQL NoSQL Dev jenkins (performance unit tests) incident report & model correlation Users DIA Node 2 S3 continuous monitoring and enhancement

16 DICE Integrated Solution DICE DICE IDE Platform Description MARTE Deployment & Continuous Integration Platform-Indep. Model Architecture Model Platform-Specific Model Domain Models Data Awareness Continuous Enhancement QA Models Data Intensive Application Continuous Monitoring DICE 09/22/2015 DICE RIA - Overview 16

17 Year 1 Milestones Milestone Baseline and Requirements - July 2015 [COMPLETED] Architecture Definition - January 2016 Deliverables State of the art analysis Requirement specification Dissemination, communication, collaboration and standardisation report Data management plan Design and quality abstractions DICE simulation tools DICE verification tools Monitoring and data warehousing tools DICE delivery tools Architecture definition and integration plan Exploitation plan DICE 09/22/2015 DICE RIA - Overview 17

18 DICE 09/22/2015 DICE RIA - Overview 18 Thank you