DIAS Information Day. DIAS Information Day. DIAS project & technical details EUMETSAT, ECMWF and Mercator-Océan DIAS

Transcription

1 DIAS Information Day DIAS Information Day DIAS project & technical details EUMETSAT, ECMWF and Mercator-Océan DIAS 1 EUM/TSS/VWG/17/955046, 10/11/2017

2 Agenda Development schedule Targeted user community DIAS use cases Building blocks and responsibilities Planned industrial procurements Current status of technical developments List of elements (infrastructure, architecture, designs, interfaces, software) Timeline for implementation 2 EUM/TSS/VWG/17/955046, 10/11/2017

3 Copernicus Eco-System (Context) Data Distribution Services S1 S2 S3 S4 S5 S6 Sentinels Data/Information Access Services (DIAS) Sentinel Data Service Information Service Information Land Data Marine Distribution Core Atmosph Services Emergen ere Security cy Climate Sentinel Data ESA Sentinel Data EUMETSAT Contributing Missions In-Situ Data Sentinel Data ESA Sentinel Data EUMETSAT 3 EUM/TSS/VWG/17/955046, 10/11/2017

4 Key Principles the basis of the DIAS The key underlying principles of the 3-partite DIAS are: Architecture on the basis that underlying Data, Resources and Knowledge are and will remain distributed. open & replicable architecture Capitalises on existing Capabilities, skilled service experience, science infusion - Funds are spent where it counts: robust, high performing, future proof platform needed to absorb the uptake 3 partners with complementary but similar operational experience, with organised, pre-existing users communities Procurement approach with strong industrial involvement, securing healthy competition, while opening for a variety of future business models in the mid-term future An initial version (V0): proof of concept and risk reduction An evolutionary implementation approach (V1, V2, V3) minimising the risks, and capitalising on user feedback 4 EUM/TSS/VWG/17/955046, 10/11/2017

5 Development schedule - heritage EUMETSAT - Data Services Roadmap DIAS V0 Establish Information Centric Service Infrastructure (ICSI) New data services & tools for EUMETSAT missions/ users -Online Data Access (OLDA) -Licensed WMS (LWMS) -Hosted Processing -Format Toolbox Federated ICSI => Distributed Cloud Service Infrastructure (DCSI). Maximise re-use of already existing elements (Roadmap, partner) DIAS V1/2/3 ITTs Address all use cases Address all services Complete Data Portfolio API for industry services & market place 5 EUM/TSS/VWG/17/955046, 10/11/2017

6 Development schedule - timeline Authorisation to Proceed Design V0 release V0 impl., test & Ops Existing industrial support framework (EUMETSAT) Procurements Industry KOMs V1 release V1 impl. & Test V2 impl. & Test V2 release V3 impl. & Test V3 release Operations preparation, Operations, User uptake support Target industrial Framework (ECMWF- EUMETSAT-Mercator) 6 EUM/TSS/VWG/17/955046, 10/11/2017

7 Development schedule - Lifecycle 7 EUM/TSS/VWG/17/955046, 10/11/2017

8 Targeted user community - Actors & Roles End User Application User Consumes services (data, processing) Consumes services (data, processing); hosted upon Other Systems Consumes services DIAS data to Data Provider data, processing and connectivity DIAS Platform 8 EUM/TSS/VWG/17/955046, 10/11/2017

9 Target User Community Small Medium Enterprises (SMEs) 3rd party application developers Collaborative Ground Segments Member States/ International Organisations Sectorial industry H2020 projects Science projects/ user 9 EUM/TSS/VWG/17/955046, 10/11/2017

10 DIAS Use Cases Use Case 6* User support, registration and service desk Use Case 1 * User can discover and view data, information and products and use data access services Use Case 7 Use Case 2 * User can upload data and apply existing tools and processing functions Use Case 5 User can set up his/her own development environment Marketplace/incubator for science to operation or third party start-ups Use Case 4 * Provide hosted processing services to the user s application software or algorithm Use Case 3 User can set up a front office and interact (with the back office) through DIAS interfaces * - Use Cases covered in V0 10 EUM/TSS/VWG/17/955046, 10/11/2017

11 Use Case 1 - Data and Information (Discovery and Access) UC1.1 - User can discover satellite data and service information (collection catalogue); UC1.2 - User can subset satellite data and service information (narrow by time/aoi); UC1.3 - User can access satellite data and service information (access -> read where it is) UC1.1 User can discover data and information End User UC1.2 User can subset data and information UC1.3 User can access data and information (via GUI, VM, Interactive Notebook) 11 EUM/TSS/VWG/17/955046, 10/11/2017

12 Use Case 2 Use of Existing Tools Tools can be applied to satellite data and service information: An Interactive tool on the Desktop, launched in a Virtual Machine on the Hosted Processing computing environment (SNAP, QGIS) Web based Tools such as Jupyter Notebook From a Virtual Machine of the Hosted Processing computing environment. Pre-installed tools such as SNAP, programming scripting languages such as Python/R, and format readers for shapefile, netcdf and GRIB Web Processing Services (WPS) End User UC2 User can upload data and apply existing tools and processing functions 12 EUM/TSS/VWG/17/955046, 10/11/2017

13 Use Case 4 Hosted Processing (Orchestration) Hosted processing services: Virtual Machine created from templates, including access to data User are granted access to resources for bulk processing Physical resource and usage accounting Virtual Machine access based on a hierarchical model (Organisation, Project) End User UC4 Provide hosted processing services to the user s application software or algorithm 13 EUM/TSS/VWG/17/955046, 10/11/2017

14 Use Case 6 Web Portal (RA) User support, registration and service desk: User online self-registration via Web Portal User authentication -> data access, hosted processing, forum and Jupyter Notebook services Portal services with catalogue for data/ information and service registry User Support to include tutorial for data, information and services Forum for knowledge exchange UC6.1 User online self-registration UC6.2 User authentication End User UC6.3 Portal Services UC6.4 User Support 14 EUM/TSS/VWG/17/955046, 10/11/2017

15 EUMETSAT & Partner DIAS Report to EC DIAS Building Blocks and Responsibilities Marketing Service validation User Management User mgt support Web Portal Monitor, Report Mercator- Ocean Access Services Processing & Data Processing Tools ECMWF Interfaces Compute, Storage, Network Infrastructure Elasticity System Integration Verification EUMETSAT 15 EUM/TSS/VWG/17/955046, 10/11/2017

16 Mercator Ocean User Management User Management blocks will be covered by upcoming ITTs (see later slides) User uptake and marketing Web Services Validation & Operations 16 EUM/TSS/VWG/17/955046, 10/11/2017

17 ECMWF Data, Processing & Tools Data Access, Processing and Tools 17 EUM/TSS/VWG/17/955046, 10/11/2017

18 Data access service Large amount of data Different protocols, different formats Some data may be off-line (tape) Data not organized to fulfil all use-cases efficiently Data subject to licensing Statistics, audit trail of who access what 18 EUM/TSS/VWG/17/955046, 10/11/2017

19 Data access service (cont.) A single, unified REST API to access all data HTTP(s) based, JSON requests Programing language agnostic Data agnostic Object stores will be considered A catalogue Queriable via an API 19 EUM/TSS/VWG/17/955046, 10/11/2017

20 Tools and processing services Based on existing tools Libraries and CLI Also interactive GUIs Leverage big data technologies Hadoop, Apache-Spark Python/R based Numpy, scipy, matplotlib, pandas, xarray Jupyterhub, dask, 20 EUM/TSS/VWG/17/955046, 10/11/2017

21 Tools and processing services (cont.) A catalogue of pre-defined VM images Pre-installed with tools relevant for use cases Libraries Desktop applications Pre-defined elastic clusters for Big-data processing Pre-installed with Apache-Spark, Dask, Hadoop, etc All images will contain data access API clients Images can be specialized Satellite, Climate, Oceanography, Image will be created to cater for different use cases User will instantiate VMs from these images Accounting and billing 21 EUM/TSS/VWG/17/955046, 10/11/2017

22 Everything is accessible via a REST APIs Everything is a resource Data and product catalogue Data access Image catalogues VMs, virtual disks, virtual network connectivity Everything is managed by REST APIs When possible, API must fit in existing frameworks (e.g. OpenStack) Web frontends will use the same APIs 22 EUM/TSS/VWG/17/955046, 10/11/2017

23 Tools and processing services (cont.) Catalogue of Images Satellite VM Climate VM Instantiate Ocean VM. VM Hadoop cluster VM VM VM Apache-Spark 1 2 cluster 3 VM 4 VM 4 VM 5 VM 5 ECMWF EUMETSAT MERCATOR Public Cloud Climate VM VM 1 VM 2 Apache-Spark cluster VM 1 VM 2 VM 3 VM 3 VM 4 VM 5 Common Data Access API EUMETSAT ECMWF MERCATOR ESA Data and Products Catalogue 23 EUM/TSS/VWG/17/955046, 10/11/2017

24 EUMETSAT Infrastructure & System 24 EUM/TSS/VWG/17/955046, 10/11/2017

25 EUMETSAT - DIAS V0 System -Built on EUMETSAT Data Services Roadmap concepts (DCSI) Elasticity Provider DIAS Users -Sites are high speed interconnected (LAN speed) CMEMS DaaS SaaS PaaS IaaS PoP & Elasticity Portal DIAS V0 -Central Cloud Broker to manage *aas Layers DIAS V0 (DCSI) -Data access via PoP (Point of Presence) CAMS C3S PoP DaaS SaaS PaaS IaaS DaaS SaaS PaaS IaaS PoP S-3 -Harmonised data access services -Central portal for federation elasticity -An industrial cloud provider provides inner elasticity 25 EUM/TSS/VWG/17/955046, 10/11/2017

26 DIAS OpenStack based Cloud Management & Brokerage Layer Cloud Operation OpenStack Orchestration OpenStack Orchestration OpenStack Orchestration OpenStack Orchestration Nova Nova Nova Nova Cloud Infrastructure Neutron Neutron Neutron Neutron KVM KVM KVM KVM Hardware Infrastructure Compute Storage Network Compute Storage Network Compute Storage Network Compute Storage Network EUMETSAT ECMWF Mercator-Ocean Commercial Elasticity provider 26 EUM/TSS/VWG/17/955046, 10/11/2017 EUMETSAT, ECMWF, Mercator Ocean DIAS

27 DIAS V0 DCSI + DCSI Platform EUMETSAT ECMWF Mercator Ocean Elasticity Cloud Provider DCSI Portal User Support & Registration Processing Services and Tools Data Access Infrastructure (Cloud Brokerage) DCSI Platform Infrastructure (Dedicated Resources) Infrastructure (Elasticity Cloud Provider) 27 EUM/TSS/VWG/17/955046, 10/11/2017

28 DIAS V0 Data/ Service Information and Resources Satellite Data S3 Marine Data L1/L2 (EUMETSAT) S1/S2/S5P (Access/ harvest from CNES/DLR) Service Information Arctic Zone (from all partners 80 products) Sentinel (Sea Level, Observation SST, Models) altimetry, temperature, ocean colour, sea ice, atmospheric characteristics Initial start with Physical Resources TB elasticity space, beta users, 500 pcore, 15TB RAM Ramp up during V0 operations Enlarge dataset coverage, IaaS resources 28 EUM/TSS/VWG/17/955046, 10/11/2017

29 DIAS APIs DIAS V1 Eco-System User Support, HelpDesk, Marketing DaaS SaaS PaaS IaaS PoP DaaS SaaS PaaS IaaS Services Marketplace Community/ Commercial Portal DaaS SaaS PaaS IaaS Elasticity Portal H2020, SMEs, 3rd Party Apps, Marketplace, Coll Ground Segment(s) Users PoP DaaS SaaS PaaS IaaS DaaS SaaS PaaS IaaS PoP External Data Provider 29 EUM/TSS/VWG/17/955046, 10/11/2017

30 Current Status of technical developments EUMETSAT Data Services Roadmap (Pathfinder): internal architecture and design review passed platform infrastructure procurement in progress on site acceptance of platform at end of January 2018 DIAS V0 Functional architecture and design review passed Deployment readiness review under preparation DIAS V1/2/3 ITT preparation 30 EUM/TSS/VWG/17/955046, 10/11/2017

31 Foreseen V1/2/3 procurement areas Procurement area Description 1. Commercial Cloud services Elasticity and virtual data store (including COMMs) 2. Data access functions Harmonised data access layer to access Earth Observation data, Copernicus information services, and future third-party data 3. Virtual environment and toolboxes Providing virtual environment for communities to run/develop their SW using toolboxes offered by the DIAS 4. Web Services Portal, interoperability services. Software engineering 31 EUM/TSS/VWG/17/955046, 10/11/2017

32 Foreseen V1/2/3 procurement areas Procurement area Description 5. User Uptake and marketing User uptake and marketing of DIAS 6. System Integration & Verification Overall DIAS system level activity (e.g. HW, Networks, Security, Engineering, System Verification, Maintenance) 7. Service Validation and Operations Preparing the DIAS for operations, validating the services and supporting in the operations phase until end of V3 Notes: Tenders will be distributed across the partners and areas potentially be grouped Scope will be V1-V3 for each block Tenders are planned to be issued by Q Dedicated industry day will be planned for complex ITTs (platform & exploitation) 32 EUM/TSS/VWG/17/955046, 10/11/2017

33 DIAS V0 API (interoperability) Front Office Third Party Platform End User Legacy System Everything based on REST APIs for usage as DIAS API and Portal 33 EUM/TSS/VWG/17/955046, 10/11/2017

34 Timeline for implementation DIAS V1 ITTs (Q1 2018) DIAS V0 Release Q1-Q2/2018 DIAS V0 Operations from Q2/2018 DIAS V1 Implementation in 2018 DIAS V1 Operations from Q1/2019 DIAS V2/3 beyond 2019 (every 9 month) 34 EUM/TSS/VWG/17/955046, 10/11/2017

35 Access to DIAS Information DIAS Website (News and Information) Planned to release technical information/ packages through dedicated website. First design information will be released by end of EUM/TSS/VWG/17/955046, 10/11/2017

36 Q&A Session Questions? 36 EUM/TSS/VWG/17/955046, 10/11/2017