SCFE final meeting - Newport. Agenda. Day 1: Wednesday 29 November. Day 2: Thursday 30 November. Day 3: Friday 1 December

Transcription

1 2018/01/25 08:23 1/9 SCFE final meeting - Newport SCFE final meeting - Newport Agenda The main objective of the meeting is to present the final results of the ESSnet. Day 1: Wednesday 29 November Time Item Topic Presenter 09:00 1 Welcome ONS 09:30 2 Introduction Eurostat - Insee 10:00 3 WP1 ONS 11:15 Coffee break 11:30 4 WP2 Statistics Lithuania 12:45 Lunch 14:00 5 WP3 Insee Destatis SURS CASD 15:15 Coffee break 15:30 6 WP4 SURS 16:45 End of day 1 Day 2: Thursday 30 November Time Item Topic Presenter 09:30 7 WP5 INE 10:45 Coffee break 11:00 8 WP6 Insee 12:00 Lunch 13:00 9 Future All 14:30 10 Service demos 15:45 Coffee break 16:00 11 Service demos 17:30 End of day 2 Day 3: Friday 1 December Time Item Topic Presenter 09:30 12 Service demos 12:00 End of meeting

2 Last update: 2018/01/24 08:13 Minutes wp6:meetings:newport Introduction The objectives of this last meeting is to present the fnal results of the Work Packages and to decide the status of these deliverables. What will be done with the legacy of the project must also be decided. The next SCFE Essnet will be,launched very soon. Invited external experts are from Hungary and Malta. WP1 - Guidance The objective of the WP is to provide guidances and best practices about how to share services amongst the NSI community. The idea is that all sharing is good but there are different level of resistance/effort : Sharing applications > low effort containers > moderate effort code > moderate effort (shyness about the code) replicated services > moderate effort shared services > high resistance/effort (legislation about where the data live ) guidance, concepts and technology choices > low effort documentation and metadata > moderate effort data and infrastructure > high resistance (for shared services you need to share data and infrastructure) D1-1 : final guidelines for the identification, devlopment, sharing and implementing statistical serices in local information system architecture and on an ESS SOA platform D1-1 presents frameworks, especially about interoperability (organisational, semantic, technological, legal). Legal interoperability is a real problem to tackle, as was seen in WP3, this subject should be discussed in the ESS. For sharing and reuse there are organisational barriers, legal barriers, technical barriers and communication barriers The deliverable Identifies 4 main types of barriers, 10 key recommendations, 13 recommended measures for central bodies and 16 supporting instruments. There are many frameworks (mainly from UNECE and Eurostat), they partly overlap and the complete panorama is really complex. A paragraph for each framework to explain for what it can be used would be very useful. WP1 proposes to foster : Printed on 2018/01/25 08:23

3 2018/01/25 08:23 3/9 SCFE final meeting - Newport Open source standards & licences Easy access to repositories (inc. Service Catalogue) Explicitly defined interfaces and well understood patterns Investment in documentation and packaging Addressing blockers: legal and linguistic Taking part: plan, promote and socialise The question now is the question of the maintenance of the guidelines which are pretty comprehensive. Eurostat's answer : it will be the mission of the future CoE. UNECE's answer : CSPA update next year, the results of WP1 will be an input for this. Eurostat : there are links between part 2 and WP4, and what it is said on open source and WP5 Guidance for ESS central hosting : NSI proposes the service, Eurostat approves and configures the service, DIGIT hosts the service, ESS uses the service. The DIGIT has provided a document on the technology stacks which are available (LAMP/T, WebLogic, ColdFusion, Oracle RDBMS, Tomcat, Oracle SOA/BPM, MS SQL Server, Win app housing, Unix app housing). Insee and Eurostat are in the process of installing the QGS at the DIGIT (it is installed bot not yet exposed). Until now there is no funding issue for configuring, deploying and hosting. But what about maintenance and new developments? Status of the deliverable : Version 0.9 on CROS Version 0.93 on the cloud Version 1.0 will be delivered before the 10th of December in order to receive feedbacks to upload the final version on the cloud on the 20th of December. This will be version 1.0, version to be approved by Eurostat. D1-2 Final report : templates for describing statistical services in conceptual, logical and physical services in conceptual, logical and physical levels Decision : only one global set of CSPA templates. It is suggested to include applicable standards, used architectural patterns, to provide recommendations on approach to reuse and to formalise/include the methodologies implemented. The templates could be updated during the next CSPA update. If it helps in usability, the templates could (should?) be updated. Next step : write a note to provide recommendations about the need for specific templates above CSPA templates. The note will be send by the 20th of december. D1-3 final report : Hands on How to expose a statistical service on an ESS

4 Last update: 2018/01/24 08:13 wp6:meetings:newport SOA platform and how to integrate a service exposed on an ESS SOA platform in a local business processes Mzny examples come from WP3. The how-to guide is more practical than the guidances (slides, step by step) and complemented by reuse posters. There will be a final update taking in the lessons learned from WP3-5. It would be useful to provide a check-list. Other material have been provided by WP1 : guidance on service granularity, guidance on types of reuse, presentation at SCFE Wiesbaden Workshop. Slides could be provided, the idea is to provide also a checklist. Eurostat : if there are only slides, it is OK if comments are provided with the slides. On the 10th of december the slides with comments will be circulated. If possible a pictogram and/or checklist will be provided. WP2 - Centre Of Excellence Status of the deliverable : Year 2016 : first version of the document. Year 2017 : update of the document. Version D2-1bis is on CROS. The final version including the last comments will be provided on the 8th of December on the cloud in both versions (PDF and doc) WP3 - Reusing services The objective of the WP3 is to put into practice the reuse of a service between a developing organisation (DO) and a reusing organisation (RO). Time Series Service (TSS) : DO > NBB and Insee, RO > Destatis Questionnaire Generation Service (QGS) : DO > Insee, RO > SURS Metadat Dissemination Service (MDS) : DO > Insee, RO > CASD TSS in Destatis : TSS is a wrapping of some JDemetra+ functions into a REST web service. The input data model was GSIMify, and some securisation was made. The modifications are integrated int the JDemetra+ codebase. The service was integrated into an existing and rather complex workflow. The data are converted from SAS to Excel then into XML, then they are send to the Time Series Service. The result is returned in XML, transformed into SAS. It was necessary to have a CSPA adaptor, the National Bank of Belgium developed a Java component for this. Done so far : Printed on 2018/01/25 08:23

5 2018/01/25 08:23 5/9 SCFE final meeting - Newport Roadmap : First workflow (1 time series) OK Second workflow (1 time series + Calendar regression) almost OK 3rd workflow (multiple time series + Calendar regression) still a work in progress 01/01/2018 department testing 01/03/2018 in production. The result will be reported to the Seasonal Adjustment CoE in order to have their reactions. A further step could be to simplify the Destatis seasonal adjustment process. Questionnaire Generation Service : DDI as input, Eno module transform it into XForm in order to obtain a web questionnaire. Developed as part of a more global project of collect platform revamping. The QGS module is now standalone, open source and internationalized. And it has been completed with a design tool. The tool is linked with the metadata repository where is stored the DDI questionnaire description. Used in Insee for all business surveys. Will be extended for households surveys and french ONAs surveys. SURS has an existing web data collection portal (ESTAT). Pogues and ENO can replace the SDM questionnaire creator and the SDM questionnaire generator used in SURS. Next steps : review of internal questionnaire design standard make SDM module ENO DDI compliant integrate Pogues and SDM module Challenges : DDI is complex ENO DDI profile and XSD schema have to be developed New technologies (XSLT, XForms, Orbeon) Pogues is available only recently The review of internal questionnaire design has not yet begun SDM module upgrade has not started yet It is planned to generate with Eno the specification for Blaise developers. Next steps : redesign the metadata model (but after the ESSNet) Metadata Dissemination Service : MDS is a subpart of Insee's metadata repository. The part submitted for reuse is based on datalift and stores concepts, classifications, and statistical operations. Problems encountered by CASD :

6 Last update: 2018/01/24 08:13 wp6:meetings:newport metadata collected manually name mismatch between data users and producers no validation from producers of available products The objective is to avoid this burden. Insee has 4 levels (based on DDI 3) : family, series, operation, product. CASD has 3 levels : program, source, product Followed Roadmap : Benefits : model convergence common operation list common code list and operation categorisation defined reuse architecture CASD webpage specifications and development coherent views for data users and producers easier maintenance for CASD machine actionable metadata Next steps : mode detailed metadata (variable level) reuse by other data producers share the model with other stakeholders Lessons learned : Reuse is costly but beneficial : costs : have to document, translate, open source, and there are additional constraints (harmonise schedules between DO and RO) benefices : documentation, bug discovery, software architecture quality improvement Standards help a lot. But it is necessary to go further and agree on profiles. And there is a need of conversion between standards. Reusability is not only a technical subject : semantic, organisational, legal Good application packaging (e.g. Docker) facilitates re-implementation. WP4 - Initial list of services that are candidates for re-use in ESS Deliverable D4-1 subtasks : Analysis of work done in the UNECE CSPA project,ess EA TF and ESS work on standardisation as well Printed on 2018/01/25 08:23

7 2018/01/25 08:23 7/9 SCFE final meeting - Newport as work already done by other statistical organisations Development of the tool for assessing statistical services costs/benefit Tool will be circulated within other members of the consortium as well as with the Members of the Task Force shared SERVices before distributed within statistical community Data collection and analysis of results Setting up initial list of shareable statistical services The result is a final list of 32 services mapped to the GSBPM. Most of the services are in the Process part. The Questionnaire for the Survey was updated based on the feedback received in Ljubljana and discussion in the group. Simplification was proposed : less questions (replacing pairwise comparison between atributes), still based on AAA model. The questionnaire was sent to ITDG and DIME members to evaluate these services. Respondents selected at least 5 most attractive services for their NSIs. For each selected service respondents evaluate achievability and affordability with three grades (no, some, very). For EU Members, the first three chosen services are : record linking, disclosure control, content data validation. Some countries named their solutions that could be shared. Results of the WP4 deliverable D4-1 : Initial list of the services = PREPARED Ranking services with cost/benefit analysis = DONE CB methodology that can be used for selection and implementation of the service = ESTABLISHED Additional: mapping services with implementation tools (existing / solutions in development) = DONE Status of the deliverables : On CROS portal : D4-1, a last updated version will be delivered in december (11th) On the cloud : D4-2, a last updated version will be delivered in decmber (11th) WP5 D5-1 : Assess the current and planned Open Source Solutions (OSS) and projects for statistical production in the European Statistical System (ESS) Survey conducted during February Input from seventeen NSI s revealing a broad range of experience and level of maturity in OSS solutions usage.survey was issued to managers, directors and IT specialists of every NSI in the ESS. A list of predefined contacts was used. Conclusions : Although all NSI's are familiar with CSPA Application Architecture, only one potentially reusable/shareable statistical product is already aligned with CSPA Application Architecture : RMéS. The majority of NSI's conduct research/benchmark/comparative studies to include open source

8 Last update: 2018/01/24 08:13 wp6:meetings:newport products when evaluating solutions for statistical products. These studies are mainly economic and financial studies. No NSI declared to have an opensource strategy. historically, little relevance has been given to development interaction and licence type Only one NSI has open/community development interaction, and half of statistical products have proprietary license type With exception of jdemetra and SDMX Framework, the remaining statistical products only have a minor usage. Status of the deliverable : final (and the CROS) D5-2 : Criteria for assessing/benchmarking Open Source Solutions (OSS) and projects for statistical production in the European Statistical System (ESS) In alignment with the task force on VIP.SERV ESSnet it was decided that the cost benefit analysis should be derived from Attractiveness, Achievability and Affordability (AAA) method. Conclusions : Choosing metrics requires considerable thought and care to support specific business goals. A prior analysis, aligned with the context of Sharing Common Functionalities in the ESS, must be done in order to define the weight/rating/grades for Attractiveness, Achievability, Affordability method. Although some aspects of software development, applied to statistical products, are measurable, some kind of subjectivity is always present. It was possible to define criteria for assessing/benchmarking Open Source Solutions (OSS) and projects for statistical production in the European Statistical System (ESS). Status of the deliverable : final (on CROS) D5-3 : Assessing open source software solutions for statistical production and for their capability to become available in the ESS as a CSPA compliant statistical service Status of the deliverable : draft (on the cloud), final before the 20th of december Method for assessing : triple A. There is a need to compare with the triple A method defined in WP4. One is for demand, the other for supply, is it possible to harmonize the two methods? The Serv TF and external expert opinions are welcome. The method uses business metrics and software metrics. Recommendations, follow specific guidelines and phases : design the service be consistent and standard keep the service simple and task-oriented test and maintain the service document the service Printed on 2018/01/25 08:23

9 2018/01/25 08:23 9/9 SCFE final meeting - Newport register the service Many web-standards exist, it is not necessary to invent a new one for the statistical community. D5-4 : Guidelines and recommendations for development of training materials and for open source solutions and projects Status : draft (on the cloud), final before the 20th of december Posters It is decided to have a poster for each WP. For WP other than WP3, A1 is an appropriate format. For WP3, the proposed posters don't have all the same models. It is proposed to use the same model for all services. HLG - MOS feedbacks Main challenges for th directors (expressed during a sprint ) Go-ahead : resilience in transformation communication access to data and new data sources standard technology and infrastructure, keep up with the changes Standards : review of GSBPM and GSIM ; more integrated view of GAMSO and CSPA Sharing tools : producing a new version of CSPA ; architectural patterns Capability an communication : organisational resilience ; skill and training frameworks ; strategic communication framework ; issue and reputation management From: - SCFEWiki Permanent link: Last update: 2018/01/24 08:13