ITDG 2012/3.1/EN IT DIRECTORS GROUP 27 TH AND 28 TH NOVEMBER 2012 Item 3.1 of the agenda Presentation of Enterprise Architecture
ESS EA: state of the art 1. Recommendation for action by the ITDG The ITDG is invited to take note of the ESS Enterprise Architecture developments so far to discuss the "to be state" from the angle of solution and technical architecture to suggest ways and priorities to proceed with further developments of the ESS EA 2. Background and brief history An EA is a framework for describing the "organisation of a system embodied in its components, their relationships to each other and to the environment, and the principles governing its design and evolution" (ANSI/IEEE definition). It shows how the business strategy can be expressed through IT and proposes a high level view of the organisation, comprising a picture of the current state, a to be state which is a blueprint or a vision for the future and a road-map on how to get there. The ESS EA aims at providing a framework for the implementation of the Joint Strategy adopted in May 2010 by the ESSC, in particular for aligning a series of initiative at ESS level ensuring the overall coherence and efficiency of the approach. The ESS is seen as a complex system of closely tied statistical authorities which have autonomy and where subsidiarity is strongly enshrined. The business case for an ESS EA was discussed at the 2011 ITDG meeting. There was a common understanding that the "to be state" providing more clarity of the target infrastructure is a key component of an ESS EA. In March 2012, a first sketch of the conceptual framework was discussed with the directors of methodology. To structure the "to be state", Eurostat presented at the ESSC in May 2012 4 strategic directions to further develop the common infrastructures (see Annex 1). A first list of enabler projects was identified and organised in a programme (see agenda item 3.2). This has led to a refinement of the "to be state" which was presented in September 2012 to the DIME Steering Group including few ITDG and Sponsorship on Standardisation members. Annex II describes the "to be state" and the conclusions of DIME SG discussion. 3. Policy context Since 2010, several Vision Infrastructure Projects (VIPS) and ESSnets have been launched bringing valuable results to the whole ESS. Despite the positive progress of these initiatives, there is a need for further streamlining the Joint Strategy implementation and strengthening the common ESS infrastructure. The aim is to build up common infrastructures with more consistent approaches. Instead of starting large number of projects, the idea is to focus on a few projects bringing high added value the Joint Strategy implementation. These new projects are called ESS-VIPs, because Eurostat emphasizes the need for combining the two streams of projects conducted so far Eurostat VIPs and ESSnets. The requirement for an ESS VIPproject is that its results are generic enough to be reusable in different contexts (e.g. in MSs or Eurostat 1 ITDG 2012/3.1/EN
in other statistical domains) to common ESS infrastructure and to Joint Strategy implementation. 4. Consequences for the NSIs The ESS EA aims at providing directions and clarity on the target common ESS architectures. MS are expected to contribute actively to its elaboration. In parallel, MS will have to evaluate the cost and benefit to connect/use the common infrastructure stat and to set up their strategy of convergence. MS should be ready to take active participation in the development of the common architecture building blocks. 5. Outstanding problems The governance for adapting the ESS EA has still to be put in place, in particular for guiding the development of the common infrastructure. 6. Risk assessment EA is still perceived as a too conceptual framework and as a costly overhead to standard project management. The current EA developments are still very much focused on technical development and lack a strong business representation which is still very much organised according to statistical domains. ESS EA should continue developing to meet expectations of efficient steering of ESS corporate projects 7. Next steps The next steps consist in. Continuing development the different ESS architectures (business, information, solution and technical infrastructure) in relation with the planned change and developments; Setting up an operational governance based on EA principles to accompany the development of the ESS VIP programme; Providing an adequate framework (function vs service matrix) allowing mapping the MS and the common infrastructure Eurostat 2 ITDG 2012/3.1/EN
Annex 1: ESS Entreprise Architecture : scope, basic principles and strategic directions 1. Business case for ESS EA ESS enterprise architecture aims at providing a framework to build ESS common infrastructures with more consistent approaches supporting the implementation of the Joint Strategy endorsed by ESSC in May 2010. The ESS is seen as a complex system of closely tied statistical authorities which have autonomy and where subsidiarity is strongly enshrined. Since then, several Vision Infrastructure Projects (VIPS) and ESSnets have been launched bringing valuable results to the whole ESS. Despite the positive progress of these initiatives, there is a need for further streamlining the Joint Strategy implementation and strengthening the common ESS infrastructure. Instead of starting large number of projects, the idea is to focus on a few projects bringing high added value the Joint Strategy implementation and to combine and prolong the two streams of projects conducted so far Eurostat VIPs and ESSnets into a limited set of new projects the called ESS-VIPS. The ESS VIP-projects should ensure the development of generic building blocks of the future common architecture to allow the system to become more agile and cost efficient in developing new projects. 2. Architectural principles and strategic directions A first sketch of the conceptual framework presented below was discussed with the directors of methodology in March 2012. A preliminary list of clusters of enabling projects was then conceived. These concepts and the list of projects was presented to DGs at the ESSC workshop back to back with the ESSC meeting in May 2012. The ESS Vision entails two fundamental principles related to the information and the process components of Official Statistics production. These principles will structure the development of common ESS architectures. These principles give rise to four strategic directions for the strengthening of common architecture, namely the network & communication infrastructure, information stores and services, a service oriented architecture for statistical production based on common building blocks/modules and protocol of communications allowing better vertical integration/cooperation of processes steps. Eurostat 1 ITDG 2012/3.1/EN
Information principle Information is meant to be a shared asset, where information flows are optimally organised. Information should be understood as set of data, metadata, and process descriptor objects. The information principle implies that Member states and Eurostat have secure access to shared information. The data sets are accompanied by information on processes that have generated those sets and their status with commonly agreed requirements. The data sets are submitted in standard format so they can be readily integrated in subsequent steps, including dissemination, and shared with other external stakeholders. Process principle Processes are meant to be interoperable and built on shared components. ESS will be provided with common reference architecture for building statistical production processes and component to be shared. Production processes are built as a set of standardised steps which interact smoothly and efficiently together leading to higher degree of automation. The reference architecture defines standards for designing processes, developing services and information objects to be shared and reused in processes and among partners. The collaboration among different stakeholders along the same production processes is organised in an efficient way. Eurostat 2 ITDG 2012/3.1/EN
Strategic direction: Network Organizing and supporting the flow of statistical information objects between national statistical institutes and Eurostat in network infrastructure. Information is flowing freely through secured connections and the information is modelled with international standards. Status (quality, history) of information objects can be obtained and information flows can be triggered by both sender and receiver. The network service ensures a good synchronisation of information throughout the stakeholders' network. Eurostat 3 ITDG 2012/3.1/EN
Strategic direction: Information stores Developing common platforms to share, process and redistribute information inside the ESS. Technology enables sharing of data and remote processing inside the ESS. Networks of databases can support the access to wider and interlinked sources of information (data warehouse) or access to shared data to be processed and cross validated. The information is to be organised according to common standards. Methods and related services should be standardised. Strategic direction: Modular production Eurostat 4 ITDG 2012/3.1/EN
Providing modular reference architecture to ESS. Modular reference architecture means statistical instruments and production processes, which are obtained by chaining a set of shared tools/services of Member states and Eurostat. This reference model for information and process should be compliant with the Generic Statistical Business Process Model (GSBPM) and the future Generic Statistical Information Model (GSIM) Strategic direction: Optimal collaboration Organising the collaboration efficiently along the production process; production of EU statistics requires involvement of various stakeholders from Member states, other public administration bodies and Eurostat. 3. ESS VIPS principles For each of the four strategic directions, a series of projects/programmes have been identified. They will pilot the implementation of the required common infrastructure. The work and deliverables of previously launched VIPs and ESSnets are utilized in the new ESS VIPS. The new ESS VIPs should Focus on the realisation of ESS business process integration and have a strong business case. Benefit ESS as a whole. Have ESS involvement and commitment. Have ESSC as High level sponsor. Be guided by enterprise architectures approaches to ensure coherence of development across projects. Be aligned with related initiatives, like High Level Group for Strategic Developments in Business Architecture (HLG-BAS) and Sponsorship on Standardization. Efficiently use resources. Eurostat 5 ITDG 2012/3.1/EN
Appropriate mechanisms should be put in place to ensure that cross-cutting issues like information models, common platform for shared services, network and communication infrastructure are appropriately handled and remains aligned with business requirements Eurostat 6 ITDG 2012/3.1/EN
Annex 2: ESS To be state WHERE DO WE WANT TO BE IN 10 YEARS TIME? In 10 years' time, the ESS will rely on a flexible architecture able to produce high quality statistics with fewer resources, responsive to the new statistical demands while not imposing additional burden to respondents and compilers. To successfully match this challenge, the ESS needs to set up a development strategy towards an "integrated" statistical system for European Statistics and selected national purposes, founded on subsidiarity and on common infrastructures and minimum standards for using the shared elements of the system. The basic idea is to conceive the ESS production system, and its sub-systems, around three metafunctions: collection, elaboration/computation (process), sharing/ dissemination of data and metadata. The system itself is organised around a set of services (functions) of statistical nature services covering operations to process and disseminate the data and logistic services to store, share the data and monitor the processes. The overall management (use and access) of services, including access to and sharing of information, will lean on a network infrastructure (BUS), with several layers, to insure the physical connectivity of the system, but also the interoperability of the different parts as well as the proper functioning and automation of the operations through process manager facilities The functioning of the system is granted by a governance structure and supported by the corresponding legal and methodological frameworks. Tab. 1 summarises the main characteristics of the "to-be-state" of the ESS in 10 years' time. What? (Targets) Basic sources 1. A common approach on how to integrate the different data sources in the data collections while ensuring comparability 2. ESS data sources fed by relevant micro-data from MS 3. A fully coordinated and consistent set of European registers Infrastructural elements - technical 4. A set of common standards, models and rules that facilitate the interoperability of the system e.g. DSDs; classifications; other internationally agreed standards; data and metadata models (SDMX, DDI,..); development principles; Service Level Agreements (SLAs) common dissemination standards; security 5. A common approach to data warehousing in the ESS interconnecting the relevant national and Eurostat data stores and a number of statistical areas having implemented this function Eurostat 1 ITDG 2012/3.1/EN
6. A set of common services to provide the functionalities of the system 7. A reference architecture and platform for sharing and interconnecting statistical services and a set of ESS statistical processes built on a sequence of common services operated by ESS partners 8. A system, where data are collected, validated and stored only ones and are readily accessible by authorised stakeholders and where appropriate final users 9. A number of statistical production systems that have been re-engineered using the solutions and services developed by the ESS programme Infrastructural elements - general 10. A governance structure for the system and its sub-domains able to ensure sustainability and flexibility in time. 11. A legal framework adapted and supportive of the system 12. A methodological framework establishing the foundations of the system 13. A collaborative approach to the use of resources in the system 14. A strategic framework for the development of the system and its monitoring 15. A communication strategy to support the development of the system addressing different stakeholders in an appropriate and inclusive way Table 1: The ESS production system in 10 years' time In order to have the system working, there is a need to define: Architectural elements: the architecture will be designed to balance the required integration with the subsidiarity principle. Some minimum standards will be fixed (regarding the way to structure the information for instance); some common infrastructures will be made available (regarding the network to use to exchange data in a secure way); a platform to share services will be proposed. The basic idea is that a full integration is not needed or even desirable; local developments subsidiarity - will continue to exist but the local systems will be interconnected with the target system. For example, solutions developed by ESS partners could be "wrapped" in a way that local systems can make use of them without having to change their legacy environment. It can, however, be expected that the local systems will connect at different speed, to the standard systems allowing them to fully profit from the common services offered by the target system this is particularly true for production solutions. Migration will be supported and corresponding resources duly integrated in the business case for common infrastructure and standards. The architectural elements include: Information models: data model for micro and aggregated data; standard for classifications and metadata, models for classifications, identifiers, documentation of process of quality, rules; system metadata models to allow entities to be identified, located and processed in the system. Eurostat 2 ITDG 2012/3.1/EN
Protocols for sharing information: common rules and procedures to share/disseminate data and metadata (statistical and system metadata) in the system (including confidentiality aspects). Governance elements: it is paramount in setting-up a new collaborative way for producing ESS official statistics to establish "who does what" and "who is responsible for what". Therefore, the governance of the programme has to be well defined taking into account a collaborative approach that makes use of the available knowledge/expertise/innovation in the system. The idea of competence centres can be developed, pooling resources together to develop and provide services to the entire system. Achievements by overarching standardisation initiatives at ESS level and beyond - provide effective mechanisms to translate into concrete actions the above governance principles and framework. Legal framework: in order to allow the target system to work in an efficient and optimised way, an adequate legal infrastructure has to support it (allowing for example, the access for production purposes to national micro-data by all the relevant actors, etc.). Methodological framework: the developments in the specific statistical areas as well as in the general way of producing statistics and the underlying paradigms have to be framed. Of particular importance would be the progressive shift in emphasis in the above mentioned standardisation framework as a vertebrating mechanism for the future Statistical System from common solutions to common/interoperable, as appropriate, methods and approaches. Resources: a collaborative approach among ESS actors to pool resources will be one of the key facilitating factors. Of particular importance for the success in the mid and longer term of this approach will be the due attention paid to boost transitional absorption capabilities by making available both appropriated mechanisms and temporary resources. It may also be considered to lobby the programme with the SG and other DGs and make a bit for additional resources at Commission level. Implementing strategy: the achievement of the overall targets has to be translated in a detailed roadmap which describes the role of the different actors, the milestones of the different projects and sub-projects, the intermediate targets and the way to monitor progress. As a golden rule, all actions will be polarised towards enhancing the genericity of the methods, approaches or solutions. Their domain independency will boost re-use and interoperability of sources, systems and production process facilitating the escalation to industrialisation, yet purpose tailored, of statistical production process. Communication strategy: the establishment and implementation of the Programme requires an important communicational effort towards stakeholders at different levels: technical, managerial, strategic. An adequate internal and external communication strategy will have to be conceived and set up coupled with a management of change approach. Respective communication aids have to be developed in a professional way. Change management: the set-up and implementation of the target system implies the establishment of a new culture for the compilation of European statistics. The transition from the old paradigm to the new one has to be planned and steered. Hence, it is paramount to prepare and manage this major change that will affect the daily work of ESS statistical producers. Professional methods have to be found how to take the ESTAT internal staff and the other ESS stakeholders and partners with us. The proposed model will have also to promote a cooperative approach with other major players such as the ECB and the ESCB, other national statistical authorities and selected private statistical actors. Eurostat 3 ITDG 2012/3.1/EN
In short, the ESS in 10 years time will look like a system with selected competence centres in place providing ESS services, with ESS data stores and data warehouses functioning at dissemination and production levels, mutually accessible by users, ESS producers and partners (according to different level of accessibility). Selected sets of micro-data and administrative data will be shared in the system. In order to achieve this target, all underlying operational and conceptual issues of data sharing, data storage, system communication, security, have to been solved in a timely fashion. This overall target system is illustrated in Fig. 1. Figure 1 - To-be-state of the ESS production and dissemination ARE WE THINKING BIG ENOUGH? The paradigm shift from national to integrated European official statistics is an ambitious project that the ESS VIPs Programme translates into a concrete development and implementation phase through the combination of business cases (ESS VIPs) and horizontal cross-cutting issues. The ESS VIPs Programme will progressively pave the way towards a real integrated system for European statistics based on Service Oriented Architecture. What are the ambitions of the ideal target system? Eurostat 4 ITDG 2012/3.1/EN
The paradigm shift from national to truly European statistics is not limited to specific statistical domains, as the example of the reform of Intrastat may suggest. There is a wider horizon and highlighting it will even help to communicate the shift. Therefore, any ESS VIP should have the wider horizon in mind in order to allow subsequent extensions to other statistical areas. Which areas could be concerned? The backbone of statistics will be a dedicated set of basic registers. For statistics with businesses as reporting units, the reference will be a European System of Business Register (or an integrated network of registers - ESBRs). Micro-data will be exchanged and validated due to their complex nature; there is a need for co-ordination and reconciliation, which only can be managed at the ESS level. For statistics with individuals as reporting units, the situation appears for the moment more difficult, but not less urgent. The difficulty stems from the high sensitivity of data protection issues as regards individuals, while the urgency stems from the unprecedented levels of intra-eu mobility and migration. A unique identifier for EU enterprises within a European business register would drastically facilitate cross-country compilation and analysis of statistics. The concept of "unique identifier" could be extended to other areas such as products and individuals. Imagining that those elements were centralised somehow, i.e. a European Business Register (including the ESBRs) and (some) micro-data were available; a huge potential for statistical analysis and validation at national and EU level would become available. In the area of business statistics, statistics on globalisation, global value chains and even European Accounts would become feasible, just by exploiting this potential. Many problems of mirror data could be solved (BOP, FDI, migration statistics). The further integration of social and economic statistics would then be easier. Issues that by nature have to be dealt with at EU level (PPPs, European Sampling, direct EU/EA accounts, data on European multinationals; environmental questions based on EU/EA input-output tables) could also be included in such a system to exploit synergies to the maximum extent. Users could be given transparent, well organised and secured access to anonymous microdata. Micro-data or very disaggregated data are already today sent to Eurostat in different areas such as social statistics, agriculture, prices and PPP, Balance of Payments, FDI and inputoutput tables. Why should not all micro-data form a system of specific data warehouses with appropriate confidentiality and access treatment ensured across the board? Why should not all micro data be shared, where applicable, between Member States for cross-checking and validation purposes? FIRST REACTION OF MSS AT THE DIME SG IN SEPTEMBER 2012? There was a general support on the progress made on the "to be state". However, MSs made following recommendation: 1. The strategy must be agreed at appropriate political level and take a careful and stepwise approach with focused experimental projects bringing European added value. Furthermore, the ESS governance must address flexibly the diversity of National approaches. The relevance, role and organization of Competence Centers must be clarified; Eurostat 5 ITDG 2012/3.1/EN
2. The "To Be State" should be process oriented, include a more explicit metadata and methodology layer and embed other highly relevant drivers, like Quality of Statistics. Eurostat 6 ITDG 2012/3.1/EN