Statistics Netherlands. AIS ESSnet Projectplan. Version 1.0.p3

Statistics Netherlands AIS ESSnet Projectplan Version 1.0.p3

Content 1 References... 2 2 Project definition... 3 2.1 Framework... 3 2.2 Aim of WP 4: AIS Data... 3 2.3 Pre-conditions... 4 2.4 Deliverables and milestones... 5 3 Project approach... 5 3.1 Task 1 Data access (Participants: NL, DK, EL, NO, PL)... 5 3.2 Task 2 Data handling (Participants: NL, DK, EL, NO, PL)... 6 3.3 Task 3 Methodology and Techniques... 7 4 Risks... 9 5 Project team... 9 ANNEX1: formal description WP4 by ESSnet Big Data programme...10 Confidential CBS 2012 Page i

1 References 1. Peter Struijs, ANNEX II STANDARD GRANT APPLICATION FORM 2015, ESSnet Big Data SGA-1, 26 February 2016, version 2.1 Confidential CBS 2012 Page 2

2 Project definition 2.1 Framework The ESSnet BIG DATA is part of the Big Data Action Plan and Roadmap 1.06 and it was agreed to integrate it into the ESS Vision 2020 portfolio. The related business case Big Data received the support of the ESSC at its meeting on 20 June 2015 in Luxemburg. The overall objective of the project is to prepare the ESS for integration of big data sources into the production of official statistics. The award criteria mentioned that the project has to focus on running pilot projects exploring the potential of selected big data sources for producing or contributing to the production of official statistics. Aim of these pilots is to undertake concrete action in the domain of big data and obtain hands-on experience in the use of big data for official statistics. Taking into account these objectives, this ESSnet is meant to go from exploration to exploitation of big data for official statistics. SGA-1 covers the first phase of the realisation of these objectives. A consortium of 22 partners, consisting of 20 National Statistical Institutes and 2 Statistical Authorities has been formed in September 2015 to meet the objectives of the project. According to the Framework Partnership Agreement (FPA) between the consortium and EuroStat, the project runs between February 2016 and May 2018. To concentrate the work as much as possible on the pilots, the consortium has organised its work around the pilots. More specifically, the consortium has subdivided its work into work packages (WP), each WP dealing with one pilot and a concrete output. This document further concentrates on WP 4: AIS Data. Below we describe the different tasks and deliverables for WP4 SGA 1. Estimate emissions (part of task 3) and task 4 (future perspectives) aren t described in this document, because the budget for these activities is part of the budget for SGA-2. The intention is starting SGA-2 around January 2017. 2.2 Aim of WP 4: AIS Data Aim of WP 4 is to investigate whether real-time measurement data of ship positions (measured by the so-called AIS-system) can be used 1) to improve the quality and internal comparability of existing statistics and 2) for new statistical products relevant for the ESS. Improvement of quality and internal comparability can be obtained e.g. by developing a reference frame of ships and their travels in European waters and then linking this Confidential CBS 2012 Page 3

reference frame, by ship number, to register-based data about marine transport from port authorities. These linked data can then be used for emission calculations. New products can be developed for e.g. traffic analyses. The added value of running a pilot with AIS-data at European level is that the source data are generic worldwide and data can be obtained at European level. Challenges ahead with this dataset are: obtaining the data at European level, processing and collecting the data in such way that they can be used for multiple purposes, and visualising the results of the traffic analyses. A part of this work package is also to look into AIS analyses done by others and to investigate the possibility of obtaining already processed data as input for creating comparable official statistics. Especially it is important to make contact with other public authorities. WP 4 is a cooperation of five National Statistical Institutes. In concrete these are the national statistical institutes of The Netherlands (Work package leader), Denmark, Greece, Norway and Poland. 2.3 Pre-conditions For delivering the results mentioned in section 3.4 in budget and time it s essential that AIS data on the European level is available for this pilot within a short time (before May 2016). It s also important the scheduled resources are really available for this project. This project is cooperation between 5 National Statistical Institutes. For communication it s very important that good WebEx facilities are available. There also has to be budget for two physical meetings between February and October 2016. This project is only one part of the ESSnet Big Data programme. It s also very important to stay in touch with the other workpackages. Because of this it s important that the workpackageleader can participate in the general meetings of the Co-ordination Group by WebEx (every 4 weeks) and at face to face meetings in Tallinn (June 2016), Brussels (November 2016) and Boedapest (February 2017). Confidential CBS 2012 Page 4

2.4 Deliverables and milestones In the first phase (SGA-1) of WP 4 we focus on delivering the products below: 1. Report on creating a database and creating a database with AIS-data for official statistics: possibilities and pitfalls (July 2016) 2. Report about deriving harbour visits and linking data from port authorities (January 2017) 3. Report about sea traffic analyses using AIS data (July 2017) Furthermore there are two important milestones for this phase of WP4: 1. Progress and technical report of first internal WP-meeting (May 2016) 2. Progress and technical report of second internal WP-meeting (October 2016) 3 Project approach SGA-1 is divided into 4 main tasks. These tasks are described below. 3.1 Task 1 Data access (Participants: NL, DK, EL, NO, PL) Aim of this task is to decide how European AIS data could be used for this project. The definition of European AIS data in this work package will be: AIS data from whatever base stations in Europe catch. There are several types of AIS messages such as position messages (current geographic position, speed, course, navigation status etc), statistic messages (ship s name, IMO number, call sign, ship type etc). In this task we have to investigate which of the AIS messages we can use for improving our statistics or to develop new statistics. Further more we have to investigate the possibilities of acquiring data from EMSA (to be coordinated with Eurostat) and, if European data are too costly or too hard to obtain, how national datasets can be obtained. This task will involve: exploration of the possibilities to collect the AIS data at a European level. For the pilot we focus on getting European data from: Dirkzwager We can get 6 months of AIS data from Dirkzwager for 2000,-. We expect that this data can be delivered in time for this pilot. In the future these data from Dirkzwager will cost a lot more, about 10.000 every month. That s why we also explore other alternatives for getting AIS: European Maritime Safety Agency (EMSA) We expect that delivering AIS data by EMSA will take a while. There are also some difficulties because EMSA prescribes that we aren t Confidential CBS 2012 Page 5

allowed to keep the data. After using the data we have to delete it. That s why we will use the Dirkzwager data for this pilot. But we start the process voor obtaining EMSA data. The EMSA data is free. Other alternatives Beside exploring the possibilities of getting EMSA data we also have to explore other possibilities for obtaining AIS data on European level. Each of the members of WP4 will look for alternatives. For example national AIS data can be obtained by the Hellenic Coast Guard. Maybe they also can provide AIS data on European level. In Norway another governmental institution, Kystverket (The Norwegian Coastal Administration, http://www.kystverket.no/en ), handles AISdata. They already make a lot of statistics based on AIS-data, also emission statistics. In this field the role of Statistics Norway (SSB) should be supplementary to Kystverket instead of trying to overlap what they are doing. On this background the contribution of Statistics Norway to WP4 won t be to analyse AIS data from scratch. Instead Statistics Norway can do the following in task 1: Make an overview of already public available AIS-based statistics in Norway (traffic statistics and emission statistics). Investigate the possibility of obtaining raw and processed AIS-data at a European level. 3.2 Task 2 Data handling (Participants: NL, DK, EL, NO, PL) Aim of this task is to process and store the data in a database in such a way that they can be used for consistent multiple outputs, like linking AIS-data with data from port authorities traffic analyses Inference of journeys from AIS data. Key elements of this task are: which programming language and environment should be used for transformation where will the data be processed (in each NSI, by NSIs at European level, by data holders) how can we create an environment which is easily accessible for all partners The SANDBOX by the UNECE seems to be a great platform for storing and analysing the AIS data. The SANDBOX is available for this project. After getting the data we store the data on the SANDBOX and explore what the possibilities and impossibilities are from using this SANDBOX. Confidential CBS 2012 Page 6

3.3 Task 3 Methodology and Techniques Develop traffic statistics: Linking with data from port authorities (Participants: NL, NO, PL) AIS-data may be linked to data from port authorities. Added value of linking AIS-data to data from port authorities is that the same reference population (= ship number) is used in all harbours. As the journeys and harbour visits of ships can be derived from AIS this linking provides the ESS information about the origin/destination of the cargo, too. Aims of this task are: to build a reference frame of ships in European water (based on AIS- data) to find out for each participant on this task how data from port authorities can be linked to AIS-data To check whether information improves the quality of current national and European statistical outputs (for example maritime statistics and Inland Waterway transport statistics) and provides more information about the origin/destination of the cargo. We mainly focus on the prescribed European statistics by Eurostat. First we make an overview of statistics (from each of the five Statistical Institutes participating in WP4) that may be controlled and/or improved using (processed) AIS-data. Secondly we will analyse processed AIS-data in order to study possible improvements of statistics in the list mentioned above and/or new statistics based on AIS-data. Finally we will look at the European dimension: are there comparable statistics? Note that implementation of the improvements and/or new statistics based on AIS-data isn t in scope of SGA-1. Traffic analyses (Participants: NL, NO, PL, EL) The number of ships during a certain time interval at certain coordinates (like inland waterways or at certain points at sea) can be calculated by AIS-data. This possibility will be explored because this information could be interesting for traffic analyses and economic analyses. Aims of this task are: calculate the number of ships at certain coordinates visualise the results to analyse variations in time Confidential CBS 2012 Page 7

Estimate emissions is also part of task 3, but the budget for this activity is part of the budget for SGA-2, that s why we didn t describe this task yet. For the same reason we also didn t describe task 4 (Future perspectives) yet. The intention is starting SGA-2 around January 2017. Table 1: activities WP4 Nr. Activities Deadline Hours planned for each participant NL DK EL NO PL Total: 960 480 320 536 320 Task 1: data access 80 30 30 86 20 1. Getting AIS Data by 1-5-2016 Dirkzwager Milestone: progress and 31-05-2016 technical report of first internal WP-meeting 2. Getting AIS Data by EMSA 1-10-2016 Task 2: data handling 150 150 50 50 50 3. Programming language and environment 4. where will the data be processed 5. how can we create an environment which is easily accessible for all partners 6. Deliverable 1: report on creating a database with AIS data for official statistics 31-07-2016 Task 3: Methodology and 630 300 240 400 250 Techniques 7. Develop traffic statistics 300 150 0 200 100 Milestone: progress and 31-10-2016 technical report of second internal WP-meeting 8. Deliverable 2: report about deriving harbour visits and linking data from port authorities 31-01-2017 9. Traffic analyses 330 150 240 250 150 10. Deliverable 3: report about sea traffic analyses using AIS data 31-07-2017 Work package leader 100 Confidential CBS 2012 Page 8

4 Risks In ANNEX II STANDARD GRANT APPLICATION FORM 2015, ESSnet Big Data SGA-1 [1] the significant risks identified by the ESSnet big data programme are described. The main risk for WP4 will be that the AIS data on European level won t be available on time. 5 Project team Project team Name Anke Consten Marco Puts Maarten Pouwels Olav Grøndal Christina Pierrakou Øyvind Langsrud Dominik Rozkrut Role Work package leader Data Scientist statistics Netherlands expert on maritime statistics statistics Netherlands Member Denmark National Statistical Institute Member Greece National Statistical Institute Member Norway National Statistical Institute Director of Statistical Office in Szczecin (PL), expert on maritime statistics The members of the project team will meet physically in May (Heerlen) and September (Copenhagen) 2016. Meanwhile they contact by WebEx (every 4weeks), phone or e-mail. Confidential CBS 2012 Page 9

ANNEX1: formal description WP4 by ESSnet Big Data programme Work package number 4 Start date: End date: 1.2.2016 31.7.2017 Title AIS Data Partner/co-beneficiary NL 120 DK 60 EL 40 NO 67 PL 40 Aim of this work package is to investigate whether real-time measurement data of ship positions (measured by the so-called AIS-system) can be used 1) to improve the quality and internal comparability of existing statistics and 2) for new statistical products relevant for the ESS. Improvement of quality and internal comparability can be obtained e.g. by developing a reference frame of ships and their travels in European waters and then linking this reference frame, by ship number, to register-based data about marine transport from port authorities. These linked data can then be used for emission calculations. New products can be developed for e.g. traffic analyses. The added value of running a pilot with AIS-data at European level is that the source data are generic word wide and data can be obtained at European level. Challenges ahead with this dataset are: obtaining the data at European level, processing and collecting the data in such way that they can be used for multiple purposes, and visualising the results. A part of this work package is also to look into AIS analyses done by others and to investigate the possibility of obtaining already processed data as input for creating comparable official statistics. Especially it is important to make contact with other public authorities. This work package may require data acquisition in collaboration with Eurostat. Methodological, quality and technical results of the workpackage, including intermediate findings, will be used as inputs for the envisaged WP 8 of SGA-2, in case SGA-2 will be realised. When carrying out the tasks listed below, care will be taken that these results will be stored for later use, by using the facilities described at WP 9. Task 1 Data access. AIS-data are available for national territories and the entire European territory. For example, AIS-data in the Netherlands are provided by RijksWaterStaat (a government agency which is part of the Ministry of Infrastructure and the Environment). It is expected that similar agencies in other countries have the AISdata for their national territories. At the European level, a dataset of AIS data is available at the European Maritime Safety Agency (EMSA). The advantage of using one AIS-dataset for the entire European territory is a) a better comparison of international traffic between the countries and b) more synergy as all participating countries work on the same dataset. A disadvantage is that these data are stored by private companies and handling fees have to be paid. Aim of this task is to decide how European data could be used for this project, to investigate the possibilities of acquiring data from EMSA (to be coordinated with Eurostat) and, if European data are too costly or too hard to obtain, how national datasets can be Confidential CBS 2012 Page 10

obtained. This task will involve: Exploration of the possibilities to collect the data at a European (or worldwide) level. Participants: NL, DK, EL, NO, PL Task 2 Data handling Aim of this task is to process and store the data in such a way that they can be used for consistent multiple outputs, like linking AIS-data with data from port authorities traffic analyses Inference of journeys from AIS data. Key elements of this task are: which programming language and environment should be used for transformation where will the data be processed (in each NSI, by NSIs at European level, by data holders) how can we create an environment which is easily accessible for all partners Participants: NL, NO, PL, and possibly DK (if national sources are used) Task 3 Methodology and Techniques Develop traffic statistics: Linking with data from port authorities. AIS-data may be linked to data from port authorities. Added value of linking AIS-data to data from port authorities is that the same reference population (= ship number) is used in all harbours. As the journeys and harbour visists of ships can be derived from AIS this linking provides the ESS information about the origin/destination of the cargo, too. Aims of this task are: to build a reference frame of ships in European water (based on AIS-data) to find out how data from port authorities can be linked to AIS-data to check whether information improves the quality of current statistical outputs and provides more information about the origin/destination of the cargo. Participants: NL, NO, PL Traffic analyses The number of ships during a certain time interval at certain coordinates (like inland waterways or at certain points at sea) can be calculated by AIS-data. This possibility will be explored because this Confidential CBS 2012 Page 11

information could be interesting for traffic analyses and economic analyses. Aims of this task are: calculate the number of ships at certain coordinates visualise the results to analyse variations in time Participants: NL, NO, PL, EL Estimate emissions (envisaged under SGA-2) This task involves following individual vessels through time. Consequently we can infer the journeys from the data. Combined with a model to estimate the emission of vessels (which depends on travel distance, speed, draught, weather conditions and characteristics of the vessel itself), emissions of e.g. CO 2 and NO x can be estimated per ship and per national territory. An advantage of doing these analyses on a European scale instead of the national level is that more precise estimates for emissions at national territories can be made. Aim of this task is 1) to infer journeys from AIS-data, 2) visualise the results, 3) combine these journeys with a model to calculate emissions and 4) estimate the impact of carrying out these calculations at the European level on the quality of emissions calculations. Task 4 Future perspectives (envisaged under SGA-2) Aim of this task is to summarise the project results and perform a qualitative cost-benefit analysis of using AIS-data for official statistics. These analyses should include aspects like sustainability of the data source, possibilities of improving international comparability, possibilities of datasharing (at micro- or aggregated level), quality improvement of current statistics and a sketch of a possible statistical process and needed infrastructure. Deliverables (SGA-1 only): 4.1 Report on creating a database with AIS-data for official statistics: possibilities and pitfalls month 6 4.2 Report about deriving harbour visits and linking data from port authorities month 12 with AIS-data 4.3 Report about sea traffic analyses using AIS-data month 18 Milestones (SGA-1 only): 4.4 Progress and technical report of first internal WP-meeting month 4 4.5 Progress and technical report of second internal WP-meeting month 9 Confidential CBS 2012 Page 12