Analytics and Big Data Management FTA use case

Transcription

1 and Big Data Management FTA use case Ilkka Kotilainen, Project Manager, Finnish Transport Agency (FTA) 7 June 2017, The Nordic Road Association, Digitalization Committee meeting

2 Content 1 and Big Data Management 2 FTA Roadmap and Operating Model FTA and Big Data examples Material modified from Pekka Kinnunen s presentation, FTA, Project Manager, FTA Team, pekka.kinnunen@fta.fi Ikonit: licensed by Creative Commons BY 3.0 Pekka Kinnunen 2

3 1 and Big Data Management Pekka Kinnunen 3

4 Growth of Data Big Data Big Data 3 Vs: Volume (amount), Velocity (speed) and Variety (structure) New Information Sources Possibilities for new analytics solutions Pekka Kinnunen 4

5 Data Methods, systems, technologies, practices and applications that analyze critical business data to help a firm to better understand its business and market Descriptive Diagnostic Predictive Suositteleva Prescriptive (Prescriptive) (& Business Intelligence) Human D e c is s i o Recommendation Decission Making Automation Parellel development of different levels A c t i o n Basic level: - Knowledge Management - Take advantage of the existing registry - Ad hoc analysis supporting decisission making Big data level: - Automated information analysis - Big data challenges old methods - Flexible information utilization - Fact based decission making Intelligent : - Artificial Intelligence makes decissions and makes recommendations e.g. for the road maintenance - Machine learnings for new environment adaption Pekka Kinnunen 5

6 ??? We have lot s of data, how do we use it? Where and how can I save data? We have lot s of business problems, how do I analyze those? Where and how can I find right kind of data for my business questions? Picture: Wikimedia Commons: 6

7 Use data to understand business CRISP-DM example 7

8 Maturity Picture: Davenport T. & Harris J., (2007). Competing on 8

9 2 FTA Roadmap and Operating Model Pekka Kinnunen 9

10 ??? We have lot s of data, how do we use it? Where and how can I save data? We have lot s of business problems, how do I analyze those? Where and how can I find right kind of data for my business questions? Research in FTA Methods to collect, store, share, search, analyze and present data by taking advantage of statistics and computer science Information needsexamples from the core business: Predicitive maintenance and analytics, HR travel data, planning,.. Picture: Wikimedia Commons: 10

11 FTA Big Data and roadmap between 2015 and Big Data pilots for data needs Big data platform: IBM Big Insights Data for the first time into the production system Big Data environment instructions service center starts Development Framework Contract is finalized Pekka Kinnunen 11

12 FTA Internal Resources Substance (business) Road nw Road tr contact team 1. Information sharing 2. Development actions - recognize - define - propose 3. Coordination 4. External networks coordination External Network PublicICT network Railway Rail tr contact coordinator Research Waterways Maritime contact service centre Data Architect FTA Operating Model and Network Development Framework contract Internal Process (Financial, HR, ) contact team resources Private vendors (e.g. benchmarking) - Expert knowledge - Business action based on knowledge and business questions - Business Intelligence and development Methods Processes Procurement models Practices for data mining, analytics, knowledge sharing and presentation - Collecting information needs and solving problems in the analytics network - Provide and offer analytics services to core business - Sharing of analytics knowledge between projects and facilities - Implementation of information needs - Solutions made possible by the network - Different solution models - New technologies and trends Two roles: 1) heavy users 2) development 12

13 FTA Internal Resources Substance (business) FTA Operating Model and Network Internal Process KNOWLEDGE contact contact contact contact DEVELOPMENT team coordinator Data Architect External Network IMPLEMENTATION 13

14 FTA and Big Data examples Pekka Kinnunen 14

15 Examples Big Data service architechture target state Big data environment project examples Ground Penetrating Radar (data) Road maintenance and monitoring (Road) Traffic flow induction data storage and publishing (Open data, production ongoing, hadoop) Automatic Identification System (AIS) data storage and publishing (Maritime) Reaaliaikadatan analysointi: LOU Ratamittaripilotti (Rail) Helsinki railway yard traffic situation issue analysis (Rail, predictive analytics) Social Media analysis: Ideas to benefit social media analysis (Road) Proof of Concept: Rail network malfunction consequence analysis (Rail) Proof of Concept: Rail network damage analysis (Rail) Data visualisation examples Pekka Kinnunen 15

16 Big Data service architechture target state Data Sources Acceleration and other sensor data Picture and video data Measurements: laser and radar Other big data * Not yet ready made solutions, small trials and pilots to find answers Raw Data Service Provider E - Storage - Analysis - Adaptation Cloud services* Organized Information FTA environment Big Data environment IBM Big Insights BigData Hadoop Operational Systems Open Data APIs Reporting environment IBM Cognos BI External utilisers/ developers Internal utilisers 16

17 Data source Traffic flow induction data storage and publishing (Open data, production ongoing, hadoop) Raw data FTA environment Big Data environment IBM Big Insights Open Data APIs External Measurement point BigData Hadoop Repoarting environment IBM Cognos BI Measurement data Internal 17

18 Helsinki railway yard traffic situation issue analysis (Rail, predictive analytics) Clustering Delay data Delay distribution Railway yard baseline - what is normal delay? Exceptions - correlation with the delay - not fully explaining Railways with highest effect on delays Delay data! Real-time data Delay the delays? Prediction - Replace the mode with real-time data What is the situation in 2 hours Modified model Delays are predicted based on delays from past 2 hours Causal relationship - Reason: staying too long on track - Effect: delay - If conditions above, then delays - Overall of six (6) rules like above Processed data: one hour average delay data - Over 10min = late - Below 10min = not late 18

19 Social Media analysis: Ideas to benefit social media analysis (Road) AWS Amazon Web Services, Hadoop Cluster Social media content Futusome - Service provider - APIs for Finnish Social Media channels Python Client - Timeline for data JSON storage, Hadoop (HDFS) Apache Hive - to Cognos ODBC FTA reporting portal 4) Social media messages listed by topic 1) Amount of search words 2) Most common adjectives and substantives 3) Timelines and trends 5) News headlines with comments - most commented - Most common substantives 6) What else? Cognos - Social media analysis visualisation 19

20 Data visualisation examples Time series Maps Processed time series Trends, time series Heat maps 20