KYTC Real-Time Data. Big Data for Incident Management

Transcription

1 KYTC Real-Time Data Big Data for Incident Management

2 Current and Future Use Cases Title 23 CFR Situational Awareness Incident Detection Snow and Ice Management Monitoring: Work Zone / Corridor Incident and Recovery Times Secondary Crash Detection Automated Bookkeeping* Environmental* Highway Safety Improvement Program* Signal Timing* Planning* Automation/ML*

3 Getting Started: 2014 Costly Winter(s) Ranging from $45mil to $80mil Downsizing of Public Sector, Do More With Less Management Wanted to Try Something Different Provide Comprehensive Understanding(s) Eliminate Silos and Ownership Concept Simply Data for End Users and Public Motto/Mantra of Meet Users Where They Are

4 IT Development Challenges Centralized IT Architecture Non-Standard Platforms and Languages Open Source Java/Scala/Python IT Decisions Have Cascading Customer Consequences Always Store Raw Data!!! Timestamp!!! LRS History Location

5 Business Customer Challenges Debates Over Authoritative Data Doubting the Results/Data Resentment at the Lack of Control/Input Fear of the Unknown Resistance to Change

6 Data Challenges: Third Party and KYTC HERE Traffic Speeds Waze Incident Reports Waze Traffic Speeds icones Speeds KYMesonet CoCoRahs Doppler Radar TOC Reports TRIMARC Reports Snow Plows (AVL) Roadway Weather Stations County Activity Reports Dynamic Message Signs Twitter

7 KYTC Architecture Harmonization of 3 rd Party and KYTC Data Using Open and Proprietary Software

8 General Data Workflow Data Source

9 ITS Real-Time Data Warehouse Architecture Cloudera.com Kafka: Data Aggregation Spark: Processing, Harmonization Hadoop: Cold/Warm Storage, Processing Historic Data Elastic.co Elasticsearch: Hot Storage / Indexing Kibana: Dashboards ESRI.com ArcGIS Online: Mapping ArcGIS Online: Data Sharing ArcGIS Server: Spatial Analytics

10 Performance: Legacy vs. Current Architecture Aggregation of Data From Source RabbitMQ: 2,300 / second Kafka: 600,000+ / second Harmonization / Processing (Dynamic Segmentation) SQL/SDE Server: 6 million points in 6 hours Spark: 6 million points in 5 minute Visualizations / Mapping ArcGIS: 10,000 records Kibana: 100 million records

11 Data Processing: Added Fields Matching all incoming records with Linear Referencing System 11 for Tabular Location Data District, County, Route, MP, etc. 12 for Time Stamp(s) Capture Time, EST, UTC, 2min UTC, etc. 6 for Geometry (GIS) Point/Line/Polygon; Longitude, Latitude, Snap, etc. 10 for Route Specific Information Route Classification, SNIC Priority, ADT, etc.

12 Future of Transportation Data Machine Learning and Internet-of-Things

13 Future of KYTC Data Current (for Perspective) 4 TB / year Raw/Unparsed/Unprocessed Data Files 6 TB / year in HDFS for Warm/Cold Storage 6 TB / year in Elastic for Dashboards/Reporting Future: DSRC for 2,500+ Signals 1,500,000 Records per minute (1.5GB/min) 90,000,000 Records per hour (90GB/hr) 786,240,000,000 Records per year (768TB/yr)

14 Future: Machine Learning

15 Future: 3-D Modeling of Congestion

16 Apps / Data Derivatives Situational Awareness, Incident Detection, After Action Review, Snow and Ice Decision Support Systems and Work Zone Monitoring

17 Situational Awareness: GoKY

18 Incident Detection: Minor & Major

19 After Action Review Dashboard

20 Decision Support: Snow And Ice

21 Decision Support: Snow and Ice

22 Work Zone Monitoring: I-64, VSL

23 Work Zone Monitoring: I-65, icones

24 Technical Team Venkata Chaganti, Application Development John Fowler, Application Development Jeremy Gould, Geographic Information Systems Chris Lambert, Systems Consultant, ITS

25 Reality Check: This is why! I-71 NORTHBOUND 8:39am Waze users report a Minor Accident. 8:52am HERE speed data starts to show a relevant decline in speeds (59mph vs mph historic average) with a Jam Factor greater than 2. 9:12am Waze Jams data shows a delay of 1.8 minutes and traffic moving at 45mph. I-71 SOUTHBOUND 9:03am Waze users report a Minor Accident 9:19am HERE speed data starts to show a relevant decline in speeds (57mph vs mph historic average) with a Jam Factor greater than 2. 9:24am Waze Jams data shows a delay of minutes and traffic moving at 7.35mph. 10:30am - GALLATIN COUNTY, KY (FOX19) - One person died in a crash on Southbound Interstate 71 near the Kentucky Speedway Thursday morning. The crash happened off the side of the road near the 57-mile marker around 10:30 a.m., according to Gallatin County dispatchers.

26

27 CONTACT INFORMATION Chris Lambert, Systems Consultant IT Facebook: /kytc120