High Level Design Review. High Level Design v1.1 Release

Transcription

1 High Level Design Review High Level Design v. Release

2 Agenda Scope Review Key Elements and Concepts Design Walkthrough Comments and Responses Questions?

3 Scope

4 High Level vs. Detailed Design 4 High Level Design System Architecture Primary Components Data Flows Processes Databases Primary Interfaces Detailed Design Component Architecture Interface Details/ Design DB Schemas AWS Design " EC " Security " Services

5 5 Key Elements

6 Key Elements 6 Microservices Architecture Amazon Web Services

7 Microservices 7 Persistence Worker

8 Microservices 8 X Apply resources only when needed, Scale on demand Scale only where needed Speed - near real time performance Persistence Worker

9 Microservices 9 X Independence of processes Resiliency to failure failure will occur, minimize impacts of failure Security of individual processes, least privilege Persistence Worker

10 Microservices 0 X Persistence Worker Upgrade system components with minimal impact to system

11 Microservices X Persistence Worker Cost efficiency/optimization

12 Microservices X Persistence Worker Flexible add new services with limited impact

13 Amazon Web Services Flexibility Ability to scale Scale as needed Add new corridors Add new components/functions Security Certifications ( Granular control Large selection of production tested capabilities

14 AWS Services 4 Strategy use commodity services available outside of AWS whenever possible, strategically use AWS specific services that provide significant development time savings Compute (EC, EBS) EMR (Spark) RDS (Postgres) Development Tools (CodeCommit, CodeBuild, CodeDeploy, Code Pipeline) Networking and Security Monitoring Deployment

15 5 Design Walkthrough

16 Objectives, Constraints, and Principles 6 Real time operation Speed to decision Flexibility for future with incremental improvements Security Maintainable, operable by Caltrans Scalable Resilient, reliable

17 Core System High Level Design 7

18 Data Hub Functions 8 Receive information Process information Data Quality Common Metrics/Analysis Predictive Analytics Standardized formatting/content Persist information Secure information Provide data communications bus Orchestration of services between the DSS and CMS

19 Data Hub High Level Design 9

20 Data Hub High Level Design 0

21 Data Hub High Level Design

22 Data Hub High Level Design

23 DSS Functions Provide response plans Evaluate response plans Provide one or more recommendations to corridor operators Provide response plan evaluation results to the Corridor Management System Recommend response plan updates when appropriate

24 Decision Support System Design 4

25 Corridor Management System 5 Primary Functions Capture or receive incident information View corridor status assets and traffic Review response plans Manage response plan lifecycle receive, select, approve, execute, update, terminate Send commands to execute response plan elements to local TMC systems Reporting/Review/Analytics

26 Sources Decision Support Data Hub Command Gateway (Camel) Data Hub ICM System Monitoring DSS Interface Data Gateway Monitor Details 7 PEMS Reader FW Sensors (Currently PEMS FTP Source) CT.D7.0.DSS.Interface.IntersectionDetectorState Detector Inventory/Maintenance CT.D7.0.DSS.Interface.FreewayDetectorInventory Persistence Worker CT.D7.0.FreewayDetector.ProcessedStatus Sensor Inventory/ Maintenance Intersection Signal Control Schedule CT.D7.0.Detector.Maintenance Intersection Signal Plans CT.D7.0.RampMeter.State Intersection Signal State Ramp Meter Inventory CMS Inventory 5 Ramp Meter Plan Inventory Pasadena Intersection Signal Reader Arcadia Intersection Signal Reader CT.D7.0.RampMeter.ProcessedPlanInventory Intersection Signal Plan Inventory 5 Cassandra 5 CT.D7.0.DSS.Interface.CMSInventory CMS Inventory CT.D7.0.DSS.Interface.CMSMessageInventory CMS Message Inventory CMS State CT.D7.0.DSS.Interface.Request Request Local Travel Time Reader LCS Schedule Transit Routes CT.D7.0.LCS.ProcessedSchedule CT.D7.0.DSS.Interface.TravelTimeDetectorInventory CT.D7.0.DSS.Interface.TravelTimeDetectorData Transit State CT.D7.0.LCS.ProcessedState Travel Time Inventory CT.D7.0.TravelTimeDetector.ProcessedInventory Travel Time CT.D7.0.TravelTimeDetector.ProcessedData GoldlineTransit Reader HAR MessageInventory HAR Inventory 5 HAR Status Environmental Data Environmental Detector Inventory HAR Center Status Reader CT.D7.0.DSS.Interface.TransitState CT.D7.0.DSS.Interface.HARMessageInventory CT.D7.0.DSS.Interface.HARInventory HAR Inventory CT.D7.0.DSS.Interface.HARStatus HAR Message Inventory Mongo (Transform only) CT.D7.0.RPM.LCSSchedule CT.D7.0.RPM.OrganizationInformation CCTV Inventory 6 5 CT.D7.0.Video.ProcessedInventory 5 Mongo (Persist) (st gen to be replaced with Postgres) CT.D7.0.Organization.CenterActive CCTV State Predicted State Org Information Organization Information 6 CT.D7.0.Organization.Information Organization Rules Importer CT.D7.0.RPM.HARStatus 4 CT.D7.0.DSS.Interface.OrganizationInformation Response Management CenterCT.D7.0.RPM.Status Rules Parameters Rules CT.D7.0.RPM.RouteStatus CT.D7.0.DSS.Interface.CenterStatus Center Status Event Service CT.D7.0.DSS.Interface.Events Event CT.D7.0.DSS.Interface.RoadNetworks Road Network CT.D7.0.RPM.Routes Routes CT.D7.0.RPM.ResponsePlan RULES REPOSITORY Event Service Road Network Information Service Road Network Information Service CT.D7.0.DSS.Interface.Routes CT.D7.0.Network.RouteInventory Route Inventory Postgres CT.D7.0.DSS.Interface.RouteStatus Route Status Response Plan Route Status Route Status Persistence Worker Cassandra Prediction Metrics Dynamic endpoint Estimation Metrics CT.D7.0.Estimation.Metrics Estimation Metrics ActiveMQ Endpoint Replay Request Mongo (Persist) (st gen to be replaced with Postgres) Persistence Worker 8 CT.D7.0.DSS.Interface.ResponsePlans Camel (Stateless Routing/Transformation) Prediction Metrics CT.D7.0.Prediction.Metrics Corridor Management System Freeway Detector Data Freeway Detector State Ramp Detector Data Ramp Detector State Intersection Detector Data Intersection Detector State CT.D7.0.Persistence.ReplayRequest Replay Request Detector Inventory/Maintenance Ramp Meter State Ramp Meter Inventory Ramp Meter Plans ActiveMQ Queue * Replay Request Response to Dynamic Endpoint Intersection Signal Inventory Intersection Signal Ctrl Schedule Persistence Worker Intersection Signal Plans Intersection Signal State Response Plan Service CT.D7.0.ResponsePlans.ResponsePlans Response Plan Service CMS Inventory CMS Control CMS Message Inventory CMS Message Appearance CMS State LCS Inventory LCS Schedule CT.D7.0.ResponsePlans.CommandLog Event HAR Control LCS State Travel Time Detector Inventory CMS Status Environmental Detector Inventory Section Control Response Plan Transit Routes Response Plan Approval Travel Time Detector Data Environmental Detector Data Transit State HAR MessageInventory HAR Inventory HAR Status CCTV Inventory Camel (Stateless Routing/Transformation) CCTV Control Intersection Control Response Command Log CCTV State Org Information Center Active Event Service Road Network Information Route Inventory Route Status Prediction Metrics Estimation Metrics Replay Request Response Plan Response Command Log CMS Status Rules Engine CT.D7.0.RPM.Events Center Active CT.D7.0.Event.DescriptionAndUpdates Static ActiveMQ/SOAP combined endpoint NOTES:. Data Hub Intersection Readers and s and Data Gateway endpoints implement both Intersection and Section TMDD messaging. Asset State CT.D7.0.RPM.LCSState Estimated State Route Inventory Kafka Topic Routes CT.D7.0.RPM.LCSInventory 5 CT.D7.0.HAR.ProcessedInventory CT.D7.0.Network.RouteStatus ActiveMQ Topic Persistence Worker 5 Persistence Worker MongoDB CT.D7.0.RPM.CMSState 6 CT.D7.0.Transit.ProcessedStatus Video Mongo (Transform only) Response State CT.D7.0.RPM.CMSMessageInventory HAR Message Inventory CT.D7.0.RPM.TransitState CT.D7.0.Network.NetworkInfoLinkAndNode Cassandra CT.D7.0.RPM.CMSInventory Transit State CT.D7.0.Transit.ProcessedRoutes Key EC Scaled Component via CC Scaling CT.D7.0.RPM.SignalInventory CT.D7.0.RPM.SignalPlan CT.D7.0.RPM.TransitRoutes 7 CT.D7.0.DSS.Interface.TransitRoutes CT.D7.0.RPM.HARMessageInventory CT.D7.0.HAR.ProcessedState Organization Information Reader CT.D7.0.RPM.RampState CT.D7.0.RPM.RampPlan Transit Route 5 CT.D7.0.Video.ProcessedState Response Plan Management CT.D7.0.RPM.HARInventory Transit Status HAR Reader PredictionState CT.D7.0.EnvironmentalDetectorInventory CT.D7.0.HAR.MessageInventory Video Inventory/ State Reader AimsunResponsePlan Travel Time Detector Data CT.D7.0.EnvironmentalDetectorProcessedData MODEL REPOSITORY Travel Time Detector Inventory Mongo (Transform only) Aimsun AimsunEvents CT.D7.0.DSS.Interface.EnvironmentalDetectorInventory Workflow Worker Transit Route Dumper CT.D7.0.RPM.SignalState RIITS Transit Reader Dumper Wrapper AimsunLCSState CT.D7.0.DSS.Interface.EnvironmentalDetectorData 6 LCS Status Environmental Sensing Aimsun License Server AimsunLCSSchedule LCS State Environmental Detector Inventory 5 Mongo (Transform only) Environmental Sensing Reader Aimsun State Persistence AimsunSignalPlan LCS Inventory CT.D7.0.DSS.Interface.LCSState 6 Cassand AimsunSignalState CT.D7.0.DSS.Interface.LCSSchedule Travel Time Detector Data CT.D7.0.LCS.ProcessedInventory 7 RIITS Video Feeds Caltrans and Local Video AimsunRampState CT.D7.0.DSS.Interface.Status LCS Schedule Travel Time Detector Inventory Data Gateway CT Travel Time Reader SignalPlan Road Network CT.D7.0.DSSRoadNetwork CT.D7.0.RPM.SignalControlSchedule LCS Control Schedule City Lane Closure Reader SignalState Prediction CT.D7.0.DSS.Interface.CMSState CT.D7.0.CMS.ProcessedState Unknown Unknown Highway Advisory Radio CT.D7.0.DSS.Interface.IntersectionSignalState CT.D7.0.CMS.ProcessedMessageInventory Mongo (Transform only) SignalControlSchedule Signal Plan CT.D7.0.DSS.Interface.LCSInventory Environmental Detector Data IntersectionDetectorState Signal Control Schedule Signal State 5 LCS Inventory ArterialRunRequest IntersectionDetectorData Signal Inventory CT.D7.0.DSS.Interface.IntersectionSignalPlan FW Lane Closure Reader 7 Research UI Ramp Plan LCS Inventory LCS State CT.D7.0.CMS.ProcessedInventory NextBus Custom Goldline Transit CT.D7.0.DSS.Interface.IntersectionSignalInventory CT.D7.0.DSS.Interface.IntersectionSignalControlSchedule 6 6 CMS State/Priority Queue CT.D7.0.DSS.Interface.RampMeterPlan AiimsunSignalControlSchedule DSS Status Persistence Worker CMS Message Inventory 7 RIITS Custom Environmental Sensors, Transit Persistence Worker Ramp Meter State AimsunRampPlan DSS Request CT.D7.0.IntersectionSignal.ProcessedState CMS Inventory Arterial CMS Reader Local Bluetooth Travel Time Custom Arterial Travel Times Cassand PersistResultsRequest Postgres 5 Mongo (Transform only) SensorDataRequest Project Manager Application Persistence Worker CT.D7.0.IntersectionSignal.ControlSchedule Intersection Signal State FW CMS Reader 7 CT.D7.0.DSS.Interface.RampMeterState CT.D7.0.IntersectionSignal.ProcessedPlanInventory CT Intersection Signal Reader CMS Message Inventory CMS State 5 CT.D7.0.IntersectionSignal.ProcessedInventory Intersection Signal Control Schedule Mongo (Transform only) County Intersection Signal Reader Mongo (Persist) (st gen to be replaced with Postgres) Intersection Signal Inventory SystemStatus Intersection Detector State 6 CT.D7.0.RampMeter.ProcessedInventory 7 7 Status CT.D7.0.DSS.Interface.RampMeterInventory 5 CT.D7.0.Detector.Inventory Ramp Meter State Intersection Signal Inventory CT.D7.0.IntersectionDetector.ProcessedStatus CT.D7.0.IntersectionDetector.CTStatus Sensor Inventory/ Maintenance Reader CMS Ramp Meter Plans CT.D7.0.IntersectionDetector.CountyStatus 7 Ramp Meter Inventory CT.D7.0.RampDetector.ProcessedStatus CT.D7.0.IntersectionDetector.CTSource CT Intersection Detector Reader RunRequest Ramp Detector State Intersection Detector CT.D7.0.IntersectionDetector.ArcadiaSource CT.D7.0.IntersectionDetector.ArcadiaStatus County Detector Reader 7 LCS Schedule CT.D7.0.IntersectionDetector.CountySource Ramp Meter Reader Caltrans Bluetooth Travel Time Custom FW Travel Times Intersection Detector State Ramp Meter State Cassandra Spark CT.D7.0.IntersectionDetector.PasadenaStatus HSR Lane Closure System Custom Arterial Lane Closures Events CT.D7.0.IntersectionDetector.ProcessedData CT.D7.0.IntersectionDetector.PasadenaSource Lane Closure CT.D7.0.DSS.Interface.RampDetectorState CT.D7.0.DSS.Interface.IntersectionDetectorData CT.D7.0.RampDetector.Source Arcadia Detector Reader CT.D7.0.RampDetector.ProcessedData CT.D7.0.FreewayDetector.RawStatus 7 Lane Closure CT.D7.0.DSS.Interface.RampDetectorData Intersection Detector Data CT.D7.0.FreewayDetector.ProcessedData CT.D7.0.RampDetector.Status Pasadena Detector Reader FW/Ramp Sensors Inventory Lane Closure System Custom FW Lane Closures command_status Estimation ReplayRequest CT.D7.0.FreewayDetector.RawSource Ramp Detector Reader CMS Trailblazer Unknown Arterial Wayfinding Signs Ramp Detector Data Ramp Detector State Incident 6 PeMS State CT.D7.0.Cmd.HubTask CT.D7.0.Cmd.HubTaskStatus Intersection Signals CT.D7.0.DSS.Interface.PemsState Ramps Caltrans TSMSS Transcode Transuite CT Intersection Signals Freeway Detector State LCS State 8 Command Listener Task CT.D7.0.Cmd.HubTask Caltrans ATMS Parsons ATMS FW Sensing, Ramp Meters, FW CMS, FW Incident PeMS Raw 4 Event Listener 8 Intersection Signals CT.D7.0.DSS.Interface.PemsData ScenarioBuilderInputQueue Camel (Stateless Routing /Transformation) 4 Intersection Signals County TMC Kimley Horne KITS Intersection Signals (Duarte, Monrovia, LA County) Freeway Detector Data REST Services Arcadia TMC Transcode Transuite Intersection Signals Modeling Conductor (Workflow Management) Intersection Signals Pasadena TMC McCain Transparity Intersection Signals LCS Control Ramp Meter Control

27 Details 7 System Interface Design Specification

28 Information Resources 8 Project information Documentation library Team share drives available released and pre-release development documentation brian.peterson@berkeley.edu

29 Questions 9

30 0 Comments and Responses Review