Uncertainty in transport models. IDA workshop 7th May 2014

Transcription

1 Uncertainty in transport models IDA workshop 7th May 2014

2 Presentation outline Rationale Uncertainty in transport models How uncertainty is quantified Case study General comments 2 DTU Transport, Technical University of Denmark

3 Rationale Transport models consist of equations combining exogenous variables (input) and coefficient (parameters) that express how the model output depends on the exogenous variables (De Jong et al. 2007) Transport models play a prominent role in many decision making processes; the output they generate, transport demand, is a key input for a wide range of analyses including: Cost Benefit Analysis (CBA) / Multi Criteria Analysis (MCA), e.g.: urban development planning strategy, sustainable mobility polices evaluation, etc. financial analysis for new (tolled) infrastructures Thus, wrong transport demand (traffic) forecast might have considerable socio/economic consequences (e.g. misplacement of public resources) 3 DTU Transport, Technical University of Denmark

4 Rationale Transport models have been criticized for lack of accuracy; an extensive literature has shown that often there is a considerable inaccuracy between forecasted and observed traffic volumes Source: Flyvbjerg et al DTU Transport, Technical University of Denmark

5 Rationale The main sources of inaccuracy can be broadly divided between technical and political/decision process Project selection positive bias toward projects with high forecasted traffic Big projects, more difficult to assess, preferred to small ones Disagreement among stakeholders might cause delays making the model results not up to date Political Inaccuracy Uncertainty Technical Standard model used for a problem it is not developed for, then overlooking core causal relationships Wrong assumptions on the transport system, e.g. (i) expectations on how competitors would react to new fare and service (strategy ) (ii) other projects being postponed (network) 5 DTU Transport, Technical University of Denmark

6 Rationale Transport models have inherent uncertainty : reproduce complex systems generating transport supply (i.e. services, infrastructure, regulation) and transport demand (i.e. traffic) receive inputs from other complex system (models): economics, sociology, psychology, technology, etc. The consequence is that transport models output is unpredictable, so it is not possible to use a deterministic approach In fact uncertainty can be defined as any departure from the unachievable ideal of complete deterministic knowledge of the relevant system (Walker et al. 2003) (...) modelled output is better expressed as a central estimate and an overall range of uncertainty margins articulated in terms of values and likelihood of occurrence (Boyce 1999), rather than as a point value 6 DTU Transport, Technical University of Denmark

7 Uncertainty in transport models Adapted from Leleur ) Model definition Uncertainty related to the definition of the boundaries of the system being modelled (context) e.g. the geographical size of the area of interest, future changes in the transport demand environment 2) Model specification Uncertainty about the assumed model structure and the technical implementation of the model (e.g. the methodology chosen) 3) Model calibration Uncertainty about input (data collected) and parameters (calibrated variables) 4) Model validation Uncertainty propagated to the transport model output (sequential frameworks) 7 DTU Transport, Technical University of Denmark

8 How uncertainty is quantified Different levels of uncertainty require/allow different uncertainty calculation techniques: Recognised ignorance 1) Scenario hypoteses + sensitivity tests Scenario uncertainty Ignorance (Indeterminacy) Adapted from Walker et al Determinism Statistical uncertainty 2) Confidence intervals + sensitivity tests 3) Stochastic sampling methods + sensitivity tests (Monte Carlo Simulation, Bootstrap) 8 DTU Transport, Technical University of Denmark Uncertainty in transport models

9 How uncertainty is quantified Deterministic approach Scenario analysis Confidence intervals Stochastic simulation 9 DTU Transport, Technical University of Denmark

10 Case study: Næstved model Uncertainty analysis (model input and parameters) through MCS combined with sensitivity analysis Analysis focuses on: uncertainty propagation pattern effects on output uncertainty of uncertainty location (input, parameters and total) effects on output uncertainty of network congestion three levels of generated traffic (1,1.5 and 3 times base traffic) Stochastic sampling (multivariate LHS,100 draws): variables mean values: from the model variability: Coefficient of Variation (CV) 0.1 and 0.3 (CV=St Dev /Mean) probability distribution: input (Triangular) parameters (Log-normal) Sensitivity tests (100 runs) 10 DTU Transport, Technical University of Denmark

11 Case study: Næstved model 11 DTU Transport, Technical University of Denmark

12 Case study: Næstved model 12 DTU Transport, Technical University of Denmark

13 Case study: Næstved model 13 DTU Transport, Technical University of Denmark

14 Case study: NTM forecasts Uncertainty analysis (model socio-economic input growth rate forecasts 2015/2025) through MCS combined with sensitivity analysis : population, employment, GDP (real) and fuel prices (both petrol and diesel) Analysis focuses on: uncertainty propagation pattern over time aggregated and for different input This approach is meant to reflect an increased level of uncertainty throughout the forecasted period Stochastic sampling (multivariate LHS,100 draws): variables mean values: official forecasts variability: standard deviations calculated by 5 years intervals from time series probability distribution: Normal distribution 14 DTU Transport, Technical University of Denmark

15 Case study: NTM forecasts 15 DTU Transport, Technical University of Denmark

16 Case study: NTM forecasts 16 DTU Transport, Technical University of Denmark

17 Conclusions Transport models output, demand of transport, is a key input for policy decision making processes; problems arise from the fact that transport models refer and describe complex systems, thus showing unpredictability in their output The final output of the transport models is no more a point value but a range of possible outcome (with a given probability to occur) Assessing uncertainty inherent to transport models is important to produce informative output (demand of traffic) However, analyzing uncertainty of a whole transport model requires a relevant work effort; the question then is how expensive are decisions based on too uncertain forecasts and how time-consuming is implementation of uncertainties analyses? 17 DTU Transport, Technical University of Denmark

18 Extra Transport models uncertainty has a temporal dimension and occur in both static and dynamic (forecast) modelling However, with respect to the influence on the model output, the prevalence of some uncertainty location relatively to the others is affected by the temporal and geographical dimension of the system modelled The influence of model and parameters uncertainty is higher in static or short run forecasts whilst context and input uncertainty influence increases in long run forecasts. The same happens for geographical dimension of the area modelled, respectively small or big size 18 DTU Transport, Technical University of Denmark

19 Extra 19 DTU Transport, Technical University of Denmark

20 Extra Definition Programming Calibration (input) Calibration (parameters) Validation Generation Subdivision into categories? Number and size of zones? Which socio economic variables? Model type? (Category analysis, linear regression, Logit, Probit) How does data and zone structure match? When is data collected? How well does data fit? How many parameters? Estimated, calibrated parameters? When is data collected? Are data representative? Model output (propagated) uncertainty Distribution Which variables describe travel resistance? Time intervals? Peak hours etc. Pivot table? Size of zones? Model type? (gravity, growth factor, Logit, Probit) Travel resistance function? Linear travel resistance function? Which matrix adjustment? How old is data? Is the generalized cost matrix the result of feedback from traffic assignment? How many parameters? Estimated or calibrated? Which calibration method? When is data collected? Are data representative? Model output (propagated) uncertainty Mode choice Which modes? Is travel resistance included? Which variables are used Is it 2nd or 3rd model step? Model type? (Logit, Probit, mixed Logit) Is travel resistance a result of traffic assignment? Which characteristics are included? How many parameters? Estimated or calibrated? Which calibration method? When is data collected? Are data representative? Model output (propagated) uncertainty Assignment Which variables are included in the utility function? Different trip purposes? Different link types and speed-flow functions? Number of iterations with feed back? Linear utility function? Correlation? Model type? (Logit, Probit, mixed Logit, nested Logit) Deterministic or stochastic? Capacity constraint? Solution algorithm? Level of detail? Which characteristics are associated with the links? How many parameters? Estimated or calibrated? Which calibration method? When is data collected? Are data representative? Model output (propagated) uncertainty 20 DTU Transport, Technical University of Denmark Uncertainty in transport models

21 Case study: Næstved model - Extra 21 DTU Transport, Technical University of Denmark

22 Case study: Næstved model - Extra 22 DTU Transport, Technical University of Denmark