Testing Activity-Based Model Component Transferability at the Individual Component and System Level Martin Milkovits, Corresponding Author Cambridge Systematics, Inc. Phone: (617) 354-0167 E-mail: mmilkovits@camsys.com Thomas Rossi Cambridge Systematics, Inc. Phone: (617) 354-0167 E-mail: trossi@camsys.com Word Count = 1,830 words + 3 * 250 (tables and figures) = 2,580 equivalent words Prepared for presentation Innovations in Travel Modeling 2016 Conference Submitted: October 9, 2015
Abstract Transferring model parameters from one region to another is an attractive alternative to conducting a local survey, processing the data, and estimating a new set of model parameters. Transferring a complex model, such as an activity-based model (ABM) presents opportunities and challenges. The greater number of components, parameters, and implementation complexity makes an ABM especially tempting to transfer as opposed to develop anew. ABMs are typically more highly specified, include more segments, and model more aspects of travel than a trip-based model, so the risk of unrepresented regional differences could be lower. But, ABMs also represent more decisions, e.g. trip chaining and intra-household coordination, and so the opportunity for non-represented differences is higher. This paper presents a transferability test between activity-based models developed for the Houston- Galveston and the Twin Cities regions. These regions have obvious differences, but the similarity in the implemented model structures and availability of rich survey data for testing present a good opportunity to contribute to transferability testing and activity-based modelling. The transferability tests examine individual component predictive power as well as the propagation of deviations through the entire model system. Transferability is evaluated by comparing the aggregate and cross-classified transferred model results with the estimated model results and household survey data. Statement of Innovation This research brief provides a background on the opportunities and challenges to transferring activitybased models and the value of comparing the transferability of not only individual components, but the entire model system. A case study transferring activity-based models from the Twin Cities to the Houston-Galveston region is presented and the transferability approaches, tests, and expected results are described.
Introduction Transferring model parameters from one region to another is an attractive alternative to conducting a local survey, processing the data, and estimating a new set of model parameters. Surveys are very resource intensive, and traditional phone-based surveys are becoming harder to undertake because of the trend away from land-lines towards mobile phones. Estimating new models from a data set is an important means to advance the state of the practice, but that is a luxury that many regional planning agencies cannot afford. But, transferring model structures and parameters is risky if the two regions differ substantially in aspects that are not well represented in the model. Transferring a complex model such as an activity-based model (ABM) presents opportunities and challenges. The greater number of components, parameters, and implementation complexity makes an ABM especially tempting to transfer as opposed to develop anew. ABMs are typically more highly specified, include more segments, and model more aspects of travel than a trip-based model, so the risk of unrepresented regional differences could be lower. But, ABMs also represent more decisions, e.g. trip chaining and intra-household coordination, and so the opportunity for non-represented differences is higher. Motivation and Challenges A model transferability test is a valuable exercise for the model being transferred and a test of an activity-based model transferability is valuable to the state of the practice. An evaluation of the predictive performance in a new region will assess the unobserved aspects of the model and also if the model is over-calibrated. Beyond this, the test should also offer insights for the type of models, many of the activity-based models are recent constructions and have not had the level of use as trip-based models. The recent Guide for Travel Model Transfer explained that most transferability research has focused on trip generation and mode choice, but not destination choice and posits that destination choice models may be inherently non-transferable [1]. A previous ABM transferability study evaluated the transferability of individual models by estimating an additional model term and examining the statistical significance [2]. Models representing activity planning and sequencing were found to be more transferrable than models representing specific travel (e.g. mode) and destination decisions. This paper examines the transferability of ABM components based with a different structure than those examined in the previous work and considers the model at the system level. ABMs are still evolving and there is not a consensus as to which aspects of daily activity planning are critical and how they should be represented. ABMs contain many components with both downstream and circular dependences (where logsums are involved). Upstream component errors may propagate through the system, or they may be stifled. In some ways, a transferability test is similar to a backcasting test where a calibrated model is run with data from the previous model update and the predictive power is evaluated. Transferability tests also involve many of the same challenges as backcasting. Differences in data formats, socio-economic data, modal skim data, and travel demand software all present obstacles to transferring model parameters, and the model results to be compared are often summarized differently and involve a lot of effort to normalize. Another reason why it is unsurprising that few models have been transferred for the sake of testing is the availability of validation data. After all, if the target region has a household survey with
activity diaries collected and summarized, it makes more sense to estimate a new set of models rather than transfer parameters and risk running with the wrong regional biases. Model Region Overview The two recent activity-based model deployments by the Metropolitan Council of the Twin Cities (Met Council) in Minnesota and the Houston-Galveston Area Council (H-GAC) in Texas present a unique opportunity to conduct a transferability test of an ABM. Both models were developed using TourCast and Cube and have similar input file requirements. The two models were estimated using local survey data and, while there are some similarities in the model structures, there were no model parameters transferred from one region to the other. A key advantage to using these two models for transferability testing is that they both underwent a comprehensive validation and thus have a complete set of validation spreadsheets with survey data summaries attached for comparison [3]. Table 1 shows a comparison between the two regions. There are notable differences: Houston is larger, has a single central city and substantially warmer in the winter. The Twin Cities have a higher non-auto mode share and lower per capita vehicle-miles traveled (VMT). Some of these differences will be represented in the input data, such as the better transit service in the Twin Cities, but others will not (e.g. temperature) though the impacts of such differences may be reflected in the behavioral data used to estimated the models. Table 1: Comparison between Houston and the Twin Cities HOUSTON TWIN CITIES METRO AREA POPULATION (2011) 6,051,850 3,389,049 CENTRAL CITY POPULATION (2011) 2,145,146 387,753 / 288,448 ESTIMATED VMT 160M (2010) 66.5M (2005) PUBLIC TRANSIT PASSENGERS (2012) 77.6M 81.1M BIKE TOUR MODE SHARE 0.6% 1.3% AVG. TEMPERATURE - JAN. (F) 63 / 43 24 / 8 AVG. TEMPERATURE - JULY (F) 94 / 75 83 / 64 AVG. ANNUAL SNOWFALL (INCHES) 0.1 54 Methodology The transferability test focuses on the activity-based model components. The activity-based model component flow chart for the H-GAC model is shown in Figure 1. The Met Council ABM includes an additional component that models the household ownership of a transit pass and toll transponder, which is required to access the MnPASS managed lane system in the Twin Cities. The Met Council pass ownership model outputs are used in mode choice to identify the appropriate highway skim (including managed lanes or not) and to adjust the transit fare.
The assignment procedures differ between the two regions. Met Council segments auto trips by mode, two income categories (lowest and all other income categories), work / non-work tour purpose, and toll transponder ownership, which is used in assignment to exclude single-occupancy vehicles without transponders from using the managed lanes. H-GAC segments auto trips by mode, work / non-work tour purpose, and has varying levels of household income segmentation depending on the mode and tour purpose. We are most interested in the transferability of the ABM components so the local assignment routine and trip table segmentation will be used. Producing trip tables with different segmentation is a trivial change in the TourCast configuration files. Figure 1: H-GAC Activity-Based Model Components The transferability test takes input data from region A and runs the activity-based model components with parameters estimated from region B; the results from the intermediate components and the highway and transit assignment are then compared with the results from region A s model. Initially, the calibrated parameters from region B are used to run the model and potentially the unadjusted
parameters from region B will be used as well. The input data required are the zonal data, which includes socio-economic data and aggregate accessibilities, synthetic population, and highway skim data. Zonal data is used in several of the components, particularly the long-term components and the tour destination choice components that use socio-economic data as part of the size functions. The H-GAC model utilizes employment data with more segmentation than Met Council, and so H-GAC data is aggregated to Met Council input formats rather than disaggregating the Met Council employment for each zone, i.e. H-GAC is region A and Met Council is region B. The synthetic population is generated by a different process in each region, but produces the same eight person types: three child types, two worker types, two non-worker types and adult students. Household income is grouped into five categories and the category thresholds differ slightly between the two regions, as shown in Table 2. However, the inconsistencies of the middle-income ranges are not very concerning because most income-related model parameters are related to the highest and lowest ranges, which are more similar. Based on the initial model results, we may rerun the population synthesizer and adjust the income ranges. Table 2: Income Categories by Region Income Categories Houston Twin Cities Income 1 <$20,000 <$25,000 Income 2 $20,000 - $39,999 $25,000 - $49,999 Income 3 $40,000 - $69,999 $50,000 - $74,999 Income 4 $70,000 - $99,999 $75,000 - $99,999 Income 5 >=$100,000 >=$100,000 Once the input data is reformatted, the results of each ABM component are examined in sequence to identify where the transferred model results deviate from the original model. Considering the overall results are also interesting, but each step needs to be examined to see where the deviation begins and how it propagates. The reports from the comprehensive validation are used to identify the deviations. The validation reports compare multiple cross-classifications and help identify if a particular aspect of the component is deviating. Table 3 shows an example comparison of average vehicles by household size and income from the vehicle availability validation spreadsheet, which also includes comparisons by county, number of workers, and children.
Table 3: Example Validation Report From H-GAC Vehicle Availability Component Expanded Household Survey HHSize Model HHSize HHIncome Total <$20000 20,000- $39,999 $40,000- $69,999 $70,000- $99,999 >$100,000 Total 2.05 1.35 1.88 2.37 2.52 2.64 1 1.10 0.88 1.19 1.29 1.43 1.48 2 2.05 1.59 1.93 2.16 2.33 2.31 3 2.57 1.86 2.29 2.82 3.00 3.01 4+ 2.54 1.79 2.37 2.80 2.80 2.92 HHIncome 20,000- $40,000- $70,000- Total <$20000 $39,999 $69,999 $99,999 >$100,000 Total 2.05 1.20 1.73 2.15 2.40 2.59 1 1.13 0.85 1.13 1.33 1.39 1.47 2 2.12 1.51 1.88 2.15 2.27 2.39 3 2.23 1.43 1.88 2.30 2.47 2.65 4+ 2.59 1.61 2.15 2.66 2.85 3.00 Percentage Difference (model - survey)/survey HHIncome 20,000- $40,000- $70,000- Total <$20000 HHSize $39,999 $69,999 $99,999 >$100,000 Total 0% -15% -15% -22% -12% -5% 1 3% -3% -6% 4% -3% -1% 2 7% -8% -5% -1% -6% 8% 3-34% -43% -41% -52% -53% -35% 4+ 5% -18% -22% -15% 5% 8% Two metrics are used to compare transferability for individual components: root mean square error (RMSE) and relative aggregate transfer error (RATE). RMSE is used to measure the difference between the predicted and observed shares from the household survey. RATE is a ratio of the RMSE between the estimated and transferred components and is used to compare the change in predictive power between the transferred and estimated model. To evaluate the propagation of deviations through the model system, we take advantage of the TourCast plug-and-play model design and replace transferred components with ones from the original model and compare the RMSE and RATE of downstream components. If a transferred component has a particularly large RMSE, we may swap that component with the estimated component so that downstream comparisons are worthwhile. Expected Outcomes An ABM transferability test will provide valuable insights not only into the specific models that are tested, but also into how well an ABM represents the factors influencing how people make travel decisions and thus the potential for transferring ABMs. These insights will be described through an examination of model result deviations and the propagation of deviations throughout the model.
Moreover, the identification of unobserved differences between these two regions will inform model transferability in general. Acknowledgements The authors would like to acknowledge the staff from H-GAC, particularly Chris van Slyke and Chi-Ping Lam, and the staff of Met Council, particularly Jonathan Ehrlich and Mark Filipi, for their support and permission to use the regional models for this study. References [1] T. Rossi and C. Bhat, "Guide for Travel Model Transfer," Federal Highway Administration, Washington, DC, 2014. [2] J. L. Bowman, M. Bradley and J. Castiglione, "Making advanced travel forecasting models affordable through model transferability," Prepared for Federal Highway Administration, US Department of Transportation,, Washington, DC, 2013. [3] M. Milkovits, A. Kuppam, D. Kurth and T. Rossi, Comprehensive Validation of an Activity-Based Model - Experiences from Houston-Galveston Area Council's Activity-Based Model Development, Washington, DC: TRB 94th Annual Meeting, 2015.