A PRACTICAL PROCEDURE FOR CALIBRATING MICROSCOPIC TRAFFIC SIMULATION MODELS

A PRACTICAL PROCEDURE FOR CALIBRATING MICROSCOPIC TRAFFIC SIMULATION MODELS By John Hourdaks 1 Panos G. Mchalopoulos 2 Jj Kottommannl 3 1 Department of Cvl Engneerng Unversty of Mnnesota 500 Pllsbury Dr. SE 500 Pllsbury Dr. SE Mnneapols, MN 55455 Mnneapols, MN 55455 Phone: (612) 625-8832, Phone: (612) 625-1509, 2.Department of Cvl Engneerng Unversty of Mnnesota Fax: (612) 626-7750 Fax: (612) 626-7750 Emal: hourd001@tc.umn.edu Emal: mcha001@tc.umn.edu 3 Department of Cvl Engneerng Unversty of Mnnesota 500 Pllsbury Dr. SE Mnneapols, MN 55455 Phone: (612) 626-1648, Fax: (612) 626-7750 Emal: kott0048@umn.edu Submtted for presentaton and publcaton Transportaton Research Board 2003 Annual Meetng January 2003 Washngton, D.C. # WORDS: 7490 July 2002

Hourdaks, Mchalopoulos, and Kottommannl 2 ABSTRACT As employment of smulaton s becomng wde spread n traffc engneerng practce, questons about the accuracy and relablty of ts results need to be addressed convncngly. A major crtcsm related to ths s proper calbraton of the smulaton parameters as well as valdaton whch s often not performed, or dealt wth n an ad-hoc fashon. Ths paper presents a complete, systematc and general calbraton methodology for obtanng accuracy needed n hgh performance stuatons. A technque for automatng a sgnfcant part of the calbraton process through an optmzaton process s also presented. The methodology s general and s mplemented on a selected smulator to demonstrate ts applcablty. The results of the mplementaton n two freeway sectons of reasonable sze and complexty n whch detaled data were collected and compared to smulated results, demonstrate the effectveness of the manual calbraton methodology. For nstance, through calbraton the average volume correlaton coeffcent on 21 detectng statons mproved from 0.78 to 0.96. Comparable results were obtaned wth the automated calbraton procedure wth sgnfcant tme savngs and reduced effort.

Hourdaks, Mchalopoulos, and Kottommannl 3 INTRODUCTION Traffc smulaton s ncreasngly beng used n practce as the sophstcaton and requrements pror to deployment of ATMS systems ncreases along wth the complexty of problems engneers are faced wth n daly practce. The effectveness of a traffc smulator n evaluatng traffc management strateges les n ts ablty to accurately replcate actual traffc condtons; ths requres proper calbraton of ts parameters rather than usng default values. Calbraton s the process n whch the model parameters of the smulator are optmzed to the extent possble for obtanng a close match between the smulated and the actual traffc measurements, whch prmarly nclude volume, speed and occupancy. Generally, calbraton s an teratve process n whch the engneer adjusts the smulaton model parameters untl the results produced by the smulator match feld measurements; the comparson part s often referred to as valdaton. There are two man ssues related to calbraton. Frst, a systematc procedure for the calbraton process s lackng. Typcally, a hgh performance smulator has numerous parameters that must be calbrated to obtan accurate results. In the absence of earler calbratons at a ste, the best-suted values of the smulator parameters are currently determned teratvely by traland-error and n an ad hoc fashon; ths makes calbraton a tme-consumng and neffcent process, and as a result t s usually not performed or treated only superfcally n most practcal applcatons. The second ssue related to calbraton s that the goodness-of-ft tests usually employed to assess the effectveness of calbraton do not provde suffcent nformaton for assstng the user to dentfy weaknesses durng the course of the calbraton. Exstng tests measure only the magntude of the percentage error and assess trends.e. mean square error or regresson coeffcent. For a more rgorous accuracy assessment of the smulator there s a need

Hourdaks, Mchalopoulos, and Kottommannl 4 to use approprate test statstcs that can measure lnear bas, as well as systematc and unsystematc error therefore provdng the user wth more nformaton about the nature of the error. Ths paper presents a complete and systematc general calbraton methodology that addresses these ssues and was mplemented and tested at several stes for assessng freeway ramp meterng performance. Its mplementaton was greatly smplfed by an optmzaton technque also presented here along wth results from a 20-Km freeway secton n Mnneapols, Mnnesota. The latter s related to a recently completed ramp meterng evaluaton study [1] n whch very accurate results were obtaned followng the calbraton methodology presented. The optmzaton technque produced comparable results faster,.e. whle the manual calbraton requred about 2 months for mplementaton of the frst stage, the smlar stage through automated calbraton requred only 6 hours. BACKGROUND The general requrements of a smulaton calbraton procedure have been dscussed n only a few publcatons [2,3,4,5,6] along wth Goodness-of-ft tests for valdatng traffc smulators. However, rgorous traffc smulator calbraton methodologes are stll lackng. Most of the publshed methodologes are not general but rather applcable only to a partcular smulator; n addton ther statstcal analyss and verfcaton of goodness-of-ft s not suffcently detaled as mentoned earler. Ths secton provdes a revew of the most wdely known calbraton procedures. INTRAS [7], the mcroscopc traffc smulator developed by FHWA, was employed to smulate traffc operaton on Southern Calforna freeways for evaluatng ncdent detecton

Hourdaks, Mchalopoulos, and Kottommannl 5 algorthms and for tranng artfcal neural network models to detect freeway ncdents [8]. The calbraton procedure adopted n that study was performed n two stages. Frst, through tral-anderror, the parameters that nfluence vehcle movement durng ncdent-free condtons were calbrated wth ncdent-free data. The parameters were calbrated sequentally,.e. whle the best value of any partcular parameter was beng calbrated, the remanng parameters were treated as constants. When a parameter was beng calbrated, the objectve was to ncrease the slope and r 2 (regresson coeffcent) of the smulated vs. actual staton volume and occupancy plots wth greater emphass on volume. After a sutable combnaton of the non-ncdent parameters was found, through tral-and-error, parameters related to ncdents were calbrated aganst ncdent data sets. The RMS percent error between the smulated and actual occupancy durng and after the ncdent was used as a performance measure durng ths stage. Even though, accordng to the authors, ths calbraton methodology produced satsfactory results, ts shortcomng s that t faled to seek the optmum combnaton of parameters n a systematc way. MITSIM [9], the mcroscopc smulator developed by the Massachusetts Insttute of Technology, was recently employed to evaluate traffc management schemes nvolvng coordnated traffc control systems, bus prorty at sgnals and bus-lane operatons n Sweden [10]. The calbraton process was performed n 2 stages; n the frst stage the drver behavor parameters were calbrated whle n the second the travel behavor parameters were calbrated. The objectve durng drvng behavor parameter calbraton was to mnmze the sum of squares of errors between the smulated and actual sensor speeds. Calbraton of the travel behavor parameters nvolved the calbraton of the route choce model parameters followed by OD estmaton. The objectve durng route choce model calbraton was to match the splt of trps between two sensors through ether one of whch all trps pass. The objectve durng OD

Hourdaks, Mchalopoulos, and Kottommannl 6 estmaton was to mnmze the devaton between the estmated and the observed sensor counts and also mnmze the devaton between the estmated OD and the seed matrx. The effectveness of the calbraton was evaluated by comparng 3 types of observed and smulated measurements: traffc flows, travel tmes and queue lengths. The two goodness-of-ft measures used for ths purpose were the root mean square percent error and the mean percent error. The calbraton methodology adopted n ths study s qute complcated and laborous. In addton, all three measurements used are general and do not assst n calbratng local model parameters. Calbraton of the drvng behavor parameters and the OD estmaton process can be automated by the ncorporaton of approprate optmzaton technques. PARAMICS [11], s a mcroscopc smulator developed by Quadstone Lmted n Ednburgh and was recently employed to evaluate freeway mprovement strateges (ramp meterng strategy, auxlary lane addton, HOV lane addton) n the San Francsco Bay Area [12]. Accordng to the calbraton methodology followed n ths study, several parameters such as lnk speed, vehcle top speed, smulaton tme step and sgnpostng dstances were calbrated based on engneerng judgment or experence. In order to calbrate the mean headway and mean reacton tme, smulatons were performed wth multple combnatons of these parameters usng the average network speed and maxmum vehcle throughput as performance ndcators. Approprate target values of these performance measures were determned for the partcular test ste selected. As a fnal measure of the effectveness of calbraton, a ch-square test was performed to compare the smulated vs. actual speed contour graphs. Smlarly to the prevous study, the measurements used are too general to allow detaled calbraton of local parameters. As dscussed later such a hgh level calbraton mght prove msleadng.

Hourdaks, Mchalopoulos, and Kottommannl 7 FRESIM [13], another smulator developed by the FHWA, was employed to smulate expressway traffc operaton n Sngapore [14]. An automated calbraton procedure was followed n ths case. A Genetc Algorthm [15] optmzaton technque was used to search for the best combnaton of 12 smulator parameters. The objectve functon used n the optmzaton algorthm was a combnaton of the average absolute error (AAE) between the smulated and the actual 30 sec volume and speed averaged across all lanes. The AAE between the smulated and actual manlne average volume and speeds were used to measure the effectveness of calbraton. The calbraton methodology adopted n ths study, nvolvng an optmzaton technque, s effcent as t s searchng systematcally for the best combnaton of smulator parameter values and as such t represents a sgnfcant mprovement over conventonal calbraton methodologes. However, the calbraton process nvolved only global parameter optmzaton.e. no attempt was made to calbrate local parameters. METHODOLOGY FOR PRACTICAL CALIBRATION AND VALIDATION As mentoned earler, the relablty of any smulator depends on ts ablty to produce results close to realty. The process of determnng whether the smulaton model s close enough to the stuaton beng smulated s generally acheved through an teratve tral-and-error process nvolvng calbraton of the model parameters, comparng the model to the actual system behavor and usng the dscrepances between the two to mprove the results untl the accuracy s judged to be acceptable. The behavor of the actual system s usually defned n terms of measurable traffc varables such as volumes, speeds, occupances, queue lengths, etc., whch for practcal purposes are measured by detectors or observed manually. To valdate the smulaton

Hourdaks, Mchalopoulos, and Kottommannl 8 model, the smulator should be able to emulate actual measurements and produce a seres of matchng smulated values. Goodness-of-ft measures used for valdaton Regardless of the exact calbraton procedure employed ts success and effcency depends on the measurements used durng the valdaton as well as the goodness-of-ft measures employed. The measurements used to compare realty wth smulaton can not be easly defned because they depend on the gven ste to be modeled and the avalable nstrumentaton. In freeways the most common measurements are volume, speed or occupancy, and rather nfrequently densty whch can be derved from occupancy. In some cases where entrance ramps are metered, an mportant valdaton measure s the queue sze. The methodology descrbed n ths paper deals prmarly wth freeway sectons where volume and speed are the prmary valdaton parameters as n most cases n practce. However, further refnements are also presented for cases where demandng objectves, such as ramp meterng, need to be evaluated. In order for the calbraton methodology to be effcent and robust the goodness-of-ft tests used should not just provde a metrc descrbng the ft but they should nclude nformaton as to what s the nature of the dscrepancy between realty and smulaton so the user can target specfc model parameters for calbraton. A typcal statstcal procedure for comparng two sets of data for a close match s through a hypothess test such as the t-test. The null hypothess n ths context could be that the mean of the smulated traffc measurements s equal to that of the actual traffc measurements. However, there s a lmtaton of applyng the t-test to traffc measurements. To apply ths test, the observatons should be dentcally and ndependently dstrbuted (..d.) but smulated and actual traffc measurements are tme seres that are not

Hourdaks, Mchalopoulos, and Kottommannl 9 necessarly..d. Therefore the valdaton of a smulator cannot be based on such a hypothess test. A wdely used error measure that can provde a farly good ntal estmate of the degree of ft between the smulated and the actual traffc measurements s the Root Mean Squared Percent Error (RMSP), defned n Eq. 1. Ths error measure gves an estmate of the total percentage error and s defned as: RMSP = 1 n n = 1 x y y 2 (1) where RMSP s the root mean squared percentage error x s the smulated traffc measurement value at tme y s the actual traffc measurement value at tme The correlaton coeffcent (r) s another popular goodness-of-ft measure used to measure the strength of the lnear assocaton between the smulated and the actual traffc measurements and s defned as: 1 r = n 1 n = 1 ( x x)( y σ σ x y y) (2) where r s the correlaton coeffcent x s the mean of the smulated traffc measurement values y s the mean of the actual traffc measurement values σ x s the standard devaton of the smulated traffc measurement values σ y s the standard devaton of the actual traffc measurement values

Hourdaks, Mchalopoulos, and Kottommannl 10 n s the number of traffc measurement observatons The RMSP has an nherent defcency n consderng the dsproportonal weght of large errors whle r although beng a good measure does not provde any addtonal nformaton to the modeler as to the nature of the error (dfference) between real measurements and smulaton. Thel, n hs work on economc forecastng [16] developed a goodness-of-ft measure called Thel s Inequalty Coeffcent, ths s more senstve and accurate than the RMSP or r and t can also be decomposed nto three other metrcs that provde specfc nformaton about the nature of the error. Thel s Inequalty Coeffcent [16] s defned as: U = = = = + n n n x n y n x y n 1 2 1 2 1 2 1 1 ) ( 1 (3) The square of the numerator n Eq. 3 can be decomposed nto the three components of the equaton: x y y x n r x y x y n σ σ σ σ ) 2(1 ) ( ) ( ) ( 1 2 2 1 2 + + = = Based on ths and Eq. 3, three components of U can be derved, namely Um, Us and Uc, whch can be used to measure dfferent aspects of the error between the smulated and the actual traffc measurements. These components are defned as: = = n m x y x y n U 1 2 2 ) ( ) ( = = n x y s x y n U 1 2 2 ) ( ) ( σ σ

Hourdaks, Mchalopoulos, and Kottommannl 11 U c 2(1 r) nσ σ = n = 1 ( y y x ) 2 x where U m s the bas proporton, whch s a measure of systematc error that can be used to determne consstent over-countng or undercountng caused by an excess/loss of vehcles. U s s the varance proporton, whch can be used to measure the smulated measurements ablty to replcate the degree of varablty (fluctuatons) n the actual measurements. U c s the covarance proporton, whch s a measure of unsystematc error. r s the correlaton coeffcent of the smulated and actual data The other varables are as defned earler. Methodology for Practcal Calbraton and Valdaton The calbraton methodology presented n ths paper was prmarly developed for freeway smulaton. Snce the most common freeway measurements are volumes, occupances and speeds, the methodology s llustrated by usng 5-mnute measurements collected from detector statons 1 but can be easly appled to any set of measurements or tme slces. Calbraton s enabled by usng manlne staton smulated and actual measurements and attemptng to obtan the best match between the two by adjustng the smulator parameters through tral-and-error n 1 Each detector staton agregates counts from all ts lane detectors and reports the total volume and average occupancy.

Hourdaks, Mchalopoulos, and Kottommannl 12 the manual process or through optmzaton. The smulator parameters to be calbrated for ths objectve fall nto two man categores: global (those that affect the performance of the entre model) and local (those that affect only specfc sectons of the roadway). Examples of global parameters are the vehcle characterstcs (Length, Wdth, Desred speed, Max Acceleraton/Deceleraton, and mnmum headway). Speed lmts of sectons of the freeway model are local parameters. Durng the calbraton process, the global parameters are calbrated frst followed by local parameter calbraton. The calbraton process s performed n two man stages based on volume and speed, followed by an optonal stage n whch the control varable depends on the specfc purpose for whch the smulaton s performed. For example, f the objectve of the smulaton s to test the effectveness of an adaptve ramp-meterng algorthm, ramp queues could be used as the approprate valdaton varable n the thrd stage. Smlarly, f the objectve s to smulate accdents, the congeston backup can be used as an approprate varable n a smlar way as presented here. Volume-based calbraton s performed frst as t s less complcated. Speed s a more senstve measure to the fluctuatons of traffc and progresses the calbraton further. The optonal 3 rd stage s used to fne-tune the smulaton model for the specfc purpose of the smulaton. The step-by-step procedure to be followed n each of the 3 stages of the calbraton process s descrbed below. Stage 1: Volume-based Calbraton. The objectve durng ths stage s to obtan smulated manlne staton volumes as close as possble to the actual manlne staton volumes whch are used for ground truth as they are not nput to the smulator and are routnely measured n freeway survellance systems. The global smulator parameters to be modfed n ths stage are those related to vehcle characterstcs lke

Hourdaks, Mchalopoulos, and Kottommannl 13 speed, acceleraton, deceleraton rates and other parameters related to nteracton between vehcles. Pror to begnnng the calbraton an ntal run should be performed usng default or best estmates of the smulator parameters and the results checked for general reasonableness and resemblance to manlne detector staton data. In some nstances [1] dscrepances are not caused by the model parameters but rather erroneous demand patterns ether n the data collecton stage or n the data entry. Once ths possblty s ruled out, the user should check f the demand patterns randomly generated by the smulator are close to the specfed ones. In the case of freeways, demand patterns nclude entrance and ext volumes or percentage of manlne volumes extng. Small devatons from ther specfed values can easly be accounted for by replcatng the smulaton runs a number of tmes; usually 10 replcatons were found to be suffcent by the authors. Followng ths prelmnary checkng, the aforementoned model parameters are adjusted through tral-and-error over several teratons. For manual calbraton the systematc procedure presented next s suggested as a gudelne. Startng from the frst upstream staton the goodness-of-ft statstcs for that staton are calculated and the vehcle model parameters startng wth the desred speed are sequentally adjusted. It s recommended not to proceed wth subsequent statons unless good ft s reached wth the one n hand. The frst couple of upstream statons n a freeway secton usually have lttle nterference from nput traffc patterns (just one or two ext/entrances) therefore the response to changes n the model varables should be strong. The calbraton exercse proceeds from upstream to downstream untl the end of the freeway secton s reached. If acceptable accuracy has not been reached n all the statons the cycle starts agan from upstream. The goal s to gradually change and fne-tune the smulaton parameters untl all statons have acceptable ft. It has been observed that the change n the global parameters dmnshes as the process moves

Hourdaks, Mchalopoulos, and Kottommannl 14 downstream. For example, for a 10-mle secton wth about 15 detector statons, the global parameters change n magntude reduces beyond the frst 4 statons. Therefore, after the smulated volumes of the frst few statons attan a close match wth the actual ones, the global parameters may have lttle effect and the engneer should focus on local parameters lke speed lmts and lane changng parameters. In the rest of ths secton, gudelnes on how to nterpret and use the nformaton gven by the statstcs are presented. Whle calbratng the global parameters, the focus should be to attan satsfactory values for the three statstcs, namely RMSP, r and U. At the frst stage generally the possblty of a hgh accuracy score s small but one can am for numbers lke RMSP below 15%, r above 0.8, and U lower than 0.3. An unsatsfactory value of RMSP or r can be attrbuted to napproprate values of global smulator parameters lke vehcle speeds, acceleraton and deceleraton rate, whch requre calbraton. The more senstve statstcs namely Us, Uc, and Um provde hnts as to the nature of the dscrepancy between the smulated and actual staton volumes. An unsatsfactory value of Um along wth satsfactory values of Uc and Um (Fg.1a) ndcates a consstent loss/excess of vehcles that could be a result of erroneous demand data at the prevous entrance or due to an error n the number of vehcles extng before that staton. The latter can be caused by mproper weavng that mght prevent vehcles from reachng the proper lane on tme for the ext or due to wrong turnng percentages. A valuable observaton n a number of smulatons, where the demand s entered as nput and ext flows updated over short perods (5 to 15 mnutes) s that the ext volumes mght not be the ones expected smply because the correct amount of vehcles does not reach the ext at the proper tme durng the smulaton. In such a case, although the correct percentage of vehcles exts the actual resultng volume s wrong. Ths problem s not observed

Hourdaks, Mchalopoulos, and Kottommannl 15 n smulatons where the demand s descrbed through tme dependent orgn-destnaton nformaton. An unsatsfactory value of Uc (less than 0.9) (whch s often accompaned by an unsatsfactory value of Um (more than 0.1)) at a partcular staton (Fg.1b) wth the value of Uc at the staton downstream beng satsfactory ndcates the exstence of a bottleneck between the two statons ether n realty or generated by the model. Through calbraton of the local parameters, ths bottleneck should be ether generated or suppressed accordngly. If unsatsfactory values of Uc and Um are observed (Fg.1c), t s ndcatve ether of an error n the vehcle behavor, whch can be attrbuted to the acceleraton / deceleraton rates or due to ncorrect extng volumes at the prevous ext, for the reasons mentoned earler. An unsatsfactory value of Us (more than 0.1), often accompaned by unsatsfactory values of Um and Uc, (Fg.1d) reflects large varablty n ether the smulated or the actual volumes whch may be caused by vehcles drvng close to each other; n such nstances the acceleraton/deceleraton rates need to be approprately adjusted. Stage 2: Speed-based calbraton The objectve durng ths stage s to obtan smulated manlne speeds as close as possble to the actual manlne speeds, and to match the actual breakdown condtons of known bottleneck locatons. The smulator parameters to be calbrated durng ths stage comprse mostly of local speed lmts though global parameters related to vehcle characterstcs such as desred speed, acceleraton and deceleraton rates, mght need some further fne tunng. To compare the smulated and the actual manlne speeds, contour/3-d graphs can be used wheren bottleneck locatons can be vsualzed easly. If pared loop detectors are not

Hourdaks, Mchalopoulos, and Kottommannl 16 avalable to measure actual manlne speeds, the speeds can be derved from the volume and occupancy measurements from sngle loop detectors usng the followng equaton: S = (0.11*Vol*(Lv+Ld)) /(Occ) where S s the actual speed n kmph Vol s the volume over the detector n 5 mnutes Lv s the average vehcle length n m Ld s the detector effectve length n m Occ s the detector occupancy n %. The process of adjustng the speed lmts, as wth the volume, should be performed startng upstream and proceedng downstream. In order to suppress a false bottleneck, often generated n an uncalbrated smulaton, the speed lmts should be ncreased n the regon after the bottleneck locaton so that vehcles leave the locaton faster. Conversely, n order to generate a mssed bottleneck, the speed lmts before the regon should be lowered; an ncrease n the grades wll also produce a reducton n avalable gaps therefore vehcles wll have to create gaps and subsequently cause congeston. Modfcaton of the acceleraton and deceleraton rates also tends to affect the speed of vehcles to a certan extent. Calbraton of the approprate vehcle speed (global parameter) along wth the local speed lmts produce varablty n speeds that are closer to those observed n realty. Through extended observatons of freeway traffc, we found two major categores of bottlenecks. The frst category descrbes the bottlenecks generated through weavng, be that from a nearby entrance/ext or smply due to a lane drop. The second category descrbes bottlenecks attrbuted to drver behavor at complex geometres. For example, a well known bottleneck locaton n Mnneapols, MN s on freeway I-94 eastbound before the ext to I-35W north. Ths

Hourdaks, Mchalopoulos, and Kottommannl 17 four lane locaton has substantal dstance between successve entrances/exts to allow for weavng to be completed well before the observed bottleneck locaton. Regardless of ths, a sharp speed drop s observed on all lanes. When ths locaton was vsually observed we notced that t contans two turns n close proxmty, the second of whch s located under an overpass and all of ths durng a consderable downhll grade. The lack of good vsblty n conjuncton wth the constrctve envronment of the underpass prompts the drvers to sharply reduce ther speed hence creatng the ground for the generaton of a bottleneck. Although bottlenecks due to weavng are relatvely straght forward to model snce most smulators have enough parameters avalable to control the lane changng, gap generaton and acceptance behavor, the second category of bottlenecks, attrbuted solely to drver behavor, needs a lot of observatons and famlarty wth the ste, n order to be modeled accurately. Stage 3 (Optonal): Objectve-based calbraton Whle the frst two stages are well defned, ths thrd stage depends on the objectve of the smulaton. The need for ths stage was revealed durng a real project. Specfcally, n ths project the objectve was to evaluate an adaptve ramp meterng algorthm. Although the calbraton results from the frst two stages resulted n hgh accuracy wthout control, when ramp control was mplemented the accuracy dropped to unacceptable levels on the manlne and especally n reproducng the ramp queue szes. The reason was that small ntal dscrepances n the model gradually amplfed due to the adaptve nature of the control algorthm. The algorthm gradually forced ramp rates a lot dfferent than the ones observed n realty. Therefore, a thrd calbraton stage was devsed and mplemented. In ths stage the Queue szes were the valdaton varable and very specfc local varables lke speed lmts and manlne secton grades where further adjusted.

Hourdaks, Mchalopoulos, and Kottommannl 18 Even though the detals of the thrd stage calbraton may vary based on the objectve of smulaton, smlar procedures can be followed as ponted out at the begnnng of ths secton. One should be careful not to be msled by the good results of the frst two stages but valdate the smulaton wth at least one addtonal measurement. IMPLEMENTATION The calbraton methodology presented was used n a real lfe project for a number of freeway sectons n Mnneapols, Mnnesota. The objectve of the smulatons was to test the effectveness of adaptve ramp meterng [1, 17] followng a perod of publc controversy. Ths secton descrbes one of the test stes followed by results of the mplementaton; the smulator employed n ths case was a well respected mcroscopc one called AIMSUN [18] Test ste and data The test ste s a 20 km (12 mles) long secton of TH 169 northbound startng from the nterchange wth I-494 and endng at I-94. The ste s a crcumferental freeway of average geometrc complexty and carres moderate traffc volumes.e. approxmately 50,000 vehcles daly. The ste conssts of manly two lanes wth 10 weavng sectons, 24 entrance ramps and 25 ext ramps. The detector data used for calbraton comprsed of 5-mnute volumes and occupances for March 23rd from 14:00 hrs to 20:00 hrs. 5-mnute volume and occupancy from 14:00 hrs to 20:00 hrs for March 21st, 2000 and March 22nd, 2000 was used for valdatng the calbrated smulaton model. It s mportant to note that n order to replcate real-tme ramp control all

Hourdaks, Mchalopoulos, and Kottommannl 19 demand patterns and boundary condtons had to be collected smultaneously for each day of smulaton.e. average values would not realstcally emulate the ramp meterng strategy employed. Results Stage 1: The volume-based calbraton process requred about 300 smulaton teratons n whch the parameters were successvely changed as descrbed earler. The calbraton process proved to be very effectve as ndcated by the test statstcs shown n Table 1. As can be observed from the table, there s sgnfcant mprovement n the values of the test statstcs compared to those obtaned usng the best ntal estmates of the smulaton parameter values. The smulator parameter values pror to and after each stage are shown n Table 2. As can be observed from the table, the smulator parameters that had to be modfed durng the frst stage were the parameters related to vehcle characterstcs and the local speed lmts. Durng ths stage, a number of rregulartes n the nput data were observed. Specfcally, n two locatons the placement of the entrance ramp loop detector was not the one reported n the plans. Because of the sensor msplacement, the smulaton results devated substantally from the actual measurements n spte of careful screenng of the data entry process promptng an nvestgaton. After some analyss and vsts to the feld the true locaton of the detectors and the nature of the problem was revealed. Mn/DOT was not aware of these dscrepances untl that tme.

Hourdaks, Mchalopoulos, and Kottommannl 20 Stage 2: At the begnnng of ths stage, the actual manlne speeds were compared wth the smulated manlne ones through contour graphs as shown n fgures 2(a) and 2(b) respectvely. As can be seen from the contours, there was sgnfcant dscrepancy between the speeds n spte the stage 1 calbraton; moreover the bottleneck locatons dd not match the observed ones. Hence, the second stage of the calbraton process was performed whch requred approxmately 100 addtonal smulaton teratons durng whch the local speed lmts, manlne secton grades, and lane changng parameters were adjusted through tral-and-error. Followng ths, the matchng of the bottleneck locatons mproved consderably as can be seen from the speed contour graphs n Fgure 2(c). The smulator parameter values obtaned at the end of ths stage are shown n Table 2. As can be observed from the table, the parameters related to vehcle characterstcs dd not have to be altered much, but the local speed lmts had to be modfed substantally. Stage 3: After stage 2, when the smulaton was performed wth ramp meterng, t was observed that the smulated and the actual merge detector counts, at almost all the entrance ramps dd not match durng the ramp-meterng perod. As a result, the smulated and the real ramp queues dd not match. An example of ths on a sngle ramp s depcted n Fgure 3. Ths problem was consdered major and prompted development and mplementaton of the 3rd stage n the proposed calbraton methodology. Ths stage requred approxmately 100 addtonal smulaton teratons. At the end of ths stage, the smulated ramp queues on all the entrance ramps had a close match wth the actual ones as shown for the TH-55WB example ramp of Fgure 3. The smulator parameter values dd not generally change n ths stage, as can be seen n Table 2.

Hourdaks, Mchalopoulos, and Kottommannl 21 What changed were the speed lmts and grades on specfc sectons (less than 10% of the total ste). After the 3 stages of the calbraton process the smulaton model was valdated based on the remanng two days,.e. March 21st and March 22nd, 2000. The results were very satsfactory as ndcated by the values of the test statstcs shown n Table 1. It can therefore be seen that the adopted calbraton methodology s very effectve. Snce the calbraton process nvolved modfcaton of the smulator parameters teratvely by tral-and-error, t was a very tme-consumng procedure. The volume, speed and queue-based calbratons requred a total of about 4 months to complete, whle the volume alone requred 2 months. AUTOMATION OF THE CALIBRATION PROCESS As the prevous secton suggests, the number of teratons, effort and tme nvolved n a rgorous calbraton can be substantal. Clearly, there s a need to automate the teratve process, of manually modfyng the smulator parameters, to the extent possble. Typcally ths s acheved through optmzaton technques whch seek the best-suted values of the model parameters through effcent search procedures. Such an approach was followed and presented here. The optmzaton problem n the context of the problem at hand s to calbrate the smulator parameters so that an objectve functon s mnmzed. The sum of squared errors of the manlne staton volumes s defned here as the objectve functon to be mnmzed, subject to bounds on the smulator parameters. Mathematcally, the optmzaton problem can be stated as: Mnmze F = st m j= 1 = 1 j ( v s v j a ) 2

Hourdaks, Mchalopoulos, and Kottommannl 22 Subject to L xp < x p < U xp,p = 1, 2,, n where F s the objectve functon to be mnmzed v s j s the smulated traffc measurement of staton j durng tme nterval v a j s the actual traffc measurement of staton j durng tme nterval L xp s the lower lmt of smulator parameter x p U xp s the upper lmt of smulator parameter x p n s the number of smulator parameters to be optmzed st s the number of detector statons on the freeway secton m s the number of tme ntervals. It should be noted here that the objectve functon s not an explct functon of the smulator parameters. Therefore, the optmzaton problem cannot be solved by the usual approach of dfferentatng the objectve functon and settng t to zero to obtan a soluton correspondng to the mnmum. The alternatve s to employ an approprate optmzaton technque. The problem defned above s a non-lnear unconstraned one; such problems can be solved usng nonlnear programmng technques lke the quas-newton methods, the method of steepest descent, Newton-Raphson method, Fletcher-Powell method, etc. Several computer programs are avalable that ncorporate such non-lnear programmng technques. The well-proven and state-of-the-art optmzaton program MINOS [19,20,21,22] was selected here because of ts wdespread use and ablty to solve a varety of large-scale optmzaton problems. These nclude problems that are lnear, non-lnear, bounded, unbounded, constraned or unconstraned based on the form of the objectve functon. If the objectve

Hourdaks, Mchalopoulos, and Kottommannl 23 functon s non-lnear and s unconstraned as n the problem at hand, the quas-newton algorthm s used n MINOS. Implementaton A computer program was wrtten that ntegrates MINOS as a subroutne wth AIMSUN and facltates the data transmsson between these two modules. From the manual calbraton methodology only stage one has been automated n order to demonstrate the technque; n addton, automaton of the frst stage alone sgnfcantly reduces the calbraton effort as ths stage s the most tme consumng one. Ths secton descrbes the results of the mplementaton of the automated calbraton n the selected smulator. Senstvty analyss and prelmnary expermentaton Pror to mplementaton of the automated calbraton process a senstvty analyss of the crtcal smulator parameters was conducted n order to determne the behavor of the objectve functon and the soluton that can be expected from the program. Space lmtatons do not allow presentaton of the detals here; suffce t to say that the objectve functon resultng from changes to all the smulator parameters was non-smooth due to nteracton effects of these parameters. The objectve functon whch consders the nteracton of all the parameters can therefore be expected to be hghly non-smooth. Hence, the gradent-based optmzaton technque can be expected to provde a soluton that les n one of the local mnma of the objectve functon. In any optmzaton algorthm, specfcaton of the approprate step sze s a requred step. In MINOS, the step sze s represented by the dfference nterval denoted here as h. In order to

Hourdaks, Mchalopoulos, and Kottommannl 24 estmate the gradent of the objectve functon wth respect to a varable x, the varable s perturbed by h(1+ x ). Several values for the dfference nterval were tred; the one that provded the best value of the objectve functon was 0.03 and was therefore used subsequently. The effect of usng dfferent ntal smulator parameter values was also analyzed. It was found that even though the fnal values of the smulator parameters obtaned through the optmzaton depend on ther ntal estmates, the fnal objectve functon values were comparable. Ths further smplfes the calbraton task as good ntal smulator parameter values need not be specfed n order to obtan a satsfactory soluton. Moreover, ths suggests the exstence of multple solutons, at least for the selected smulator, all of whch mght be equally acceptable. Results The mplementaton results of the optmzaton technque are shown n Tables 1 and 2. The automated calbraton process requred about 9 teratons of the smulator. In each teraton, approxmately 100 dfferent combnatons of the smulator parameters were tred. It can be seen from the Table 1 that the values of all the goodness-of-ft measures obtaned usng the automated calbraton process are very close to the manual process wth sgnfcant savngs n tme and effort; for nstance, not only the number of teratons was reduced substantally but also the tme to obtan the desred results was reduced to 6 hours, compared to 2 months of the manual process for stage one of the calbraton. CONCLUSIONS In ths paper, a three-stage general and systematc methodology for manually calbratng mcroscopc traffc smulators was presented. Its mplementaton on a selected smulator proved

Hourdaks, Mchalopoulos, and Kottommannl 25 very effectve. For example, an average correlaton coeffcent of 0.961 between the smulated and the actual manlne staton volumes was obtaned when all calbraton stages were completed. Ths s an unusual hgh ft whch can be explaned by the detaled data collected as nput and the qualty of the smulator whch resulted n hgh accuracy even pror to calbraton ( r = 0.78). The Thel s goodness-of-ft statstcs presented here were effectve n dentfyng dscrepances between smulated and actual volumes that would not have been accounted for by commonly used tests. Furthermore, the correct bottleneck locaton dentfcaton was enabled n the second stage of the calbraton. Fnally, the thrd stage of the calbraton process also proved very effectve n obtanng a close match between the smulated and the actual entrance ramp queues. The procedure for automatng a sgnfcant part of the calbraton process through optmzaton yelded comparable results as the manual calbraton process wth substantal savngs n tme. For nstance, the automated volume-based calbraton process requred about 6 hours for 9 teratons (plus 2 months for manual stage 2 & 3 calbraton) resultng n a fnal average correlaton coeffcent of 0.946 whereas the correspondng fully manual calbraton requred about 4 months for 300 teratons to obtan an average correlaton coeffcent of 0.961. Ths suggests that even though the gradent-based optmzaton procedure employed here does not ensure attanment of the global optmum, t s suffcent for practcal purposes. It s worth mentonng that although optmum smulator parameter estmaton depends on ther ntal values, the fnal objectve functon values obtaned usng rough ntal parameter values through the unconstraned non-lnear optmzaton proved to be satsfactory. Ths suggests that unlke the manual procedure, the automated calbraton technque does not rely on very good ntal parameter estmates, whch further smplfes the calbraton task. It also ndcates the exstence

Hourdaks, Mchalopoulos, and Kottommannl 26 of multple solutons, all of whch are equally acceptable at least for the smulator employed and the example at hand. Fnally, the automated calbraton procedure used here s general and allows employment of any optmzaton technque one wshes to use; such technques nclude genetc algorthms, smulated annealng, Nelder-Mead, and others that could possbly result n better parameter estmates. Before concludng t s worth mentonng that the proposed methodology s not restrcted to freeways only but t can be used for arteral streets as well. Ths was recently demonstrated n another study [23] n whch ths calbraton was mplemented n a freeway corrdor that ncluded 5 major arteral streets and 250 ntersectons n addton to the freeway and ts ramps. ACKNOWLEDGMENTS Ths research was supported jontly by Mnnesota Department of Transportaton and The Center for Transportaton Studes. The authors wsh to thank Mr. Frank Llja, Mr. James Aswegan, and Mr. Rchard Lau at the Mn/DOT Metro Dvson for ther nvaluable help n obtanng data that was crtcal for performng ths research. The authors also wsh to thank Dr. Jame Barceló for hs gudance and support n the development of the calbraton methodology.

Hourdaks, Mchalopoulos, and Kottommannl 27 REFERENCES 1. Hourdaks, J., Mchalopoulos, P.G. (2002), Evaluaton of ramp control effectveness n two twn ctes freeways, Paper presented at the 81 st Transportaton Research Board Meetng. 2. Klejnen, J.P.C. (1995), Verfcaton and valdaton of smulaton models; Theory and methodology, European Journal of Operatonal Research 82, 145-162. 3. Hellnga, B.R.(1998), Requrements for the calbraton of traffc smulaton models, Proceedngs of the Canadan Socety for Cvl Engneerng, Vol IVB, 211-222. 4. Rao, L., Owen, L. (1998), Development and applcaton of a valdaton framework for traffc smulaton models, Proceedngs of the 1998 Wnter Smulaton Conference, IEEE. 1079-1086. 5. Rakha, H., Hellnga, B., Van Aerde, M., Perez, W. (1996), Systematc verfcaton, valdaton and calbraton of traffc smulaton models, Paper presented at the 1996 Transportaton Research Board Annual Meetng. 6. Jayakrshnan, R., Oh, J., Sahraou, A.(2001), Calbraton and path dynamc ssues n mcroscopc smulaton for advanced traffc management and nformaton systems, Paper presented at the 80th Transportaton Research Board Meetng. 7. Wcks, D.A. and Andrews, B.J. (1980) Development and testng of INTRAS, a mcroscopc freeway smulaton model, Vol. 2: User s Manual, Report FHWA/RD- 80/107. 8. Cheu, R.L., Recker W.W., Rtche, S. G. (1994), Calbraton of INTRAS for Smulaton of 30-sec loop detector output, Transportaton Research Record 1457, 208-215.

Hourdaks, Mchalopoulos, and Kottommannl 28 9. Yang, Q. and Koutsopoulos, H.N. (1996) A mcroscopc traffc smulator for evaluaton of dynamc traffc assgnment systems, Transportaton Research, 4C, pp. 113-129. 10. Ben-Akva, M., Davol, A., Toledo, T., Koustopoulos, H.N. (2002), Calbraton and evaluaton of MITSIMLab n Stockolm, Paper presented at the 81st Transportaton Research Board Meetng, January 2002. 11. Smth, M., Drutt, S., Cameron, G. and MacArthur, D. (1994) Paramcs Fnal Report, Techncal Report EPCC-PARAMICS-FINAL, Unversty of Ednburg. 12. Gardes, Y., May, A.D., Dahlgren, J., Skabardons, A. (2002), Freeway calbraton and applcaton of the Paramcs model, Paper presented at the 81st Transportaton Research Board Meetng. 13. FHWA (1995) The FRESIM User s Gude, Offce of Traffc Safety and Operatons R&D, Federal Hghway Admnstraton 14. Cheu, R.L. (1998), Genetc Algorthm approach n FRESIM calbraton, Proceedngs of the Internatonal Conference on Applcatons of Advanced Technologes n Transportaton Engneerng, ASCE. 191-198. 15. Goldberg, D.E. (1989) Genetc algorthms n search optmzaton and machne learnng, Addson-Wesley. 16. Thel, H. (1961) Economc forecasts and polcy, North-Holland Publshng Company- Amsterdam, 1961. 17. Employment of the Traffc Management Laboratory (TRAMLAB) for the evaluaton of Ramp Meterng n the Twn Ctes. Project Report. Mn/DOT, 2002. 18. Transport Smulaton Systems (2001) Amsun verson 4.0 User manual, TSS. Aprl 2001.

Hourdaks, Mchalopoulos, and Kottommannl 29 19. Murtagh, B. and Saunders, M. (1983) MINOS 5.5 User s Gude, Stanford Unversty. 20. Murray, W. (1972), Numercal methods for unconstraned optmzaton, Academc Press. 21. Wolfe, M.A.(1978), Numercal methods for unconstraned optmzaton, Van Nostrand Renhold Company. 22. Murtagh, B.A, Saunders, M.A.(1978), Large Scale Lnearly constraned optmzaton, Mathematcal Programmng, v14, 41-72. 23. Shekar, A. Issues n freeway corrdor smulaton, Thess. Unversty of Mnnesota. 2002

Hourdaks, Mchalopoulos, and Kottommannl 30 LIST OF TABLES TABLE 1: Volume statstcal measures pror to and after calbraton TABLE 2: Smulator parameter values pror to and after calbraton LIST OF FIGURES FIGURE 1: Examples of unsatsfactory U m, U s and U c FIGURE 2: Average manlne speed contours FIGURE 3: Smulated and actual queues at TH-55WB ramp

Hourdaks, Mchalopoulos, and Kottommannl 31 Root Mean Square Error % Intal 40 Correlaton coeffcent Thel s Inequalty Coeffcent Thel s Bas Proporton Thel s Varance Proporton Thel s Covarance Proporton 0.78 0.3 0.40 0.31 0.29 Mar 21 st 10.62 0.98 0.00426 0. 30877 0.01052 0. 68070 Mar 22 nd 6.42 Mar 23 rd 7.39 Automated 8.84 0.97 0.00154 0.12352 0.05365 0.82281 0.96 0.00238 0.08826 0.03098 0.88075 0.95 0.004 0.078 0.011 0.91 Table 1: Volume statstcal measures pror to and after calbraton.

Hourdaks, Mchalopoulos, and Kottommannl 32 Parameter Intal After stage 1 (manual) After stage 2 (manual) After stage 3 (manual) After stage 1 (automated) Max. desred 100.000 110.000 110.000 110.000 104.249 speed (kmph) Max. acc. rate (m/s2) 2.800 3.000 3.000 3.000 2.838 Normal dec. rate 4.000 4.000 4.000 4.000 3.983 (m/s2) Max. dec. rate (m/s2) 8.000 7.000 7.000 7.000 6.901 Reacton tme (sec) 0.750 0.590 0.590 0.590 0.512 Percent overtake 0.950 0.950 0.940 0.940 0.950 Percent recover 1.000 1.000 0.990 0.990 1.000 Max. speed dfference (kmph) Max. speed dfference on-ramp (kmph) Av. secton speed (regular secton, kmph) Av. secton speed (weavng secton, kmph) 40 40 60 60 40 50 50 70 70 50 110 100 105 105 110 90 75 70 72 90 Av. secton speed (ramp secton, kmph) 60 60 55 55 60 Table 2: Smulator parameter values pror to and after calbraton

Hourdaks, Mchalopoulos, and Kottommannl 33 200 200 volum e 1 50 Volume 150 actual volume smulated volume actual volume smulated volume 100 100 50 1 7 13 19 25 31 37 43 49 55 6 1 67 50 1 7 13 19 2 31 37 43 49 55 61 67 tme n 5-mn ncrements tme n 55-mn ncrements (a): Illustraton of unsatsfactory Um (b): Illustraton of unsatsfactory Uc 200 200 Volume Volume 150 150 100 actual volume smulated volume 100 smulated volume actual volume 50 1 7 13 19 25 31 37 43 49 55 61 67 tme n 5-mn ncrements mdnght 50 1 7 13 1 25 31 37 43 49 55 61 67 9 tme n 5-mn ncrements (c): Illustraton of unsatsfactory Uc and Um (d): Illustraton of unsatsfactory Us Fgure 1: Examples of Unsatsfactory Um, Us, and Uc.: Lane Volume vs. Tme

Hourdaks, Mchalopoulos, and Kottommannl 34 40-50 50-60 60-70 767 70-80 769 80-90 90-100 443 100-110 441 439 437 434 432 430 428 426 169 171 173 175 177 179 181 183 185 187 189 191 193 195 197 199 201 203 205 207 209 211 213 215 217 219 221 223 225 227 229 231 233 235 237 239 169 171 173 175 177 179 181 183 185 187 189 191 193 195 197 199 201 203 205 207 209 211 213 215 217 219 221 223 225 227 229 231 233 235 237 239 (a): Actual speed contour Speed (smulaton) 767 769 443 441 439 40-50 50-60 60-70 70-80 80-90 90-100 100-110 437 434 432 430 428 426 (b): Smulated speed contour before stage 2 Speed (smulaton) 40-50 50-60 60-70 76770-80 769 80-90 90-100 443 100-110 441 439 437 434 432 430 428 426 169 172 175 178 181 184 187 190 193 196 199 202 205 208 211 214 217 220 223 226 229 232 235 238 (c): Smulated speed contour after stage 2 Fgure 2: Average Manlne speed contours used n calbraton.

Hourdaks, Mchalopoulos, and Kottommannl 35 80 70 V e h c l e 60 50 40 30 20 10 actual queue smulaton queu after e stage 3 smulaton queue before stage 3 0 s 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 tme (5-mn ncrements from mdnght) Fgure 3: Smulated and actual queue at TH-55WB ramp.