Development of TDM Impact Measurement Guidelines

Transcription

1 Development of TDM Impact Measurement Guidelines Canadian TDM Summit Halifax, 22 October 2008 David Kriger, itrans Consulting Inc. Geoff Noxon, Noxon Associates Ltd. 1 Objectives As part of a Transport Canada study on the development of guidelines for measuring TDM impacts, we want to: Explore best practices in measuring TDM Gain your inputs on: Your needs What should the guidelines be for measuring the impact of TDM initiatives? 2 1

2 Format Overview: why is this important? Best practices: evaluation frameworks What the experts say Scenario exercise Next steps (wrap up, user needs survey) Discussion welcome! 3 Overview Why is measuring TDM so important? 4 2

3 Premise What gets measured tends to get done. If you don t measure results, you can t tell success from failure. If you can t recognize success, you can t reward it. If you can t recognize failure, you can t learn from it. David Osborne and Ted Graebler Reinventing Government, What Does TDM Do? TDM is intended to impact the demand for travel or transportation infrastructure, by decreasing: Travel demand overall Demand for one mode by shifting it to another Travel demand during a specific time period Travel demand for a specific mode during a specific time period but how do we know how to plan our programs, or if they are working, or if we have reached the intended audience? How do we know if we have succeeded? 6 3

4 What is Included as TDM? Branding and positioning Information and education Targeted marketing Special events Recognition and rewards Ridematching Guaranteed ride home Traveller information services Road or motor vehicle use pricing Transit pricing Workplace-based programs School-based programs 7 Why Measure Impacts? Understand what s working and what s not: Definition of working depends on the goals Specific applications: Identify who are the target audiences Help to plan the project Be able to complete cost-benefit analysis Compare investment in TDM to other types of transportation investments Secure project approvals and funding Get buy-in from target audiences Benchmark and track progress over time (evaluation) Report back to funders (what their money got them) All of these require measurements that are consistent locally and nationally 8 4

5 Challenges What should I measure? What are the dimensions of what I am measuring (e.g., frequency, permanence)? What data are available? How much do they tell us? How reliable? What is the purpose of the measurement (e.g. program evaluation v. benchmarking)? How easy is it to collect data? 9 What is Success? To understand success must define success Different programs have different goals: Reduce congestion in peak periods Improve transit ridership Increase physical activity through active transportation Improve road safety Decrease costs to individuals and businesses Decrease infrastructure costs Improve quality of living Reduce greenhouse gases (GHG) or Criteria Air Contaminants (CAC) 10 5

6 Spatial Focus Programs can be area-wide (e.g. cycling promotion) or site-specific (e.g. employer-based commuter program) Different types of programs require different measurement strategies All strategies should fall under consistent measurement framework Site-specific programs are easier to measure: Boundaries are defined Willing participants in program (in most cases) TDM is less likely to get lost among other changes 11 Evaluation Frameworks Methods of organizing measurements 12 6

7 What is a Best Practice? Applied innovation Practices proven successful 13 Evaluation Steps MOST MET = European Mobility Management (MM) Strategies for the next Decades - Toolkit Includes a step-by-step guide for measuring impacts Form objectives and mission statement Identify target groups Choose MM (TDM) instruments and services Apply assessment levels Specify indicators Monitor Evaluate 14 7

8 Need Different Levels of Information MOST MET uses 4 Assessment Levels applicability depends on initiative: Framework Conditions (all projects): S Spatial Framework Conditions P Personal Framework Conditions Mobility Management Services: A Knowledge of MM Services B Usage of MM Services Travel Services: C Satisfaction with MM Services D Acceptance of Travel Options E Experimental Individual Travel Behaviour F Satisfaction with Travel Option Mobility Behaviour: G Permanent Individual Travel Behaviour H System Impact 15 Need Different Levels of Information TCRP Report 107 (US): Analyzing the Effectiveness of Commuter Benefits Programs (transit-based) Defines categories for measurable outcomes: Awareness Participation Travel behaviour changes Agency impacts Regional impacts 16 8

9 The Basics of Measurement Applications determine data collection design: Statistically significant can be used to measure impacts, detect changes beware bias! Informative used to understand results, develop plans, make improvements, adjust programs, Two sets of data: Before data what is the baseline? After data what changes have occurred due to the program? Changes due to TDM can be small and statistical errors combine (before and after): The error has to be smaller than the change in order to be statistically valid (50% ± 2% transit share increased to 52% ± 2% reveals a change of 2% ± 4%) 17 Creating a Baseline Need information on: The site itself services available, parking, pricing, transit, bicycle facilities, working hours, etc. Basic transportation data Specific data on participants (schools / employers or individuals): Allows for future categorization into groups when data are combined Can help tailor the plan to the individual or site (not statistical measurement, but information for the development of the plan) Travel patterns of potential clients 18 9

10 Understanding Participation Every agency should understand their participation rates and keep complete and up to date records The Auckland Regional Transport Authority (ARTA) requires all participating schools and employers to complete a Commitment Contract: Completed contract = participant Incomplete contract = NOT participant Document this information in a database. Contract includes basic site information (number of employees, hours, etc.) When a travel plan is complete, add to the file 19 Categorization of Indicators Every assessment level has its own indicators Some indicators can only be calculated using other indicators TCRP 107 uses three categories: Activity or output indicators: relate to activities, processes of the initiative e.g., number of marketing calls, amount of media attention Outcome indicators: program results, tied to goals e.g., participation rates, ridership, emissions reductions Cost-effectiveness indicators: quantify economic benefits i.e., cost of achieving outputs or outcomes 20 10

11 Possible Indicators (1) Awareness, participation, satisfaction: Percentage of population that has heard of program Percentage of population that has tried program Number of participants Number of calls Customer satisfaction (for consultation service) Satisfaction with trial mode 21 Possible Indicators (2) Travel behaviour changes, system / regional impact: Mode share Vehicle trips per 100 employees Vehicle-kilometres travelled (VKT) is an important measure: e.g., 1 vehicle travelling 0.5 km = 0.5 VKT VKT can be used to estimate: Fuel savings GHG / CAC emissions Number of accidents reduced Number of kilometres walked or biked Transit passenger-kilometres 22 11

12 Cost / Benefit Measurement (1) Economic benefits to calculate / measure: Health and safety Congestion (user delay) Emissions Land use impacts Infrastructure (public and private) 23 Cost / Benefit Measurement (2) Employer benefits to calculate / measure: Employee travel expenses Parking cost Office space Recruitment / retention Productivity 24 12

13 Predicting Impacts Want to predict impacts for use in planning Want to ensure that the predicted impacts are also the same impacts that will be measured Experience-based: Basic expect what we ve got before, more or less Model-based: Predict changes in mode share and/or VKT Calculate expected cost / benefit 25 Prediction Models EPA COMMUTER Model MOST MAX European model (under development) WTRM (Worksite Trip Reduction Model) Center for Urban Transportation Research, University of South Florida Transfund New Zealand Trip Reduction Impacts of Mobility Management Strategies (TRIMMS) EPA Business Benefits Calculator 26 13

14 Getting Meaningful Data From Surveys Surveys are useful to find out both the numbers (counts, statistically-valid travel surveys, etc.) and the motivation (opinion surveys, consultation, etc.) Statistical significance important School roll surveys 27 Measuring Distance If VKT (or dependent indicators) are important, need accurate distance travelled: People do NOT accurately estimate their own travel distance, even for routine trips Understand how difference varies by mode and trip type: Contra Costa 511 uses average trip distances to calculate the impacts of measures maintain accuracy by: Building averages for very specific modes/trip types (i.e. walk to school, walk to work, commuter rail, light rail, bus rapid transit, bus, drive alone, drive with others, etc.) 28 14

15 Geo-Coding Addresses for Distance Picture place holder New Zealand s national base survey Questions can be add or removed by agencies Geo-coding capability built-in Uses addresses to calculate distance, then throws them away to protect privacy Ties distance to mode 29 Geo-Coding for Accuracy Picture place holder 30 15

16 What do the experts say? 31 What the Experts Have to Say Interviewed several experts and practitioners from Canada, USA and New Zealand There is a wide variety of initiatives happening on a range of sub-topics All agree that developing guidelines is the right direction: Build a set of standardized methods and tools Set basic requirements and practices Lay the foundation for future work All had a shortlist of most important things 32 16

17 The Most Important Thing Know who and how many are participating Pick a couple of important indicators and measure those really well (e.g., VKT as the basis for GHG, CAC) Be aware of biases Understand how to accurately measure distance Understand that there are no perfect tools Save time and money by sharing resources Work questions into existing surveys, where applicable Simplify the process, but maintain significance in the numbers 33 Some Great Ideas! Record odometer readings for all vehicles annually as part of the vehicle registration process this could provide great data for year-to-year changes in VKT Count more modes, more often Use Roll surveys at school teachers are doing roll anyway, can add a question about mode Create base tools that everyone can make their own Record and maintain data in a consistent way regionally, or even nationally use these data to develop predictive tools in the future (e.g. SARA) Provide service after sales for the guidelines themselves 34 17

18 That was the experts but what do YOU say? 35 Scenarios Purpose: Apply some key principles & ideas Illustrate issues that arise in real world application Demonstrate the need for creativity Process: Assign 1 scenario to each table Discuss 5 focus questions for 2 minutes each Report back & discussion on each scenario: responses, problems, uncertainties 36 18

19 Scenarios 1. Key outputs: What will be done? e.g. number of free passes distributed, number of promotional events held 2. Key outcomes: What will change? e.g. transit ridership, carpooling activity 3. Key indicators: What will be measured? e.g. transit trips/week/employee, persons/vehicle 4. Methods of data collection & analysis: How will it be measured? e.g. questionnaire distributed at staff meetings, driveway counts 5. Challenges & weaknesses: What are the concerns or limitations? e.g. endurance of effects, chance of low response rate 37 Wrap Up: Next steps 38 19

20 Next Steps Web-based survey of user needs Synthesize user needs, practices (from surveys) and best practices (from literature and interviews) Identify candidate best practices Prepare draft guidelines Finalize into report (end November 2008) 39 What do you have and what do you need? Please take 5 minutes to fill out the short survey provided 40 20

21 Survey of User Needs 1. What TDM initiatives does your organization deliver? 2. Have you quantified these initiatives? 3. If so, what indicators do you use? 4. Are there any indicators that you would like to use, but cannot? 5. If so, why not? 6. What types of data are available to your organization for general use? 41 For more information: