This draft Framework is intended to guide LSTF-funded local authorities on the monitoring and evaluation of LSTF.

Transcription

1 This draft Framework is intended to guide LSTF-funded local authorities on the monitoring and evaluation of LSTF. DfT would welcome any comments LSTF-funded authorities and key stakeholders have on this document. We are seeking comments particularly on questions posed in bold. Local Sustainable Transport Fund Draft Monitoring and Evaluation Framework July 2012 (1) Overview 1.1 Introduction This framework sets out guidelines and good practice recommendations for what projects receiving funding through the Local Sustainable Transport Fund (LSTF) will be expected to monitor and evaluate. This framework has been designed to be flexible and reflect local variation in schemes and data collection. It is based on a light-touch, resource efficient approach which seeks to balance the needs of accountability and developing the evidence base with the burden of additional monitoring and evaluation activity. The DfT expects to work collaboratively with local authorities and others to deliver this framework. It aims to build on the existing evidence base to inform future local and national decision making in enabling sustainable travel choices. Whilst the framework sets out DfT's priorities for monitoring and evaluation, it recognises that other stakeholders may have additional priorities and supports the consideration of these by local authorities. This is a draft framework for which DfT is seeking views on how best to balance various views and constraints to maximum effect. It has been developed as a result of conversations with LSTF Tranche 1 project teams, academics and specialists in sustainable travel monitoring and evaluation, Large Project bidders (through a workshop on 14 May 2012), and contributions from in-house DfT analysts. The intention in circulating this draft to funded project teams, academics and other key contacts is to gauge feasibility of delivery, test the anticipated quality of results from this work, and to improve the evaluation framework overall before publication in late Once published, this document will form the basis for how monitoring and evaluation will be collaboratively delivered by funded project teams with the Department. Please provide feedback by 14 September to DfT at lstf@dft.gsi.gov.uk 1

2 1.2 Policy objectives of the Fund Funded LSTF projects are expected to deliver on the following primary objectives: support the local economy and facilitate economic development, for example by reducing congestion, improving the reliability and predictability of journey times or enhancing access to employment and other essential services; reduce carbon emissions, for example by bringing about an increase in the volume and proportion of journeys made by low carbon, sustainable modes including walking and cycling. Projects may also deliver on some or all of the following secondary objectives: help to deliver wider social and economic benefits (e.g. accessibility and social inclusion) for the community; improve safety; bring about improvements to air quality and increased compliance with air quality standards, and wider environmental benefits such as noise reduction; and actively promote increased levels of physical activity and the health benefits this can be expected to deliver. 1.3 Objectives for monitoring and evaluating Monitoring seeks to check progress against planned targets and can be defined as the formal reporting and evidencing that spend and outputs are successfully delivered, milestones met and changes in outcomes tracked over time. Monitoring data plays a key part in evaluation by providing valuable evidence throughout the lifetime of the initiative. Evaluation is the assessment of the effectiveness and efficiency of the initiative during and after implementation. It seeks to measure and attribute outcomes and impacts generated by the initiative in order to assess whether the anticipated benefits have been realised and whether any unanticipated impacts have occurred. National-level objectives Demonstrate the contribution of the Fund to delivering economic growth and carbon reduction Provide accountability to taxpayers and the Public Accounts Committee Develop evidence to inform the case for future local or national funding Show how the LSTF can help deliver secondary objectives (e.g. social impacts, air quality, health, safety) 2

3 Local objectives Learn what works in enabling sustainable travel choices, in which context, for whom and why Better understand how to most effectively and efficiently design, target and deliver sustainable travel initiatives Test innovative approaches where the evidence base is underdeveloped Further objectives indicated by Large Project bidders Generate evidence to persuade local elected representatives Improve appraisal, to inform future proposals Identify which measures to continue post-2015 Demonstrate local impact, with local data Benchmark outcomes against other local authorities Influence of other funding sources on measures is taken into consideration 1.4 Principles underpinning this framework Proportionality - Developing a light-touch, resource efficient approach to monitoring for local authorities (where possible utilising existing data sources and annual outputs monitoring) Partnership - Working collaboratively with local authorities and other organisations Prioritisation - Developing the evidence base to inform future local and national decision making (targeting key evidence gaps) 1.5 An introduction to the evaluation framework Overall approach The evaluation framework contains three components: (1) Detailed research on a few key priority questions where the evidence is relatively weak (e.g. the impact of sustainable transport on the economy and carbon) this will be progressed as case studies with a small selection of project teams (still to be selected). This will provide a detailed account of how specific sustainable travel measures work and how these impacts vary e.g. by location or type of individual. (2) Outcome monitoring within large project teams e.g. changes in traffic levels and use of sustainable modes. This will provide evidence across a number of areas as to whether the programme is having an impact on those factors which drive economic growth and carbon reduction e.g. reduced congestion, increased uptake of lower carbon modes. (3) DfT is already collecting annual outputs reports from all funded project teams, which monitor what the investment has been spent on. The first 3

4 set of these reports (from tranche 1 projects) were submitted in July The proportionate approach which is being taken means that fully evaluating all funded projects would not be cost effective and would risk small effect sizes which may not be distinguishable from other factors influencing behaviour. Therefore, the tiered framework aims to direct effort and resources to the areas which provide the most value and to minimise burdens on the project teams. Responsibilities The DfT will support and co-ordinate the monitoring and evaluation programme, and the responsibility of ensuring progress and delivery of monitoring and evaluation plans will be through the local authority project teams. In line with the Government s approach to decentralisation, the framework does not fully prescribe how projects will be monitored and evaluated, but sets out objectives and models of best practice. The local authority project teams are responsible for developing the approaches to meet the Department s objectives as well as ensuring their own requirements are met. Department for Transport Large project teams (section 2.2) Small project teams Local Government Association (LGA) Ensure programme of evaluation is delivered and provides fit for purpose evidence to demonstrate accountability and learn lessons for future decision making Support project teams through sharing best practice and ensuring consistency of evaluation of projects Co-ordinate dissemination of evaluation findings Record and report to the DfT annually on inputs and delivery of outputs Routine monitoring of core outcomes Lead the development and delivery of case study evaluations (minimum 1 per large project team) Record and report to the DfT inputs and delivery of outputs If selected, contribution to case study evaluations Provide website/information platform through existing mechanisms to share best practice 4

5 1.6 Budgetary issues Large Project bidders were encouraged to budget for monitoring and evaluation when developing business cases, to enable adequate participation in delivering this framework. Unless otherwise negotiated (e.g. for in-depth analysis for case studies), large project teams are expected to meet the costs of engaging with the monitoring and evaluation requirements set out in this framework. Where small projects are identified for case studies, DfT will negotiate with the relevant local authority what budget is required, and whether DfT is able to meet these costs. 1.7 How should evaluation be conducted and the lessons learned and disseminated? We believe it is up to Local Government to decide, supported by the LGA and in agreement with DfT and projects. Question: Are LSTF-funded authorities content with this approach? 1.8 Expected outputs from this framework We believe it is for Local Government to decide on the reporting process and dissemination. For DfT, we require a monitoring and evaluation plan which should be published by all large projects. 5

6 (2) Details 2.1 Detailed description of outputs monitoring Project teams are invited to complete an Annual Progress Report at the start of each financial year, for submission to the DfT in July, to cover the previous financial year (where funding was approved part way through the previous funding year, this reporting requirement starts from the date funding was announced). The report covers: 1. Project Description, Spend Profile (including variations description) and Progress Summary 2. Scheme Element Outputs 3. Case Studies (optional) These reports will form the basis for information exchange between projects on good practice and lessons learnt. Project teams are encouraged to make these reports publicly available via the local authority s website. These reports will provide an ongoing record of outputs during delivery, which will then form the basis for any subsequent analysis of delivery as required. All project teams are strongly recommended to keep a project diary and assign a team member to complete it regularly, to keep track of what the project is delivering, but also to capture information about external events that may impact on the outputs and outcomes (either positively or negatively). Project teams will find such project diaries invaluable when carrying out basic programme monitoring and any associated evaluation activities at a later stage. 2.2 Detailed description of outcomes evaluation (for primary objectives) This section sets out the metrics that will be monitored and analysed by all large project teams to test the extent to which sustainable travel measures can help support local economic development and reduce carbon. Whilst small scale project teams are not required to collect this data, they may find these metrics useful in setting out their own monitoring programmes. Collecting these metrics is one part of our approach to improving our understanding of the impact of the LSTF and variables which may influence the success of sustainable travel measures in supporting local economies and reducing carbon. These metrics will provide information on the impact of a range of sustainable transport measures applied in many different contexts. When viewed in isolation this information is likely to provide an indication of trends. To allow more robust conclusions to be drawn we propose to combine the results of this monitoring programme with the in-depth case studies (which 6

7 will explore the impact in more detail in some areas) and other contextual data. LSTF recipients should be aware that the metrics have been selected to monitor the main expected impacts of the programme. The metrics are not intended to be an exhaustive list of all potential impacts and as such may not reflect all aspects of individual packages or answer specific research questions at a local level. In these circumstances the onus is on individual recipients to consider what additional data or contextual information should form part of their monitoring programme and/or reports. When selecting metrics to include in the monitoring programme we have considered the extent to which data is already collected. Local authorities collect a variety of information and data to monitor the performance of their transport networks and the impact of their investments e.g. Local Transport Plan. LSTF-funded authorities are encouraged to consider a range of potential data sources to inform the monitoring programme and to set baselines. The 2011 Census may provide a useful baseline, particularly for small geographic areas. A detailed prospectus for the release of Census data is available from the Office for National Statistics (ONS) website 1. Useful data items include main mode of travel to work, car ownership and, for tourism, numbers of overnight visitors. We do not propose specifying how the metrics should be collected as local authorities use a number of methods to collect data and we are keen to minimise costs by drawing on, and possibly adapting, existing series. However, to ensure that there is sufficient consistency across the programme we have identified some minimum requirements which we would expect to be met for each metric. We will also encourage recipients to describe how they undertook their data collection so that data quality and robustness is understood. Project teams should compare their monitoring data against suitable controls. In some cases, this could be done using national trend data but in other cases, it may require some data for areas or roads which are unaffected by LSTF interventions. There may be scope for project teams to join up with those in neighbouring authorities to share control data. Project teams will be expected to collect and report data for each year in which they are in receipt of funding and for 2015/2016 (i.e. the year after which LSTF funding ceases) unless otherwise specified below or agreed in advance (e.g. this might not be proportionate with surveys). Question for LSTF-funded authorities, in particular large project teams: What is your view on what should be required, by whom, and to what timeline? To cover data analysis and interpretation responsibilities

8 2.2.1 Monitoring changes in travel patterns A pre-requisite for LSTF to have an impact on the economy and carbon is for the programme to change how people travel. The objective of this section is to set out what monitoring information should be collected to help identify the extent to which the programme has led to changes in travel behaviour. Framework and metrics - principles The aim is to determine whether there has been a statistically significant change in the use of different modes within the areas covered by large projects. This will be assessed at an aggregate level and should provide an indication of trends within the intervention area. The case studies will be used to provide a more robust estimate of changes in travel patterns and to consider more detailed questions e.g. the longevity of impacts, variation in the impact by geographical area etc. The monitoring data recommended here draws heavily on the secondary data sources analysed during the evaluation of the Sustainable Travel Towns (STTs) 2. These have been selected because: - It will make it easier to benchmark the results of the LSTF against the STTs; - These metrics were originally selected for the STT evaluation as they were readily available and it would appear reasonable to assume that most other local authorities regularly collect this kind of data so the additional costs are minimised; - As this data is collected by other authorities it also reduces the cost of collecting information from comparator areas; - The direction and order of magnitude of impacts estimated in the STT evaluation using the monitoring data were broadly consistent (with some exceptions) with the results of the household surveys; and - It is possible to draw on practical experiences of using these metrics for monitoring purposes. The report on the evaluation of STTs identified a number of practical issues which potentially limited their use in that evaluation. These included amongst other things count location, timing and continuity. To help reduce the risk of the same problems impacting on this monitoring programme, we have identified a number of recommendations about how the data should be collected. 2 The Effects of the Smarter Choice Programmes in the Sustainable Travel Towns (Sloman et al, 2010). arterchoiceprogrammes/pdf/summaryreport.pdf 8

9 Metrics Given the costs associated with household travel surveys, DfT suggests that the monitoring requirements placed on all large project teams is limited to count data. We think that in most cases local authorities are collecting this type of data and should be able to meet the requirements at minimal costs. The data should provide an indication of overall trends in highway traffic, cycling, public transport use, and walking. However, it will not give a deeper understanding of changes in travel patterns (e.g. trip lengths, destinations, journey purpose) and getting robust/unbiased results will require careful consideration of the location of counts. Question: Do LSTF-funded authorities, in particular large project teams, consider this to be correct? What role can surveys play in providing greater granularity? Every large project team will be required to monitor and report changes in: Vehicle flow Bus and (where applicable) light rail patronage Number of cycle journeys Number of pedestrians journeys It will not be possible to measure all movements within the areas covered by LSTF projects and so the impact should be monitored at a sample of locations. This sample will need to be carefully selected to ensure that it is representative, sufficiently large and targeted to provide meaningful and unbiased data. The value of the monitoring data will be strengthened if it can be compared to data collected on similar routes which have not benefitted from LSTF investment. Local authorities are encouraged to explore the options for collecting comparator data in their areas. Given that many of the LSTF projects intend to influence travel behaviour across an area we would anticipate that monitoring should occur across the affected area. Cordon counts and screenlines (e.g. on approaches to town centres) may provide useful secondary data but should only be considered as the primary data collection method if costs are prohibitive and/or there are practical issues which cordon counts can help overcome (e.g. obtaining ticket data from bus operators). Each of the metrics is described in more detail in the boxes below. We would expect these data to be reported for each year although monthly/quarterly data should be provided where possible (e.g. from automatic traffic counters (ATCs)). This includes a high level definition of the metric along with suggestions about how the data might be collected. We have also identified 9

10 further optional disaggregation of the results which local authorities may want to consider but are not required to collect or report. In many cases, centrally collected local authority level official statistics exist for the metrics and these have been highlighted in the tables below. In many cases, the sample size/geographic level of these statistics will be insufficient for LSTF intervention monitoring, especially interventions which are aimed at specific routes/areas/demographic groups. However, in some cases, the official data may be sufficient, or at least provide a useful supplement/means of validation for data collected by local authorities. As every intervention is different in nature and scope, it is for local authorities to judge how applicable these official statistics would be and highlight them in their monitoring returns as appropriate. Relevant statistics have been highlighted in the table below, referenced by the series name. Further details can be found on Measure Definition Potential data collection methods/existing sources Optional disaggregation Measure Definition Potential data collection methods/existing sources Optional disaggregation Vehicle flows Average 24 hour figures for seven day, two way flows National Road Traffic Estimates data DfT TRA data series and underlying count point data (Note: these are unlikely to be located in sufficient numbers or in the right place to provide an accurate representation of traffic trends and we would expect this to be supplemented by additional observations. Furthermore, most of the count points are not surveyed every year.) Automatic Traffic Counters Manual Classified Counts (this may need to be combined with ATC data to scale up results to 24 hour figures) CCTV/Automatic Number Plate Recognition although care will need to be taken to make sure these are not moved around over time The merits of different approaches and recommended sample sizes are discussed in This will depend on the methods used to undertaken the vehicle counts but in some instances it may be possible to breakdown changes in flow by vehicle type. This might be useful as certain vehicle types (e.g. cars and taxis) are likely to be more susceptible to change given the nature of interventions funded by the LSTF. Bus and light rail patronage Average number of bus/light rail passenger journeys DfT Bus and Light Rail statistics series annual bus and light rail passenger journeys, by local authority and PTE. Passenger boardings from bus/light rail company reported patronage data Bus/light rail occupancy at cordon/screenline May want to consider by route or length of journey, particularly if the LSTF interventions are very route/area specific, but scope to undertake this may be limited by commercial confidentiality. 10

11 Measure Definition Potential data collection methods/existing sources Cycle journeys Cycle journeys Active People Survey - household survey, administered by Sport England, including questions about how often respondents cycle and the purpose; Sample size ~ per local authority. National Road Traffic Estimates manual counts (cyclists) although minor only and used on STT evaluation to corroborate. Manual counts Automatic Cycle Counts Optional disaggregation Measure Definition Potential data collection methods/existing sources Optional disaggregation Whatever method is used, care should be taken to minimise the influence of external factors when reporting and interpreting the results (e.g. weather, temporary road works). If data is collected continuously or more than once a year it might be useful to consider whether there is any difference in trends at different points in the year. Walking journeys Pedestrian journeys average number of walking journeys Active People Survey - household survey including questions about how often respondents cycle and the purpose; Sample size ~ per local authority. Manual counts Whatever method is used, care should be taken to minimise the influence of external factors when reporting and interpreting the results (e.g. weather, temporary road works). If data is collected continuously or more than once a year it might be useful to consider whether there is any difference in trends at different points in the year. Data from counts can provide a rich data source with which to monitor the impact of LSTF on travel patterns. However, the results obtained from these sources can be sensitive to decisions about how the data is collected and analysed. In developing monitoring plans authorities should consider the following: Sites of counters should be reviewed to ensure that they are a representative sample of the intervention sites/areas. Where counts are required to be undertaken at new locations it is recommended that these are in addition to, rather than a substitute for, existing sites. Count data should be provided for locations/routes that are in areas likely to be affected by the scheme and should be considered for control areas. To avoid any potential bias these should be identified prior to the scheme being implemented. 11

12 Count data should be analysed by the recipient and significant spatial differences identified the evaluation of the sustainable travel towns found a significant difference in traffic reductions between inner and outer areas. Counts should only be included if the data series they provide is of sufficient length to be of use. Automatic Traffic Counters and Automatic Cycling Counters should be routinely checked to make sure they are working (it is recommend that this is done every 3 months) to limit any gaps in the data series e.g. to malfunctioning equipment. Missing values in the data series of individual counters should be investigated and interpolation considered. Reporting should include: a map showing the location of counts; a description of how data was collected at these sites e.g. manual counts, ATCs (and if manual the dates of counts should be recorded); a description of how each site has been allocated to different geographical areas for analysis (if applicable); and monthly/annual (as applicable) counts (in absolute terms) should be provided for each site or route. Anomalous values or sudden shifts in counts should be investigated and explanations reported (e.g. road works, opening of new supermarket, occupation of new housing development, and change in bus services). If interpolation or other adjustments are applied to the raw data these should be noted and recorded. To help identify whether the introduction of LSTF-funded measures has had an impact on trends we recommend that historical data on sites is reported where available. LSTF large project teams should not feel constrained by the minimum reporting requirements identified above and are encouraged to analyse this rich dataset. Chapter 17 of Sloman et al (2010) 3 provides useful examples of how data can be presented and analysed which recipients may find useful. It is recognised that the proposed monitoring metrics will not provide a complete picture of changes in travel patterns which result from implementation of LSTF projects e.g. changes in average trip distance, destinations. Local authorities are encouraged to draw on other available data, including any additional data to be collected for the LSTF, when compiling the monitoring report and use this to cross-check the results they obtain. 3 ogrammes/pdf/chap17.pdf 12

13 The section above encourages large project teams to undertake analysis of the monitoring data themselves. We have specified the format in which we would expect local authorities to report raw data and set out the contextual information we would expect them to provide. Setting these common standards should help ensure that other local authorities can compare the results against their own (or other authorities ) performance. This should also allow DfT or another third party to review the monitoring results, or pool the results to calculate the overall impact across the programme (for large project teams). Question: Are LSTF-funded authorities, particularly large project teams, content with this approach? Question: What data do you consider you have or could easily obtain regarding vehicle (mainly car and bus) occupancy and whether they think such a metric could usefully and feasibly be included? Economy The proposed metrics have been selected to monitor the direct impact of the LSTF programme on outcomes which are believed to influence economic development e.g. the cost of accessing a particular site, journey times, and footfall. The scope for monitoring data on economic impacts (e.g. number of people in employment, retail expenditure) was considered but rejected on the basis that it would be difficult to isolate the impact of LSTF investment amongst the various other factors that influence high level economic indicators. Framework and metrics - principles The LSTF guidance was not prescriptive about how funded authorities should support their local economy and facilitate economic development. As a result there is a wide variation in the packages that will be implemented as part of the programme which support economic development in different ways. The metrics that we propose to collect are based around five categories of intervention which cover a majority of funded projects: - Congestion relief e.g. improving the frequency and reliability of public transport services; - Increasing the use of non-car modes e.g. increasing rates of cycling and walking from workplace travel plans; 13

14 - Helping get the unemployed into work e.g. cycle/scooter loans; - Encouraging town centres and high streets e.g. giving greater priority to pedestrians ; and - Supporting tourism e.g. new off-road cycle routes for leisure purposes. Large project teams are not expected to collect data for all five categories of intervention listed above. They should only collect data on those metrics which relate to the objectives of their scheme. Metrics Congestion relief Many project teams are using the LSTF to apply sustainable transport measures to help reduce the impact of congestion on local economies. Left unchecked, congestion can increase costs to business due to longer journey times (e.g. leading to higher fuel costs, lost productivity whilst in transit) and reduced reliability (e.g. leading to the costs of holding higher stock levels to mitigate against later deliveries). Congestion may also reduce the size of the labour market local businesses may be able to draw from particularly if high and unpredictable journey times put potential employees off commuting to some locations. We propose that schemes that are using the LSTF to reduce congestion to support economic growth monitor the following indicators: Average AM peak journey time per mile Variation in journey times on key corridors In assessing the impact of the programme on congestion, large project teams are also encouraged to review these metrics alongside data on traffic flows (see section 2.2.1). Measure Definition Average AM peak journey time per mile Average AM peak journey time per mile (minutes and seconds) Requirements Reported across whole intervention area (e.g. local authority level) and on key corridors targeted for investment. Data collected on at least an annual basis. Potential data collection methods Optional disaggregation Trafficmaster data Automatic Number Plate Recognition DfT CGN statistics series vehicle speeds on locally managed A-roads during the morning peak, quarterly by local authority. Recipients may want to consider changes in peak journey times by link (e.g. on key corridors) although they should be aware that it will be more difficult to establish statistically significant changes. 14

15 Measure Definition Variation in journey times on key corridors Variation in journey times (7AM to 7PM) on key corridors targeted for investment [ideally would want to analyse as day to day variability within time bands but may need to settle with variation in average journey times across the day] Requirements There should be a sufficient number of observations to ensure that any differences reported are statistically significant. Existing data collection methods Optional disaggregation Trafficmaster data 4 Automatic Number Plate Recognition Recipients may want to consider changes in peak journey times by link (e.g. on key corridors) although they should be aware that it will be more difficult to establish statistically significant changes. Increasing the use of non-car modes Encouraging the use of alternatives to the car can contribute to economic growth in a number of ways: Increasing the pool of labour that businesses have access to, allowing them to expand and find the best people (20% of adults live in a household without access to a car 5 ). Improving the resilience of the transport network by providing feasible alternatives for when things go wrong e.g. road closures. If the promotion of non-car modes leads to an increase in physical activity, this may reduce the level of absenteeism. Allow firms to expand at existing sites without having to provide additional car parking. Reducing the use of car by promoting alternatives can also lead to reduced congestion and may also help make an area more attractive these impacts are captured elsewhere in the framework. The previous section set out the data to be collected by all large project teams to monitor the impact of LSTF on a range of modes. We propose the following additional data for those schemes that are using the LSTF to support economic growth by promoting the use of alternative modes for access to specific sites e.g. business parks, schools, and city/town centres: Households able to access site within 20/40 minutes using public transport/walking, car and cycle and/or households with access to site within a reasonable time period by public transport/walk, car and cycle Modal split at specific site either cordon counts or survey 4 The current Trafficmaster contract will provide data to the end of DfT will be awarding a new contract for the Journey Time Data Service in September This contract is for two years with the option to extend for up to a further two years. Under the new contract, provision of data to local authorities will continue, although the data may not be the same source or format as the current provision 5 National Travel Survey, 2010 (DfT). 15

16 Measure Definition Total number of households able to access site within 20/40 minutes using public transport/walking, car and cycle Total number of households that are able to access site within 20 minutes/40 minutes using public transport/walking, car and cycle. Note: we would only anticipate this data being collected and reported if the recipient is improving access. This indicator is unlikely to provide a good indication of the success of schemes based on soft measures (e.g. travel plans). Requirements Separately reported for public transport/walking, car and cycling. Separately reported for each lower super output area (LSOA) in which there is a significant site(s) targeted as part of the programme. Figures should be reported for each year of the LSTF programme. Historical information should be reported (where available) to provide indication of trends. Accessibility indicator dataset is available back to Should report both absolute number of people with access and percentage of households within local authority. Existing data collection methods Optional disaggregation Project teams may want to make use of existing data collected for the accessibility statistics published by DfT. Teams may also want to investigate options for producing their own bespoke accessibility measures and travel time calculations using off-the-shelf software Recipients may want to analyse changes in accessibility to site from different LSOAs particularly if targeting improvements on particular corridors/routes e.g. new bus services. 16

17 Measure Definition Households with access to a site within a reasonable time period by public transport/walking, car and cycle This measure will use the continuous indicators applied in developing DfT s accessibility statistics. This is a measure based on the sensitivity of users to the travel time for each service, i.e. the longer it takes to get to a particular service, the fewer people will undertake the journey. However, instead of measuring the number of households within a geographical area that can access a service (regardless of where that service is located) within a reasonable time we would seek to estimate the number of households that can access a specific site (defined as the LSOA in which the site is located) in a reasonable time using the same underlying indicator. Note: we would only anticipate this data being collected and reported if the recipient is improving access. This indicator is unlikely to provide a good indication of the success of schemes based on soft measures (e.g. travel plans). Requirements Separately reported for public transport/walking, car and cycling. Separately reported for each LSOA in which there is a significant site(s) targeted as part of the programme. Figures should be reported for each year of the LSTF programme. Historical information should be reported (where available) to provide indication of trends. Accessibility indicator dataset is available back to Should report both absolute number of people with access and percentage of households within local authority. Existing data collection methods Optional disaggregation Project teams may want to make use of existing data collected for the accessibility statistics published by DfT. Teams may also want to investigate options for producing their own bespoke accessibility measures and travel time calculations using off-the-shelf software. Recipients may want to analyse changes in accessibility to site from different LSOAs particularly if targeting improvements on particular corridors/routes e.g. new bus services. 17

18 Measure Definition Modal split at specific sites Number of trips to/from site or area split by following modes: car (multiple occupancy); car (single occupancy); public transport; walking, cycling, motor cycle, goods vehicle, other. Requirements Reported for as many sites/areas in the intervention area as possible, but prioritising the busiest sites/largest employment centres. Data to be collected before scheme/travel plans are delivered and 1 year after scheme completed or travel plans fully rolled out (for areas and existing sites), 1 year after occupation for new sites. Survey/count should be undertaken for a single day on the same day of the week and same month of the year (during a neutral month). [Some useful pointers and a template are available at : and data/assets/word_doc/0004 /169438/Monitoring-Report-template.doc] Absolute number and percentage of total trips should be recorded. Existing data collection methods Optional disaggregation Staff questionnaire Cordon count Travel to work Areas from the 2011 Census. This would provide a good baseline measure and this could be monitored through repeat surveys with a sample of the population Results could be further analysed by time of day (e.g. with cordon counts) and journey purpose (e.g. with staff questionnaires). Helping unemployed people access work A number of LSTF project teams are seeking to support particular groups of individuals or communities (e.g. long term unemployed) in accessing employment opportunities. Examples of interventions include free public transport travel for a limited period, making people aware of travel options, subsidised cycle hire etc. Whilst these schemes could potentially promote additional economic activity (by expanding the effective supply of potential employees), these impacts are likely to be difficult to observe across the LSTF area. Therefore, the scope of monitoring activity will be to measure changes to the economic status of the target group at regular intervals. There is no published set of indicators which allow the success of programmes in helping people access work to be easily monitored. DfT plans to liaise with the Department of Work and Pensions (DWP) over the summer to explore methods for assessing the efficacy of employment measures. Our emerging thinking is that the following indicators are monitored for those initiatives which have significant components whose aim is to support individuals or communities in accessing employment opportunities: 18

19 Employment/benefit status of target population Proportion of unemployed people reporting transport as a barrier to getting employed. Question: How could data on the impact of schemes targeted at the unemployed be collected in a way which allows comparisons across the LSTF programme? Measure Definition Requirements Existing data collection methods Optional disaggregation Employment/benefit status of participants at [13] weeks Recording trends in employment / benefit status before, during and after the intervention period. This is seeking to observe the extent to which employment levels have increased amongst the populations being targeted, and for those who have moved into employment, how long this is sustained for. The data should be collected for the whole target population including those who chose not to participate in the initiative The measure of sustained employment should be consistent with the Department of Work and Pension's definition of 13 weeks Benefit status may need to be used as a proxy for employment given the difficulties of obtaining accurate employment data. We are inviting views from LSTF-funded authorities, in particular large project teams, in response to this draft framework We are inviting views from LSTF-funded authorities, in particular large project teams, in response to this draft framework Measure Definition Requirements Existing data collection methods Optional disaggregation Proportion of unemployed people reporting transport as a barrier to employment We are inviting views from LSTF-funded authorities, in particular large project teams, in response to this draft framework We are inviting views from LSTF-funded authorities, in particular large project teams, in response to this draft framework We are inviting views from LSTF-funded authorities, in particular large project teams, in response to this draft framework We are inviting views from LSTF-funded authorities, in particular large project teams, in response to this draft framework 19

20 Encouraging town centres and high streets A number of LSTF project teams are seeking to promote town centres and high streets as a destination. The methods being pursued to achieve this objective include improving transport links (e.g. additional bus services) and enhancing the place function of certain areas (e.g. by removing or redirecting traffic to create a nicer environment). It is proposed that at a minimum large project teams monitor the impact on the town centres and high streets that have been targeted for improvement as part of the programme. Large project teams are also encouraged to consider monitoring other sites to check whether any displacement of economic activity has occurred although we anticipate this can only be determined through a more detailed evaluation. Potential measures include: Number of households with access to town centre within a reasonable time. Footfall: the measurement of footfall should gauge the number and frequency of visitors to the area defined locally as the high street or town centre, not just visitors to shops or particular attractions. Vacancy rates: percentage of retail unit vacant. Consumer and business satisfaction: to measure whether high streets are serving the social and community purposes envisaged for them as well as their traditional retail functions, we need to measure users satisfaction. Do they like what they are finding and will they come back for more? 20

21 Measure Definition Number of households with access to the town centre within a reasonable time This measure will use the continuous indicators applied in developing DfT s accessibility statistics. This is a measure based on the sensitivity of users to the travel time for each service, i.e. the longer it takes to get to a particular service, the fewer people will undertake the journey. However, instead of measuring the number of households within a geographical area that can access a town centre within a reasonable time we would seek to estimate the number of households that can access a town centre or high street (defined as the LSOA in which it is located) in a reasonable time using the same underlying indicator. Note: we would only anticipate this data being collected and reported if the large project team is seeking to support town/city centres by improving access. This indicator is unlikely to provide a good indication of the success of schemes based on soft measures (e.g. travel plans) or that seek to improve the environment within town centres and high streets (e.g. widening of pavements). Requirements Figures should be reported for each year of the LSTF programme. Historical information should be reported (where available) to provide indication of trends. Accessibility indicator dataset is available back to 2007 Should report both absolute number of people with access and percentage of households within local authority. Existing data Project teams may want to make use of existing data collection methods Optional disaggregation collected for the accessibility statistics published by DfT. Can be broken down by mode: Public Transport/Walk, Car and Cycle. 21

22 Measure Definition Number of households with access to the town centre within a reasonable time This measure will use the continuous indicators applied in developing DfT s accessibility statistics. This is a measure based on the sensitivity of users to the travel time for each service, i.e. the longer it takes to get to a particular service, the fewer people will undertake the journey. However, instead of measuring the number of households within a geographical area that can access a town centre within a reasonable time we would seek to estimate the number of households that can access a town centre or high street (defined as the LSOA in which it is located) in a reasonable time using the same underlying indicator. Note: we would only anticipate this data being collected and reported if the large project team is seeking to support town/city centres by improving access. This indicator is unlikely to provide a good indication of the success of schemes based on soft measures (e.g. travel plans) or that seek to improve the environment within town centres and high streets (e.g. widening of pavements). Requirements Figures should be reported for each year of the LSTF programme. Historical information should be reported (where available) to provide indication of trends. Accessibility indicator dataset is available back to 2007 Should report both absolute number of people with access and percentage of households within local authority. Existing data Project teams may want to make use of existing data collection methods Optional disaggregation collected for the accessibility statistics published by DfT. Can be broken down by mode: Public Transport/Walk, Car and Cycle. We are inviting views from LSTF-funded authorities, particularly large project teams, on how to monitor footfall, vacancy rates and consumer/business satisfaction in response to this draft framework. Supporting tourism A few project teams are pursuing proposals to support the local economy by encouraging tourism e.g. improving off-road cycle tracks primarily for leisure use. Given that the number of authorities promoting such schemes is relatively low we do not propose including any specific metrics as part of the monitoring programme. Whilst we do not propose including any specific metrics relating to the impact on tourism as part of this monitoring programme, all relevant project teams 22

23 (small and large) are encouraged to consider how they demonstrate whether their schemes have had the intended impact e.g. by monitoring: Changes in total number of day and overnight visitors. Changes in the use of cycling/walking facilities e.g. cycle paths. The number of people reporting having cycled or walked whilst visiting. Local authorities may want to consider using existing visitor surveys as a way of collecting monitoring data. Question: Are there any other measures LSTF-funded authorities consider should be included? Carbon The LSTF guidance was not prescriptive about how funded authorities should reduce carbon emissions. As a result there is a wide variation in the packages that will be implemented. The proposed framework keeps to the main factors that effect carbon emissions from local travel. Framework and metrics principles Large project teams are expected to assess the carbon impacts of their scheme. In doing so the key factors for determining carbon emissions (outlined below) should be considered and converted into carbon impacts. This could be done, for example, using the Local Authority Carbon Tool produced by DfT. Suitable controls should be used to provide a comparison of the impact of the scheme against what would otherwise have happened. The key factors for assessing the carbon impacts of changes in local travel are: Distance travelled a greater distance travelled by a carbon emitting mode of transport would increase carbon emissions, other things being equal. o Proposed metric: the vehicle count data outlined in the Monitoring changes in travel patterns section should be used. For assessing carbon impacts, knowing the mode of transport is important since a kilometre (km) travelled by a bus and a car can generate rather different emission impacts. Vehicle speed carbon emissions per km vary with the speed a vehicle is travelling. For each type of vehicle, more CO 2 is emitted per km at low speed and high speed than at moderate speed. o Proposed metric: the congestion metrics outlined in the Economy section should be used where a project is intended or expected to affect vehicle speeds. For assessing carbon impacts, the mode of transport is important since speed emission relationships vary by mode. 23