REVIEWING THE BENEFITS: POST COMPLETION EVALUATION IN PLANNING AND PRACTICE

Transcription

1 SARAH LOWE Senior Transport Engineer WSP REVIEWING THE BENEFITS: POST COMPLETION EVALUATION IN PLANNING AND PRACTICE In recent years, VicRoads has developed the VicRoads Investment Evaluation Framework to provide a consistent approach to planning, undertaking and delivering post completion evaluations. WSP has been involved in VicRoads journey to transform evaluations by assisting VicRoads during the development stage of the Investment Evaluation Framework (the Framework); and completing full and partial post completion evaluations using the Framework. This conference paper discusses the importance of completing the investment cycle by reviewing and learning from completed projects; the development and key components of the Framework; and the experience and learnings gained so far from undertaking post completion evaluations. Undertaking post completion evaluations has been shown to be useful to assist with learning and improving, though there are still some limitations and challenges. The results and benefits of undertaking post completion evaluations will continue to improve over time through training and learning, and as it becomes imbedded into the everyday practice of VicRoads projects and programs. 1. Introduction The transport planning and engineering profession has strong skills in developing and delivering projects. We are generally well adept at planning for, designing and constructing/implementing projects. However, areas that have room for improvement include demonstrating that completed projects have achieved their intended benefits, and learning from our completed projects. VicRoads responded to this need within the organisation by strengthening its approach to evaluations, through enhancing its ability to measure and evaluate the outcomes of their investments. This resulted in a change to how VicRoads plans, delivers and learns from investments, to deliver the best value outcome for the community. This paper gives an overview of post completion evaluation with a focus on the VicRoads context and evaluation framework. This paper looks at: what VicRoads did to address the need to improve their approach to evaluations the VicRoads context the development and key components of the overarching document for post completion evaluation the Investment Evaluation Framework the experience and learnings gained so far from undertaking post completion evaluations of several VicRoads projects following their completion evaluations in practice. 1.1 What is post completion evaluation? Post completion evaluations review the outcomes and performance of an initiative after it has been implemented. A post completion evaluation should look at all aspects of an investment (project or program) including its planning, design, delivery and benefit identification processes, as well as the outputs and benefits it produces. Undertaking an evaluation at the completion of a project or program not only reports on performance and confirms outcomes achieved, but can also provide valuable learnings to improve processes and operations as well as improve future decision making by providing feedback into the planning stage of future investments. As such, post completion evaluations are generally undertaken for: Accountability: to demonstrate the worth or value of an investment

2 Best practice: to contribute to the knowledge of what works and why and to compare with similar projects Learning and improvement: to understand how a project or program can be improved Interest: to see and share the outcomes of a project. Furthermore, into the future, demonstrating the achievement of projected benefits is likely to become more important in securing funding for projects. For example, if it can be shown that a project of a particular type repeatedly delivers very well on its intended benefits, then that type of project may be looked on favourably for future funding. Post completion results can also give organisations confidence in the results they are delivering on projects, which can help staff to confidently engage with the community on current or future investments. 2. VicRoads context An effective organisation measures and learns from its past performance and uses the lessons to improve (VAGO, 2011) In recent years, VicRoads has strengthened its approach to evaluations and shown a desire for continuous improvement. Traditionally, VicRoads evaluations were conducted following the completion of a project and focused on delivery aspects of the project, such as time and cost to deliver. Some VicRoads programs, such as Safer Road Infrastructure Program, also undertake routine evaluations; however these were generally only concerned with reporting on the benefits to safety. As such, most VicRoads evaluations were only focussed on one aspect of a project, rather than the overall success of an investment. A high level of data was being collected for monitoring the performance of the network, and for use in developing new projects (including economic appraisals), but this data was generally not used for evaluation. Therefore, despite the amount of monitoring, reporting and evaluation approaches already used in VicRoads, there were still further improvements to strive for. The Victorian Auditor General s Office emphasised this following their examination of how effectively VicRoads managed a sample of major road projects in the report Management of Major Road Projects (VAGO, 2011). Part of the definition of effective management in the report was demonstrating that completed projects have achieved their intended outcomes and using this experience to better manage future projects (VAGO, 2011). The VAGO report indicated that VicRoads needed to do more to strengthen its ability to track and report on all outcomes of a project, and to review lessons learned from major projects. The recommendations provided an immediate imperative for VicRoads to invest in outcome management and used this as a significant opportunity to improve investment evaluation and promote continuous improvement across the organisation (Narayan, 2015). In response to this, the Investment Management Approach (IMA) was developed by VicRoads and built on the Victorian Department of Treasury and Finance s (DTF s) Investment Management Standard (IMS), which are promoted across the Victorian Government and detailed in its Investment Lifecycle and High Value High Risk Guidelines. DTF has overall processes for investment to provide guidance on realising investments and on useful processes that will help agencies frame their thinking as they evaluate investments. The development of a solution to a problem using the IMS is illustrated in Figure 1. The IMS includes the Investment Logic Maps (ILM) and Benefit Management Plans (BMP), which are now required for all infrastructure projects and programs within VicRoads. The focus of the evaluation of investments for DTF is outcome-driven, evidence-based reporting. The intention of the IMS is to drive a consistent approach to investment evaluation across government. Each agency must then develop methods that are useful to them within the government processes.

3 Figure 1 The DTF Investment Management Approach These drivers were something of a paradigm shift in how (VicRoads) plans, delivers and learns from it investments so it always maximises the outcomes that matter most to the community (Narayan, 2015). In making this shift it has been important for VicRoads to embed evaluation at each stage of the investment cycle by making changes on a number of fronts. To do this has involved implementing policies, maintaining specific budgets for evaluations and gaining support from senior leadership. It has also been necessary to develop supporting tools, guides and processes. This has included the development of the Benefit Management Framework, which enables a consistent approach to identifying, monitoring and evaluating the success of VicRoads investments. It provides a line of sight from investment-level indicators to the benefits and outcomes that VicRoads and ultimately government aims to achieve (VicRoads 2016). Importantly, the Investment Management Approach has also included the development of the VicRoads Investment Evaluation Framework. 3. Investment Evaluation Framework As part of the Investment Management Approach (IMA) adopted by VicRoads, the Investment Evaluation Framework (the Framework) was developed to provide an overarching framework for post completion evaluation. It provides a stand-alone document for principles, policy positions and guidance for undertaking post completion evaluation of projects and programs in VicRoads and use of the evaluation findings. WSP assisted VicRoads to develop and refine the Investment Evaluation Framework such that it captured evaluation concepts, while being practical and easy to follow for transport engineers. This challenge meant tailoring language to allow transport professionals to be able to understand and undertake post completion evaluations, without the need to burden themselves with typical language used in evaluation, such as fuzzy logic and counter-factuals. Similarly, technical engineering language needed to be simplified and generalised such that it could be understood by evaluation professionals who were not familiar with transport planning and engineering. In order for evaluation concepts to be understood by engineers, and for transport principles to be understood by evaluators, evaluation language was suggested by evaluation experts and then tested and reviewed by WSP.

4 Determining the aspects to include in the evaluation itself, such that the evaluation would provide a holistic review of an investment, was another key challenge during the development of the Framework. To do this, WSP worked closely with managers and team leaders at VicRoads to arrive at a set of Key Evaluation Questions (KEQs) and supporting questions which both aligned with DTF requirements and could be used as the basis for all investment evaluations. 3.1 Key Evaluation Questions (KEQs) and rating the performance The Key Evaluation Questions (KEQs) are the core of the Framework, and are designed to guide and frame the evaluation, as well as to ensure the efficient use of resources for data collection required for an evaluation. The KEQs are designed to cover the following five aspects of any investment: KEQ1 (Appropriateness): To what extent was the planning, design and delivery of the investment the most appropriate way to address the problem? KEQ2 (Efficiency): How efficiently was the investment delivered? KEQ3 (Effectiveness): How effective has the investment been in delivering the expected benefits and outcomes? KEQ4 (Unintended outcomes): What are the unintended outcomes of the investment? KEQ5 (Impact sustainability): To what extent are the benefits achieved likely to endure beyond the timeframe of the project? In general, the idea behind the KEQs is to cover important aspects of an investment from initial planning through to the long term endurance of benefits. The KEQs have been tailored and organised as a logical set of queries for transport and road projects, supported by a list of specific considerations and guidance for responses. The intention is that all investment evaluations will answer all KEQs, if possible. The level of detail provided for each KEQ however, may differ depending on the size and complexity of the investment and the purpose of evaluation. Responses to KEQs can be either quantitative, qualitative or a mixture of both, where: Quantitative responses involve collecting and analysing measurable data (e.g. traffic data). Qualitative responses involve collecting and analysing descriptions or distinctions based on some quality or characteristic (e.g. community surveys). Mixed responses involve the combination of qualitative and quantitative responses. After responses to all KEQs have been completed, the performance of the investment is able to be rated. The Framework suggests a performance rating is given to each KEQ, and then to the investment overall. The ratings range from a score of 5 where performance is clearly strong, to a score of 1 where evidence is unavailable or insufficient. All ratings are to be accompanied with a narrative explaining the rating. The ratings are intended to give a simple way to describe the findings of each KEQ, and allow for consistency across investments (such that a rating of 4 on one investment is comparable to another investment with the same rating). 3.2 Planning for evaluations the VicRoads approach So how are evaluations planned into the investment cycle at VicRoads? The Framework suggests that post completion evaluations should not only be thought about at the end of an investment, but at all stages of the investment cycle, such that the evaluations are properly planned, prioritised and resourced. The investment cycle (Plan-Deliver-Learn), shown in Figure 1, highlights how planning for outcomes becomes part of the whole investment cycle.

5 Figure 2 Plan-Deliver-Learn cycle At the plan stage of the investment cycle, the Investment Management Approach (IMA) plays a critical role in defining the problem; identifying the expected resulting benefits of addressing those problems as well as determining what activities, assets and resources which are required to address the problem. Defining indicators, measures and targets for those benefits is also undertaken at the plan stage to give information on what the performance expectation of an investment is. Any required before data is also collected during the plan stage, before work on the project or program starts, so that relevant benefits can be assessed later. For example, before travel time surveys should be collected if required to provide a baseline comparison, since without baseline data, it is difficult to prove the success of a project later on. During the deliver stage, any relevant data on delivery is collated. For example, information on cost, time and resources spent on the project delivery is tracked to feed into post completion evaluation questions about efficiency. Changes to scope which may affect the benefits of an investment should also be recorded, with the risk to benefits explained and managed. Post completion evaluations are then undertaken as part of the learn stage. This can be partially done immediately following the completion of the project, where a post-project review could be undertaken to capture any information the project team may have that is useful, such as changes to scope during the construction and lessons learned. Responses for some KEQs can therefore be prepared at this stage when the information is still fresh in the project team s mind (such as KEQ2, relating to time, cost and resources efficiency). More often though, a full post-completion evaluation is undertaken at least 12 months after project completion, at which point the project will generally have entered a period of normal operation where benefits of the project have manifested, or for initial benefits to have settled to a level that is likely to be sustained over time. At this time, responses to all KEQs can be prepared, however where the target for a benefit is greater than 12 months, only an indication of achieving that benefit is provided and a following evaluation should be undertaken at the time when the target is to be achieved. 4. Evaluations in practice WSP has undertaken a number of post completion evaluations for VicRoads using the Framework. These evaluations have included different types of road projects such as road duplications, road/rail grade separation and bypass projects.

6 Typically the evaluations are undertaken one year after the completion of the project. The usual starting point for an evaluation project is to go through documents on the project to find responses to the key evaluation questions. At this stage, any before traffic data and community surveys should have already been undertaken, while matching after traffic data and community surveys are organised (if not already undertaken). Project documents such as the business case, ILM and BMP are gathered, and the project team is asked to provide any available information on the delivery stage of the project. Immediately following the completion of a project, Post Project Reviews have also been completed, and have been useful for capturing insights from the project team before they disperse to other projects. In a Post Project Review, some of the responses to the appropriateness and efficiency questions can be easily and quickly prepared, since information and learnings are readily available (such as information on why the project was ahead or behind budget and schedule), whereas this information can be lost quite quickly once a project has been completed and the project team has dispersed. 4.1 Evaluation results Overall, evaluations undertaken for VicRoads so far have generally been a positive and useful exercise, since they have: Proved benefits have been met: completed post completion evaluations have provided an evidence-base response to whether or not investments are meeting their expected benefits, and if decisions made were appropriate. Confirmed the appropriateness of projects: findings from the evaluations have found that there is generally logical and sound reasoning driving projects, and proof of options considered. This kind of finding gives confidence, for example, that a particular project was the most appropriate response to the specified problems. In one evaluation it was found that travel time benefits were achieved by duplicating a road without simultaneously removing a level crossing and therefore the decision to only duplicate was appropriate to achieve that benefit. Identified strengths and weaknesses: Areas which are being done well and areas for improvement have often been highlighted in evaluations. For example, from the evaluations undertaken to date, VicRoads project management has been shown to be strong, reinforcing that their project management practices are efficient and effective. Provided information for future projects: From the evaluations undertaken, a pool of information is now available to VicRoads to draw upon to assist with various aspects of planning and delivery of future investments, such as: identifying appropriate targets, identifying appropriate delivery methods and better understanding community needs and requirements. 4.2 Current limitations and challenges While there has been good progress on post completion evaluations, there are some areas that still have limitations or pose a challenge to the evaluator: Lack of data or missing benefit data: Matching appropriate before and after data sets for comparison can be difficult if the evaluation was not planned up front. Some particular limitations include: Lack of data about public transport targets. Commonly within identified project benefits there will be a benefit about improving priority of public transport or improve connectivity of public transport services. However suitable data needs to be collected and targets need to be set in order to understand if projects are truly having a benefit for public transport.

7 Similarly, proving benefits for cyclists/pedestrians are sometimes not straightforward. Generally we compare usage of a facility before and after improvements, through traffic surveys. Cycling surveys are however typically done only for one day and may show limited use of the facility. This may not provide enough data to make suitable conclusions about the benefits of the facility. Community feedback: Community surveys provide great information on the more intangible benefits (such as amenity, attractiveness of study area, and sense of safety), however community surveys are often not considered or collected to the same degree as traditional traffic surveys. As evaluations become more common there should be better planning into surveying community for the purposes of evaluation to collect before and after data for relevant questions. Efficiency data: Resource efficiency (including people, systems and capital) is one of the specific considerations of KEQ 2 but often is not well recorded for the purposes of post completion evaluation, and what details are available is proportional to the size of the project. This consideration can prove valuable however; in one case it was found that VicRoads provided more staff for a project than was originally expected, to support a contractor who lacked experience with some project management aspects of the project, which has now become a learning for the organisation. Similar project data: Part of the efficiency KEQ is comparing the project against other similar projects. This gives an indication of whether the project was typical in its resource needs and expected benefits, or a look at what aspects of the project were different from the usual. Currently there is not a readily accessible way to check details on other similar projects, but as more post completion evaluations are undertaken, these will be a useful source of comparison for future projects. 4.3 Evaluation learnings at VicRoads While more evaluations are now being undertaken, it is still a relatively new practice within VicRoads, and staff is still becoming familiar with what is required for evaluations. Continuing to invest in training and supporting materials is therefore a focus for VicRoads currently, as they recognise it is important for staff to be knowledgeable in the evaluation process to understand what needs to be done as well as what can be expected from an evaluation. Pilot training and training manuals have recently been developed by WSP for VicRoads to assist with the learning process. Since it is possible that an evaluation may include performance ratings that show that some aspects of the project were weak, or some benefits were not met, an important aspect of the training and learning process is also a shift in mindset such that mistakes and learnings are not hidden or viewed negatively, but rather shared throughout the organisation. This means recognising that less than perfect results from an evaluation can actually provide an opportunity to learn and improve on future projects. In particular, since every KEQ is required to include learnings, the evaluation can identify opportunities for improvement, key lessons, and recommendations for the future. Similarly, evaluations are also an opportunity to highlight successes on projects and what has worked well. Thus whatever the results of the evaluation are, positive or negative, there should be something to be learned from it. The practice of post completion evaluation has also helped VicRoads staff to improve their thinking with regards to the whole investment cycle (i.e. staff are now learning and thinking more about how they might actually measure benefits at the end of the project). This is evident, as over time there has been an improvement in the information provided in ILMs and the data collected before a project. Overall however, evaluations will only achieve their full potential when results are properly disseminated such that others can learn from them. As such, VicRoads are currently working on dissemination strategies and the creation of accessible databases of learnings.

8 5. Conclusion VicRoads has made great progress in post completion evaluations, with the introduction of the Investment Evaluation Framework, which ties in with the their Investment Management Approach and the investment cycle (Plan-Deliver-Learn). The Framework has provided a stand-alone document to guide the practice of post completion evaluations. Undertaking post completion evaluations has been shown to be useful to assist with learning and improving, though there are still some limitations and challenges to overcome. The results and benefits of undertaking post completion evaluations will continue to improve over time through training and learning, and as it becomes imbedded into the everyday practice of VicRoads projects and programs.

9 References Department of Treasury and Finance. (2012). Investment Lifecycle and High Value High Risk Guidelines: Realise. Retrieved from Austroads (2005). Guide to Project Evaluation Part 7: Post-Completion Evaluation (AGPE07-05). Austroads, Sydney. Narayan, Sarita (2015). Plan-Deliver-Learn: Driving an Evaluative Mindset, presented to the AES 15 International Evaluation Conference 5-9 September 2015 Victorian Auditor General s Office (2011). Management of Major Road Projects, Available: VicRoads (2014). VicRoads Investment Evaluation Framework. Retrieved from VicRoads (2016). VicRoads Benefit Management Framework Version 3. Retrieved from

10 6. Biographies Sarah Lowe Senior Transport Engineer WSP Sarah Lowe is a senior transport engineer with 14 years experience in traffic engineering and transport planning. Sarah has worked on a wide range of transport projects from planning to post completion evaluation. She first worked on post project evaluations as a research engineer during the development of the Austroads Guide to Project Evaluation. In recent years she has been involved in many post completion evaluation projects, including completing evaluations using the Investment Evaluation Framework. Her areas of work also include major urban freeway projects, microsimulation modelling, traffic impact assessments and transit oriented developments. Reena Lall Senior Transport Engineer WSP Reena Lall has ten years experience in transport planning, modelling and engineering. She is proficient in undertaking post completion evaluations, and communicating the lessons learnt from evaluations to organisations in order to improve the planning and delivery of projects. Reena is also experienced in calibrating and validating mesoscopic transport models in SATURN, as well as planning, managing and undertaking option testing in both strategic and mesoscopic transport models, with outputs from the modelling feeding into option analysis, economic assessments, traffic impact assessments and more detailed design. Sarita Narayan - Manager Investment Outcomes - VicRoads Sarita Narayan has nineteen years of experience in project, program and strategy development, management and evaluation. As an effective change leader, Sarita has lead many roles in rationalising organisational systems, processes and services to improve efficiencies and influence continuous improvement opportunities that build on learnings from previous evaluations. In VicRoads, Sarita lead the Investment Outcomes team responsible for assisting the organisation "make better investment decisions" with a focus on delivering better value outcomes for community and customers.