Final Report Mid-Term Evaluation of the Sydney Tar Ponds and Coke Ovens Remediation Project

Transcription

1 Mid-Term Evaluation of the Sydney Tar Ponds and Coke Ovens Remediation Project January 26, 2012 Office of Audit and Evaluation

2

3 TABLE OF CONTENTS MAIN POINTS... i INTRODUCTION... 1 PROFILE... 1 Background... 1 Authority... 2 Roles and Responsibilities... 2 Activities and Resources... 5 Stakeholders... 6 Logic Model... 6 FOCUS OF THE EVALUATION... 8 FINDINGS AND CONCLUSIONS... 9 RELEVANCE... 9 Continuing Need... 9 Alignment with Federal Government Roles and Responsibilities Alignment with Government Priorities Conclusions: Relevance PERFORMANCE Immediate Outcomes Intermediate Outcomes Long-Term Outcomes Cost-Effectiveness Lessons Learned Unintended Impacts Conclusions: Performance GENERAL CONCLUSIONS MANAGEMENT RESPONSE RECOMMENDATIONS AND MANAGEMENT ACTION PLAN ABOUT THE EVALUATION APPENDIX A: EVALUATION MATRIX APPENDIX B: PROJECT WORKS DESCRIPTIONS APPENDIX C: CATEGORIES OF RISK WITH EXAMPLES APPENDIX D: CHANGES IN PROJECT WORKS APPENDIX E: AIR MONITORING Public Works and Government Services Canada

4

5 MAIN POINTS What we evaluated Mid-Term Evaluation of the Sydney Tar Ponds i. This evaluation examined the Sydney Tar Ponds and Coke Ovens Remediation Project (STPCORP), a $397.7 million ten-year ( ) federal-provincial project, for which Public Works and Government Services Canada (PWGSC) is providing federal oversight. 1 The Results-based Management and Accountability Framework created for the project committed to three program evaluations of the STPCORP. A formative evaluation of the Interim Cost-Share Agreement was completed in 2007 and a summative evaluation is scheduled to begin upon project completion in This mid-term evaluation serves as an assessment of results achieved by the project to date and aims to provide additional assurance that the project is proceeding as expected. Why it is important ii. The STPCORP is designed to remediate two of Canada s most contaminated sites. The Tar Ponds and Coke Ovens are located in urban Sydney, Nova Scotia's second largest city, with 25,000 people living within four kilometers of the site. The contamination was the result of nearly a century of steel production. The Tar Ponds and Coke Ovens sites pose a significant threat to the environment and the public. iii. The federal government has a responsibility to contribute to the cleanup of the sites, as it operated the facilities through the Cape Breton Development Corporation into the 1990s. As a steward of the environment, the Government of Canada has acknowledged that it has a legislative responsibility under the 1996 Canadian Environmental Protection Act to contribute to the cleanup. What we found iv. As noted in the 2002 Report of the Commissioner of the Environment and Sustainable Development, industrial chemicals, pesticides, and their by-products can pose serious environmental and health risks. Based on extensive existing research in the scientific community, there is a continuing need for remediation of the Sydney Tar Ponds and Coke Ovens sites as a result of its continued contamination by volatile hydrocarbons and heavy metals (by-products of steel-making and coke production at the sites). Federal involvement is consistent with federal environmental legislation and with the federal government s past role as an operator and underwriter of the steel mills from 1986 to v. STPCORP governance mechanisms are effective, with an active and efficient committee system; significant federal-provincial co-operation; effective, independent 1 While the allocated funds total $436.7 million, the scope of this evaluation only covers the $397.7 million set-aside for remediation activities. Public Works and Government Services Canada i Office of Audit and Evaluation January 26, 2012

6 oversight from the Independent Engineer; and good communication, with timely information being disseminated to the project teams and the public. vi. While budget and expenditure management is effective and subject to third-party review by the Independent Engineer, higher than expected costs for some project construction works and for engineering design and oversight have consumed most of the project s contingency funds. As a result, there is limited flexibility in the remaining project budget. vii. As of December 31, 2011, 62% of the $397.7 million budget has been spent. In the same period, 72% of the contaminants to be solidified and stabilized had been treated. With two construction seasons remaining, the Independent Engineer has forecasted that the project will finish on time and within budget. viii. The selected remediation method (solidification/stabilization) is internationally recognized as efficient and effective. To date, remediation costs have been lower than documented costs of alternative technologies and within the average cost range of other solidification/stabilization projects. ix. The Sydney Tar Ponds Agency (STPA), the provincial agency established to manage the project, is responsible for monitoring and mitigating threats to the health and safety of the workers on site, the general public, and the environment. Results have shown very low levels of risk to human health and the environment from the remediation work. x. The project has brought economic benefits to the Cape Breton region and contributed to the employment of African Nova Scotians. 2 The project has also funded several initiatives in support of employment and training for women in trades and technology and has employed 30 women in non-traditional occupations. First Nations participation has been a particular success. A set-aside strategy guaranteed First Nations firms a share of available contracts. Participation by First Nations firms has far exceeded initial projections, with First Nations firms competing for, and winning, non-set-aside contracts. xi. Future land use for the Tar Ponds and Coke Ovens sites is a major component of the project. A master plan for future land use has been developed and funding requirements for this purpose have been finalized. Plans are to use the remediated sites for public green space and new commercial developments. Management Response PWGSC-Atlantic concurs with the findings. We will continue to monitor the project closely, as we have in the past, in accordance with our agreements and established governance, which is appropriate and functioning effectively. PWGSC, as also opined by the Independent Engineer, is confident that the project will be delivered on time, on budget, within scope in a manner that is protective both of health and environment. 2 The community of people of African descent within the province self-identify as African Nova Scotians. Public Works and Government Services Canada ii Office of Audit and Evaluation January 26, 2012

7 Ultimately, as outlined in the logic model for the Project, PWGSC is confident that the Government of Canada's priority of a sustainable Canadian society will be supported by the outcomes of the Project upon its successful completion. Public Works and Government Services Canada iii Office of Audit and Evaluation January 26, 2012

8 Recommendations and Management Action Plan Recommendation 1: The Director General, Atlantic Region, should continue to monitor engineering design and oversight costs, with a specific emphasis on ensuring that future costs are in line with current estimates. Management Action Plan 1.1: The Director General, Atlantic Region, PWGSC, along with the Project s Management Committee (PMC) Co-Chair, the Deputy Minister of Nova Scotia Transportation and Infrastructure Renewal, will continue to monitor the engineering construction oversight costs, with a specific emphasis on ensuring that future costs are in line with current estimates. This has been monitored as part of the PMC responsibility on an ongoing basis, and these costs will continue to be scrutinized and challenged. This is being done through the existing standard operating procedures as outlined in the project management framework, such as the Independent Engineer s daily, weekly, and monthly activities and reports to the PMC. Recommendation 2: The Director General, Atlantic Region should monitor implementation of the recent Change Management Review to ensure that the Independent Engineer (IE) is provided with timely, accurate information by the Design Engineer and the STPA on areas within the IE s purview, including proposed changes to design or construction and information necessary to advise on the reasonableness of design and construction costs being incurred on the project. Management Action Plan 2.1: The Director General, Atlantic Region, PWGSC, along with the PMC Co-Chair, the Deputy Minister of Nova Scotia Transportation and Infrastructure Renewal, will continue to closely monitor the implementation of the recent Change Management Review. Results of implementation will be reported to, and monitored by, PMC to ensure effective implementation of actions which also meet their intended purpose. This monitoring is considered to be part of the ongoing responsibility of the PMC. This is being done through the existing standard operating procedures as outlined in the project management framework, such as the Independent Engineer s daily, weekly, and monthly activities and reports to the PMC. Public Works and Government Services Canada iv Office of Audit and Evaluation January 26, 2012

9 INTRODUCTION Mid-Term Evaluation of the Sydney Tar Ponds 1. This report presents the results of the mid-term evaluation of the Sydney Tar Ponds (STPCORP) (hereinafter referred to as the project in this report). The Audit and Evaluation Committee of Public Works and Government Services Canada (PWGSC) approved this evaluation as part of the Risk-Based Multi-Year Audit and Evaluation Plan The project s Results-based Management and Accountability Framework committed to three program evaluations of the STPCORP. A formative evaluation of the Interim Cost-Share Agreement was completed in 2007 and a summative evaluation is scheduled to begin upon the project s completion in This mid-term evaluation serves as an assessment of results achieved by the project to date and aims to provide additional assurance that the project is proceeding as expected. As a result, the evaluation places greater reliance on data provided by the program (validated by the Independent Engineer where possible) and on information obtained from key stakeholders; and less on analysis of new data generated by the evaluation. PROFILE Background 2. Steel-making was established in Sydney in 1901 by the Dominion Iron and Steel Company to take advantage of abundant coal and iron ore, as well as access to a deepwater harbour. By 1912, the mills accounted for over half of Canada s steel output. In 1967, the government of Nova Scotia took over the facility from the private sector and established the Sydney Steel Corporation in an effort to protect the local economy. In 1968, the federal government acquired ownership of the Coke Ovens and operated these facilities under the auspices of the Cape Breton Development Corporation until 1973, when they were purchased by the Sydney Steel Corporation. The federal government also provided $177 million in funding from 1986 until 2001, which allowed the steel plant to continue operations. 3. During the 1980s, Fisheries and Oceans Canada shut down lobster fisheries in Sydney Harbour due to contaminant migration from the Tar Ponds and Coke Ovens sites. Studies conducted by the federal government and other organizations have confirmed the environmental and health risks posed by the contamination. 4. As shown in Exhibit 1, the Tar Ponds cover an area of 31 hectares, divided into the South Pond and the North Pond, with an estimated 700,000 tonnes (550,000m 3 ) of sediments contaminated with polycyclic aromatic hydrocarbons (PAHs) and metals, of which 5% also contain polychlorinated biphenyls (PCBs). The Coke Ovens (including the Coke Ovens Brook and the Tar Cell) cover 68 hectares, with an estimated 586,000 tonnes (293,500m 3 ) of soil contaminated with petroleum hydrocarbons, PAHs, and metals. Public Works and Government Services Canada 1

10 5. Under the 1996 Canadian Environmental Protection Act, PAHs and PCBs are classified as toxic substances. According to the National Classification System of Toxic Substances, the Sydney sites required action. 6. By the 1990s, the federal government had contributed $253.1 million to health and environmental studies and cleanup initiatives, but these efforts had resulted in little progress. In 1996, the Joint Action Group, a community-based advisory group, was created to recommend a cleanup method. In 1998, a Memorandum of Understanding was signed by the Joint Action Group and the federal, provincial, and municipal governments, establishing a framework for participation in the cleanup. 7. In 2002, the Commissioner of the Environment and Sustainable Development criticized the federal government for not having addressed the issue of federal contaminated sites and cited Sydney as being among Canada s worst. The 2004 Federal Budget acknowledged federal responsibilities and policies on shared-responsibility contaminated sites, and committed $500 million over ten years for remediation activities, noting Sydney as a priority. In April 2004, the federal Cabinet approved support for the STPCORP. Authority 8. Following the 2004 Federal Budget and federal government approval of the project, the Governments of Canada and Nova Scotia signed a Memorandum of Agreement in May 2004, pledging their commitment to the cleanup of the Sydney Tar Ponds and Coke Ovens sites. The agreement included commitments to an Environmental Assessment, the hiring of an Independent Engineer, community liaison activities, consultations with First Nations, and an emphasis on sustainable development activities. It also committed the parties to a future agreement that would outline the major elements of the project. 9. Project activities were scheduled to be completed over ten years ( ). An Interim Cost-Share Agreement for the first phase was signed and implemented in This was followed by a Final Cost-Share Agreement covering the second phase, from 2007 to Roles and Responsibilities 10. The federal and provincial governments share responsibility for funding and oversight. The 2007 Final Cost-Share Agreement defines the cost-sharing arrangement, budget, and reporting requirements; payment procedures; governance and implementation structures; roles and responsibilities; and the project work breakdown structure. 11. The STPCORP governance and management structure is extensive, with participation by several government departments, federal and provincial committees, and the private sector. 12. PWGSC s Atlantic Region was assigned the lead role for the federal government based on its experience with major Crown projects and its proximity to the sites. Its role Public Works and Government Services Canada 2

11 is to oversee federal interests and ensure that the project is delivered on time, on budget, and within scope. Funding was also authorized for Environment Canada and Health Canada to carry out their roles on the project. The Department of Justice, Natural Resources Canada, and Fisheries and Oceans Canada also participated in the project but did not receive funding. 13. The Nova Scotia Department of Transportation and Infrastructure Renewal (NS-TIR) has the responsibility for project implementation and operations. The province established the Sydney Tar Ponds Agency (STPA) as a special operating agency to manage and implement the project. The provincial Department of the Environment is involved in the project as the environmental and occupational health and safety regulatory authority. 14. The STPA is responsible for daily project management activities, support and contract services, work breakdown, documentation of activities, adherence to health, safety and environmental requirements, and communications. The STPA is accountable to the federal and provincial governments through the joint federal-provincial Project Management Committee. As the governing body responsible for strategic direction, the Project Management Committee has ultimate decision-making authority. Co-chaired by federal and provincial representatives, it must approve all budget and design changes. It meets quarterly, receiving reports from the Independent Engineer on current issues; from the STPA on the status of construction works, economic benefits, scheduling and budget, and current issues; and from the Environmental Management Committee on environmental issues. 15. Several senior-level committees 3 are charged with oversight of specific project components and report to the Project Management Committee. These committees are: i. The Operational Advisory Committee, which oversees implementation; ii. iii. The Environmental Management Committee, which advises on environmental matters; and The Community Liaison Committee (led by a community representative and comprised of leaders from the Sydney community), which facilitates information flows between the project and the community. 16. First Nations are not involved in the committee structure; however, as established by the 2005 Protocol Agreement, they meet directly with federal and provincial officials to discuss First Nations participation in the project. 17. A key part of the governance and management team is the Independent Engineer. This office, which reports directly to the Project Management Committee, exercises a monitoring and validation role to ensure that technical requirements are addressed; effective management controls are in place; and to review and report on costs and contract compliance, as well as other aspects of the project (e.g., risk management). The 3 All committees include federal and provincial participation. Public Works and Government Services Canada 3

12 Independent Engineer produces monthly reports that address such issues as progress against the schedule, deliverables, budgets and expenditures, and others. 18. Exhibit 1 outlines the STPCORP governance structure and the reporting relationships among the committees, departments, groups, and individuals involved in the project. Public Works and Government Services Canada 4

13 Exhibit Mid-Term Evaluation of the Sydney Tar Ponds Project Governance Structure Provincial Lead NS - TIR Federal Lead PWGSC - Atlantic Project Management Committe Community Liaison Committee Sydney Tar Ponds Agency (Implementing Agency) Project Management Committee Secretariat Environmental Management Committee Operational Advisory Committee Design Engineer Independent Engineer Oversight Contractors and Consultants Source: Final Cost-Share Agreement, Schedule E, IV, Chart 1. Activities and Resources 19. Under the Final Cost-Share Agreement, the federal government committed $280 million to the STPCORP and the Government of Nova Scotia $120 million. The first $300 million in funding is shared in a 3:2 ratio between the federal and provincial governments, respectively, with the last $100 million provided by the federal government. As such, any final cost savings will accrue to the federal government. The Agreement also stipulates that there would be no cost overruns. Should these occur, the province will be responsible. Currently, the estimated cost of project management, engineering and construction, and related activities is $397.7 million over ten years (2004 to 2014). 20. The federal government also authorized funding for operating expenses for the three federal departments with significant ongoing involvement in the project. PWGSC was authorized funding of $25.9 million to enable it to carry out its role as the lead federal oversight agency. Environment Canada was authorized $7.6 million and Health Canada $5.5 million to provide technical and scientific advice in the areas of environment and human health The activities of the Sydney Tar Ponds are being undertaken in two phases. The first phase involved implementation of preliminary and preventative works, establishment of the project governance system, an environmental assessment, the Results-based Accountability Framework, and the Risk Assessment Strategy. PWGSC s 2007 evaluation of the Interim Cost-Share Agreement concluded that this phase operated efficiently. 4 While the allocated funds total $436.7 million, the scope of this evaluation only covers the $397.7 million set-aside for remediation activities. Public Works and Government Services Canada 5

14 22. The second phase of the project involves the construction of on-site facilities and the remediation of the sites. This phase constitutes the focus of this evaluation. Stakeholders 23. The stakeholders include the residents of Sydney, Nova Scotia; the Province of Nova Scotia, which will take ownership of the land when the remediation is complete; and the Government of Canada, which has exercised its responsibility for good stewardship of the environment through its participation in the project. 24. Cape Breton is home to five Mi kmaq First Nations communities (population: 6,750) that have a historical interest in the site. Aside from their interest in the cleanup from an environmental perspective, they are interested in the economic benefits and marketable skills to be derived from participation in the project. Logic Model 25. A logic model is a visual representation that links a program s activities, outputs, and outcomes; provides a systematic and visual method of illustrating the program theory; and illustrates how a program, policy, or initiative is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the evaluation matrix. 26. A logic model for this project was developed as part of the project s Results-based Management and Accountability Framework. The logic model is presented in Exhibit 2. As shown in the logic model, the key activities of Phase II of the project are remediation support, remediation of the Tar Ponds and Coke Ovens sites, and post-remediation monitoring and maintenance plans. These activities lead directly to the project outputs: remediation engineering; project facilities; surface and groundwater controls; solidification/stabilization of contaminants; post-remediation maintenance and monitoring plans; and transfer of ownership to the provincial government. 27. The immediate outcomes of these activities and outputs are: effective governance and management of the project; containment or elimination of contaminants; mitigation of environmental issues; protection of the public and workers; and effective operation of all facilities. Intermediate outcomes include: the elimination of all pathways to exposure; readiness of sites for use; and citizen support for the cleanup efforts. These intermediate outcomes will ultimately result in the protection or enhancement of human health, the environment, and land assets, as well as enhanced community benefits. Public Works and Government Services Canada 6

15 Exhibit 2. Logic Model Mid-Term Evaluation of the Sydney Tar Ponds Objective Remediate the contaminated areas of the Sydney Tar Ponds and Coke Ovens sites Activities Manage interim Cost-share Agreement and Remediation Project Undertake Preventative Works Undertake Preliminary Works Remediation Support Remediate the Tar Ponds Remediate the Coke Ovens Post-Remediation Outputs Accountability/governance, Strategic management Project management Preventative Works Environmental Assessment North Pond Coffer Dam Stabilized Cooling Pond Realigned Coke Ovens Brook Engineering design for leachate collection system Remediation Environmental Assessment Preliminary engineering Site preparation works Health and safety plan Environmental management plan Detailed remediation engineering Site infrastructure Water treatment plant Remediation monitoring Surface and ground water controls Solidified/stabilized sediment Contaminant sediment Surface and ground water controls Soil bioremediated Contained contaminants Long-term monitoring and maintenance plan Deed of transfer to the provincial government Immediate Outcomes Project is well managed Community is engaged Progress with respect to cleanup is demonstrated Government commitment to citizens is demonstrated Environmental quality is improved Contaminants on and entering the site are reduced Contaminants leaving site are eliminated/controlled Potential environmental issues are mitigated Workers/general public are protected Potential environmental issues are mitigated Contaminants in waste water leaving the site are minimized Workers and general public are protected Contaminant migration is eliminated Contaminant levels on site are reduced/contained All facilities and structures are maintained and operating effectively Intermediate Outcomes Sites are ready for remediation Citizens/Aboriginals support the cleanup efforts All pathways to exposure are eliminated Sites are ready for use Citizens support the cleanup effort Long-Term Outcomes Human health, ecological, and land use assets are protected and/or enhanced Community benefits Ultimate Outcomes A Sustainable Canadian Society Sound Stewardhip Healthier Environment and Stronger Economy Public Works and Government Services Canada 7

16 FOCUS OF THE EVALUATION 28. While the focus of the evaluation is to determine the relevance and performance of Phase II of the STPCORP, its primary objective is to report on early results and provide additional assurance that the project is proceeding as expected, through an examination of the effectiveness of the project governance and management structure, the progress of project works in relation to the project schedule, and progress towards the achievement of other intended outcomes (e.g., community engagement, protection and enhancement of human health and the environment, local economic benefits). The evaluation focused primarily on activities carried out under the Final Cost-Share Agreement as a previous evaluation was conducted by PWGSC s Office of Audit and Evaluation of the Interim Cost-Share Agreement. 29. An evaluation matrix including evaluation issues, questions, indicators, and data sources was developed during the planning phase (see Appendix A). Multiple lines of evidence were used to assess the project. These included: a) Initial Document review: An initial document review provided an understanding of the project and its context to assist in the planning phase. Documents reviewed included Memoranda to Cabinet, Treasury Board Submissions, Memoranda of Agreement, reports to Treasury Board, and budget and financial reports provided by the Project Team. b) Detailed Document Review: During the research phase of the evaluation, a large number of STPCORP reports and STPA documents were reviewed. These documents included budgets; selected monthly reports of the Independent Engineer; environmental monitoring reports; committee proceedings; reports from the Design Engineer and project works managers; and agreements and reports on protocols, budgets, health and safety monitoring, communication activities, local economic benefits, and construction progress. c) Literature review: A literature review examined technologies used in other jurisdictions for similar projects; the factors that influenced the technology selection; and the costs of these alternative approaches. d) Key stakeholder interviews: The evaluation team conducted 21 interviews with representatives of PWGSC, the government of Nova Scotia, the STPA, the office of the Independent Engineer, First Nations communities, and the local community. The interviews provided information about the project s governance system and its activities, outputs, and results, including benefits to the local and First Nations communities. They also informed other lines of enquiry, such as the review of documents and the financial analysis. e) Financial analysis: The evaluation team initially reviewed budgetary information up to June 2010 to assess changes in expenditures and forecasts from original estimates and to assess the reasonableness of expenditures. Due to delays in the Public Works and Government Services Canada 8

17 completion of the evaluation, financial data up to March, 2011 was requested from the project authority. FINDINGS AND CONCLUSIONS 30. The findings and conclusions presented below are based on the multiple lines of evidence identified above and are presented by evaluation issue (relevance and performance). As previously noted, this is a mid-term evaluation that addresses early results and serves to provide additional assurance that the project is proceeding as expected. While the evaluation relied on financial and other data provided by the program, the data were reviewed and analyzed by the evaluation team and validated by the Independent Engineer, which provided an additional level of assurance regarding their accuracy. RELEVANCE 31. Relevance was assessed based on the degree to which a program or project addresses a demonstrable and continuing need; the extent to which it is aligned with federal government priorities and departmental strategic outcomes; and whether it is an appropriate role and responsibility for the federal government. Continuing Need 32. Continuing need is assessed by three primary elements: relevance of original program rationale, legislative or policy requirements, and use of the program s services. In this case, the continued need is based on the first two criteria, as use of program services is not relevant to this type of project. 33. In the case of the contaminated sites in Sydney, Nova Scotia, the need for remedial action is clear. Concerns regarding health and environmental protection played a key role in ensuring that the cleanup activities proceeded. The Environmental Assessment for the project completed in July, 2006 confirmed the need for the project to reduce risks to people and the environment and highlighted the potential to create social and economic benefits for the community. It also made specific recommendations regarding remediation methodology, environmental protection, and local economic benefits. 34. The 1996 Canadian Environmental Protection Act defines PAHs and PCBs both present at the Sydney sites as toxic substances, requiring remediation action. The National Classification System for Contaminated Sites rated the Sydney sites as Class 1 (Critical), requiring action. 35. The 2002 Report of the Commissioner of the Environment and Sustainable Development criticized the federal government for not having addressed the issue of federal contaminated sites and cited Sydney as among Canada s worst. In Budget 2004, the federal government acknowledged its responsibility and committed funds for this and other cleanup projects. Public Works and Government Services Canada 9

18 36. In addition to its responsibilities under the Canadian Environmental Protection Act, the federal government has responsibility for the contamination at the Tar Ponds and Coke Ovens sites, as it operated the Coke Ovens site for five years and contributed $177 million through an economic development agreement with Nova Scotia from 1986 until Alignment with Federal Government Roles and Responsibilities 37. To determine if the remediation of the Sydney Tar Ponds and Coke Ovens sites is an appropriate role for the federal government and PWGSC, three elements were considered: devolution of the project to another level of government, the private sector, or other federal departments and agencies. The evaluation found that the remediation activities are an appropriate role for the federal government and PWGSC. 38. Provincial Devolution: The province of Nova Scotia attempted to remediate the Tar Ponds and Coke Ovens sites in 1986 and The federal government, specifically PWGSC, was brought in to contribute financing and expertise. Given the previous attempts and the large amount of federal funding attached to the project, it would likely prove unfeasible to completely divest PWGSC of any involvement. 39. Private Sector: While project delivery has largely been outsourced to the private sector, overall responsibility for environmental remediation activities must remain with the federal and provincial governments. PWGSC must provide monitoring and oversight of these activities to meet its accountability and stewardship responsibilities under the Financial Administration Act and other federal legislation and policies, such as the Federal Accountability Act, the Department of Public Works and Government Services Act, and the Transfer Payment Policy. 40. Federal Decentralization: PWGSC was chosen specifically for this project to leverage its strengths in construction, design, and management of major Crown projects, as well as its experienced personnel. It would be impractical to decentralize this function across the federal government. Alignment with Government Priorities 41. The 2004 Speech from the Throne committed the federal government to work with its partners to build sustainable development into decision-making. 42. The 2004 Federal Budget identified the Sydney Tar Ponds and Coke Ovens as a priority and committed $4 billion over ten years to remediate contaminated sites across Canada. 43. At the departmental level, this project contributes to PWGSC s strategic outcome of sound stewardship by facilitating the federal government cleanup of the contaminated sites. Public Works and Government Services Canada 10

19 CONCLUSIONS: RELEVANCE 44. There is a continuing need for cleanup of the Sydney Tar Ponds and Coke Ovens sites due to the health and environmental risks posed by the contaminants. Federal government involvement in the cleanup is consistent with federal environmental legislation and the federal government s past role as an operator and underwriter of the steel mills from 1986 to The project also aligns with federal government priorities on remediation of contaminated sites. PERFORMANCE 45. Performance is measured by the extent to which a program or project is achieving its expected results and the degree to which it is able to do so in a cost-effective manner. As previously stated, the focus of this mid-term evaluation is to report on early results and assess progress. 46. This evaluation examined the achievement of immediate outcomes, early indications of intermediate and long-term outcomes, cost-effectiveness, lessons learned, and unintended impacts. This mid-term evaluation (designated in the Results-based Management and Accountability Framework) will report on early results and provide additional assurance that the project is proceeding as expected. Immediate Outcomes 47. One of STPCORP s immediate outcomes is that the project is well managed. To assess this outcome the evaluation team examined governance and management of the project, based on the following aspects: federal-provincial co-operation, committees, the Independent Engineer, audit and evaluation oversight, information management, budget and expenditure management, and risk management. The section on remediation efforts examines implementation of project components, outcomes for the reduction, elimination, control, or minimization of contaminants, whether planned facilities are operating, and whether the federal government has demonstrated its commitment to the project. In addition, the evaluation team examined health and safety efforts, including monitoring and mitigation of environmental and public health risks and the protection of worker health and safety. Governance and Management 48. This section reviews the effectiveness of the governance and management structure of the project, including federal-provincial co-operation, governance committees, the Independent Engineer, audit and evaluation, budget and expenditure management, information management, and risk management. 49. Given the project s size and complexity, the governance and management structure functions effectively. Most interviewees were of the opinion that federal-provincial cooperation has been excellent and that governance and management mechanisms have Public Works and Government Services Canada 11

20 functioned well. While some interviewees, notably STPA officials, indicated that the extent of management and oversight mechanisms is excessive, most interviewees expressed a more positive view of governance and management. It is noteworthy that the Auditor General of Nova Scotia decided following a preliminary assessment that a performance audit of the project was not necessary, given the degree of management and oversight of the project. i. Federal-Provincial Co-operation 50. Federal-provincial co-operation on the project is high. The Interim and Final Cost- Share Agreements outlined the respective roles and responsibilities of the federal and provincial governments, thereby reducing the potential for conflict. The Final Cost-Share Agreement also outlined the project budget and the cost-sharing arrangement, governance and management structures and processes, reporting requirements, payment procedures, and the work breakdown structure. The majority of interviewees from the federal, provincial, and private sector indicated that federal-provincial co-operation, communication, and collaboration have been strong. Provincial officials expressed satisfaction with the federal role on the project. In addition, First Nations representatives indicated that federal-provincial co-operation contributed to their successful participation in the project. ii. Committees 51. The four main committees included in the project governance structure the Project Management Committee, the Operational Advisory Committee, the Environmental Management Committee and the Community Liaison Committee are effective, based on interviews with stakeholders and on a review of the mandates, membership meeting frequency, and levels of participation of the above-mentioned committees. 52. Based on a review of their Terms of Reference, all committees have clear mandates. The organizations represented on these committees are appropriate, given their mandates and the roles and responsibilities of participating organizations. The committees meet regularly, with a frequency appropriate to their role: the Project Management Committee, which is strategic in nature, meets quarterly; the Operations Advisory Committee, which deals with numerous issues affecting implementation, meets monthly. Participation levels are high, with most or all members attending the majority of meetings. 53. The vast majority of stakeholders interviewed for the evaluation almost all of whom were members of one or more of these committees indicated that these committees are functioning smoothly and that no improvements are necessary to enhance their effectiveness. iii. Independent Engineer 54. The Independent Engineer has been an effective element of the governance structure that has contributed to effective project and expenditure management. However, the Independent Engineer has experienced difficulty obtaining timely and comprehensive Public Works and Government Services Canada 12

21 information from the STPA. This hinders the Independent Engineer s monitoring and validation role. In addition, the requirement that the Independent Engineer approve all change orders over $25,000 has been circumvented on a number of occasions. 55. To ensure its independence from the STPA, the Independent Engineer reports directly to the Project Management Committee on a range of topics: budget and expenditures; technical issues; progress on construction and other activities; adherence to the project schedule and the critical path; health and safety; risk management; and compliance of implementation by the STPA with the federal-provincial Memorandum of Agreement. 56. The Independent Engineer s monthly reports provide detailed, relevant, and comprehensive information on many aspects of the project. In particular, the evaluation team used these reports to assess construction progress against budget utilization. 57. In addition, the Independent Engineer meets regularly with the STPA and the Design Engineer to discuss issues, obtain clarifications, and provide advice on technical issues, costs, and other matters. 58. Key stakeholders indicated that the Independent Engineer has been key to the success of the project and that the costs of the Independent Engineer have been recouped several times over. One interviewee stated that [the Independent Engineer costs were] probably the best money we ve spent on the entire project. In one case, as a result of the Independent Engineer s intervention, the project team was able to reduce a projected cost increase of over $62 million in one Project Work to $23 million. 59. The independence of the Independent Engineer is essential to its effectiveness. However, the Independent Engineer is dependent on the STPA and the Design Engineer to provide critical information. Based on a review of a sample of Independent Engineer monthly reports, the Independent Engineer experienced difficulty in obtaining updated scope, schedule, and budget details for construction oversight work carried out by the Design Engineer. 60. An example of this difficulty is change orders. In order to control costs, all changes to construction/design activities with a cost greater than $25,000 must be approved by the Independent Engineer before work can proceed. However, this control has been circumvented in a number of instances by designating changes as urgent and seeking approval only after construction has already begun (or in some cases, after it has been completed). This has resulted in billing disputes, with contractors submitting invoices for work that had not been approved. The Project Management Committee has taken steps to remedy this issue, most recently as part of a Change Management review. 5 5 The results of this Change Management Review have yet to be determined. Further investigation of this issue will need to be addressed in the following summative evaluation, due to commence in Public Works and Government Services Canada 13

22 iv Mid-Term Evaluation of the Sydney Tar Ponds Audit and Evaluation 61. Audit and evaluation are integrated into the governance and management structure; two evaluations and a contribution audit of the Interim Cost-Share Agreement have been completed. As well, several operational reviews have been carried out by the project. Annual financial statements are audited by an external auditing firm and budgets, expenditures, and procurement contracts are reviewed by the Independent Engineer. In addition, several audits of procurement recipients have been carried out since 2007 by independent auditors. 62. A Results-Based Management Accountability Framework was prepared for the STPCORP at the start of the program, as was a Risk-based Audit Framework. The Results-based Management Accountability Framework called for an evaluation of the Interim Cost-Share Agreement in 2007, a mid-term evaluation of the Final Cost-Share Agreement in 2010, and a summative evaluation upon completion of the project. The evaluation of the Interim Cost-Share Agreement is complete. This report presents the results of the mid-term evaluation of the Final Cost-Share Agreement. 63. Although no internal audits have been conducted, STPCORP s Risk-based Audit Framework contains a commitment to undertake audits to address program risks. The monthly Independent Engineer reports partly fulfill these commitments as do operational reviews that have been carried out of communications activities, the Environmental Management Plan, and operations and management of the project. A Change Management review recently carried out also addresses this commitment. As well, the project has sponsored a contribution audit of the Interim Cost-Share Agreement by Audit Services Canada. 64. In addition, the Risk-Based Audit Framework commits to procurement audits of First Nations recipients of contracts on an as required basis and procurement audits of recipients, in general, in the first year and every second year thereafter. Two procurement audits of First Nations recipients and one of general recipients have been completed. v. Information Management 65. Based on the evaluation team s review of various reports and on information provided by the program, information necessary for effective management, oversight, and accountability is effectively managed. 66. The Web Warehouse, a Web-accessible file storage system operated by the federal government through a private contractor, is effective. It houses a wide array of documents, including government agreements, minutes of committee meetings, financial records, records of monitoring activities and results, data from the 1998 cost-share agreement, design documents, and Geographical Information System data. 67. Reporting on the project is extensive, ranging from daily reports on air monitoring and water quality to progress reports to the Treasury Board. The progress reports cover all aspects of the project, including First Nations participation, local economic benefits, cash flow analysis, risk assessment, assessments of public opinion, and emerging issues. Public Works and Government Services Canada 14

23 These reports provide a complete account of project progress and the degree to which objectives and results are being achieved in the areas of remediation, health and safety, environmental monitoring, and local economic benefits. vi. Budget and Expenditure Management 68. The total estimated cost for remediation activities is $397.7 million over ten years ( ). The budget is divided into major sub-components. As shown in Exhibit 3, actual expenditures to March 31, 2011 amount to $207.5 million (52% of the project budget). During this same period, almost all of the project components are at or above 51% complete. Progress on major construction components of the project (Project Works) constituting 74% of the project budget is estimated at 51% complete, while 52% of the budget has been utilized in the same period. Exhibit 3. Project Budget and Expenditures and Project Component Status to March 31, 2011 Project Components Budget Allocation (July, 2011 Budget) ($ millions) Actual Expenditures to March 31, 2011 Percentage of Budget Utilized to March 31, 2011 Component Status Public Engagement $5.9 $4.2 71% 75-99% Comments/Observations Sydney Tar Ponds Agency $25.8 $ % 51% Based on Progress of Project Works Project Works (PW) $258.6 $ % 51% Independent Engineer $12.8 $8.1 67% 51% Based on Progress of Project Works Preliminary Works $28.9 $ % Complete Environmental Assessment Joint Panel Impacts (JPI): Other PW and JPI Contingency $4.9 $ % Complete $17.5 $ % 75-99% $8.1 $0 0% N/A Future Land Use $20.4 $0 0% 0% Long-Term Monitoring $15 $0 0% 0% Total $ $ % Source: Office of Audit and Evaluation analysis based on July, 2011 Semi-Annual Budget Update, Independent Engineer s reports, and program data. 69. Variances in project component budgets between 2008 and 2011 were examined as part of the evaluation. While the budget is aligned with the overall rate of progress, current budgets for some individual Project Works and other components vary significantly from original estimates. For example, the cost of the water flow diversion channels was originally estimated at $38 million in the Final Cost-Share Agreement. According to interviewees, by the late engineering design stage this had increased to almost $100 million due to a number of unanticipated factors. Although the channels were subsequently redesigned based on a lower cost solution, costs still increased significantly from the original estimate. Public Works and Government Services Canada 15

24 70. Numerous factors, such as lack of familiarity with the costs of raw materials or contracted services, soil conditions, and environmental and health and safety factors can result in modifications to estimates. The project also had to budget for several elements committed to as part of the Final Cost-Share Agreement, but for which budgets had not been established in the agreement, including future land use, women s employment, and recruitment of African Nova Scotians. 71. Some budget areas raise concerns: i. Engineering design and oversight costs up to March 31, 2011 have amounted to 22% of applicable Preliminary Works, Projects Works, and Joint Panel Impacts budgets. According to a joint report from the United States Environmental Protection Agency and the U.S. Army Corps of Engineers, remedial design and oversight expenditures should be approximately 12% of remediation costs. According to STPCORP Independent Engineer, who has worked on remediation projects in the United States, these costs should be in the range of 12% to 17%. The Independent Engineer has expressed concern in its reports that the project is over budget for engineering design and oversight costs. At present, total estimated costs for engineering design and oversight for the entire project are $50 million, or 12.5% of the total budget. ii. The Materials Handling Facility was constructed at a total cost of $10.8 million. This facility is designed to decontaminate workers, equipment and vehicles before they leave the sites and to treat contaminated debris on the site. According to the Independent Engineer, these facilities are generally temporary; however the STPCORP facility was designed as a permanent building in order to accommodate the decontamination of material not suitable for the standard solidification/stabilization process. While a significant quantity of this debris was anticipated, realized deposits were substantially lower than original estimates. Several interviewees were of the opinion that the facility was over-designed and could have been constructed for a fraction of the cost. 72. Contingency funds have played a major role in ensuring that the project remains within budget. A total of $57 million (14.3% of the total budget) was earmarked for contingency purposes. This includes $45 million to accommodate design and construction changes for Project Works and $12 million for contingencies within the Project Works Joint Panel Impacts budget. Any cost savings on project components are recycled back into the contingency fund. As of March 31, 2011, $49 million in contingency funds has been used to cover change conditions on specific Project Works and Joint Panel Impacts components. Although the appropriate use of contingency funds was not examined during the course of the evaluation, governance mechanisms were examined to ensure that sufficient checks and balances were in place (including independent engineering oversight) to provide reasonable assurance that use of the contingency funds was warranted. Public Works and Government Services Canada 16