WATER SERVICES ASSOCIATION OF AUSTRALIA AUDIT PROTOCOL FOR THE AQUAMARK ASSET MANAGEMENT PROCESS BENCHMARKING PROJECT DECEMBER 2007
TABLE OF CONTENTS COPYRIGHT:... 3 1.0 INTRODUCTION:... 4 1.1 OVERVIEW... 4 1.2 APPLICATION:... 4 1.3 OBJECTIVES:... 4 2.0 OBLIGATIONS OF THE PARTIES:... 6 2.1 OBLIGATIONS OF PARTICIPANTS:... 6 2.2 OBLIGATION OF AUDITORS... 6 2.3 OBLIGATIONS OF WSAA:... 6 3.0 REQUIREMENTS FOR AUDITORS:... 7 3.1 FORMAL QUALIFICATIONS:... 7 3.2 TRAINING AND DEVELOPMENT:... 7 3.3 PROFESSIONAL INTEGRITY:... 7 3.4 CONFIDENTIALITY AND INTELLECTUAL PROPERTY:... 8 4.0 AUDIT PROCEDURES:... 9 4.1 AUDIT METHODOLOGY... 9 4.2 PRE AUDIT:... 9 4.3 RISK BASED SAMPLING:... 9 4.4 COMMENCEMENT OF SITE AUDIT:... 10 4.5 VALIDATION AND DATA MANAGEMENT:... 10 4.6 PROFESSIONAL SKEPTICISM:... 11 4.7 OTHER STATUTORY OBLIGATIONS:... 12 4.8 INTERNAL AUDIT:... 12 4.9 PEER AUDIT:... 12 4.10 PRODUCTION OF THE AUDIT REPORT:... 12 4.11 PARTICIPANT COMMENTS ON THE AUDIT REPORT... 13 5.0 DISPUTE RESOLUTION... 14 5.1 DISPUTE VS. DISAGREEMENT:... 14 5.2 DISPUTE RESOLUTION:... 14 DECEMBER 2007 WSAA 2
Copyright: This document is protected by copyright. Apart from any use as permitted under the Copyright Act 1968, and except for where provided below, no part of this document may be reproduced or transmitted in any form or by any means, electronically or mechanical, for any purpose, without the express written permission of the Water Services Association of Australia Inc. The Water Services Association of Australia will permit copies of parts or all of this publication, without payment of a royalty or advice from the Water Services Association of Australia Inc, where it is used exclusively in house by the Associations full financial members Water Services Association of Australia, 2007 DECEMBER 2007 WSAA 3
1.0 Introduction: 1.1 Overview The Asset Management process benchmarking project (incorporating the Aquamark software) has been developed by WSAA for the urban water industry. The overall objective of the Asset Management process benchmarking project is to improve the standard of Asset Management performance within the water industry; principally through the identification and promotion of best practice within the industry. The Aquamark software is the primary tool used in assessing a businesses process performance. The software is designed to: Allow participant businesses to objectively assess their strengths and weaknesses in Asset Management process capacity and execution; Provide a mechanism for businesses to assess their performance against their peers in the international urban water industry; and Provide an indication of specific areas of process improvement for consideration by each participating water business. The outcomes of the Aquamark Asset Management process benchmarking exercise will be identification of a standardized measure of business Asset Management process capability as well as industry best practice across a range of businesses in a variety of jurisdictions (e.g. Australia, New Zealand, US, Europe and the Asia Pacific). To achieve this goal, it is imperative that the assessment undertaken by each business is a fully objective reflection of that business s capabilities. It is also important that the assessments between businesses are comparable. Concurrently, the Water Services Association of Australia (WSAA) will engage an independent party to undertake an accredited audit of the businesses Aquamark assessment. 1.2 Application: This protocol only applies to audit of the agencies self assessment undertaken using the Aquamark software. This audit protocol should not be used for any other purpose. The auditor shall conduct an audit in accordance with these audit protocols. However, information contained in text boxes is provided for information only. These comments are designed as illustrative examples to assist auditors in the application of the audit protocols. 1.3 Objectives: The objective of the audit protocol is to: Clearly establish the requirements for undertaking an audit of agencies as part of the Aquamark Asset Management process benchmarking project; Outline the responsibilities of the parties involved in the project; Ensure that the process is objective and repeatable; and Provide assurance as to the comparative nature of the data sets (i.e. application of a consistent understanding or interpretation of key issues). The application of the audit protocols should assist the auditor in expressing an opinion on whether the agencies self assessment has been prepared, in all material respects, in a manner which is objective and repeatable and that there is sufficient consistency between agencies to allow effective benchmarking. DECEMBER 2007 WSAA 4
The secondary objectives of the audit protocol are to ensure that the auditors demonstrate an adequate quality control procedure (consistent with best practice) which ensures data quality and objectivity in the assessment. DECEMBER 2007 WSAA 5
2.0 Obligations of the Parties: 2.1 Obligations of Participants: As a condition of participating in the audit process, all participants shall Be professional in their conduct; Be cooperative, open and objective in their self assessment; Provide ready access to the relevant personnel for each component of the audit; and Be responsive to the needs of the auditor and WSAA. Where, in the opinion of WSAA, a participant is uncooperative or demonstrates a lack of commitment to the process, following attempts to achieve cooperation, as a last resort WSAA reserves the right to remove the participant from the process and may consider recovery of costs. 2.2 Obligation of Auditors The audit organization is to exercise sound professional judgment in objectively assessing participant s responses. The overall objective of the auditor is to ensure consistency in the assessment both within the business (e.g. between the water and sewerage components of the business), between businesses and between different countries. The specific requirements of the auditors are outlined in greater detail in section 3.0. 2.3 Obligations of WSAA: Having pioneered the development of the Aquamark software, and as the overall program manager, the obligations of the Water Services Association of Australia include: Overall management of the project (specifically management of the audit process); Management of the production of the Industry report; and Provide a role as independent arbiter on disputes as they arise between parties (refer section 5.0 for dispute resolution procedures) DECEMBER 2007 WSAA 6
3.0 Requirements for Auditors: 3.1 Formal qualifications: The companies engaged by WSAA to undertake the audit are required to have: Recognized internal quality control and quality assurance processes (preferably under a third party accredited system); Flexibility to accommodate new arrangements and requirements; Demonstrated commitment to professional development of their audit team members; and Strategies for ensuring that the audit is undertaken in a fair, objective and independent manner (refer Clause 3.3) The companies engaged to undertake the audit shall ensure that all individuals who participate in the process have: Qualifications appropriate to their role in the project; Demonstrated capabilities and competence in the areas of the audit for which they are responsible; High degree of professional ethics; and Can demonstrate their independence (subject to clause 3.3 below) Companies engaged to undertake the audit shall not change the audit team members without the prior written approval of WSAA. 3.2 Training and Development: Those individuals appointed as lead auditors are required to undertake formal Aquamark Accreditation. Prior to the commencement of the audit, the lead auditors are required to brief support personnel on the Aquamark tool, audit process, and project objectives. Ongoing training throughout the project will include processes to ensure that the interpretations made by different auditors and in different jurisdictions are consistent. 3.3 Professional Integrity: The companies and individuals engaged to undertake the audit must deal equally with all participants. The auditing company must be independent of the businesses self assessment team. That is, the auditing firm and individuals are not permitted to assist businesses undertake their self assessment. This would potentially result in the auditor reviewing its own work and would be unacceptable to WSAA. It is recognized that the auditor may have a pre-existing professional relationship with some agencies. This previous involvement should not affect the auditor s professional judgment on this project. Any conflict of interest should be reported immediately to WSAA. DECEMBER 2007 WSAA 7
3.4 Confidentiality and Intellectual Property: The auditors are to ensure that all information contained within Aquamark remains confidential and protect the anonymity of the project participants. It is anticipated that, during the project, the auditors may require access to system and information that are the intellectual property of either the participating business or other, third party businesses. The consultant will respect the intellectual property rights of participants at all times. Use of participant s intellectual property shall only be allowed where appropriate commercial arrangements have first been put in place. All intellectual property developed as result of this project will be the property of WSAA. This will include any reports and data collection and analytical tools; WSAA will retain ownership of all data provided by the participants in the project. The consultants shall not use the results of the study, Aquamark, nor the data collected in it in any way without the express written permission of WSAA. DECEMBER 2007 WSAA 8
4.0 Audit Procedures: 4.1 Audit Methodology Immediately prior to commencement of the project, the participants engaged to undertake the audit will provide a detailed audit methodology. The audit methodology will reflect the requirements of this protocol and will include: An outline of the businesses its processes and procedures for undertaking the audit. A copy of the proposed sampling procedure which will be applied by all auditors; The businesses strategy for ensuring objectivity and replication of the outcomes; and The businesses strategy for managing the risk of systematic bias or errors in the field determinations Changes to the proposed auditing procedures are to be submitted to WSAA for review. 4.2 Pre Audit: The auditor is to make contact with the participant a minimum of two (2) weeks prior to the site visit. To ensure that the agencies staff are both available and prepared for the audit, the auditor is to discuss with the participant its requirements including: Overview of the proposed audit process; Probable schedule of interviews (scope); Data requirements (if known); Anticipated length of interviews; and Key personnel to be interviewed Any relevant supporting documentation that they require The auditor may provide the participant with a questionnaire to assist the auditor in determining high risk areas to be subject to detailed audit (refer section 4.3 on sample size). The auditor must provide the participant with contact details of the lead auditor as well as none other accredited party who is available to respond to questions prior to the audit proceeding. 4.3 Risk Based Sampling: The auditor is expected to adopt a risk based approach in establishing the extent of the audit, sample size and key areas of interest. The assessment of the scope of sub processes and measures to be audited should be based on the likelihood of and consequence of noncompliance or material error in the result. In assessing the scope of the audit, the auditor should consider the agencies asset management profile (in comparison with the businesses stated drivers) and anomalies or inconsistencies in the businesses self assessment (e.g. very high score in one function compared with low scores in all other areas). DECEMBER 2007 WSAA 9
4.4 Commencement of Site Audit: It is recommended that the site audit commence with a start up meeting between the agencies senior staff and the audit team. This initial meeting will provide the opportunity for the service provider to provide an overview of its operating environment and issues it faces in delivery of its services. The meeting can also provide an opportunity for the auditor to outline its audit processes and its expectations from the participant. Throughout the audit process, the auditor needs to accommodate: Differences in the strategies, work practices, cost drivers and other elements that contribute to an organizations relative performance; Account for differences which may arise between different delivery strategies (e.g. in house vs. contracted activities and may be required to speak to these parties); and Allow the results of the targeted activities to be used as a basis for establishing the performance of the whole Asset Management function. Outcomes from 2004: In the 2004 project, a presentation was made to the agency s key managers. The objective of this presentation was to: Ensure that there was a consistent understanding of the Aquamark process; Ensure that the participant managers were aware of the time commitments and expectations required for the audit. Provide an opportunity for the participant to comment on key issues or the context within which the business operate that may be unique to that business (e.g. some agencies are subject to strict regulation whilst others aren t; some are restricted by government policy objectives; some simply have a high number of legacy issues (e.g. old pipes, corrosive soils etc) This presentation provided the audit team with further insight into the operations of the business. By comparing the anecdotal feedback with the general outcomes form the self assessment, the auditor was then able to further refine its approach (e.g. if the participant identified its asset renewal processes as a strength (possibly driven by an ageing infrastructure) yet the self assessment indicated that the participant was underperforming in this function, further audit time was allocated to this function). 4.5 Validation and Data management: During the audits, the auditor would be expected to have access to supporting information (where available). However, the structure of the Aquamark software is such that complete documentation is not always required. In these instances, the auditor is to use their professional judgment in determining that the business has demonstrated a level of understanding appropriate to their assessment of each measure. Where documentation is available, the auditor should consider the following: DECEMBER 2007 WSAA 10
On relatively minor issues, the auditor may accept the agencies advice regarding availability of documentation and/or demonstrating execution compliance; Wherever practical, the auditor should sight relevant documentation and/or field notes that verify process execution. Where documentation is sighted in the audit process, the auditor should identify this fact by stating documentation sighted in the accredited version of the aquamark software. Given the time pressures, the auditor is not required to validate the effectiveness of the information provided (i.e. if a process exists, is documented and evidence is produced which demonstrates execution in the field then the participant will score highly in this regard. The auditor is not required to comment on whether the outcomes of the process are effective or efficient) Where there is a significant issue or process, the auditor should make a note of their report title and date. This is required to ensure an auditable data trail. 4.6 Professional skepticism: In undertaking the audit, the auditor should consider the information provided by the participant in terms of: Objectivity does the response clearly and concisely address the issue? To what extent may the agencies response be affected by subjective interpretation (e.g. watermain break may be a break in the pipe itself; a leak at a joint; problems with valving etc. Is the procedure well defined?) Replication If the current practitioners within the participating agency were to leave, would the process be clear to a qualified person from outside the agency Materiality do the responses from the field audit differ in a material way from the information contained in the self assessment Staff Knowledge is the matter fully (and consistently) understood by staff? Outcomes from 2004: At the outset of an audit interview, the participants managers were often asked the following open questions: How do you rate your performance against this particular function; and What are your main issues in managing this activity? The first of these questions was designed to provide an initial assessment of information contained in the self assessment and the field response. Where a participant responded that their processes were average or good and yet their overall score for the function was 30 or 90 then it suggested that there was a material difference between the self assessment and actions at the coalface. The second question then allowed the auditor to undertake a broad cross check against the process level responses. DECEMBER 2007 WSAA 11
4.7 Other Statutory Obligations: Several of the processes undertaken by the participant may be subject to other legislative, professional or legal requirements. In the event that these other requirements differ from the requirements under Aquamark, the outcomes of any audit or review conducted by other agencies to meet these external requirements will not in itself represent compliance with the Aquamark standards. In performing the audit, the auditor is to ensure that all aspects of the agencies response are consistent with the specific requirements of the software. Outcomes from 2004: A participant is subject to specific regulation with regard to its operational performance. The participant has been subject to a separate independent audit on its performance against these regulatory requirements. This audit has identified that the agencies performance meets its regulatory obligations. This outcome does not imply that the business therefore meets the relevant aspects of the Aquamark framework. It may well be the case that the regulatory audit is outcomes based which may conflict with the process based nature of the Aquamark project. In this instance, while the auditor may well consider the content and findings of the previous regulatory audit, the auditor is not bound to accept the outcomes of that audit as a basis for the agencies Aquamark response. 4.8 Internal Audit: As part of the contracted auditor s QA processes, the auditor may undertake internal auditing of the project outcomes. This would be undertaken at the discretion of the consultant but may assist the participant in the event of an external audit being undertaken by WSAA (refer section 4.9) 4.9 Peer audit: WSAA reserves the right to undertake an audit of the auditor s processes and practices at any time. Such an audit will be undertaken at WSAAs cost. However, if such an audit indicates deficiencies in the auditors approach (as measured against these protocols), the auditor will be required to address these deficiencies at their cost. WSAA also requires the right to request auditor s previous Asset Management engagements with all participants. 4.10 Production of the Audit report: The outcomes from the audit process will be summarized in a report to each participant. Development of the individual reports will follow a staged process including: The Initial Report: Within two (2) weeks of completion of the site audit, the auditor is to provide the participant with its initial report. This is to be provided in PDF format and MS PowerPoint (to enable presentation within the company) and will highlight the relevant findings for that organization and identifying areas for improvement and how it could bridge the gaps identified. The Benchmarked Report DECEMBER 2007 WSAA 12
Once auditing is completed for all participants, the auditor shall issue an updated version of the initial report. This report will incorporate comparison with the participant s peers. The report will identify where the organization stands relative to other participants and may provide further information on the key features which make any leader stand out in terms of performance Both the Initial report and Benchmarked report will be provided to the participant only. The auditor will not provide a copy of the audit report to any other party without the written permission of the participant. 4.11 Participant Comments on the Audit report On completion of the Initial Report, the business will have the opportunity to comment on the audit findings. Protocols for managing comments will include the following: If the participant identifies factual errors in the audit report (which may arise through mis understanding, or mis communication) then the auditor may adjust the audit report; Where an participant comments are more subjective (i.e. disagreement with the auditors assessment) then the auditor will not be obliged to update the report; and/or Where a participant feels strongly about its comments, it may request that the auditor include their comments in an appendix to the report. Where such a request is made by a participant, the auditor must oblige. DECEMBER 2007 WSAA 13
5.0 Dispute Resolution 5.1 Dispute vs. Disagreement: Disputes are defined as arising where: There is a risk that the difference between the views of the auditor and views of the self assessor would result in a material difference in the overall assessment. If an issue would have relatively a minor impact on the outcome of the assessment, the auditor s assessment will be taken as correct. There has been an issue of conflict of interest or professional integrity WSAA will not consider simple disagreement with the auditor s interpretation, personality differences or other subjective issues as grounds for dispute. 5.2 Dispute Resolution: Where a dispute arises between the auditor and the agency, the following process will be used to resolve the issue: The auditor will advise WSAA of any dispute as it arises; Minor disagreements are to be resolved by negotiation between the parties and will not require involvement by WSAA; Where this negotiation fails, the dispute will be referred to WSAA s Project Director for resolution. This process will include the following: o both parties are to provide the WSAA Project Director with an agreed statement which outlines: The nature of the dispute; Key issues on which the parties agree; Key issues on which the parties disagree; and An assessment of the materiality of the disagreement. o Each agency will then provide a brief (<2 page) statement on their proposed options for resolution of the issue o Both parties will then be given an opportunity to discus the dispute with WSAA o The WSAA Project Director will make a final determination on the issue. This determination will be issued in writing to the parties and not will be subject to further negotiation. Outcomes from 2004: One issue that arose during the 2004 audit was the relative importance of an accredited QA system in obtaining full credit for process documentation. On this issue, there were two opinions: On one hand, it was considered that a businesses documentation could not be assessed as Complete unless the documentation was integrated into the DECEMBER 2007 WSAA 14
businesses overall QA procedures (specifically controlled documentation); The alternative view was that the nature of the businesses overall QA system was assessed under Function 1 (specifically item 1.8). It was considered that the above interpretation would double count the effect of the QA system and effectively mark down any organization which did not have comprehensive QA accreditation (no matter how complete the documentation for a specific activity). The final determination of the issue was that it was not necessary for a business to have a comprehensive QA system for it to score very highly (i.e. complete on documentation for each specific measure (i.e. the alternative view). DECEMBER 2007 WSAA 15