How mature are electricity distributors asset management practices?

Transcription

1 How mature are electricity distributors asset management practices? Authors: Nick Russ and Tobias Maugg, Commerce Commission 1 Presenter: Nick Russ EEA Conference & Exhibition June, Auckland April The authors can be contacted at nick.russ@comcom.govt.nz and tobias.maugg@comcom.govt.nz. The views in this paper are those of the authors and not the Commerce Commission.

2 Abstract For the first time, each distributor has self-assessed the maturity of its asset management practices and published the results from the assessment on its website. This paper explains the asset management maturity assessment tool (AMMAT) and provides initial observations on electricity distributors self-assessment of the maturity of their asset management practices. We summarise our observations from reviewing a sample of distributors asset maturity assessments. We also provide a brief assessment of the level of support for the AMMAT among distributors and how they intend to use the results from the AMMAT. We find that there is generally support for the AMMAT among electricity distributors and many businesses are planning to use the findings from their assessment. Many distributors found themselves on many assessed areas as mature. We find it encouraging that many distributors appear to have assessed themselves critically against the reference standard, found areas for improvement and given themselves lower scores where they considered this was appropriate. We explain why we think that it is not very useful to compare results from the first AMMAT across distributors and attempt to interpret differences in results as differences in actual maturity. The AMMAT is part of a wider set of information that electricity distributors have to publicly disclose under requirements set by the Commerce Commission. We point out that with the information that will become available later this year it should be possible to consider the likely relationship between assessed maturity and actual network performance, cost and quality.

3 1. Introduction Providing electricity distribution services to customers involves a range of different physical assets. The way electricity distributors create, maintain, renew and enhance these assets has implications for the life time cost of these assets, and the quality of service consumers receive from these assets. Each electricity distributor is responsible for choosing and delivering the investments on its network. In doing so distributors need to consider what service customers want and the most efficient way of delivery. Ultimately it is customers that pay for any investment. Self-assessed maturity of asset management practices For the first time, each distributor has self-assessed the maturity of its asset management practices and published the results from the assessment on their website. The asset management maturity assessment tool (AMMAT) is an evidence-based self-assessment questionnaire that distributors are required to complete twice every five years. The results are intended to help stakeholders understand the maturity of their electricity distributor s asset management compared with an asset management reference standard. The AMMAT is part of a wider set of information that electricity distributors have to publicly disclose under requirements set by the Commerce Commission. 2 The purpose of information disclosure regulation is to give transparency about how regulated businesses are performing and to help stakeholders understand whether regulation is working. This transparency should also create incentives for the distributors to improve their performance. 3 Initial observations on the 2013 self-assessments of asset management maturity This paper explains the AMMAT and provides initial observations on electricity distributors self-assessment of the maturity of their asset management practices. Section 2 explains the main features of the AMMAT. Section 3 summarises our observations from reviewing a sample of 23 distributors asset maturity assessments. 4 Section 4 provides a brief assessment of the level of support for the AMMAT among distributors and how they intend to use the results from the AMMAT. 2 Distributors published their AMMAT results as part of their asset management plan for the ten-year period starting on 1 April The requirement to disclose the AMMAT is part of the Commerce Commission s new information disclosure requirements which are available at 3 For a discussion of our role and why we regulate electricity distribution services refer to Commerce Commission, Overview of electricity distributors performance from 2008 to 2011, available at 4 At the time of writing this the Commission had only received the assessments of 23 out of 29 electricity distributors. Our analysis is therefore not representative of the electricity industry overall. 1

4 Section 5 discusses our initial observation on what we can learn from the first set of results from applying the AMMAT. Section 6 concludes and provides some thoughts for further analysis. 2. The asset management maturity assessment tool Asset management comprises systematic and coordinated activities and practices through which a business optimally and sustainably manages its assets and asset systems, their associated performance, risks and expenditures over their life cycles for the purpose of achieving its organizational strategic plan. 5 The goal of infrastructure asset management is to meet a required level of service, in the most cost effective manner, through the management of assets for present and future customers. 6 The quality of asset management affects the lifetime cost and benefit of assets. Having a good understanding of areas of strength and weakness compared with good asset management practices and using this understanding to maintain good practices in some and improve practices in other areas is essential for delivering efficient outcomes. What are the questions in AMMAT? The AMMAT consists of 31 questions designed to cover the full range of asset management activities. These questions are a subset of the 121 questions contained in the Institute of Asset Management s PAS 55 Assessment Methodology (PAM). 7 The AMMAT does not include all 121 questions because aspects that are central to some assessment areas are already included in another PAM question and because some of the areas in PAM are covered in detail in distributors asset management plans. 8 Each question contributes to one of six assessment areas, and each focuses on the way that the organisation manages either its processes or its people. The following are the capability assessment areas: Asset strategy and delivery controls, and review Systems, integration and information management Communication and participation 5 From the Asset Management Specification PAS 55, published by the British Standards Institution. 6 International Infrastructure Management Manual, v3.0, 2006, Association of Local Government Engineering New Zealand Inc (INGENIUM) and the Institute of Public Works Engineering of Australia (IPWEA). 7 See theiam.org/products-and-services/pas55-methodology 8 For a description of the development of the AMMAT refer to Parsons Brinkerhoff, Asset Management Maturity Assessment Tool, 27 September 2011, Review/Asset-Management-Maturity-Assessment-Tool-Study-Report-27-September-2011.pdf The AMMAT including guidance notes and ratings can be found at Assessment-Tool-Study-Spreadsheet-27-September xls 2

5 Structure, capability and authority Competency and training The individual AMMAT questions are reproduced in Attachment 1. What does completing the AMMAT involve? In completing the self-assessment, distributors have to give themselves a score, provide a short explanation to help the user interpret the score, and list the evidence that was used to determine the maturity level. For each question the AMMAT explains why the question is being asked, who in the business should be able to provide an answer and suggests what evidence could be used to support the assessment. The assessment is expressed by a numeric score that indicates the level of maturity; the AMMAT explains for each question what each level of maturity entails. What do the different levels of maturity mean? The AMMAT contains, for each question, a detailed description of each of five levels of maturity when assessed against a reference asset management standard (such as PAS55). The figure below provides a generic description of the different maturity levels and how they can be interpreted. Figure 1 The different asset management maturity levels in the AMMAT Learning Applying Embedding Optimizing and integrating Beyond the reference standard Awareness Development Competence Excellence Maturity level 0 Maturity level 1 Maturity level 2 Maturity level 3 Maturity level 4 The elements required by the reference standard are not in place. The organisation is in the process of developing an understanding of the reference standard. The organisation has a basic understanding of the reference standard. It is in the process of deciding how the elements of the reference standard will be applied and has started to apply them. The organisation has a good understanding of the reference standard. It has decided how the elements of the reference standard will be applied and work is progressing on implementation. All elements of the reference standard are in place and are being applied and are integrated. Only minor inconsistencies may exist. Using processes and approaches that go beyond the requirements of the reference standard. Pushing the boundaries of asset management development to develop new concepts and ideas. Source: The figure is based on the description of maturity levels in The Institute of Asset Management, PAS 55 Asset Management Methodology. 3

6 The Electricity Engineers' Association (EEA) has developed useful guidance to assist and encourage electricity distributors to follow a consistent approach across the industry when undertaking their AMMAT assessments. 9 The EEA notes that while the purpose of its guidance is to assist electricity distributors to prepare their AMMAT disclosures on a consistent basis, it recognises that the guide will be useful for infrastructure businesses in general, not just electricity distributors Initial observations from assessing the results from the AMMAT Most distributors have used PAS55 as the reference standard in their assessments. Industry average scores for each capability assessment area On the whole, the electricity distribution industry considers that its current asset management maturity, compared to a reference standard, is between maturity level 2 ie, it understands the reference standard, has decided how to apply the elements and working on implementing them and maturity level 3 ie, all elements considered to be good practice are in place and are integral part of asset management practices. The figure below shows the average scores for each of the capability assessment areas. Figure 2 Industry average AMMAT scores for each of the capability assessment areas Asset strategy and delivery Communication and participation Competency and training controls, and reviews Structure, capacity, and authority Systems, integration and information management Source: Authors analysis based on information disclosed by electricity distributors. Note: The figure reflects scores by the 23 (out of 29) electricity distributors whose results were available in spreadsheet format at the time of writing. Assessing the dispersion of scores within each capability assessment area The industry average scores hide a considerable amount of dispersion in distributors average scores for each of the capability assessment areas. The figure below marks each of the 9 Electricity Engineers Association, Guide to Commerce Commission Asset Management Maturity Assessment Tool (AMMAT), October Electricity Engineers Association, Guide to Commerce Commission Asset Management Maturity Assessment Tool (AMMAT), October 2012, p

7 distributor s average score as a dot and industry average as a triangle. (Some of the dots are not visible because many of the scores are similar or identical.) Figure 3 Dispersion in distributors average scores for each capability assessment area Asset strategy and delivery Communication and participation Competency and training controls, and reviews Structure, capacity, and authority Systems, integration and information management Source: Authors analysis based on information disclosed by electricity distributors. Note: The figure reflects scores by the 23 (out of 29) electricity distributors whose results were available in spreadsheet format at the time of writing. The figure shows that distributors score themselves highest on the structure, capacity and authority questions, and this area has the lowest dispersion among distributors. The assessment area with the lowest average score and the widest dispersion is competency and training. 11 The level and dispersion of scores varies across distributors Slicing the average results for each competency assessment area by distributor reveals differences in the assessed maturity scores and the spread of scores. The figure below shows the average score for each of the six capability assessment areas for each distributor. 11 For details of the four questions that make up each of the assessment areas refer to Attachment 1. 5

8 Figure 4 Average scores for each capability assessment area for each distributor Alpine Aurora Eastland Electra Electricity Ashburton Electricity Invercargill Mainpower Nelson Network Tasman Network Waitaki Northpower Orion Otagonet Powerco Scanpower Lines Company The Power Company Top Energy Vector Waipa Wellington Electricity WEL Networks Westpower Asset strategy and delivery Competency and training Structure, capacity, and authority Communication and participation controls, and reviews Systems, integration and information management Source: Authors analysis based on information disclosed by electricity distributors. Note: The figure reflects scores by the 23 (out of 29) electricity distributors whose results were available in spreadsheet format at the time of writing. The figure above draws out a number of features. For example, some distributors scores for different capability assessment areas are tightly clustered together, others are relatively widely dispersed. There are clear differences between distributors overall level of self-assessed maturity: Some distributors assess themselves to be close to or exceeding the reference standard (ie, scores of 3 or above). 6

9 Others have scores closer to maturity level 2 (ie, the distributor understands the reference standard, has decided how to apply the elements and is working on implementing them) or below (ie, the distributor has a basic understanding of the reference standard, is in the process of deciding how to apply elements from the reference standard and has started to apply them). The dispersion varies across individual questions We also assessed the dispersion of individual answers across distributors. 12 The table below shows the three questions with the widest dispersion, as well as the number of scores with each level of maturity. We found that a distributor that gave itself a low score, eg 0 or 1, often did not give itself low scores on many other areas. Table 1 The 3 questions with the widest score dispersion between distributors Number of scores with maturity level: Capability assessment area and question Communication and participation: To what extent has an asset management policy been documented, authorised and communicated? Competency and training: How does the organisation ensure that the results of risk assessments provide input into the identification of adequate resources and training and competency needs? Asset strategy and delivery: What plan(s) and procedure(s) does the organisation have for identifying and responding to incidents and emergency situations and ensuring continuity of critical asset management activities? Source: Authors based on qualitatively assessing electricity distributors asset management plans Note: The table reflects scores by the 23 (out of 29) electricity distributors whose results were available in spreadsheet format at the time of writing. We rounded non-integer scores using Swedish rounding. 4. How are distributors planning to use the findings from the AMMAT? Completing and disclosing the AMMAT is a regulatory requirement. Distributors are not required to act on the findings. For example, a distributor with low scores is not required to lift its capabilities, but the decision to settle for a lower standard should be conscious and made transparent. 13 Below we summarise the degree of support for the AMMAT and how distributors intend to use the findings from the AMMAT as stated in asset management plans. While we think this 12 We measured dispersion using the coefficient of variation of responses. 13 Commerce Commission, Information Disclosure of Electricity Distribution Businesses and Gas Pipeline Businesses: Final Reasons Paper, 1 October 2012, para

10 provides some useful insights, we need to be cautious in what we can infer from these on actual support and how the results will actually be used. These results are based on our qualitative assessments of statements in distributors asset management plans and we have not asked distributors to confirm whether our interpretation is complete or correct. 14 There appears to be support among distributors for the AMMAT Electricity distributors have signalled in their asset management plan different degrees of support. The table below summarises the different levels of support. Table 2 Level of support The support for the AMMAT stated in the asset management plans differs between distributors Number of distributors Explicitly supports AMMAT 1 Implicit support - plans to use results 18 No support mentioned 8 Questions the value of the AMMAT 1 Source: Authors based on qualitatively assessing electricity distributors asset management plans. Note: The assessment is based on the 28 (out of 29) distributors whose reports were available at the time of writing. We allowed only one level of support for each distributor. Distributors have indicated different uses for the results from the AMMAT Electricity distributors have signalled in their asset management plan ways in which they plan to use the results from their assessments. The table below summarises the different intended uses. 14 The AMMAT was generally well received during its development. For a brief summary of distributors views and concerns with the AMMAT refer to Commerce Commission, AMMAT Workshop: Assessing the maturity of asset management in NZ s EDBs a new tool, 27 June 2011, Workshop Minutes. Services-Asset-Management-Tool-Workshop-Minutes-June-2011.pdf 8

11 Table 3 Stated use Distributors intend to use the results from the AMMAT in different ways Number of times mentioned Use insights for driving change 17 Set targets to improve score 6 Embed the AMMAT as part of asset management plan 3 Move towards PAS55/ get full PAS55 certification 7 AMP does not explain how will use 9 Source: Authors based on qualitatively assessing electricity distributors asset management plans. Note: The assessment is based on the 28 (out of 29) distributors whose reports were available at the time of writing. Where applicable, we allowed for multiple stated uses for each distributor. 5. What can we learn from this first assessment of asset management maturity? We found it encouraging that many distributors appear to have found going through the AMMAT exercise useful, and many distributors are planning to respond by implementing changes to the processes that support their asset management. Many distributors appear to have gone through considerable effort in completing their self-assessments, or are planning to do so over the coming years. We also found it encouraging that many distributors found themselves on many assessed areas as being close to or exceeding maturity level 3. However, we think it is just as encouraging that many distributors appear to have assessed themselves critically against the reference standard, found areas for improvement and given themselves lower scores where they considered this was appropriate. We do not think that it is very useful to compare results from the first AMMAT across distributors and attempt to interpret differences in results as differences in actual maturity. This is the first time distributors have undertaken this self-assessment and some may not yet be fully familiar with their reference standard for good asset management practices. 9

12 Counties Power provides a helpful explanation why there may be differences in views regarding the level of maturity within and between distributors: 15 The initial assessment of maturity ratings for each question resulted in a range of scores by the participating group. Group discussion narrowed this range but did not eliminate it, since the AMMAT maturity level descriptions are, in many cases, broadly defined and subject to interpretation that will differ with organisational perspective. [ ] Scoring inconsistencies were found to be associated with problems in maturity level definitions, not major disagreements about performance in each functional area. Differences were therefore less significant for the purposes of identifying asset management practice improvement priorities and initiatives. Inconsistency in quantitative scoring between electricity lines businesses is likely in the early stages of this regime and until less subjective criteria for maturity scoring are developed. [ ] We expect that collaborative work within the industry, such as the EEA s work that is underway to provide more detailed guidance on the AMMAT will help promote greater consistency across the industry. 6. Conclusions Our high level analysis of the first application of the AMMAT suggests that it is a valuable tool for electricity distributors and their stakeholders. We expect that its repeated application will further increase its value. Good asset management promotes the long term benefit of consumers. Electricity consumers care about the price and quality of the electricity they receive and electricity distribution services contribute to that. The price for and quality of electricity distribution services is a consequence of how well the network performs and how much it costs to create, maintain, renew and enhance the network. The way the owner of electricity distribution assets chooses to manage those assets is a key driver of network performance and the lifetime cost of assets. The AMMAT assesses the maturity of asset management practices against a good practice reference standard. Distributors are required to publicly disclose by August 2013 information on their performance in the year ending March This disclosure will include information on asset condition, expenditure and reliability. Over time, it should be possible to consider the likely relationship between assessed maturity and actual network performance, cost and quality. 15 Counties Power Asset Management Plan 2013, p

13 Attachment 1 The 31 questions in the AMMAT The table below sets out the 31 questions in the AMMAT. Table A1 The 31 question in the AMMAT Assessment category Asset strategy and delivery Asset strategy and delivery Asset strategy and delivery Asset strategy and delivery Asset strategy and delivery Asset strategy and delivery Asset strategy and delivery controls, and reviews controls, and reviews controls, and reviews controls, and reviews Question What has the organisation done to ensure that its asset management strategy is consistent with other appropriate organisational policies and strategies, and the needs of stakeholders? In what way does the organisation's asset management strategy take account of the lifecycle of the assets, asset types and asset systems over which the organisation has stewardship? How does the organisation establish and document its asset management plan(s) across the life cycle activities of its assets and asset systems? What plan(s) and procedure(s) does the organisation have for identifying and responding to incidents and emergency situations and ensuring continuity of critical asset management activities? How has the organisation documented process(es) and/or procedure(s) for the identification and assessment of asset and asset management related risks throughout the asset life cycle? How does the organisation ensure that process(es) and/or procedure(s) for the implementation of asset management plan(s) and control of activities during maintenance (and inspection) of assets are sufficient to ensure activities are carried out under specified conditions, are consistent with asset management strategy and control cost, risk and performance? How does the organisation instigate appropriate corrective and/or preventive actions to eliminate or prevent the causes of identified poor performance and non conformance? Where the organisation has outsourced some of its asset management activities, how has it ensured that appropriate controls are in place to ensure the compliant delivery of its organisational strategic plan, and its asset management policy and strategy? What documentation has the organisation established to describe the main elements of its asset management system and interactions between them? What procedure does the organisation have to identify and provide access to its legal, regulatory, statutory and other asset management requirements, and how is requirements incorporated into the asset management system? How does the organisation establish implement and maintain process(es) for the implementation of its asset management plan(s) and control of activities across the creation, acquisition or enhancement of assets. This includes design, modification, procurement, construction and commissioning activities? 11

14 Assessment category controls, and reviews controls, and reviews controls, and reviews Systems, integration and information management Systems, integration and information management Systems, integration and information management Systems, integration and information management Communication and participation Communication and participation Communication and participation Communication and participation Structure, capacity, and authority Structure, capacity, and authority Structure, capacity, and authority Question How does the organisation measure the performance and condition of its assets? What has the organisation done to establish procedure(s) for the audit of its asset management system (process(es))? How does the organisation achieve continual improvement in the optimal combination of costs, asset related risks and the performance and condition of assets and asset systems across the whole life cycle? What has the organisation done to ensure that appropriate arrangements are made available for the efficient and cost effective implementation of the plan(s)? (Note this is about resources and enabling support) What has the organisation done to determine what its asset management information system(s) should contain in order to support its asset management system? How does the organisation maintain its asset management information system(s) and ensure that the data held within it (them) is of the requisite quality and accuracy and is consistent? How has the organisation's ensured its asset management information system is relevant to its needs? To what extent has an asset management policy been documented, authorised and communicated? How has the organisation communicated its plan(s) to all relevant parties to a level of detail appropriate to the receiver's role in their delivery? To what degree does the organisation's top management communicate the importance of meeting its asset management requirements? How does the organisation ensure that pertinent asset management information is effectively communicated to and from employees and other stakeholders, including contracted service providers? How are designated responsibilities for delivery of asset plan actions documented? What has the organisation done to appoint member(s) of its management team to be responsible for ensuring that the organisation's assets deliver the requirements of the asset management strategy, objectives and plan(s)? How does the organisation ensure responsibility and the authority for the handling, investigation and mitigation of asset-related failures, incidents and emergency situations and non conformances is clear, unambiguous, understood and communicated? 12

15 Assessment category Structure, capacity, and authority Competency and training Competency and training Competency and training Competency and training Competency and training Question How does the organisation seek and acquire knowledge about new asset management related technology and practices, and evaluate their potential benefit to the organisation? What evidence can the organisation's top management provide to demonstrate that sufficient resources are available for asset management? How does the organisation develop plan(s) for the human resources required to undertake asset management activities - including the development and delivery of asset management strategy, process(es), objectives and plan(s)? How does the organisation identify competency requirements and then plan, provide and record the training necessary to achieve the competencies? How does the organisation ensure that persons under its direct control undertaking asset management related activities have an appropriate level of competence in terms of education, training or experience? How does the organisation ensure that the results of risk assessments provide input into the identification of adequate resources and training and competency needs? 13