Introduction... 2 About Bristol Water... 2 About this document... 2 Regulatory Requirements... 3 Responding to this consultation... 3 Our analysis of strengths, risks and weaknesses... 5 Background... 5 Informing customers and stakeholders about our performance... 6 Our approach to providing information and reporting on performance... 6 Assurance progress during 2016/17... 9 How we have assessed our risks, strengths and weaknesses... 9 How risk assessments will impact on our assurance... 12 Feedback from stakeholders on our information reporting... 13 Results of Assessment... 14 External assessment... 20 Summary of findings of assessment... 22 Next Steps... 22 Annex 1 Assessment Methodology Scoring System... 23 Annex 2 Results of Risk Assessment... 27 Page 1
Introduction About Bristol Water Bristol Water is a water supply company, responsible for the provision of water to 1.2 million customers in the city of Bristol and surrounding area. Our vision is to meet our customers expectations by providing an outstanding water service in a sustainable and affordable way. We published our Business Plan for 2015-20, and identified six key aims that will help us to deliver our vision. We want to provide a highly reliable supply We want your water to be of excellent quality We want to be environmentally sustainable We want to be responsive to our customers We want our organisation to attract the best people and have the right culture We want to be a sustainable business About this document We regularly publish information on our performance to demonstrate to customers, stakeholders and our regulators that we are delivering the services expected of us. This document considers the risks, strengths and weaknesses of the data we assess and report on, and how we aim to ensure trust and confidence in the information we provide. We consider it important to demonstrate to our customers and other stakeholders that we report information on our performance that is transparent, reliable, relevant, complete and up-to-date. This is part of our commitment to demonstrate that we take ownership of what we report. In this document, we set out the results of our own analysis, the conversations we have had with stakeholders to date and the controls that we have in place to provide assurance on our information. We first published an assessment of our risks, strengths and weaknesses in November 2015, which we used to inform our draft and final assurance plans for 2015/16. This updated version takes into Page 2
account the outcomes of the assurance work we have carried out in the last year and feedback we have received from stakeholders. We have also extended the scope of our review to include additional data that we publish and provide to regulators, including the data we are providing to support the opening of the retail market for non-household customers from April 2017. Regulatory Requirements The Water Industry Regulator, Ofwat, requires different companies to provide different levels of assurance to support the information they publish, depending on Ofwat s confidence in the quality of the information that companies produce. Companies where Ofwat has given the most discretion to decide what assurance above the minimum is required has the most trust are categorised as selfassurance, Ofwat categorises companies as targeted where it considers companies have not consistently met the high standards customers and other stakeholders expect. Ofwat categorises companies as prescribed where companies have not provided them with sufficient confidence in their ability to deliver, monitor and report performance. In November 2015, Ofwat informed Bristol Water that it intended to move it into the prescribed category. At that point Ofwat confirmed that companies must remain in the prescribed category for a minimum of two years, and so Bristol Water would remain prescribed until at least November 2017. Ofwat s updated view of the Company Monitoring Framework published on 30 th November 2016 confirmed this position. As a prescribed company we are required to incorporate independent external assurance as part of our assurance plans. We set out in this document the role that our independent technical assurer, Atkins, and our financial auditors, PwC, will play in assurance of our key submissions. We are required to publish all of our assurance plans on our website ahead of reporting. Following the conclusion of our consultation on this statement of risks, strengths and weaknesses we will publish and consult on our draft assurance plan, we will publish a final version of our plan by 31 st March 2017, ahead of our annual reporting and assurance programme. Our draft assurance plan will set out what we will do to meet the criteria for prescribed status, and address the minor concerns noted in Ofwat s updated monitoring framework assessment. Our intention is that through demonstration of compliance with these criteria we will enable Ofwat to move us to one of the less prescriptive categories in its 2017 assessment. Responding to this consultation We would like to hear your views on the way we assess data and information and how we present our performance to customers and other stakeholders. We have published this document as a consultation on which we would like your feedback. We would ask you to read through the whole document and carefully consider the following questions, which we invite your views upon: 1. How would you use the information published by Bristol Water? Would you use it to compare Bristol Water with other water companies, or other utility service providers? Page 3
2. Is the information that Bristol Water publishes produced in a useful and easy to understand format, and sufficiently precise for your needs? 3. Is there any information that Bristol Water doesn t currently publish that you consider would be useful? 4. What do you consider to be the most important type of information that Bristol Water provides? This could be information about your bill, the work we are doing in your area, whether we are meeting our customer service targets, our financial performance or other information and data that we publish. 5. Do you have any concerns about the quality of the information that is provided by Bristol Water? Do you consider it to usually be reliable? 6. Are there lessons we could learn from elsewhere to improve the reliability of our information? 7. Are there any other risks, strengths or weaknesses in our information that you think we should address? Please let us know your thoughts on these questions or any other comments you have. You can respond using our online survey, by email to Alex.Smethurst@bristolwater.co.uk, or by post to: Alex Smethurst Bristol Water Bridgwater Road Bristol BS13 7AT Your views are requested by 31 st January 2017. Page 4
Our analysis of strengths, risks and weaknesses Background In June 2015, the water industry regulator, Ofwat, published a company monitoring framework, which set out how it intends to oversee the information that water companies in England and Wales provide to their customers. This framework is intended to provide customers and stakeholders with confidence in the information that companies produce, and allow Ofwat to take action where it feels that its requirements are not being met. As part of the company monitoring framework, Ofwat requires each company to engage with its customers and stakeholders annually to understand any issues with its information, and to publish a statement on that engagement ahead of the publication of information. As we describe above, Ofwat has categorised us as a prescribed assurance company, and as such will be looking to confirm that the level of assurance provided for our information meets its requirements. In November 2016 Ofwat outlined the results of the latest company monitoring framework. The assessment re-affirmed that Ofwat would not consider a company assessed as prescribed for an upgraded assurance rating until 18 months after downgrading and that it did not therefore consider changing Bristol Water s assurance status for 2016/17 reporting. Although minor concerns were raised about some data ambiguities, Ofwat concluded that we had met their expectations across the majority of the assessments and that our data was in most cases accurate and well-explained. Area of Assessment Financial Monitoring Framework Charges schemes assurance Financial information Final 2010-15 reconciliation data submission Outcomes Compliance with principles of board leadership, transparency and governance Risk and compliance statement Assurance plan Targeted reviews Data assurance summary Evidence from casework activities Ofwat Assessment Meets expectations Meets expectations Meets expectations Meets expectations Meets expectations Meets expectations Meets expectations Minor concerns Minor concerns Minor concerns Minor concerns Page 5
Informing customers and stakeholders about our performance It is important that customers can find out how we are performing against our targets. We are committed to providing this information on our website and have embraced a more open and accessible approach to customer communications, with a report on performance in our customer magazine Watertalk. Our company annual report provides data and commentary on our performance that is aimed at a broad audience, and we provide an update on our performance every six months. We present this information to our stakeholders through the Bristol Water Challenge Panel 1, who have the opportunity to challenge our information and performance and have already been consulted for this assurance consultation (see section 5). Our approach to providing information and reporting on performance We use a thorough system of controls to make sure that the information we report and publish is as accurate as possible. Each piece of information has a specific owner and reviewer, responsible for production and updating the reporting methodology statement. Data owners and reviewers are required to provide signed confirmation that the data has been compiled in accordance with the relevant methodology, and that the data is a true representation of the facts. This form provides the opportunity for the data owner to identify any concerns with the quality of the data, for investigation by senior managers and Directors. A committee of executive Directors reviews key data and information before it is published. Major regulatory submissions, including annual reports, tariffs, accounts and business plans are subject to Board review and approval prior to submission. We also use external expert auditors to review our methods, systems and processes for reporting key data and information. In particular, the engineering consultancy, Atkins, provides technical assurance on our regulatory submissions, and financial auditors, PwC, audit our key financial data. We also have an internal audit function, which is currently outsourced to Mazars, that reviews our compliance with our governance and assurance procedures. These auditors provide reports to our Board to provide confidence in the accuracy of the information produced. Our main regulatory submissions are subject to sign off by the Board before we send them to Ofwat. Our PR14 business plan submissions were subject to this detailed level of internal and external assurance, and were approved by our Board on the basis of the assurance received. However, we recognise the comments subsequently made by the CMA and Ofwat that the level of supporting detail and evidence to underpin the proposals in our plan could have been greater. We want all parties to have trust and confidence in the information we produce, and we will take this feedback into account in our future provision of regulatory information and business plans. The business plan that we will deliver for 2015 2020 is based on a number of outcomes, which were identified through detailed customer engagement, and agreed with our PR14 stakeholder challenge group, the Bristol Water Local Engagement Forum (LEF). To monitor the delivery of those outcomes we set out a specific performance measure for each outcome, and a target level of service. These targets were then reviewed by Ofwat and the Competition and Markets Authority, to ensure they reflected customers desired level of service. 1 The Bristol Water Challenge Panel is the successor to the Local Engagement Forum Page 6
To make sure our focus is on the delivery of outcomes that customers want, financial incentives have been applied to many of the performance measures. These incentives mean that money will be returned to customers if we don t deliver the required level of service, and in some cases mean that we can charge a little more if we perform well. Other measures do not have financial incentives, but a reputational incentive is created by our reporting of our performance against targets. The table below sets out the customer-facing performance measures against which we report. A full list of our performance measures, which also includes some internal measures, is included within Annex 2. The internal measures will be included in our annual report as we think it is important for customers and stakeholders to have trust and confidence in our performance on all of our key measures. Page 7
Outcome Performance Measure Incentive Reliable Supply Customer Minutes Lost to unplanned interruptions Financial reward and penalty Reliable Supply Assessment of asset reliability Financial penalty only Resilient Supply Population at risk from asset failure Financial reward and penalty Sufficient Supply Security of Supply Index Reputational Sufficient Supply Hosepipe ban frequency Financial penalty only Safe Drinking Water Water is Good to Drink Efficient use of resources by company Efficient use of resources by customers Sustainable Environmental Impact Sustainable Environmental Impact Sustainable Environmental Impact Sustainable Environmental Impact Efficient Use of Resources by Customers Mean Zonal Compliance with water quality regulations Negative Water Quality Contacts Leakage Meter Penetration Total carbon emissions Raw water quality of sources Biodiversity Index Waste Disposal Compliance Per Capita Consumption Financial penalty only Financial reward and penalty Financial reward and penalty Financial reward and penalty Reputational Reputational Reputational Reputational Reputational Page 8
Outcome Performance Measure Incentive Affordable Bills Percentage of customers in water poverty Reputational Satisfied Customers Satisfied Customers Satisfied Customers Service Incentive Mechanism General satisfaction from surveys Value for Money rating from surveys Financial reward and penalty Reputational Reputational Easy to Contact Ease of contact rating from surveys Reputational Bills are accurate and easy to understand Negative billing contacts Reputational Assurance progress during 2016/17 Since the publication of our risks, strengths and weaknesses in 2015/16 we have revised our stance to look at industry best practice in all areas of assurance. We have carried out additional assurance, via: An audit on the methodology used to set our customer tariffs carried out by Frontier Economics; Data required by Ofwat to support its development of cost efficiency models was subject to audit by Atkins; A 16/17 mid-year audit of data and methodologies carried out by Atkins; Additional resources have been brought in to map our business processes and embed them in the business, including updates to methodology documents to demonstrate data controls; The recruitment of two new staff in to the regulation team since July 2016 to support the reporting and assurance process; We have also identified our assurance framework to support the production of our PR19 business plan, including a Board Assurance Partner to oversee all assurance activity and report to the PR19 Board Committee. How we have assessed our risks, strengths and weaknesses We have analysed the current risks, strengths and weaknesses of the key information we report and publish and as well as the outcome performance measures outlined above; this now also includes: Internal-facing outcomes identified in our PR14 business plan The customer tariffs we publish in our charges scheme Our regulatory accounts that we publish each year Page 9
Information on compensation payments made through the Guaranteed Standards Scheme and the Bristol Water Bond information on water quality provided to the Drinking Water Inspectorate information provided to the Environment Agency information on customer complaints provided to the Consumer Council for Water data provided for opening of the market for retail services to non-household customers from April 2017 information provided to Water UK on services to developers To carry out our analysis we have examined the inherent probability of inaccurate reporting this information, the strength of the control framework we have in place to mitigate this risk and the potential impacts of any errors. This analysis has been carried out by the team in Bristol Water responsible for reporting information to customers, regulators and stakeholders. This team has detailed knowledge of our reporting processes and requirements, and the control framework in place for assurance of our information. Methodology We have based our methodology on the Data Assurance Guidance for Electricity and Gas Network Companies, published by Ofgem in February 2015. This framework provides a sound basis for assessment of data reported by utility companies. We have in some cases made alterations to the detail of our assessment to take account of differences in priorities or processes between the water and energy industries. This methodology identifies a five-stage risk assessment process, as set out in Figure 1: Figure 1 - Risk Assessment Process 1 2 3 4 5 Assess impact of risk for each data item Determine probability of inaccurate reporting Assess strength of controls for each data item Assess impact of inaccurate reporting for each data item Determine total risk rating Page 10
Probability The inherent probability of inaccurate reporting is assessed against four criteria: Complexity of data sources Completeness of data set Extent of manual intervention Complexity and maturity of reporting rules. Each data item is given a score between two and four, with four representing the highest risk of inaccurate reporting. The details of how each score is determined are provided in Annex 1 to this paper. Control Framework The control framework for each measure is assessed against three criteria: Control activities Experience of personnel Evidence of historical errors with this data. Each data item is given a score between zero and two, with zero representing the least effective control framework. The details of how each score is determined are provided in Annex 1 to this paper. No item has been awarded a score of two if Atkins has identified the internal methodology we use as needing improvement. Impact Assessment We have assessed the impact of inaccurate reporting or calculation of each measure against four criteria 2 : Impact on customers if inaccurate reporting meant that service levels were not being adequately identified as being below the levels required, how would this impact on customers Impact on company reputation would the company s reputation suffer adversely if it was found to have reported inaccurately against this measure Use as financial incentive to what extent is the measure used to influence financial penalties or rewards applied to the company Use as industry comparator to what extent is the measure used by Ofwat or other stakeholders to make comparisons between water companies. 2 We have chosen the company reputation criterion in place of the competition criterion suggested in the Ofgem paper, as this was considered to be a more appropriate consideration for assessment of our outcomes, most of which do not impact on the competitive market Page 11
Each data item is given a score between one and four, with four representing the greatest impact. Overall Assessment The probability score is taken as the maximum of any of the scores given in that assessment. The control score is taken as the average of the scores given in that assessment. A post-control probability score is taken by deducting the control score from the probability. Where this produces a result of zero, a score of one is applied. The post-control probability and impact scores are taken together to ascertain a position on a risk matrix: Figure 2 - Impact Probability Risk Matrix 3 Our target is to reduce the risk score for each data item to the lowest appropriate level. A measure can only be classified as critical where there are potential weaknesses in the control framework, our aim is to ensure that any such weaknesses are identified and addressed. Our objective is for the probability score to be no greater than 2, and that therefore each measure is no greater than medium risk, unless it has a very high impact. Probability metric scores of 3 or 4 could be due to factors such as the range of data sources used or the complexity of reporting rules and associated assumptions. In these cases we will review whether simplification is possible, and whether the control framework can be further strengthened to minimise the risk of inaccurate reporting. How risk assessments will impact on our assurance We will set out the level of internal and external assurance that we propose to apply to each piece of information in our draft assurance plan. Our Board will review our proposed audit plan, and set out 3 Taken from Ofgem Data Assurance Guidance, page 16 Page 12
the level of internal and external assurance it requires to provide comfort in the accuracy of our data. We intend that all information assigned as medium, high or critical risk would be subject to external assurance prior to publication. This would normally be from our technical assurers or financial auditors, who would provide a statement to our Board on their findings. Where risks are identified as low, we will consider the extent to which external assurance is required, or whether our internal controls will provide sufficient confidence in the quality of the data. Feedback from stakeholders on our information reporting The Bristol Water Challenge Panel reviews our performance against delivery of our PR14 business plan, including the assurance arrangements in place to support data reporting. The Bristol Water Challenge Panel is comprised of representatives from the Consumer Council for Water, the Environment Agency, Natural England, Citizens Advice Bureau, local councillors, local universities, the Avon Wildlife Trust and independent customer representatives. Figure 3 - Composition of Bristol Water Challenge Panel Education The environment and nature Regulators Health Regional and local government Retailers Others Commerce and industry Stakeholder Panel Customer Service We provided details of our 2015/16 performance to a meeting of the BWCP on 9 th June 2016. This meeting was also attended by our technical reporter from Atkins, who gave details of the assurance assessment on each outcome. Page 13
A further meeting of the BWCP was convened on 30 th November to review our assurance framework and progress against actions raised from year end reporting. Our updated data risks assessment was presented to this meeting. We intend to use this consultation as the basis for further engagement with our stakeholders as we develop our final assurance plan for2017/18, to be published by 31 st March 2017. Progress on delivery of our assurance plan will be reviewed by the BWCP at a meeting in the summer of 2017, when 2016/17 performance data will be presented. Results of Assessment The position on the assessment matrix of each piece of information we have assessed is shown in Figure 4. The full results of the assessment we carried out are set out in Annex 2. Page 14
Figure 4 - Results of Risk Assessment Critical One measure was identified as being within the critical category. Page 15
Tariffs and Charges Tariffs and charges were identified as being a critical risk due to misallocation of revenue between price controls in 2015/16 reporting. This was caused as the collation of data requires a number of assumptions and the rule set used to report the data requires significant interpretation. The impact of inaccurate reporting would be highly significant due to the impact on service to customers, the reputational impact on the Company and Ofwat s use of tariffs and charges to compare company performance. Our assessment of the impact reflects the importance of this measure. High Thirteen measures were identified as high risk. Five of these measures were identified as having a probability of inaccurate reporting score of three. Non-Household Market (Operational Performance) Data Reporting of non-household market (operational performance) data was identified as being high risk. This is due to the complexity of reporting rules and associated assumptions, which are defined in Open Water s Market Performance Framework. The data collated by Bristol Water is driven from our systems handling operational activity with external parties, such as a request to install a meter. The impact of inaccurate reporting would be significant due the impact on customers and the measure s use as an industry comparator. Developer Services Information Reporting of Developer Services Information was identified as being high risk. Since April 2015 all water companies have been required to publish information on their performance in handling enquiries from developers. This information is collated by the industry body Water UK. We reviewed our process for handling developer contacts in response to an Ofwat investigation, and now follow an agreed methodology and process. The impact of inaccurate reporting however would be significant due to the effect on service to customers and our assessment of the impact as high risk reflects that importance. Hosepipe Ban Frequency Maintaining a 1 in 15 year hosepipe ban frequency is delivered by a range of measures including leakage and metering and reporting of the frequency of hosepipe bans was identified as being high risk. This is due to the complexity of the data sources involved, which are derived from multiple systems. The impact of inaccurate reporting would be significant due the financial incentive attached to the measure and the its use as an industry comparator. CCWater complaints data Reporting of complaints data was assessed as high risk. This reflects the use of multiple data sources, as we combine billing and operational data. We also recognise the potential reputational impact of inaccurate reporting, which may in turn indicate an issue with SIM reporting. Customers would be impacted on if complaints were not being correctly classified, which could lead to non-payment of GSS. Biodiversity Index The Biodiversity Index commenced in April 2015 and the Index was identified as being high risk as it is still under development. This is due to the complexity of the data sources involved, which are Page 16
derived from multiple systems and the number of assumptions to collate the data. The impact of inaccurate reporting would be significant due the impact on customers. The other eight measures in the high risk category were identified as having a probability of inaccurate reporting score of two. Infrastructure Asset Reliability Reporting of infrastructure asset reliability is based on the total number of bursts and the number of properties identified as low pressure. The measure was identified as being high risk. Although a strong control environment is in place, the risk rating is due to the complexity of data sources, which involves multiple systems. The impact of inaccurate reporting would be significant due to the impact on service to customers, the impact on the Company s reputation and the financial incentive attached to infrastructure asset reliability performance. Non-infrastructure Asset Reliability (Turbidity at WTWs) Reporting of non-infrastructure asset reliability (turbidity at WTWs) was identified as being high risk. Although a strong control environment is in place, this risk rating is in recognition that the collation of data requires a number of assumptions. The impact of inaccurate reporting would be significant due to the financial incentive attached to non-infrastructure asset reliability and the measure s use as an industry comparator. Mean Zonal Compliance The percentage Mean Zonal Compliance is determined as per the guidance given in the Drinking Water Inspectorate publication Calculation and Composition of Indices Published in the Chief Inspector s Report. Reporting of the measure was identified as being high risk. Although a strong control environment is in place, this risk rating is because the collation of data requires a number of assumptions. The impact of inaccurate reporting would be significant due to the financial incentive attached to the index and the measure s use as an industry comparator. Leakage Leakage reporting was identified as being high risk. Although a strong control environment is in place, the risk rating is due to the calculation of the data, which involves a number of different systems, with some assumptions required. The impact of inaccurate reporting would be significant due to the financial incentive attached to leakage performance, the reputational impact and Ofwat s use of leakage to compare company performance. Leakage is particularly important to stakeholders as a key measure to compare company performance against each other and target, and our assessment of the impact reflects that importance. GSS Payments Our system for payment and monitoring of GSS payments is assessed as high risk. Although the calculation of the data involves a number of different systems, with some assumptions required, a thorough analysis and review of this system was carried out in 2006, in response to reporting issues at other companies. This review found no material issues, and we have continued to update and review our processes accordingly, which includes a strong control environment in place. Inaccurate reporting would however have significant regulatory and reputational impacts on the Company, and would impact customers if payments were not being made to which they were entitled. Page 17
Regulatory Accounts Our regulatory financial reporting was assessed as high risk. Although data systems and methodologies are well established, the calculation of the data involves a number of different systems, the collation of data requires a number of assumptions and the rule set used to report the data requires significant interpretation. A strong control environment is in place, but any inaccurate reporting would have significant impact particularly on Ofwat s regulation of Bristol Water and use of comparative data. Population at Risk from Asset Failure This performance measure is influenced solely by delivery of specific schemes so the metric has to be viewed in the context of the AMP6 investment plan (derived through PR14 investment planning). Reporting of customers at risk from asset failure was identified as being high risk. Although a strong control environment is in place, this risk rating is due to the complexity of the data sources involved, which are derived from multiple systems. The impact of inaccurate reporting would be significant due to the financial incentive attached to the measure. Unplanned Customer Minutes Lost Reporting of unplanned customers minutes lost was identified as being high risk. Although a strong control environment is in place, this risk rating is due to the complexity of the data involved, with sources derived from multiple systems, such as BWBSL s Billing System Rapid. The impact of inaccurate reporting would be significant due to the impact on customers, the impact on the Company s reputation and the financial incentive attached to the measure. Medium Twenty-three measures were identified as medium risk. Two of the measures were identified as having a probability of inaccurate reporting score of three. Raw Water Quality of Sources As the water quality of some of our sources is at risk of deterioration, due to potential ingress of nutrients and/ or pesticides from its catchments, reporting of raw water quality of sources was identified as being medium risk. This is because of the complexity of the data sources involved, which are derived from multiple systems, the complexity of collating the data and the rule set used to report the data requires significant interpretation. The impact of inaccurate reporting would be noticeable due to the financial incentive attached to the measure. Training Matrix of Compliance Reporting of training matrix compliance was identified as being medium risk. This is because of the rule set used to report the data requires significant interpretation. The impact of inaccurate reporting would be noticeable due to the measure s indirect impact on service to customers. Seven of the medium risk measures were identified as having a probability of inaccurate reporting score of two. Non-household Market (Performance) Data Reporting of non-household market (performance) data was identified as being medium risk. This is because of the incompleteness of data and the rule set used to report the data requires significant interpretation. The impact of inaccurate reporting would be considerable due to the measure s indirect impact on service to customers and its potential to be used as an industry comparator. Page 18
Per Capita Consumption (PCC) Reporting of PCC was identified as being medium risk. This is because of the complexity of the data sources involved, which are derived from multiple systems. There may be some minor adverse impact on customers from inaccurate reporting but a strong control environment is in place. Total Carbon Emissions Reporting of total carbon emissions is entered into the UKWIR Greenhouse Gas Estimator workbook, which is updated annually by UKWIR on behalf of the UK water industry and the measure was identified as being medium risk. This is because of the complexity of the data sources involved, which are derived from multiple systems. The impact of inaccurate reporting would be considerable due to the potential for the measure to be used as an industry comparator, but a strong control environment is in place. Security of Supply Index The index is calculated as the proportion of dry weather demand that can be met by the water available for use. Reporting of the index was identified as being medium risk. This is because of the complexity of the data sources involved, which are derived from multiple systems. The impact of inaccurate reporting would be considerable due to potential for the measure to be used as an industry comparator but a strong control environment is in place. EA Data The information we provide to the Environment Agency includes data on our water resource position and updates to our water resources management plan. We have well established systems and processes for providing this information and a strong control environment is in place. Inaccurate reporting would have a minimal immediate impact on customers as it largely affects the longer term resource position, but would impact on the ability of the Environment Agency and other interested parties to compare our information and act accordingly. As such, this measure was identified as medium risk. Waste Disposal Compliance Reporting of waste disposal compliance was identified as being medium risk. This is because of the number of assumptions used to collate the data. Inaccurate reporting would have a minimal immediate impact on customers, but would impact on the Company s reputation and the ability of other interested parties to compare our information and act accordingly. A strong control environment is also in place. Stakeholder Survey Reporting of the stakeholder survey was identified as being medium risk. This is due to the incompleteness of data sets and the rule set used to report the data requires significant interpretation. The impact of inaccurate reporting would be considerable due to the impact on the Company s reputation. The other sixteen measures in the medium risk category were identified as having a probability of inaccurate reporting of one. Low Two measures (pumping efficiency and staff satisfaction) were identified as being of low risk, as they are not subject to regulatory incentives or reporting. Page 19
External assessment Our reporting data and methodologies were reviewed by Atkins as part of their year-end audit of performance data. Following this audit, we began work to address concerns raised in the assessment of some methodologies. This work is following a set programme, intended to improve all methodology documents to a satisfactory standard by year end reporting for 2016/17. As part of their review of our 2016/17 mid-year performance data, Atkins again reviewed the progress in updating methodology documents. Measures which are assessed based on an annual calculation were not reviewed at mid-year. The outcome of these two audits on our data and methodology documents is as follows: Table 1 - Review of Data Quality Measure 15/16 Year End Assessment 16/17 Mid-Year Assessment A1 Unplanned Customer Minutes Lost Green Green A2 Asset Reliability Infrastructure Green Green A3 Asset Reliability Non-Infrastructure Green Green B1 Population in centres > 25,000 at risk from asset failure Green Green C1 Security of Supply Index Green Not Covered C2 Hosepipe Ban Frequency Green Not Covered D1 Mean Zonal Compliance Green Green E1 Negative Water Quality Contacts Green Green F1 Leakage Green Green G1 Meter Penetration Green Green G2 Per Capita Consumption Green Green H1 Total Carbon Emissions Green Green H2 Raw quality of sources Green Not covered H3 Biodiversity Index Green Green H4 Waste disposal compliance Green Green I1 Percentage of customers in water poverty Green Not covered Page 20
Measure 15/16 Year End Assessment 16/17 Mid-Year Assessment J1 Service Incentive Mechanism (SIM) Green Green J2 General Satisfaction Green Not covered J3 Value for Money Green Green K1 Ease of Contact Green Green L1 Negative Billing Contact Green Not covered This assessment provides us with confidence that there are no material issues with the quality of our data systems for reporting on our performance measures. Table 2 - Methodology Assessments We have reflected these findings in our assessment, to indicate that controls need to be strengthened where the status of methodology documents is shown as Amber. This assessment provides us with confidence that there are no material issues with the quality of our data systems for reporting on our performance measures. Page 21
Summary of findings of assessment Our initial assessment is that the probability of inaccurate reporting the majority of our key information is on the lower end of the scale. Although strong internal and external control frameworks are in place for some measures, a number of the items have been assessed as amber. This has been caused by a range of factors, including: By being placed within the prescribed status by Ofwat we have an increased focus on assurance and reporting processes; The Atkins review of 2015/16 and mid-year 2016/17 reporting methodologies, which has helped identify potential weaknesses promptly; Changes in assessment methodology and personnel affecting the control frameworks for some of the indicators; Identification of additional data reporting for assessment; and A shift in focus from prioritising measures related to financial incentives, to take a greater account of the impact on customers, by giving more weight on reputational incentives These factors do not reflect an expectation of an increase in any adverse impact on customers, but rather reflects the Company s ability to identify the potential for errors and act accordingly to address them. We also recognise that most of the information we have assessed carries a high impact, and could lead to adverse consequences for customers and the Company if errors occur. We take this risk into account when deciding the level of scrutiny that each piece of information requires before we provide it to others. Next Steps We will publish a draft version of our plan for assurance of our data in February 2017, which will take into account the views received in response to this consultation. We will discuss this draft plan with Ofwat and key stakeholders to ensure that it meets their expectations. We will then publish a final version of our assurance plan by 31 st March 2017. This plan will support the publication of our reports on our performance in 2016/17, which will be published in the summer of 2017. Page 22
Annex 1 Assessment Methodology Scoring System Assessment Methodology Complexity of Data Sources Completeness of Data Set Extent of Manual Intervention Complexity and Maturity of Reporting Rules Score Range 0 1 2 3 4 1-4 Single data system used One numerical and one Data derived from two to populate submission financial data system numerical systems, two used to populate financial systems, or submission more than two data 1-4 Complete data set routinely captured to populate this report for 2 years or more 1-4 Data collation and reporting are fully automated 1-4 Rule set is complete, the rules require no interpretation, judgement or assumptions, the rules have been in place for more than 12 months Data routinely captured but for less than 2 years, or some elements of reporting based on extrapolation of sample data More than 0% but less than 60% of the data is manually collated and reported The rule set is complete and has not changed for at least 12 months, but the rules require some interpretation, judgement or assumptions systems Data not routinely captured to report performance against this measure - significant level of extrapolation required Significant level of manipulation of data and application of assumptions required, - more than 60% The rule set is incomplete, the rules require significant interpretation, judgement or assumptions, or the rules have been newly developed within the last year Page 23
Assessment Methodology Control Activities Experience of Personnel Score Range 0 1 2 3 4 0-2 There are inadequate There are adequate There are extensive validation/preventative validation/preventative validation/preventative controls, or controls controls, in place for controls which have have been in place for 12-24 months, systems been in place for more less than 12 months, or and processes than 2 years, systems systems and processes substantially and process fully not document and documents and control documented and control points not points assessed, and control points fully assessed, or regulatory regulatory submissions evaluated and submissions are not subject to effective assessed, and subject to effective review or supervision regulatory submissions review of supervision process subject to processes comprehensive and effective review and supervision processes. 0-2 Submission collated by Submission collated by Submission collated by employees with no employees with no employees with prior prior experience of prior experience but experience, with doing so and no using method method statements for method statement statements for prior prior years in place available to explain submissions, or prior year approach to submission collated by completing report employees with prior experience but with not method statements for prior years available Page 24
Assessment Methodology Evidence of historical errors with data Impact on Customers Impact on Company Reputation Financial ODI Incentive Score Range 0 1 2 3 4 0-2 Material errors Material errors Audit has been identified for this data identified within the undertaken within the within the last two last two years, for last two years and no years and the issues which all issues have material errors have not been been remediated but identified, and either eliminated, or no audit not yet validated, or no no previously identified undertaken on this data audits undertaken on errors in submissions or in the last five years data within the last two audit confirmed that years but has been any previously undertaken within last identified issues have 5 years been properly 1-4 Performance has no or negligible service impact on all customers, other network operators or service providers 1-4 Level of performance has no material impact on Company reputation 1-4 Measure is not subject to regulatory financial incentive addressed Performance has limited indirect impact on service to customers, mainly through longer term effects Level of performance has impact on Company reputation with specialist regulators or relevant stakeholders Performance reflects indirect impact on service to customers Level of performance impacts Company reputation with affected customers, and/or within industry and with regulators. Potential for coverage in industry or local media. Measure is subject to regulatory reputational incentive Performance reflects direct impact on service to customers Level of performance has significant impact on company reputation, likely to attract media coverage at regional level Measure is subject to regulatory financial incentive Page 25
Assessment Methodology Industry Comparator Score Range 0 1 2 3 4 1-4 Measure is only Measure is largely only Measure may be used Measure is directly relevant to BW and not relevant to BW but by Ofwat or other used by Ofwat or other used for external comparisons may be regulators and regulators to compare comparisons drawn with other stakeholders to make and incentivise companies on similar comparisons between company performance measures companies Page 26
Annex 2 Results of Risk Assessment Page 27
Page 28
Page 29
Page 30
Page 31
Page 32