ANNUAL PERFORMANCE REPORT DATA ASSURANCE PLAN 2015/2016

Transcription

1 ANNUAL PERFORMANCE REPORT DATA ASSURANCE PLAN 2015/ INTRODUCTION 1.1. Ofwat s shared vision for the water sector in England and Wales is one where customers, the environment and wider society have trust and confidence in vital public water and wastewater services Having information that is easy to understand and navigate provides transparency and helps everyone build trust and confidence in water. Ofwat expects us to provide information to our customers and stakeholders that enables them to understand how we are performing. Ofwat also expects us to have processes in place to make sure our published information can be trusted. If there is a lack of confidence in the information that individual water companies provide about their performance, Ofwat will step in to protect customers Our vision is to earn the trust of our customers every day by adopting values such as: being trusted to do the right thing; being honest with everyone; and ensuring excellence in everything we do. Our Code of Conduct makes it clear that it is vitally important that the data and information we provide is accurate and complete, so that our customers and stakeholders can trust what we say The overarching aim of our Data Assurance Plan is to reduce the Risk to customers and other stakeholders of inaccurate or incomplete performance reporting. 1

2 2 RISK ASSESSMENT Introduction 2.1. Our approach to data assurance is one that is based on Risk. It comprises a Risk Assessment matrix that combines assessments of the impact and the probability of the Risk. While the Risk score that results from the matrix may be different for each Performance Measure in the Annual Performance Report this will help ensure we follow a consistent approach In order to provide our customers and stakeholders with a high level of confidence in our approach to assurance we will publish an Annual Assurance Plan, which will set our plans for assurance of our outcomes, performance commitments and Outcome Delivery Incentives in line with our risk based approach As part of this risk based approach, we will carry out an assessment of our risks, strengths and weaknesses in order to target our assurance most effectively. This approach will allow us to address any specific areas with the appropriate level of internal and external scrutiny to mitigate any risks, or highlight areas of best practice. We will engage with our key stakeholders on our plans for assurance to ensure that our view of risks, strengths and weaknesses is appropriate and that our plans for assurance are robust and meet their expectations. The Risk Matrix 2.4. Risk is defined as an uncertain future outcome that, if it occurs, will have negative effects on the quality and reliability of Performance Report submissions. A Risk is specified by the combination of the probability of it occurring and a measure of the impact should it occur. Risk relates to the level of expectation that inaccurate or incomplete data will be submitted to our stakeholders in the future and the possible consequences The overall Risk profile for each Performance Measure is determined by assessing both the probability of it containing an error and the impact this error would have across key drivers. Therefore, the Risk Matrix comprises two component metrics the Impact Metric and the Probability Metric. The Total Risk Rating is a combination of both metrics The probability element of Risk is proxied by the Probability Metric and the impact element of Risk is proxied by the Impact Metric. The Impact and Probability Metrics are defined as follows: Impact Metric: a measure to represent the impact of an identified Risk materialising. It relates to the expected impact of inaccurate or incomplete data on stakeholders, our finances, our reputation and our coverage in the media. It is scored by assessing each performance data measure against the specified impact categories. Probability Metric: a measure to represent the probability of error occurrence. It is scored through the evaluation of the processes for data collection, reporting and 2

3 the related control systems and processes We have adopted a five-stage process in assessing the overall Risk for each Performance Measure, which is summarised below. The details of each stage are provided in the sections that follow. The results of the Risk Assessment should inform the choice of the appropriate Data Assurance Activity for each Performance Measure. Figure 2.1: Five-stage Risk Assessment process Data Risk Identification 1 Assess impact of risk for each Performance Commitment 2 Determine overall Impact Metric Score 3 4 Assess probability of risk for each Performance Commitment Determine overall Probability Metric Score Historical Errors should inform future risk assessments 5 Determine Total Risk Rating Data Assurance Activities Impact Metric: Stages 1 and Table 2.1 overleaf sets out the criteria for assessing the Impact Metric for each Performance Measure The Impact Metric has four ratings, 1 to 4, with 4 denoting the highest level of adverse impact and 1 denoting the lowest level of adverse impact that would arise (in a realistic worst-case scenario) due to the use of inaccurate or incomplete data. 3

4 2.10. We are of the view that the three existing impact categories provide sufficient scope for us to capture any major impacts. To calculate the Impact Metric the three categories should be scored on a scale from 1 to 4. The three impact categories are: a. Financial; b. Reputational (including Media coverage); c. Stakeholders The scores are recorded in the Risk Assessment template (Excel file) under the relevant impact categories (a to c). Each category is scored separately To calculate an overall impact score for a Performance Measure, we take the highest score of all impact categories We interpret the impact assessment as being the associated impact of inaccurate or incomplete data and not the impact associated with poor performance that the data might reveal. In doing so, we assume a realistic worst-case scenario. 4

5 Table 2.1: Impact Metric: assessment of impact caused by inaccurate or incomplete data Score (a) Financial (b) Reputational (c) Stakeholders 4 Potential under/over claim of reward/penalty of 15m or more in any one year 3 Potential under/over claim of reward/penalty of 10m or more but less than 15m in any one year 2 Potential under/over claim of reward/penalty of 5m or more but less than 10m in any one year 1 Potential under/over claim of reward/penalty less than 5m in any one year Total loss of confidence from key stakeholders. Potential national long-term media coverage Loss of confidence from some key stakeholders requiring input from Senior Managers. Potential national short-term negative media coverage Confidence fears allayed through normal liaison arrangements. Potential local reputational damage Confidence unlikely to be affected. Local media attention quickly remedied Requirement that data is totally accurate Able to tolerate accuracy within 1% Able to tolerate accuracy within 5% Able to tolerate accuracy of 5% or more 5

6 Probability Metric: Stages 3 and Table 2.2 sets out the criteria for assessing the Probability Metric for each Performance Measure to represent the probability of an error occurring The Probability Metric has four ratings, from 1 to 4, with 4 denoting the highest probability and 1 denoting the lowest probability of inaccurate or incomplete data There are seven categories that will be scored for each Performance Measure in order to calculate its probability score. These are: 1. I1. Complexity of data sources 2. I2. Completeness of data set 3. I3. Extent of manual intervention 4. I4. Complexity and maturity of reporting rules 5. C1. Control activities 6. C2. Experience of personnel 7. C3. Evidence of historical errors with this data Inherent Probability Control Frameworks I1 to I4 reflect the inherent (I) probability of error where no additional controls (on top of general system or process controls) are used to reduce Risk. C1 to C3 reflect the control (C) framework in place to reduce the probability of error. Combining these gives the overall probability of error, taking into account any controls that are in place Categories I1 to I4 (inherent probability) have three ratings 2 to 4, with 4 denoting the highest probability and 2 denoting the lowest probability of inaccurate or incomplete data. These are indicators of the probability of error associated with the systems used, available data and reporting rules without any controls in place Categories C1 to C3 (control assessment) have three ratings 0 to 2, with 0 denoting the weakest control and 2 denoting the strongest control. These are indicators of the level of confidence in the control environment (i.e. confidence in the business s ability to prevent errors or decrease the probability of errors occurring). Each of these categories is scored either 0, 1, or 2 representing low, medium, and high controls The rules for calculating an overall probability score for any Performance Measure are as follows: Select the maximum of inherent probability category score. Calculate the arithmetic average rounded to nearest whole number of scores in the control framework assessment. Subtract the average control score from the maximum inherent score. If both the inherent probability and control framework assessment are scored 2 the probability score to be used is 1 (instead of 0, as 2-2=0). 6

7 2.21. The rules are based on the principle that (all other factors being constant) high inherent Risk or a weak control environment should result in a higher Risk score, while low inherent Risk or strong control environment should result in a lower Risk score The overall probability score ranges from 1 to We might expect to see greater variation between Performance Measures in the Probability Metric Scoring than we would expect for Impact Metric Scores. This is because each Performance Measures will have different reporting systems, processes, and control environments for reporting data. 7

8 Table 2.1: Probability Metric I Score I1. Complexity of data sources 4 Data obtained in full, or in part, from an external source I2. Completeness of data set A significant number of elements of the performance data submission relies on extrapolation of sample data rather than a full data set I3. Extent of manual intervention The final data set is reliant on a significant amount of manual collation and intervention I4. Complexity & maturity of reporting rules The Methodology Statement is: Incomplete or Requires significant interpretation, judgement or assumptions or Less than 12 months old C1. Control activities There are inadequate validation / preventative controls or controls have been in place for less than 12 months or systems and processes not documented and control points not assessed (i.e. any such material lacks substantial coverage) C2. Experience of personnel The performance data submission is being collated by employees with no prior experience of doing so and no Methodology Statement available to explain prior year approach to completing this exercise C3. Evidence of historical errors with this data Material errors identified for this data set within the last two years; and the issues identified have not been eliminated or no audit undertaken on this data submission in the last five years C Score 0 8

9 I Score I1. Complexity of data sources I2. Completeness of data set I3. Extent of manual intervention I4. Complexity & maturity of reporting rules C1. Control activities or C2. Experience of personnel C3. Evidence of historical errors with this data C Score 3 More than 2 data systems used to populate the data submission Some elements of the data submission relies on extrapolation of sample data rather than a full data set The final data set is reliant on some manual collation and intervention The Methodology Statement is complete and has not changed for at least 12 months but requires some interpretation, judgement or assumptions Performance Report submissions not subject to effective review or supervision processes There are adequate validation / preventative controls and controls have been in place for more than 12 months but less than 2 years and The data submission is being collated by employees with no prior experience of doing so but are using Methodology Statements for prior submissions to support them or The data submission is Material errors identified for this data set within the last two years; for which all issues actioned but not yet checked by subsequent audits or no audits undertaken 1 9

10 I Score I1. Complexity of data sources 2 Single data system used to populate submission I2. Completeness of data set No extrapolation or sampling used to populate the data submission I3. Extent of manual intervention Data collation and reporting are fully automated I4. Complexity & maturity of reporting rules The Methodology Statement is complete and has not changed for at least 12 months and requires no interpretation, C1. Control activities systems and processes substantially documented and control points assessed and Performance Report submissions are subject to effective review or supervision processes There are extensive validation / preventative controls and controls have been in place C2. Experience of personnel being collated by employees with prior experience of doing so but with no Methodology Statements for prior years available The data submission is being collated by employees with prior experience of completing it with Methodology Statements for C3. Evidence of historical errors with this data on this data submission within the last 2 years but audit has been undertaken in the last five years Audits have been undertaken on this data submission within the last 2 years and no material errors were identified C Score 2 10

11 I Score I1. Complexity of data sources I2. Completeness of data set I3. Extent of manual intervention I4. Complexity & maturity of reporting rules judgement or assumptions C1. Control activities for more than 2 years C2. Experience of personnel prior years in place C3. Evidence of historical errors with this data and either C Score and systems and processes fully documented and control points assessed there were no previously identified errors in submission data or and Performance Report submissions subject to comprehensive and effective review and supervision processes Audit confirmed that any previously identified issues have been properly addressed 11

12 Impact Metric Score Qualitative Performance Measures Some Performance Measures (e.g. Serviceability) are opinion based but nevertheless fall within the scope of this guidance. We will use reasonable endeavours to assess these Performance Measures against the Risk scoring criteria outlined earlier. Total Risk Score: Stage Impact and process scores are multiplied to arrive at a total Risk score in accordance with the Impact and Probability Risk Matrix below (Figure 2.2) There are four levels of Total Risk: low, medium, high and critical. The assessed Total Risk Rating is used to inform our choice of data assurance activities to be applied to our Annual Performance Report. It is our responsibility to demonstrate to Ofwat, our customers and stakeholders the robustness and suitability of our Data Assurance Plan and Risk reduction measures. Figure 2.2: Impact and Probability Risk Matrix Probability Metric Score Possible Total Risk Rating Low Risk Medium Risk High Risk Critical Risk Performance Measures may feature as Critical/High Risk areas but not because we consider the control frameworks in place are not robust. Indeed, for each Performance Measure we have developed an individual Risk Matrix that lists individual risks that could potentially affect the accuracy or completeness of reported data. The internal controls in place to mitigate these risks are also listed and we satisfy ourselves that the correct operation of these controls is sufficient to prevent the risks from occurring. Their inclusion 12

13 therefore is rather a combination of several factors including: the fact that in the main they are new performance measures introduced in April 2015 and the proper operation of internal controls has not yet been independently tested by our in house internal audit team or our external third party expert the Reporter; they are complex areas that require an element of judgement; and, recent regulatory reporting issues within the water sector in general. 3 DATA ASSURANCE ACTIVITIES Introduction 3.1. Each Data Assurance Activity is defined in terms of who should undertake the activity, when (i.e. under what circumstances) and what this involves. For all Performance Measures, the data assurance activities are informed by the results of the Risk Assessment. All Performance Measures (including those rated as low Risk) require a degree of Planning, Review, and Sign-off. Further details are provided in Table 3.1 overleaf. 13

14 Table 3.1: Data assurance activity options Activity When it applies Who is responsible What is the content/coverage Planning Methodology Statement - completion All Performance Measures Nominated Data Originator Explains process to produce the submission and should include details of: - responsibilities; - sign off; - management checks; - governance arrangements; - assumptions. Methodology Statement - review All Performance Measures Person(s) with reasonable understanding of requirements Checks for robustness, transparency and compliance with reporting obligations. Reporter Ensure consistency between individual Methodology Statements. Performance Report Submission Plan All Performance Reports Regulation Team Details the plan to complete the Performance Report, including details of timetable, responsibilities, sign-off and governance meetings as appropriate. Review 14

15 Activity When it applies Who is responsible What is the content/coverage Internal Expert Review All Performance Measures Regulation Team Responsible through the Due Diligence process for ensuring that Performance Measure data is complete and accurate and in accordance with any guidance issued by Ofwat/DWI/NRW. The expert reviewer satisfies him/herself that the Key control activities have been performed and any unusual findings investigated and resolved. Internal Advice All Performance Measures Compliance Team Responsible for providing advice, guidance and support to help ensure compliance with regulatory/legal obligations. Achieved through a combination of formal reviews outlined in the Annual Compliance Plan and ad hoc reviews provided as requested. Internal Data Audit As identified through Risk Assessment. Internal Audit Programme agreed by Audit Committee. Responsible for providing evidence of verification of data. Done through a sampling approach. Intends to determine the level of confidence that can be placed on the Performance Measure through testing a sample of the data. Reported/documented through formal governance channels. 15

16 Activity When it applies Who is responsible What is the content/coverage External Data Audit Internal Process Review of Performance Report External Process Review of Performance Report Essential for financial accounts and financial Performance Measures Audits carried out by independent third party auditors outside the Company Programme agreed by Audit Committee. Responsible for providing evidence of verification of data. Done through a sampling approach. Intends to determine the level of confidence that can be placed on the Performance Measure through testing a sample of the data. Formal report produced. All Performance Reports Internal Audit Programme agreed by Audit Committee. Not responsible for ensuring that Performance Report details are complete and accurate but to provide an independent challenge to the process to produce the Report. Review of the adequacy and effectiveness of the internal control systems to ensure the Performance Report is accurate, complete and timely. Formal report produced. Control gaps/areas for improvement identified and actions logged. All Performance Reports Reporter Programme agreed by Audit Committee. Not responsible for ensuring that Performance Report details are complete and accurate but to provide an independent challenge to the process to produce the Report. Review of the adequacy and effectiveness of the internal control systems to ensure the Performance Report is accurate, complete and timely. Formal report 16

17 Activity When it applies Who is responsible What is the content/coverage produced. Control gaps/areas for improvement identified and actions logged. Senior Manager Sign-off All Performance Measures Accountable Senior Manager Detailed review of data and any supporting narrative. Complete and sign a Declaration Form attesting to confidence in the accuracy of the submission. Director Sign-off All Performance Measures Accountable Director Must complete a final review prior to submission to the Board. This review must include a challenge of the Senior Manager Sign-off. Must complete and sign a Declaration Form attesting to accuracy of the Performance Measure data. Drives an overall confidence assessment for the Performance Report. Board Sign-off All Performance Measures DCC Board High-level oversight. Board reviews summary of submission and assurance activities followed, as presented by a relevant Director. Detailed review of tables and assurance processes formally delegated to Director who approves with delegated authority on behalf of the Board. Approval of submission must be minuted to enable completion of a record of evidence attesting to accuracy, to be delegated to the CEO or other Director identified by the Board. 17

18 Selection of data assurance activities 3.2. All Performance Measures identified as Critical/High Risk will be subject to a further detailed appraisal process. It is necessary to carry out this exercise each year in order to target areas to improve. The areas to be targeted may not be those of greatest underlying risk as we may be able to demonstrate that we already have strengths in these areas All data assurance activities will be completed before the Annual Performance Report is signed off by the Board and subsequently published. This means, for example if we have stated that a particular Performance Measure will be subject to an Internal Data Audit then that review must be complete before the Board s sign off. 18