Level 1 Water Audit Validation

Transcription

1 Level 1 Water Audit Validation Lucy Andrews North American Water Loss Conference 2017

2 What Is Typical Water Loss in the US?

3 Collecting Water Loss Data CA DRBC GA TN TX total audits ,646 # of unrealistic audits ,065 % of unrealistic audits 33% 25% 16% 19% 40% R. Sturm, K. Gasner, and L. Andrews Water Audits in the United States: A Review of Water Losses and Data Validity (Project 4372B). Denver, CO: Water Research Foundation.

4 Something Is Amiss What s the problem? How do I fix it? Water Research Foundation 4639: Level 1 Water Audit Validation

5 Data Quality Matters! inaccuracy & uncertainty in inputs inaccuracy & uncertainty in results Sources of error: Instruments Databases People Missing information

6 What Does Validation Look Like Abroad? IWA validation is recommended consider reliability (qualitative) and accuracy (quantitative) Australia third-party validation every three years prompted by drought (2004) Quebec minimum night flow analysis annual Ministry validation interview with all agencies Audit collection programs also exist in Austria, England, and Denmark.

7 What Does Validation Look Like in the US? Georgia level 1 validation is required started in 2012; first three years of data were validated by a state-funded third party California level 1 validation is required started in 2017; first year of data was validated by state-funded third party AWWA WADI third-party validation of annual national dataset started before validation methodology had been standardized Additional states are considering validation programs to improve the water audit data informing resource decisions.

8 Programmatic Recommendations Standardize water audit data collection and validation. Insist on transparent and neutral data validity grades. Approach regulating data validity grades and scores with caution. Focus on the role of data validity in water loss control and utility management. Establish a legislative mandate and funding.

9 What Does Validation Accomplish? Validation aims to: 1. Identify and appropriately correct errors in data and application of methodology. 2. Evaluate and communicate uncertainty in water audit data inputs.

10 How Much Effort? Level 1: data validity and methodology interview Level 2: desktop analysis and raw data work Level 3: field data collection Validation doesn t necessarily fix all errors auditing and validation are retrospective.

11 Level 1 Validation Goals: Outcomes: Limitations: Confirm interpretation of methodology Identify evident errors Assign appropriate data validity grades Appropriate data validity grades Recommendations for further validation Does not correct errors in raw data Does not study instrument performance

12 Level 1 Validation The Validator: Knowledgeable understands what can go wrong Objective keeps the goal in mind Diplomatic avoids blaming Systematic doesn t miss anything Not the auditor!

13 Level 1 Validation The Method: 1. Collect the audit and required supporting documents. 2. Examine initial performance indicators. 3. Validate audit inputs (volumes, values, validity grades). 4. Re-examine final performance indicators. 5. Document results and propose next steps.

14 Level 1 Validation: Collect The Required Materials: A water audit! Summary tables Water supplied by month and meter Authorized consumption by month and account class Supply meter test and calibration results (if applicable) Whatever else the audit compiler can pass along!

15 Level 1 Validation: Examine Financial Indicators Check NRW volume as % of Water Supplied > 0% NRW value as % of operating cost < 100% Operational Efficiency Indicators Check Apparent Losses per Service Connection per Day > 0 Real Losses per Service Connection per Day > 0 Real Losses per Service Connection per Day per PSI > 0 Infrastructure Leakage Index > 1.0 Anything funky in the performance indicators, audit, or supporting documentation?

16 Level 1 Validation: Validate For each input: How did the auditor arrive at the input? Is this corroborated by supporting documentation? How did the auditor interpret methodology? Which data validity grade describes operational practices? Does anything need to be changed to make the audit more accurate?

17 Level 1 Validation: Re-Examine After validation, did the performance indicators change? Financial Indicators Check NRW volume as % of Water Supplied > 0% NRW value as % of operating cost < 100% Operational Efficiency Indicators Check Apparent Losses per Service Connection per Day > 0 Real Losses per Service Connection per Day > 0 Real Losses per Service Connection per Day per PSI > 0 Infrastructure Leakage Index > 1.0

18 Level 1 Validation: Document People auditor and validator Initial performance indicators Validated performance indicators Recommended changes (audit inputs, data validity grades) Remaining questions Recommendations for further validation Overall impression

19 Success Story: CA Water Loss TAP 2015 CA UWMP submissions: 46% pass rate

20 What Happens After Level 1 Validation? Audits still aren t perfect! Inaccuracy can persist. Level 2 validation raw data, data transfer Level 3 validation instrument tests, leakage investigation Subsequent audits Water loss intervention

21 Thanks! Comments? Questions? Brilliant ideas? Lucy Andrews - lucy.andrews@wsoglobal.com L. Andrews, K. Gasner, R. Sturm, G. Kunkel, W. Jernigan, and S. Cavanaugh Level 1 Water Audit Validation: Guidance Manual (Project 4639A). Denver, CO: Water Research Foundation. L. Andrews, K. Gasner, R. Sturm, G. Kunkel, W. Jernigan, and S. Cavanaugh Utility Water Audit Validation: Principles and Programs (Project 4639B). Denver, CO: Water Research Foundation.